You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 12:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Dec 12, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2526922950,1sleep20-21swapper/219:05:372
3215221520,6sleep32664399cyclictest23:55:493
2901221000,3sleep029013-21ssh21:40:470
32932860,9sleep3381ktimersoftd/323:18:013
199162840,4sleep0140-21systemd-journal22:12:560
26640998250,12cyclictest0-21swapper/023:00:490
251792750,4sleep10-21swapper/119:05:351
155902720,2sleep30-21swapper/322:06:303
308832710,2sleep10-21swapper/120:50:361
2664099712,50cyclictest5295-21aten_rbpower_vo19:40:330
26640997116,30cyclictest26032-21timerwakeupswit22:20:470
2664099707,40cyclictest9-21ksoftirqd/022:55:320
2664099701,42cyclictest0-21swapper/021:55:330
2664099688,39cyclictest27373-21fschecks_time20:40:340
2664099686,35cyclictest0-21swapper/020:50:320
2664199676,26cyclictest0-21swapper/122:35:301
2664099676,52cyclictest0-21swapper/022:50:340
3432660,1sleep20-21swapper/222:30:412
2664199666,5cyclictest9550irq/80-mmc000:20:171
26640996617,19cyclictest7950irq/56-dwc_otg_23:15:220
26640996612,24cyclictest0-21swapper/019:50:310
195762660,2sleep30-21swapper/321:28:503
289252650,2sleep339-21ksoftirqd/322:25:353
2664099654,46cyclictest10958-21unixbench_multi00:10:540
2664099653,34cyclictest9395-21ssh21:15:300
2664099653,34cyclictest9395-21ssh21:15:300
22252650,2sleep12222-21ssh23:15:491
2664099647,24cyclictest0-21swapper/019:35:280
2664099635,12cyclictest0-21swapper/021:50:320
42082620,2sleep00-21swapper/000:02:230
26641996213,8cyclictest17137-21diskstats21:25:381
2664099625,37cyclictest0-21swapper/020:02:060
26640996237,11cyclictest0-21swapper/020:57:120
26640996237,11cyclictest0-21swapper/020:57:120
2664199616,4cyclictest0-21swapper/122:10:171
2664199611,16cyclictest22750-21unixbench_multi20:25:481
26640996142,8cyclictest0-21swapper/023:45:400
84052600,2sleep30-21swapper/300:08:133
26641996017,4cyclictest24195-21switchtime23:45:431
2664199601,6cyclictest21357-21ntp_kernel_pll_21:30:421
2664099601,43cyclictest21395-21open_files21:30:440
26640996014,25cyclictest11006-21grep22:45:290
26643995914,9cyclictest7950irq/56-dwc_otg_21:40:323
2664199594,4cyclictest7950irq/56-dwc_otg_21:05:401
2664199593,10cyclictest7850irq/56-dwc_otg_21:36:361
2664099599,25cyclictest10919-21fschecks_count19:55:340
2664099596,17cyclictest7850irq/56-dwc_otg_23:25:320
26640995940,4cyclictest1365-21ls19:30:170
2664099591,45cyclictest16930-21aten_rbpower_cu21:25:320
2664099591,34cyclictest17256-21unixbench_singl20:10:460
2664199588,4cyclictest0-21swapper/123:52:591
2664199588,4cyclictest0-21swapper/123:52:591
2664199587,4cyclictest7850irq/56-dwc_otg_22:10:441
2664199583,4cyclictest7850irq/56-dwc_otg_19:45:321
2664199581,21cyclictest0-21swapper/120:10:261
2664099586,35cyclictest0-21swapper/020:35:330
26640995851,5cyclictest0-21swapper/022:26:440
2664099584,42cyclictest4735-21grep21:05:510
26640995829,6cyclictest19896-21fschecks_time23:40:340
26640995821,15cyclictest0-21swapper/022:10:170
26640995813,27cyclictest16699-21taskset00:20:170
2664199577,6cyclictest9450irq/48-DMA09:28:481
2664199573,4cyclictest7750irq/56-dwc_otg19:40:421
2664199572,9cyclictest0-21swapper/100:20:371
2664199571,5cyclictest3593-21missed_timers21:50:431
2664199571,4cyclictest0-21swapper/123:20:361
2664199571,4cyclictest0-21swapper/123:20:361
26640995729,12cyclictest0-21swapper/022:38:130
295772560,2sleep0101ktimersoftd/023:10:330
2664199564,3cyclictest0-21swapper/100:31:311
2664199561,6cyclictest1215-21cron19:30:161
26640995635,15cyclictest101ktimersoftd/023:20:450
26640995635,15cyclictest101ktimersoftd/023:20:450
2664099562,50cyclictest15857-21idleruntime-cro20:10:160
26640995616,20cyclictest0-21swapper/021:35:280
26640995613,6cyclictest20125-21latency_hist00:25:170
2664099561,33cyclictest6570-21ls00:05:390
2664099561,24cyclictest0-21swapper/023:35:310
2664199557,9cyclictest0-21swapper/100:11:091
2664199554,5cyclictest7750irq/56-dwc_otg00:01:231
2664199554,4cyclictest7750irq/56-dwc_otg22:35:161
2664199553,4cyclictest20639-21/usr/sbin/munin20:20:431
26641995533,13cyclictest0-21swapper/121:55:321
2664199552,5cyclictest25316-21/usr/sbin/munin20:35:341
2664199551,4cyclictest7750irq/56-dwc_otg00:05:471
2664099557,38cyclictest0-21swapper/022:43:480
2664099557,38cyclictest0-21swapper/022:43:480
2664099557,33cyclictest0-21swapper/019:10:310
2664099553,49cyclictest0-21swapper/023:10:170
26640995531,9cyclictest0-21swapper/000:33:340
26642995421,7cyclictest7850irq/56-dwc_otg_23:20:312
26642995421,7cyclictest7850irq/56-dwc_otg_23:20:312
2664199546,8cyclictest7750irq/56-dwc_otg22:02:171
2664199543,5cyclictest7850irq/56-dwc_otg_23:05:431
2664199542,8cyclictest7850irq/56-dwc_otg_21:22:501
2664199542,5cyclictest21849-21irqstats22:15:391
2664199541,9cyclictest32106-21latency_hist22:30:171
2664199541,5cyclictest31331-21snmpd23:31:171
2664199541,4cyclictest7950irq/56-dwc_otg_23:40:171
26641995412,4cyclictest32762-21rm23:14:561
26640995438,5cyclictest101ktimersoftd/022:15:480
26640995429,13cyclictest0-21swapper/000:27:380
26640995429,13cyclictest0-21swapper/000:27:380
2664199537,4cyclictest29769-1kworker/u9:123:55:361
2664199534,4cyclictest7950irq/56-dwc_otg_21:40:491
2664199533,6cyclictest7850irq/56-dwc_otg_20:15:551
26641995314,4cyclictest7950irq/56-dwc_otg_20:45:451
2664099538,22cyclictest0-21swapper/021:10:350
2664199525,10cyclictest7950irq/56-dwc_otg_22:44:031
2664199525,10cyclictest7950irq/56-dwc_otg_22:44:031
2664199523,6cyclictest11335-21ssh21:17:101
2664199523,6cyclictest11335-21ssh21:17:101
2664199522,7cyclictest0-21swapper/120:08:031
2664199522,4cyclictest0-21swapper/121:14:131
2664099523,6cyclictest12111-21ssh23:30:290
2664099523,37cyclictest1526-21ssh23:59:400
26640995228,11cyclictest0-21swapper/021:22:230
26640995225,5cyclictest11384-21ssh22:00:460
2664099521,30cyclictest28843-21/usr/sbin/munin20:45:340
2664099521,25cyclictest28135-21ls23:50:450
2664099521,25cyclictest28135-21ls23:50:450
2664199511,15cyclictest0-21swapper/122:52:291
2664199511,11cyclictest0-21swapper/121:48:391
2664099516,40cyclictest101ktimersoftd/020:15:410
26640995129,8cyclictest0-21swapper/019:45:540
26640995123,18cyclictest0-21swapper/020:20:310
2664099511,24cyclictest0-21swapper/022:35:160
26640995111,20cyclictest23765-21fschecks_time20:30:350
2664199507,8cyclictest7750irq/56-dwc_otg22:55:381
2664199503,10cyclictest26981-21ssh22:22:251
2664099503,32cyclictest0-21swapper/019:16:260
2664099501,26cyclictest28836-21ls00:35:500
235712500,29sleep37950irq/56-dwc_otg_23:45:353
2664399491,4cyclictest7950irq/56-dwc_otg_00:25:323
2664399491,4cyclictest7950irq/56-dwc_otg_00:25:323
26642994927,4cyclictest3093050wget23:55:342
26640994933,5cyclictest1810-21runrttasks21:00:210
2664399482,8cyclictest0-21swapper/320:05:303
26643994824,5cyclictest28080-21tune2fs00:35:393
2664199483,8cyclictest7850irq/56-dwc_otg_22:46:511
2664099481,26cyclictest30565-21fschecks_count19:20:360
26640994812,19cyclictest32377-21grep21:45:390
2664199477,26cyclictest0-21swapper/119:50:221
261492460,5sleep32664399cyclictest00:32:213
2664399456,25cyclictest0-21swapper/300:00:323
2664199458,5cyclictest10638-21ssh23:27:481
2664199451,5cyclictest21240-21ntp_states00:25:441
2664199451,5cyclictest21240-21ntp_states00:25:441
2664099458,16cyclictest2818-21taskset19:32:490
2664399447,23cyclictest0-21swapper/320:30:313
2664399446,30cyclictest0-21swapper/323:05:313
52722430,2sleep20-21swapper/221:53:052
2664399432,4cyclictest0-21swapper/322:30:323
2664099431,30cyclictest0-21swapper/020:28:450
2664399417,25cyclictest0-21swapper/321:00:333
2664399417,22cyclictest0-21swapper/321:30:313
2664299419,3cyclictest0-21swapper/222:19:032
2664299412,11cyclictest0-21swapper/220:10:282
2664299404,24cyclictest11210-21ls19:55:372
2664199402,12cyclictest0-21swapper/100:38:541
2664399393,7cyclictest0-21swapper/319:45:293
2664399393,17cyclictest0-21swapper/322:35:293
26643993920,6cyclictest947050wget21:15:333
26643993920,6cyclictest947050wget21:15:333
2664399391,5cyclictest7950irq/56-dwc_otg_20:35:333
2664399391,24cyclictest0-21swapper/320:40:293
2664299394,5cyclictest7850irq/56-dwc_otg_21:50:172
2664299391,9cyclictest0-21swapper/222:10:302
2664299388,14cyclictest7850irq/56-dwc_otg_23:15:402
2664299386,23cyclictest0-21swapper/221:10:222
2664299383,5cyclictest0-21swapper/219:10:282
2664299381,20cyclictest7950irq/56-dwc_otg_22:35:302
110902381,2sleep20-21swapper/223:28:482
105752381,32sleep20-21swapper/200:10:512
26643993716,10cyclictest30503-21latency_hist23:55:173
26643993716,10cyclictest30503-21latency_hist23:55:173
2664299377,14cyclictest7850irq/56-dwc_otg_00:25:402
2664299377,14cyclictest7850irq/56-dwc_otg_00:25:402
2664299376,21cyclictest24271-21ssh21:35:282
2664299374,26cyclictest7950irq/56-dwc_otg_19:50:162
2664199371,6cyclictest0-21swapper/119:15:281
2664399361,21cyclictest9963-21latency_hist22:00:173
26643993611,14cyclictest0-21swapper/319:40:173
2664299361,29cyclictest22570-21cut23:00:382
26642993611,6cyclictest10658-21basename22:45:172
26642993611,6cyclictest10658-21basename22:45:172
26641993616,4cyclictest28236-21latency_hist19:15:161
136072361,4sleep213616-21ssh22:48:382
2664399357,19cyclictest0-21swapper/321:45:313
2664399357,15cyclictest0-21swapper/319:55:313
2664399355,20cyclictest21362-21ssh22:15:333
2664399352,14cyclictest7850irq/56-dwc_otg_20:10:383
2664399351,5cyclictest0-21swapper/300:23:343
2664399351,20cyclictest0-21swapper/323:23:293
2664399351,20cyclictest0-21swapper/323:23:293
2664299357,13cyclictest7850irq/56-dwc_otg_21:00:482
26642993515,4cyclictest29092-21if_wlan022:25:372
26642993514,4cyclictest27735-21/usr/sbin/munin20:40:392
2664199358,11cyclictest7750irq/56-dwc_otg20:30:421
2664399348,5cyclictest8093-21processes22:40:463
2664399348,5cyclictest8093-21processes22:40:463
2664399347,15cyclictest0-21swapper/322:55:323
2664399345,6cyclictest7950irq/56-dwc_otg_21:25:173
2664399343,4cyclictest10300-21apt-get22:00:293
2664399342,16cyclictest3171-21latency_hist19:35:173
26643993410,12cyclictest0-21swapper/323:27:553
2664299345,5cyclictest7850irq/56-dwc_otg_21:15:512
2664299345,5cyclictest7850irq/56-dwc_otg_21:15:512
26642993419,4cyclictest13556-21fschecks_count00:15:382
2664299341,23cyclictest14785-21ls20:05:382
2664199344,16cyclictest7950irq/56-dwc_otg_20:55:461
2664199344,16cyclictest7950irq/56-dwc_otg_20:55:461
2664199342,12cyclictest14224-21latency_hist20:05:171
2664399336,15cyclictest0-21swapper/323:35:313
2664399336,11cyclictest7950irq/56-dwc_otg_21:50:343
2664399335,5cyclictest0-21swapper/323:30:333
2664399333,15cyclictest21499-21latency_hist20:25:173
2664399332,21cyclictest25356-21timerwakeupswit21:35:443
2664299339,7cyclictest7850irq/56-dwc_otg_00:30:412
2664299337,5cyclictest7850irq/56-dwc_otg_22:06:012
2664299337,13cyclictest7850irq/56-dwc_otg_23:10:372
26642993314,10cyclictest1120-21runrttasks21:57:332
26642993313,5cyclictest1120-21runrttasks22:57:382
2664199332,15cyclictest7850irq/56-dwc_otg_19:20:391
2664199331,9cyclictest7750irq/56-dwc_otg19:55:451
2664399327,15cyclictest0-21swapper/323:00:343
2664399327,14cyclictest7950irq/56-dwc_otg_20:45:453
26643993217,5cyclictest29933-21tune2fs23:10:373
2664299327,15cyclictest0-21swapper/219:50:312
2664299327,14cyclictest7850irq/56-dwc_otg_23:39:112
2664299326,7cyclictest7850irq/56-dwc_otg_19:40:172
26642993222,7cyclictest26187-21head22:20:482
2664299322,20cyclictest0-21swapper/223:49:372
2664299321,22cyclictest0-21swapper/221:30:312
26642993212,13cyclictest26077-21cat23:05:342
2664299321,16cyclictest23114-21ls23:45:172
26641993213,6cyclictest22820-21ssh23:00:411
322212310,2sleep30-21swapper/320:55:173
2664399315,17cyclictest32079-21/usr/sbin/munin19:25:363
2664399314,4cyclictest0-21swapper/322:20:313
2664399314,15cyclictest0-21swapper/320:20:173
26643993111,4cyclictest0-21swapper/321:10:203
2664299316,13cyclictest7850irq/56-dwc_otg_00:35:412
2664299315,6cyclictest140-21systemd-journal20:35:432
2664299315,12cyclictest7850irq/56-dwc_otg_21:05:452
2664299315,11cyclictest7850irq/56-dwc_otg_20:25:482
2664299313,8cyclictest32324-21cat20:55:172
2664299313,7cyclictest7850irq/56-dwc_otg_20:00:442
2664299313,12cyclictest0-21swapper/200:00:502
2664299312,4cyclictest7950irq/56-dwc_otg_21:42:272
2664299312,17cyclictest0-21swapper/219:34:082
2504523111,12sleep325086-21fschecks_count19:05:343
2664399308,7cyclictest7950irq/56-dwc_otg_00:15:403
2664399305,14cyclictest7950irq/56-dwc_otg_20:00:343
2664399304,7cyclictest7850irq/56-dwc_otg_00:10:393
2664399302,8cyclictest7950irq/56-dwc_otg_22:45:273
2664399301,8cyclictest31340-21sed19:20:463
2664399301,5cyclictest7750irq/56-dwc_otg21:05:513
2664399301,21cyclictest17803-21ssh22:10:323
2664399301,19cyclictest0-21swapper/323:42:233
2664399301,17cyclictest0-21swapper/320:25:493
2664399301,13cyclictest7950irq/56-dwc_otg_19:20:173
2664399301,12cyclictest7950irq/56-dwc_otg_22:55:163
2664299308,4cyclictest7850irq/56-dwc_otg_19:23:252
2664299307,6cyclictest7750irq/56-dwc_otg00:20:332
2664299307,21cyclictest0-21swapper/220:30:312
2664299307,12cyclictest0-21swapper/223:50:392
2664299307,12cyclictest0-21swapper/223:50:392
2664299306,14cyclictest7950irq/56-dwc_otg_20:55:392
2664299306,14cyclictest7950irq/56-dwc_otg_20:55:392
26642993011,4cyclictest0-21swapper/200:07:182
26642993011,3cyclictest0-21swapper/221:23:032
26642993010,4cyclictest0-21swapper/223:30:302
2664199301,13cyclictest3133150snmpd19:30:291
2664399298,4cyclictest7850irq/56-dwc_otg_19:55:163
2664399294,12cyclictest7950irq/56-dwc_otg_19:10:403
2664299295,12cyclictest6355-21vmstat19:40:472
2664299293,13cyclictest7850irq/56-dwc_otg_19:15:362
26642992921,6cyclictest10297-21ssh22:00:292
2664299291,20cyclictest15310-21latency22:50:422
2664299291,14cyclictest7950irq/56-dwc_otg_19:25:342
26642992910,6cyclictest20490-21/usr/sbin/munin20:20:402
2664199292,10cyclictest2423-21missed_timers21:00:401
2664199291,14cyclictest7950irq/56-dwc_otg_20:45:171
2664299285,4cyclictest0-21swapper/221:28:032
2664299283,11cyclictest7850irq/56-dwc_otg_20:15:302
2664399274,4cyclictest7850irq/56-dwc_otg_20:55:423
2664399274,4cyclictest7850irq/56-dwc_otg_20:55:423
2664399274,4cyclictest5807-21ls19:40:393
2664299275,12cyclictest7950irq/56-dwc_otg_20:45:412
2664199271,17cyclictest0-21swapper/119:35:191
263652263,14sleep00-21swapper/019:10:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional