You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-16 - 08:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Wed Sep 16, 2026 00:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
732621730,6sleep2788799cyclictest22:59:372
78859910797,7cyclictest0-21swapper/000:22:090
7885991037,51cyclictest0-21swapper/021:35:380
104202980,3sleep013-21rcuc/023:02:570
7885999133,46cyclictest0-21swapper/019:15:370
7885999014,56cyclictest25587-21runrttasks22:40:460
7885999013,60cyclictest9877-21taskset20:45:350
323782900,6sleep1788699cyclictest00:11:381
7885998917,48cyclictest2586-21aten_rbpower_cu20:25:370
7885998915,52cyclictest32743-21aten_rbpower_po22:50:360
7885998615,55cyclictest11110-21ssh00:25:500
193962860,3sleep30-21swapper/323:15:013
788599848,48cyclictest0-21swapper/020:15:350
7885998417,48cyclictest30302-21ssh23:28:180
7885998415,48cyclictest11924-21fschecks_count20:50:380
174742820,2sleep3788899cyclictest00:35:173
7885998112,56cyclictest0-21swapper/000:18:530
7885998111,46cyclictest0-21swapper/000:00:460
7885998014,47cyclictest0-21swapper/022:55:480
7885998013,46cyclictest4447-21awk20:30:330
788599798,56cyclictest22971-21tr21:15:360
7885997916,47cyclictest27561-21ls21:20:450
7885997912,51cyclictest20139-21unixbench-2d23:55:530
788599786,44cyclictest0-21swapper/023:15:360
7885997820,44cyclictest8181-21tr20:40:370
7885997813,48cyclictest0-21swapper/021:46:300
7885997812,47cyclictest31650-21/usr/sbin/munin23:30:390
7885997714,48cyclictest18827-21grep19:40:360
7885997713,52cyclictest18397-21ssh22:31:030
7885997713,52cyclictest18397-21ssh22:31:030
788599767,46cyclictest14037-21aten_rbpower_po00:30:380
788599763,42cyclictest0-21swapper/021:00:360
7885997618,48cyclictest11560-21aten_rbpower_cu19:20:360
7885997616,45cyclictest15757-21ssh22:28:370
7885997616,43cyclictest6366-21aten_rbpower_vo20:35:360
7885997612,52cyclictest0-21swapper/022:21:390
788599755,48cyclictest0-21swapper/021:05:490
7885997516,41cyclictest20617-21aten_rbpower_po19:45:360
788599746,54cyclictest18648-21diskmemload00:39:280
7885997411,44cyclictest19731-21ntpq21:10:450
9202730,3sleep37884-21cyclictest21:28:263
9202730,3sleep37884-21cyclictest21:28:263
7885997312,52cyclictest0-21swapper/019:11:460
7885997311,13cyclictest28928-21ls22:45:370
788799728,14cyclictest7950irq/56-dwc_otg_21:15:482
788599729,40cyclictest0-21swapper/000:07:230
788599728,54cyclictest0-21swapper/022:01:360
788599722,47cyclictest0-21swapper/020:00:450
7885997219,38cyclictest14017-21ssh23:49:010
7885997215,41cyclictest25492-21ssh22:00:220
7885997211,46cyclictest0-21swapper/021:43:530
7885997211,46cyclictest0-21swapper/021:43:530
285602720,2sleep20-21swapper/220:05:482
158432720,1sleep30-21swapper/321:46:513
788599719,50cyclictest0-21swapper/020:20:340
7885997111,42cyclictest0-21swapper/021:50:460
7885997110,46cyclictest0-21swapper/019:30:430
788599703,51cyclictest0-21swapper/023:50:520
7885997010,49cyclictest3272-21basename23:35:370
7885997010,49cyclictest3272-21basename23:35:370
788899692,5cyclictest7950irq/56-dwc_otg_20:40:523
788599699,51cyclictest0-21swapper/023:44:300
788599694,48cyclictest0-21swapper/021:31:280
7885996919,38cyclictest26712-21rm23:23:570
7885996914,39cyclictest23508-21sh22:38:260
7885996912,38cyclictest13677-21taskset20:55:370
7885996911,45cyclictest32301-21ssh21:26:180
7885996911,45cyclictest32301-21ssh21:26:180
788899683,5cyclictest7750irq/56-dwc_otg23:30:523
788799687,11cyclictest0-21swapper/221:35:592
788799682,8cyclictest7750irq/56-dwc_otg22:25:352
788799682,8cyclictest379-21ssh23:30:512
788599685,24cyclictest15926-21ssh23:10:280
7885996812,41cyclictest31656-21seq22:06:550
7885996812,41cyclictest1280-21rm00:14:170
788899679,11cyclictest23615-21sed21:15:443
7888996719,11cyclictest381ktimersoftd/323:24:023
788799677,8cyclictest7950irq/56-dwc_otg_22:45:522
788799662,15cyclictest0-21swapper/221:10:382
788799661,4cyclictest7850irq/56-dwc_otg_22:00:432
7885996618,29cyclictest509-21cron22:20:210
7885996618,29cyclictest509-21cron22:20:210
7885996614,38cyclictest1872-21ls22:10:430
788899651,5cyclictest0-21swapper/321:50:383
788799655,9cyclictest7950irq/56-dwc_otg_20:25:452
788799654,5cyclictest7850irq/56-dwc_otg_20:55:482
788799651,4cyclictest0-21swapper/223:15:442
7885996529,16cyclictest7850irq/56-dwc_otg_19:35:350
788599652,31cyclictest0-21swapper/020:10:370
788599652,31cyclictest0-21swapper/020:10:370
788899641,7cyclictest15626-21/usr/sbin/munin21:00:443
788899641,5cyclictest1276-21apt-get22:10:333
788799649,10cyclictest7950irq/56-dwc_otg_22:30:472
788799649,10cyclictest7950irq/56-dwc_otg_22:30:472
788799642,11cyclictest0-21swapper/221:24:402
788799641,16cyclictest820-21ssh22:50:432
788899632,15cyclictest0-21swapper/322:25:193
788899631,7cyclictest8363-21fschecks_time23:00:373
788899631,7cyclictest7950irq/56-dwc_otg_22:55:403
788899631,11cyclictest0-21swapper/323:35:463
788899631,11cyclictest0-21swapper/323:35:463
788799635,13cyclictest7950irq/56-dwc_otg_20:30:462
788799634,13cyclictest7950irq/56-dwc_otg_23:40:032
788799634,13cyclictest7950irq/56-dwc_otg_23:40:032
788799631,6cyclictest31885-21timerwakeupswit21:25:522
788799631,6cyclictest31885-21timerwakeupswit21:25:522
7885996310,34cyclictest0-21swapper/023:05:350
788899628,32cyclictest0-21swapper/322:00:283
788899627,11cyclictest0-21swapper/322:10:003
788899624,5cyclictest19434-21/usr/sbin/munin19:40:463
788899621,4cyclictest0-21swapper/300:06:443
788799624,5cyclictest11661-21/usr/sbin/munin20:50:472
788799623,5cyclictest7850irq/56-dwc_otg_00:20:352
788799621,6cyclictest6771-21latency20:35:422
788799621,4cyclictest0-21swapper/200:15:022
788799621,13cyclictest0-21swapper/223:28:062
788899611,7cyclictest8556-21ssh00:23:073
788899611,6cyclictest18648-21diskmemload22:34:523
788899611,6cyclictest18648-21diskmemload22:34:523
788899611,4cyclictest9721-21apt-get19:15:363
788799618,11cyclictest0-21swapper/223:21:542
788799615,10cyclictest7950irq/56-dwc_otg_00:00:412
788799612,4cyclictest0-21swapper/223:46:222
788799611,6cyclictest7850irq/56-dwc_otg_19:40:472
788799611,6cyclictest0-21swapper/200:00:222
7885996112,27cyclictest28754-21cat20:05:510
788899608,5cyclictest7850irq/56-dwc_otg_23:45:383
788899602,4cyclictest0-21swapper/300:01:593
788899601,14cyclictest7950irq/56-dwc_otg_21:33:563
788799608,11cyclictest7950irq/56-dwc_otg_20:45:472
788799606,11cyclictest7850irq/56-dwc_otg_23:00:382
788799605,20cyclictest7950irq/56-dwc_otg_22:40:352
788799601,6cyclictest14657-21memory21:45:432
7885996016,32cyclictest23835-21chrt19:54:420
7885996016,32cyclictest23835-21chrt19:54:420
281472600,8sleep3788899cyclictest23:25:413
788799599,35cyclictest0-21swapper/223:05:362
788799592,14cyclictest7950irq/56-dwc_otg_21:33:572
788799592,11cyclictest7950irq/56-dwc_otg_20:40:472
7887995910,4cyclictest0-21swapper/221:58:332
7886995918,17cyclictest18648-21diskmemload22:20:341
73172592,14sleep00-21swapper/019:06:350
788899585,4cyclictest0-21swapper/300:28:473
7888995817,7cyclictest7950irq/56-dwc_otg_22:29:213
7885995811,36cyclictest14625-21seq19:27:060
7885995810,40cyclictest25613-21taskset19:59:210
788899571,6cyclictest0-21swapper/300:20:073
788899571,4cyclictest19221-21ssh21:10:373
788799578,12cyclictest7950irq/56-dwc_otg_23:42:202
788799571,5cyclictest0-21swapper/223:14:092
788899564,11cyclictest7950irq/56-dwc_otg_22:15:543
788899564,11cyclictest7950irq/56-dwc_otg_22:15:543
788899563,4cyclictest0-21swapper/322:39:373
788899561,6cyclictest3089-21runrttasks20:25:443
788799564,12cyclictest7950irq/56-dwc_otg_00:37:212
788799563,5cyclictest7850irq/56-dwc_otg_20:20:442
788799561,20cyclictest0-21swapper/200:35:042
788899555,4cyclictest7950irq/56-dwc_otg_23:56:063
788899555,17cyclictest0-21swapper/323:05:303
788799551,5cyclictest10369-21diskstats21:40:382
788799551,5cyclictest10369-21diskstats21:40:382
788799551,10cyclictest0-21swapper/200:07:332
7886995517,23cyclictest8059-21ssh23:00:331
788899541,5cyclictest9519-21ssh23:42:533
7888995412,5cyclictest2332750ssh21:56:093
788899541,12cyclictest17928-21ssh23:54:023
788699549,34cyclictest0-21swapper/121:35:381
788899538,28cyclictest0-21swapper/320:30:283
788799535,10cyclictest7950irq/56-dwc_otg_21:54:152
788899527,41cyclictest7950irq/56-dwc_otg_21:25:223
788699524,8cyclictest0-21swapper/121:55:331
788699517,20cyclictest0-21swapper/100:30:381
788799491,7cyclictest0-21swapper/219:38:452
788699498,30cyclictest0-21swapper/119:50:361
788699498,30cyclictest0-21swapper/119:50:361
788699497,26cyclictest7111-21ssh23:40:281
7886994941,6cyclictest23-21ksoftirqd/121:45:331
7886994917,8cyclictest0-21swapper/120:15:351
788899489,6cyclictest0-21swapper/300:15:223
788699479,30cyclictest0-21swapper/122:15:351
788699479,30cyclictest0-21swapper/122:15:351
788699478,29cyclictest0-21swapper/120:30:291
788699478,28cyclictest0-21swapper/120:35:351
788699478,28cyclictest0-21swapper/120:05:361
7886994717,7cyclictest16840-21sh21:05:221
788699471,30cyclictest0-21swapper/119:30:351
7886994710,27cyclictest0-21swapper/123:55:361
7886994710,25cyclictest30769-21ssh21:25:341
7886994710,25cyclictest30769-21ssh21:25:341
80712460,2sleep20-21swapper/222:18:542
80712460,2sleep20-21swapper/222:18:542
788699469,28cyclictest0-21swapper/121:05:351
788699468,28cyclictest0-21swapper/122:30:351
788699468,28cyclictest0-21swapper/122:30:351
788699467,25cyclictest0-21swapper/122:35:341
214642460,4sleep321451-21munin-run00:40:213
207932460,2sleep10-21swapper/100:39:061
118092460,9sleep1221ktimersoftd/100:26:491
7888994517,5cyclictest20847-21fschecks_time19:45:393
788899451,12cyclictest7950irq/56-dwc_otg_22:45:413
788699458,26cyclictest0-21swapper/121:10:351
7886994519,6cyclictest30357-21latency_hist00:10:211
7886994518,7cyclictest29667-21latency_hist22:05:221
7886994516,7cyclictest1180-21runrttasks20:00:371
788699451,27cyclictest0-21swapper/120:10:371
788699451,27cyclictest0-21swapper/120:10:371
788899441,24cyclictest11910-21uniq20:50:383
788699448,28cyclictest0-21swapper/119:35:341
788699448,26cyclictest0-21swapper/122:50:351
788699448,25cyclictest0-21swapper/119:40:361
7886994421,8cyclictest3145-21/usr/sbin/munin23:35:491
7886994421,8cyclictest3145-21/usr/sbin/munin23:35:491
7886994421,21cyclictest19936-21ssh23:15:361
788799431,18cyclictest0-21swapper/220:04:132
788699439,18cyclictest0-21swapper/120:55:351
788699437,25cyclictest0-21swapper/121:15:361
788699436,26cyclictest4175-21ssh22:55:391
7886994324,5cyclictest13547-21ssh23:06:231
788799421,28cyclictest24717-21ls19:55:452
788799421,18cyclictest0-21swapper/219:14:012
788699425,22cyclictest7750irq/56-dwc_otg22:25:321
788699423,7cyclictest0-21swapper/119:10:351
7886994219,6cyclictest8509-21ssh00:23:011
788699421,7cyclictest0-21swapper/100:15:321
275992420,6sleep127598-21rm21:20:461
7888994118,6cyclictest76450wget20:20:373
7888994117,6cyclictest6522-21fschecks_time21:35:423
788899411,30cyclictest0-21swapper/321:41:303
788899411,30cyclictest0-21swapper/321:41:303
788799407,14cyclictest7950irq/56-dwc_otg_19:30:402
788799401,18cyclictest0-21swapper/219:54:502
788799401,18cyclictest0-21swapper/219:54:502
788699409,7cyclictest24862-21ssh23:20:491
788699408,6cyclictest7850irq/56-dwc_otg_20:20:531
788699408,28cyclictest0-21swapper/119:45:351
788699402,17cyclictest7850irq/56-dwc_otg_19:25:221
788899397,16cyclictest7950irq/56-dwc_otg_23:15:493
788899392,14cyclictest0-21swapper/319:31:063
7886993921,10cyclictest0-21swapper/123:50:351
7886993918,5cyclictest25040-21sh22:40:371
7886993911,17cyclictest0-21swapper/123:10:281
7888993821,6cyclictest108150ssh22:50:483
7888993817,11cyclictest25444-21ssh22:40:443
788899381,26cyclictest10880-21timerwakeupswit20:45:513
7887993819,10cyclictest301ktimersoftd/222:39:572
788799381,30cyclictest0-21swapper/200:25:332
788699388,18cyclictest0-21swapper/121:50:361
788699385,16cyclictest7850irq/56-dwc_otg_19:30:211
788699381,14cyclictest7950irq/56-dwc_otg_22:45:331
7886993810,17cyclictest1180-21runrttasks20:50:181
788899376,8cyclictest0-21swapper/320:19:293
788899374,5cyclictest0-21swapper/319:25:373
7888993713,11cyclictest22335-21/usr/sbin/munin19:50:523
7888993713,11cyclictest22335-21/usr/sbin/munin19:50:523
788899371,27cyclictest0-21swapper/319:20:413
788799377,5cyclictest7850irq/56-dwc_otg_22:10:442
788799374,19cyclictest7950irq/56-dwc_otg_21:05:432
788799373,25cyclictest7950irq/56-dwc_otg_23:50:362
788699377,16cyclictest0-21swapper/122:05:351
788699372,7cyclictest11209-21aten_rbpower_po23:45:371
788699372,14cyclictest31984-21ssh23:30:401
7886993719,5cyclictest22960-21aten_rbpower_po00:00:371
7886993718,6cyclictest11841-21sh21:41:401
7886993718,6cyclictest11841-21sh21:41:401
7886993716,6cyclictest0-21swapper/120:25:401
788899368,10cyclictest7950irq/56-dwc_otg_20:55:363
788799363,11cyclictest7850irq/56-dwc_otg_22:05:522
788799362,6cyclictest0-21swapper/219:15:492
7887993611,17cyclictest21102-21/usr/sbin/munin19:45:432
788699369,7cyclictest0-21swapper/121:30:341
788699363,5cyclictest0-21swapper/123:28:181
788699363,17cyclictest25214-21grep19:55:521
788699362,25cyclictest1755-21ssh22:10:421
7888993515,5cyclictest29106-21sshd20:07:573
788899351,28cyclictest25137-21ls19:55:523
788899351,11cyclictest0-21swapper/320:10:373
788899351,11cyclictest0-21swapper/320:10:373
788799359,16cyclictest0-21swapper/200:17:532
788799356,6cyclictest29-21rcuc/219:25:412
788799352,6cyclictest0-21swapper/219:20:512
788799351,25cyclictest0-21swapper/221:01:142
7887993512,11cyclictest11776-21ssh22:23:282
7886993516,5cyclictest9707-21/usr/sbin/munin19:15:491
788699344,7cyclictest0-21swapper/120:50:391
7888993310,7cyclictest7950irq/56-dwc_otg_19:13:103
788799335,7cyclictest7950irq/56-dwc_otg_20:10:412
788799335,7cyclictest7950irq/56-dwc_otg_20:10:412
788899321,5cyclictest39-21ksoftirqd/319:35:393
788899321,4cyclictest7850irq/56-dwc_otg_21:05:443
788899312,4cyclictest0-21swapper/320:36:183
788899311,5cyclictest0-21swapper/320:00:333
788699314,15cyclictest7950irq/56-dwc_otg_20:44:161
788799307,5cyclictest7750irq/56-dwc_otg20:15:372
66412304,15sleep10-21swapper/119:05:431
60682285,15sleep36070-21apt-get19:05:323
60562254,13sleep26057-21/usr/sbin/munin19:05:312
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional