You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-24 - 05:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Jul 24, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
595423140,1sleep00-21swapper/019:05:350
2067921270,2sleep1768899cyclictest23:10:371
76879911915,87cyclictest18579-21diskmemload21:25:230
7687991152,88cyclictest8050irq/56-dwc_otg_22:50:530
7687991152,88cyclictest8050irq/56-dwc_otg_22:50:530
7687991079,77cyclictest0-21swapper/021:10:460
7687991074,15cyclictest4038-21sh22:50:250
7687999345,18cyclictest0-21swapper/023:25:490
768899912,64cyclictest19582-21wc21:10:471
7687999144,22cyclictest0-21swapper/000:25:480
768799897,43cyclictest28172-21apt-config22:40:250
768899882,79cyclictest24802-21sort21:55:511
7687998847,25cyclictest17294-21grep00:25:240
768799884,63cyclictest18921-21kworker/0:021:55:520
768899873,57cyclictest18211-21ntp_states00:25:481
768899863,54cyclictest8050irq/56-dwc_otg_22:50:541
768899863,54cyclictest8050irq/56-dwc_otg_22:50:541
768899863,51cyclictest0-21swapper/122:40:251
768899853,75cyclictest17272-21ssh00:25:241
768799837,47cyclictest5228-21ssh00:10:400
226202830,6sleep2768999cyclictest21:54:152
7689998236,27cyclictest31-21ksoftirqd/221:10:452
768899822,73cyclictest1682-21ls23:25:511
768899822,72cyclictest31021-21latency_hist21:25:231
768799822,63cyclictest0-21swapper/022:00:440
768899811,64cyclictest15352-21ssh23:43:031
768899811,64cyclictest15352-21ssh23:43:031
768799819,12cyclictest0-21swapper/022:40:260
768799813,60cyclictest0-21swapper/021:30:360
768899802,71cyclictest5230-21df_inode00:10:401
768899795,67cyclictest16127-21apt-get23:05:271
293422790,5sleep229335-21ssh00:40:222
212042790,5sleep1768899cyclictest19:45:481
768899781,60cyclictest30362-21ls20:10:491
768799787,40cyclictest0-21swapper/023:55:360
7687997851,8cyclictest7950irq/56-dwc_otg_23:05:300
768799782,63cyclictest0-21swapper/023:40:550
768799782,63cyclictest0-21swapper/023:40:550
768899762,68cyclictest31174-21ssh21:25:251
768799766,49cyclictest0-21swapper/020:25:270
768799762,54cyclictest0-21swapper/020:45:530
768799762,54cyclictest0-21swapper/020:45:530
768999752,44cyclictest1583-21ssh23:25:502
768899753,65cyclictest3853-21latency_hist22:50:231
768899741,53cyclictest18556-21apt-get19:40:361
7690997321,27cyclictest0-21swapper/322:50:243
7689997326,31cyclictest5256-21forks00:10:422
768899733,65cyclictest28303-21gpg-connect-age22:40:261
768899731,57cyclictest18872-21ssh22:28:451
768999725,26cyclictest29-21rcuc/222:40:252
768899721,49cyclictest25018-21ssh23:15:381
7687997217,46cyclictest9-21ksoftirqd/023:45:360
768999719,45cyclictest0-21swapper/221:25:222
22892710,1sleep30-21swapper/321:29:483
7690997020,26cyclictest19574-21memory21:10:473
768999705,26cyclictest0-21swapper/200:25:232
7689997010,26cyclictest29-21rcuc/222:50:542
7689997010,26cyclictest29-21rcuc/222:50:542
768899703,49cyclictest9025-21ssh22:55:471
768899701,49cyclictest12467-21ssh23:40:161
768899701,45cyclictest17703-21ssh23:45:451
768799703,55cyclictest0-21swapper/019:45:360
768899691,48cyclictest15723-21ssh23:04:411
8102680,2sleep20-21swapper/222:45:452
768899681,49cyclictest23999-21rm23:53:541
768899681,45cyclictest9456-21ssh21:37:511
768799685,50cyclictest0-21swapper/021:45:540
768799684,51cyclictest0-21swapper/022:30:520
213102680,4sleep221301-21ssh23:50:412
769099673,54cyclictest0-21swapper/300:25:473
768899671,53cyclictest7247-21ssh22:15:061
768899671,53cyclictest24721-21ssh00:34:261
768899671,53cyclictest24721-21ssh00:34:261
768899671,51cyclictest9716-21df19:15:381
768899671,48cyclictest24415-21ssh21:15:581
768899671,46cyclictest7645-21ssh22:15:251
768899671,46cyclictest29237-21ssh00:40:111
768799674,48cyclictest9-21ksoftirqd/023:20:420
768799673,53cyclictest0-21swapper/019:10:370
7687996728,10cyclictest101ktimersoftd/023:15:230
768799672,53cyclictest0-21swapper/019:30:500
768799672,52cyclictest0-21swapper/020:50:470
7687996714,38cyclictest0-21swapper/023:15:420
769099662,43cyclictest24912-21unixbench_multi21:55:533
768899661,46cyclictest11949-21ssh00:18:291
768899661,42cyclictest32482-21ssh00:04:291
768899661,42cyclictest32482-21ssh00:04:291
768999653,11cyclictest587850kworker/2:000:25:522
768899652,47cyclictest12938-21timerandwakeup22:20:521
768899651,47cyclictest13890-21ssh21:43:161
768899651,44cyclictest31622-21ssh23:23:321
768899651,42cyclictest7832-21latency_hist20:40:241
7687996542,15cyclictest101ktimersoftd/020:15:490
7687996525,8cyclictest0-21swapper/019:50:350
768799652,15cyclictest8050irq/56-dwc_otg_21:25:250
768899641,46cyclictest18579-21diskmemload00:07:141
768799648,44cyclictest0-21swapper/022:10:370
768799646,41cyclictest0-21swapper/021:15:510
768899632,36cyclictest2465-21/usr/sbin/munin20:25:331
768899631,34cyclictest0-21swapper/120:05:411
768799632,51cyclictest0-21swapper/022:26:290
768799632,35cyclictest29211-21ssh00:00:330
768799632,35cyclictest29211-21ssh00:00:330
768899622,35cyclictest0-21swapper/121:46:271
768899621,44cyclictest23282-21ssh22:34:171
768899621,43cyclictest6437-21ssh23:31:471
768899621,42cyclictest29382-21ssh22:01:241
768799623,48cyclictest0-21swapper/021:45:220
768799621,49cyclictest10812-21latency_hist21:40:230
768999613,38cyclictest0-21swapper/221:55:522
768899611,40cyclictest3097-21ssh22:10:011
768799613,49cyclictest9-21ksoftirqd/023:50:470
768799612,44cyclictest4654-21sh00:10:220
768799611,45cyclictest27337-21idleruntime-cro20:05:230
7690996033,22cyclictest31046-21latency_hist21:25:233
769099603,25cyclictest0-21swapper/323:25:503
768899607,18cyclictest0-21swapper/120:55:271
768899601,39cyclictest28358-21ssh23:59:251
768899601,38cyclictest21375-21ssh21:51:561
7690995912,31cyclictest8050irq/56-dwc_otg_22:50:543
7690995912,31cyclictest8050irq/56-dwc_otg_22:50:543
7689995930,10cyclictest14866-21kworker/2:221:25:252
768799599,40cyclictest9-21ksoftirqd/021:10:230
7687995918,26cyclictest32111-21aten_rbpower_po22:05:380
769099586,26cyclictest7950irq/56-dwc_otg_22:40:243
768899581,43cyclictest7888-21/usr/sbin/munin20:40:541
768899581,43cyclictest23600-21taskset19:53:581
768899581,42cyclictest6557-21ssh21:35:071
768899581,38cyclictest0-21swapper/119:10:331
768799582,5cyclictest12549-21munin-run23:40:230
768799581,40cyclictest0-21swapper/020:35:330
769099573,38cyclictest15182-21kworker/3:100:25:233
7690995726,24cyclictest5242-21diskstats00:10:423
7690995722,19cyclictest28264-21ssh22:40:263
769099572,10cyclictest23068-21ssh21:15:363
7690995720,25cyclictest30022-21timerwakeupswit23:20:523
768799576,41cyclictest25533-21latency_hist20:00:230
768999566,31cyclictest0-21swapper/223:05:272
768899564,33cyclictest0-21swapper/120:02:371
768799562,28cyclictest0-21swapper/000:35:230
768799562,28cyclictest0-21swapper/000:35:230
7687995610,9cyclictest101ktimersoftd/022:20:430
7690995526,23cyclictest16485-21rm23:05:283
768799551,42cyclictest15263-21aten_rbpower_en21:00:370
768799551,32cyclictest4388-21aten_rbpower_vo20:30:360
7690995322,13cyclictest19321-21rm23:47:523
768899532,47cyclictest25038-21grep19:55:521
768899532,38cyclictest0-21swapper/120:48:491
768899532,38cyclictest0-21swapper/120:48:491
768899532,37cyclictest0-21swapper/120:32:271
768899531,48cyclictest15569-21ls19:30:431
768899531,36cyclictest7850irq/56-dwc_otg19:40:231
768799533,27cyclictest0-21swapper/022:15:450
7687995328,5cyclictest1303-21proc_pri20:20:450
7687995326,5cyclictest8050irq/56-dwc_otg_23:05:220
768899521,48cyclictest15204-21apt-get21:00:361
768899521,47cyclictest31203-21/usr/sbin/munin20:15:351
768899521,36cyclictest16992-21runrttasks21:05:311
768799522,5cyclictest0-21swapper/023:00:220
768799521,29cyclictest29360-21/usr/sbin/munin20:10:510
768799521,25cyclictest20422-21ssh21:50:500
7690995113,15cyclictest24557-21latency_hist23:15:243
768799512,28cyclictest18941-21runrttasks19:40:410
769099504,10cyclictest2937-21ssh21:30:343
768899501,46cyclictest957-21if_eth020:20:391
768899501,46cyclictest13655-21/usr/sbin/munin19:25:431
768899501,46cyclictest13655-21/usr/sbin/munin19:25:431
768799501,37cyclictest0-21swapper/020:55:420
7688994922,24cyclictest13183-21latency_hist20:55:231
768899491,45cyclictest11448-21grep19:20:391
7687994924,6cyclictest0-21swapper/020:45:230
768799492,28cyclictest0-21swapper/019:35:440
768799482,9cyclictest0-21swapper/000:15:470
7687994812,28cyclictest9-21ksoftirqd/023:34:220
769099475,5cyclictest8050irq/56-dwc_otg_23:35:383
769099473,4cyclictest7850irq/56-dwc_otg00:05:373
7690994724,11cyclictest8050irq/56-dwc_otg_00:01:513
7690994724,11cyclictest8050irq/56-dwc_otg_00:01:513
769099471,38cyclictest0-21swapper/320:35:363
768999478,19cyclictest28282-21modprobe22:40:262
7689994719,6cyclictest8423-21missed_timers22:15:472
7687994729,4cyclictest9851-21tune2fs19:15:410
7687994712,19cyclictest0-21swapper/020:05:530
768999465,3cyclictest0-21swapper/222:10:232
7687994626,3cyclictest0-21swapper/019:25:230
769099459,6cyclictest19486-21ssh21:50:353
769099455,7cyclictest0-21swapper/323:00:353
7690994526,8cyclictest8050irq/56-dwc_otg_22:34:393
768799451,3cyclictest0-21swapper/019:25:480
768799451,3cyclictest0-21swapper/019:25:480
769099443,36cyclictest0-21swapper/320:15:333
7690994420,9cyclictest18556-21apt-get19:40:343
7690994411,18cyclictest8050irq/56-dwc_otg_20:40:393
7690994411,14cyclictest7850irq/56-dwc_otg00:15:293
768999446,24cyclictest0-21swapper/220:15:352
7689994419,8cyclictest7950irq/56-dwc_otg_22:20:352
768999441,30cyclictest0-21swapper/200:00:362
768999441,30cyclictest0-21swapper/200:00:362
768799442,26cyclictest0-21swapper/000:39:100
769099435,17cyclictest11982-21aten_rbpower_po22:20:383
769099434,24cyclictest10184-21rm22:17:553
7690994325,10cyclictest7950irq/56-dwc_otg_23:53:373
769099432,23cyclictest18945-21ssh22:28:523
769099432,14cyclictest0-21swapper/322:00:353
7690994312,13cyclictest9971-21irqstats19:15:433
7690994311,20cyclictest381ktimersoftd/322:12:413
768999437,24cyclictest0-21swapper/223:00:372
769099425,20cyclictest24666-21apt-config23:15:243
768999427,23cyclictest0-21swapper/221:05:382
768999424,10cyclictest0-21swapper/222:55:342
768999422,8cyclictest7950irq/56-dwc_otg_23:15:232
86912410,1sleep20-21swapper/223:35:312
7689994114,8cyclictest0-21swapper/220:45:502
7689994114,8cyclictest0-21swapper/220:45:502
769099405,23cyclictest25125-21ssh23:55:363
769099402,15cyclictest21479-21grep19:45:513
7690994017,13cyclictest8050irq/56-dwc_otg_00:40:143
768999404,12cyclictest31-21ksoftirqd/221:40:372
7689994019,5cyclictest31-21ksoftirqd/221:40:232
769099399,5cyclictest8050irq/56-dwc_otg_19:35:233
7690993922,9cyclictest8050irq/56-dwc_otg_21:46:283
7689993912,7cyclictest15993-21grep19:30:492
769099388,6cyclictest7850irq/56-dwc_otg00:30:393
769099388,6cyclictest7850irq/56-dwc_otg00:30:393
769099387,23cyclictest0-21swapper/320:25:283
769099387,22cyclictest0-21swapper/321:35:363
769099384,17cyclictest8050irq/56-dwc_otg_20:05:233
769099384,13cyclictest11393-21grep21:40:393
7690993827,4cyclictest8050irq/56-dwc_otg_19:25:503
7690993827,4cyclictest8050irq/56-dwc_otg_19:25:503
769099381,26cyclictest13969-21missed_timers20:55:443
769099381,22cyclictest24160-21ls19:55:383
7690993810,18cyclictest8049-21ssh23:34:443
768999382,5cyclictest7950irq/56-dwc_otg_21:30:382
768999382,4cyclictest8050irq/56-dwc_otg_23:30:362
768999381,10cyclictest25435-21fschecks_time23:55:402
769099377,22cyclictest0-21swapper/319:50:373
769099377,16cyclictest0-21swapper/320:30:353
769099377,12cyclictest15185-21ssh23:42:453
769099377,12cyclictest15185-21ssh23:42:453
7690993718,10cyclictest8050irq/56-dwc_otg_19:10:343
768999377,17cyclictest0-21swapper/223:20:372
768999371,5cyclictest0-21swapper/219:50:352
768999371,21cyclictest13842-21ssh23:40:522
768999371,21cyclictest13842-21ssh23:40:522
769099366,22cyclictest0-21swapper/320:50:363
769099364,13cyclictest16515-21chrt21:02:263
768999367,19cyclictest0-21swapper/200:15:382
768999362,5cyclictest0-21swapper/223:20:242
768999361,26cyclictest28595-21ls20:05:512
769099356,16cyclictest11118-21sshd20:48:473
769099356,16cyclictest11118-21sshd20:48:473
7690993515,7cyclictest8050irq/56-dwc_otg_22:05:413
769099351,13cyclictest30532-21unixbench_singl20:10:503
768999355,21cyclictest7950irq/56-dwc_otg_00:10:222
768999351,7cyclictest21370-21grep22:30:522
7689993516,5cyclictest31-21ksoftirqd/219:45:482
768999351,22cyclictest23614-21ssh21:15:452
769099348,21cyclictest12242-21cat23:00:233
7690993415,6cyclictest17903-21unixbench_singl19:35:503
768999346,14cyclictest0-21swapper/219:35:362
7689993414,5cyclictest954350wget19:15:372
768999341,20cyclictest0-21swapper/219:55:372
769099337,10cyclictest8050irq/56-dwc_otg_21:06:273
769099331,3cyclictest8050irq/56-dwc_otg_19:20:373
768999339,16cyclictest516-21in:imuxsock22:00:472
768999337,14cyclictest0-21swapper/219:20:382
768999336,12cyclictest7950irq/56-dwc_otg_00:33:182
768999336,12cyclictest7950irq/56-dwc_otg_00:33:182
768999335,6cyclictest18193-21unixbench_singl23:45:502
768999334,18cyclictest29-21rcuc/220:54:312
768999331,25cyclictest3864-21ls22:10:382
768999331,24cyclictest31-21ksoftirqd/221:49:062
768999327,15cyclictest0-21swapper/220:40:372
768999326,16cyclictest31-21ksoftirqd/220:10:502
7689993218,4cyclictest25763-21/usr/sbin/munin20:00:462
768999321,17cyclictest18239-21runrttasks22:27:332
68552323,14sleep16948-21/usr/sbin/munin19:05:501
769099312,24cyclictest8050irq/56-dwc_otg_20:20:313
769099311,27cyclictest27886-21grep20:05:403
768999315,16cyclictest0-21swapper/219:40:352
768999314,9cyclictest16771-21munin-run21:05:222
768999312,10cyclictest0-21swapper/220:35:332
768999301,4cyclictest8050irq/56-dwc_otg_19:30:242
768999301,4cyclictest8050irq/56-dwc_otg_19:30:242
768999301,19cyclictest0-21swapper/220:25:282
768999298,5cyclictest8402-21ntp_offset19:10:452
768999296,4cyclictest7850irq/56-dwc_otg20:30:362
768999284,11cyclictest1653-21grep20:20:482
768999271,14cyclictest0-21swapper/220:56:462
72262274,14sleep30-21swapper/319:07:583
72202263,14sleep20-21swapper/219:07:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional