You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 03:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Sun Aug 02, 2026 00:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1518921600,5sleep23257299cyclictest22:34:532
1405021150,6sleep33257399cyclictest22:32:453
32570991082,73cyclictest669-21grep22:15:580
32570991053,13cyclictest8050irq/56-dwc_otg_23:30:280
32570991024,62cyclictest0-21swapper/019:50:420
32570999837,19cyclictest8135-21uname00:25:470
32570999543,15cyclictest24690-21ls22:45:510
32570999512,77cyclictest9-21ksoftirqd/022:50:390
154642920,6sleep03257099cyclictest21:15:290
23602910,2sleep20-21swapper/200:18:542
187772900,2sleep00-21swapper/021:59:190
187772900,2sleep00-21swapper/021:59:190
54742890,10sleep05477-21latency_hist19:25:280
32570998636,15cyclictest101ktimersoftd/022:55:520
81102830,2sleep20-21swapper/221:00:462
32570998223,43cyclictest19748-21latency_hist00:40:280
302452820,2sleep20-21swapper/222:13:022
80042790,6sleep13257199cyclictest00:25:451
3257199767,45cyclictest7850irq/56-dwc_otg23:30:291
32572997332,18cyclictest7850irq/56-dwc_otg22:55:522
245022712,9sleep124599-21ls22:45:491
3257199686,47cyclictest0-21swapper/122:55:531
3257199673,14cyclictest0-21swapper/122:50:401
3257099666,5cyclictest0-21swapper/019:35:410
3257099661,50cyclictest3999-21fschecks_time23:40:450
3257099661,41cyclictest0-21swapper/023:50:380
3257199654,9cyclictest0-21swapper/122:15:571
3257099653,32cyclictest0-21swapper/019:10:290
3257399644,39cyclictest0-21swapper/322:45:513
3257399638,11cyclictest8252-21grep00:25:503
3257099639,22cyclictest0-21swapper/021:40:540
32570996334,13cyclictest7850irq/56-dwc_otg23:30:300
32572996216,18cyclictest7850irq/56-dwc_otg00:25:492
3257099621,29cyclictest7416-21latency_hist22:25:280
258382620,2sleep20-21swapper/223:27:102
3257399612,23cyclictest8050irq/56-dwc_otg_21:35:413
3257399612,23cyclictest8050irq/56-dwc_otg_21:35:413
32570996132,7cyclictest8050irq/56-dwc_otg_22:05:280
3257099612,8cyclictest24989-21tail22:05:570
3257399609,14cyclictest0-21swapper/323:30:283
32573996012,9cyclictest7950irq/56-dwc_otg_22:55:513
3257099609,33cyclictest23583-21modprobe23:25:270
32570996019,24cyclictest3698-21/usr/sbin/munin23:00:470
32570996019,24cyclictest3698-21/usr/sbin/munin23:00:470
3257099601,30cyclictest12161-21ssh21:50:470
3257399593,10cyclictest23361-21kworker/3:222:15:583
3257199598,29cyclictest0-21swapper/120:25:401
3257099593,30cyclictest31906-21ssh00:15:400
3257099592,9cyclictest15983-21wget22:35:440
3257099592,34cyclictest0-21swapper/021:40:280
3257099592,34cyclictest0-21swapper/021:40:280
3257099592,28cyclictest9018-21ssh23:05:580
32570995912,5cyclictest28364-21cut22:10:480
234412590,7sleep13257199cyclictest23:25:111
144842590,2sleep10-21swapper/123:13:271
32573995822,18cyclictest39-21ksoftirqd/322:50:403
32570995832,5cyclictest7950irq/56-dwc_otg_20:10:580
32570995736,5cyclictest7950irq/56-dwc_otg_22:30:540
3257099572,29cyclictest24524-21ps21:25:520
32570995721,5cyclictest30801-21cron20:35:280
32570995721,5cyclictest30801-21cron20:35:280
32570995717,24cyclictest101ktimersoftd/021:10:440
32570995717,23cyclictest23767-21latency_hist22:45:280
32570995717,23cyclictest23767-21latency_hist22:45:280
32570995714,30cyclictest9-21ksoftirqd/000:10:560
3257399563,26cyclictest27811-21apt-key23:30:313
3257099563,22cyclictest12665-21/usr/sbin/munin23:10:540
3257099561,33cyclictest2548-21fschecks_time20:45:460
3257099561,16cyclictest0-21swapper/023:55:470
3257099551,35cyclictest3419-21sed23:40:280
3257099551,28cyclictest25968-21fschecks_time20:20:450
3257099551,26cyclictest32205-21kworker/u8:021:30:460
3257399548,29cyclictest0-21swapper/320:25:403
32572995418,14cyclictest8050irq/56-dwc_otg_23:30:302
32572995410,22cyclictest29-21rcuc/222:50:422
3257199546,30cyclictest3569-21ssh22:20:411
3257099549,22cyclictest0-21swapper/023:50:280
3257099543,28cyclictest0-21swapper/021:45:540
3257099542,8cyclictest20489-21ssh00:00:520
3257099542,6cyclictest15963-21runrttasks23:15:430
32570995414,6cyclictest13-21rcuc/019:30:280
3257399538,29cyclictest0-21swapper/322:20:413
32570995319,6cyclictest27528-21wget20:25:410
3257199527,26cyclictest7950irq/56-dwc_otg_23:30:301
3257099525,30cyclictest17331-21chrt19:55:510
3257099524,32cyclictest0-21swapper/019:30:320
32570995232,5cyclictest8050irq/56-dwc_otg_20:10:270
3257099522,24cyclictest4773-21sed00:20:580
3257099521,24cyclictest12954-21unixbench_multi00:30:560
32570995210,16cyclictest0-21swapper/000:05:450
3257399518,27cyclictest0-21swapper/319:30:333
32572995112,15cyclictest0-21swapper/222:45:502
32570995112,19cyclictest19971-21munin-plugin-st20:05:270
263002500,2sleep226301-21/usr/sbin/munin20:20:492
3257299493,8cyclictest7850irq/56-dwc_otg19:10:292
32570994924,11cyclictest7850irq/56-dwc_otg21:20:450
32570994916,14cyclictest3681-21latency_hist19:20:280
32570994911,7cyclictest9550irq/48-DMA15:26:470
3257199488,28cyclictest0-21swapper/123:55:421
3257199487,26cyclictest0-21swapper/119:30:331
3257199482,28cyclictest32569-21cyclictest19:10:291
32570994815,5cyclictest518-21rs:main0
3257399477,25cyclictest0-21swapper/300:35:423
3257399477,23cyclictest0-21swapper/320:45:423
3257199477,25cyclictest3524-21ssh00:20:411
3257199476,25cyclictest0-21swapper/120:50:411
3257199472,5cyclictest0-21swapper/121:15:341
32570994710,20cyclictest0-21swapper/019:45:410
32570994710,20cyclictest0-21swapper/019:45:410
3257199467,30cyclictest7950irq/56-dwc_otg_21:55:471
3257199467,30cyclictest7950irq/56-dwc_otg_21:55:471
3257199465,29cyclictest19818-21ssh00:00:421
3257099461,5cyclictest1128-21ntpq20:40:520
3257099461,24cyclictest24420-21/usr/sbin/munin20:15:510
321682452,36sleep00-21swapper/019:08:450
321682452,36sleep00-21swapper/019:08:450
321652451,40sleep20-21swapper/222:15:452
3257399444,12cyclictest32574-21sshd19:10:293
32573994417,18cyclictest0-21swapper/323:35:563
3257399441,35cyclictest0-21swapper/321:55:283
3257199449,26cyclictest15984-21grep22:35:441
3257199441,8cyclictest8050irq/56-dwc_otg_19:20:411
3257399437,23cyclictest0-21swapper/320:50:413
3257399437,21cyclictest0-21swapper/319:20:273
3257399432,23cyclictest0-21swapper/323:45:543
3257299432,6cyclictest8050irq/56-dwc_otg_21:40:402
32571994317,7cyclictest3802-21fschecks_time21:40:451
3257199431,5cyclictest15959-21wget23:15:431
32571994311,16cyclictest8401-21/usr/sbin/munin21:45:541
3257099431,24cyclictest7277-21apt-get22:25:290
32573994223,10cyclictest17706-21ssh21:17:333
3257399421,29cyclictest4708-21ls19:20:553
3257299426,6cyclictest8050irq/56-dwc_otg_22:40:472
3257299426,6cyclictest8050irq/56-dwc_otg_22:40:472
3257299421,4cyclictest7950irq/56-dwc_otg_23:05:582
3257199426,25cyclictest31697-21ssh00:15:281
3257199421,6cyclictest12563-21ssh22:30:511
32570994217,10cyclictest0-21swapper/020:37:380
3257099421,26cyclictest0-21swapper/020:50:400
32573994120,11cyclictest18730-21ssh22:39:003
32573994117,16cyclictest3220-21kworker/u8:423:13:563
32573994116,6cyclictest20421-21sh23:20:523
32573994116,5cyclictest19990-21fschecks_time21:20:453
3257299415,22cyclictest7850irq/56-dwc_otg23:24:392
3257299413,8cyclictest0-21swapper/220:55:392
3257299412,7cyclictest7850irq/56-dwc_otg23:35:462
3257199416,23cyclictest7850irq/56-dwc_otg23:40:541
3257199415,22cyclictest0-21swapper/121:55:281
32571994117,16cyclictest3220-21kworker/u8:423:50:431
3257099412,20cyclictest0-21swapper/019:40:510
215562410,2sleep30-21swapper/322:01:523
3257399406,21cyclictest15666-21ls00:35:283
3257399402,15cyclictest0-21swapper/319:25:393
32573994015,5cyclictest23874-21aten_rbpower_po22:05:423
3257299409,11cyclictest12278-21latency23:50:482
3257299401,18cyclictest8050irq/56-dwc_otg_00:05:472
3257199406,25cyclictest23742-21aten_rbpower_en21:25:411
3257199405,19cyclictest3815-21ssh23:00:421
3257199405,19cyclictest3815-21ssh23:00:421
3257199402,13cyclictest0-21swapper/120:30:401
3257199402,13cyclictest0-21swapper/120:30:401
32571994017,5cyclictest11022-21diskmemload00:35:421
20842400,2sleep10-21swapper/120:45:271
118592400,2sleep211839-21ssh00:30:422
32573993918,4cyclictest6969-21sh23:04:353
32573993918,4cyclictest6969-21sh23:04:353
3257399391,16cyclictest0-21swapper/300:15:563
3257299392,9cyclictest0-21swapper/221:10:402
3257299392,10cyclictest11343-21ssh23:50:272
3257299391,25cyclictest8008-21ls19:30:462
3257199397,5cyclictest32709-21missed_timers23:35:511
3257199395,7cyclictest0-21swapper/120:55:381
3257199393,22cyclictest0-21swapper/123:45:481
3257199392,29cyclictest0-21swapper/121:35:411
3257199392,29cyclictest0-21swapper/121:35:411
3257199392,27cyclictest7850irq/56-dwc_otg21:10:421
32571993912,16cyclictest27303-21latency_hist20:25:281
3257099390,23cyclictest8009-21idleruntime21:00:450
3257399384,6cyclictest39-21ksoftirqd/320:40:413
3257399382,27cyclictest0-21swapper/323:57:033
32573993818,5cyclictest727-21runrttasks23:17:213
3257299387,23cyclictest0-21swapper/220:15:442
3257199386,6cyclictest24954-21grep22:05:571
3257199383,26cyclictest0-21swapper/121:31:221
3257199381,23cyclictest24613-21ls00:05:551
3257399378,16cyclictest0-21swapper/321:25:323
3257399377,17cyclictest0-21swapper/300:20:413
32573993717,12cyclictest10244-21sh21:48:343
32573993716,5cyclictest27886-21sh00:10:403
3257399371,11cyclictest824-1kworker/u9:222:25:463
3257299378,8cyclictest31-21ksoftirqd/223:15:522
3257299377,25cyclictest0-21swapper/219:45:322
3257299377,25cyclictest0-21swapper/219:45:322
3257299376,27cyclictest0-21swapper/221:25:402
3257299375,14cyclictest7950irq/56-dwc_otg_22:20:452
3257299374,20cyclictest19778-21/usr/sbin/munin00:00:582
3257299372,21cyclictest7850irq/56-dwc_otg21:20:572
32572993716,4cyclictest7697-21/usr/sbin/munin22:25:472
32572993710,14cyclictest8050irq/56-dwc_otg_21:15:302
3257199378,17cyclictest7950irq/56-dwc_otg_22:10:501
3257199377,20cyclictest0-21swapper/100:18:381
3257199372,26cyclictest7850irq/56-dwc_otg23:05:551
3257199371,23cyclictest0-21swapper/122:40:551
3257199371,23cyclictest0-21swapper/122:40:551
3257199371,22cyclictest2162-21ls19:15:431
3257199371,19cyclictest11669-21grep19:40:501
323812370,5sleep10-21swapper/119:10:271
323812370,5sleep10-21swapper/119:10:271
3257399362,20cyclictest0-21swapper/320:30:533
3257399362,20cyclictest0-21swapper/320:30:533
3257399361,5cyclictest15456-21munin-run23:55:273
3257399361,21cyclictest0-21swapper/322:10:513
3257299367,4cyclictest8050irq/56-dwc_otg_00:20:522
3257299363,6cyclictest7950irq/56-dwc_otg_19:15:392
3257299362,21cyclictest0-21swapper/221:30:472
32572993616,5cyclictest31-21ksoftirqd/219:25:422
32572993612,7cyclictest8050irq/56-dwc_otg_23:00:452
32572993612,7cyclictest8050irq/56-dwc_otg_23:00:452
3257199366,15cyclictest0-21swapper/119:35:421
3257199362,9cyclictest7850irq/56-dwc_otg21:20:501
227422361,29sleep20-21swapper/220:10:562
3257399352,23cyclictest8050irq/56-dwc_otg_21:55:413
3257399352,23cyclictest8050irq/56-dwc_otg_21:55:413
32573993515,5cyclictest727-21runrttasks19:36:443
32573993514,4cyclictest20793-21sensors22:40:543
32573993514,4cyclictest20793-21sensors22:40:543
32573993513,4cyclictest25683-21aten_rbpower_en20:20:413
32573993512,4cyclictest7950irq/56-dwc_otg_23:08:513
32573993512,4cyclictest11048-21latency_hist21:10:283
3257299356,14cyclictest7950irq/56-dwc_otg_21:35:432
3257299356,14cyclictest7950irq/56-dwc_otg_21:35:432
3257299352,19cyclictest0-21swapper/220:25:462
3257299352,16cyclictest31-21ksoftirqd/200:13:472
32572993513,5cyclictest31-21ksoftirqd/200:40:282
3257299351,22cyclictest23973-21ls22:05:432
3257299351,20cyclictest0-21swapper/219:50:422
32572993510,14cyclictest8050irq/56-dwc_otg_22:35:522
3257199356,19cyclictest7950irq/56-dwc_otg_22:25:471
3257199355,16cyclictest0-21swapper/119:45:411
3257199355,16cyclictest0-21swapper/119:45:411
3257199352,22cyclictest7850irq/56-dwc_otg19:25:491
3257199351,7cyclictest16466-21latency_hist19:55:271
3257199351,23cyclictest0-21swapper/122:02:541
32571993512,14cyclictest11090-21latency_hist21:10:281
3257399344,14cyclictest0-21swapper/321:10:583
32573993425,5cyclictest3998-21fschecks_time23:40:453
3257399341,21cyclictest13042-21/usr/sbin/munin19:45:413
3257399341,21cyclictest13042-21/usr/sbin/munin19:45:413
3257399341,21cyclictest0-21swapper/300:07:443
3257299348,5cyclictest31-21ksoftirqd/219:55:442
3257299346,7cyclictest7850irq/56-dwc_otg21:45:562
3257299344,20cyclictest2474-21grep20:45:462
32572993429,3cyclictest301ktimersoftd/222:00:392
3257299341,4cyclictest8050irq/56-dwc_otg_20:30:422
3257299341,4cyclictest8050irq/56-dwc_otg_20:30:422
3257399336,19cyclictest0-21swapper/321:31:223
3257399332,2cyclictest0-21swapper/319:50:413
32573993311,5cyclictest20139-21fschecks_time00:00:473
32573993311,5cyclictest18395-21/usr/sbin/munin20:00:463
32573993310,7cyclictest3975-21kernelversion21:40:473
3257299339,5cyclictest7850irq/56-dwc_otg19:20:452
3257299338,9cyclictest8050irq/56-dwc_otg_23:42:442
3257299337,9cyclictest7950irq/56-dwc_otg_19:35:562
3257299337,21cyclictest0-21swapper/221:50:402
3257299332,21cyclictest0-21swapper/223:13:412
3257299331,6cyclictest31-21ksoftirqd/220:40:272
3257199334,4cyclictest7850irq/56-dwc_otg20:10:451
32571993322,9cyclictest8050irq/56-dwc_otg_00:30:401
3257199331,19cyclictest0-21swapper/120:45:421
3257399325,16cyclictest6109-21cut20:55:443
32573993218,7cyclictest381ktimersoftd/320:17:163
3257299327,22cyclictest0-21swapper/221:05:412
3257299322,6cyclictest16035-21fschecks_time21:55:472
3257299322,6cyclictest16035-21fschecks_time21:55:472
3257299321,29cyclictest19998-21ls20:05:272
32572993210,5cyclictest0-21swapper/223:57:172
3257199326,17cyclictest7850irq/56-dwc_otg19:55:481
3257199326,13cyclictest19966-21latency_hist20:05:271
3257199324,18cyclictest7950irq/56-dwc_otg_21:00:441
3257199322,14cyclictest0-21swapper/120:35:391
3257199321,21cyclictest21954-21latency_hist20:10:281
3257399319,11cyclictest23085-21unixbench-2d20:10:583
32573993113,6cyclictest7973-21fschecks_time21:00:443
3257299317,14cyclictest0-21swapper/219:40:422
3257199312,5cyclictest0-21swapper/120:15:421
3257399308,5cyclictest11628-21sed19:40:493
3257399306,4cyclictest0-21swapper/320:05:553
3257299306,8cyclictest4356-21grep20:50:452
32572993021,6cyclictest29-21rcuc/220:40:412
3257299301,5cyclictest8050irq/56-dwc_otg_20:06:492
3257399295,8cyclictest381ktimersoftd/320:00:173
3257399272,15cyclictest0-21swapper/320:36:193
320302274,15sleep30-21swapper/319:06:583
320302274,15sleep30-21swapper/319:06:583
322692263,14sleep20-21swapper/219:10:052
322692263,14sleep20-21swapper/219:10:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional