You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-04 - 22:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Tue Nov 04, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
309442980,3sleep31289999cyclictest10:08:223
211202930,2sleep30-21swapper/309:54:533
87282850,2sleep00-21swapper/010:22:400
1289699817,24cyclictest0-21swapper/007:40:420
29392750,2sleep3381ktimersoftd/310:15:303
4962740,5sleep11289799cyclictest09:23:511
244182730,2sleep20-21swapper/211:32:472
165402730,2sleep30-21swapper/307:20:403
295562720,2sleep30-21swapper/309:17:583
104272710,1sleep3381ktimersoftd/310:25:573
1289699691,32cyclictest0-21swapper/011:10:390
25402680,2sleep10-21swapper/111:47:311
12896996646,10cyclictest326512chrt12:30:500
12896996638,4cyclictest0-21swapper/009:15:380
12896996636,17cyclictest7950irq/56-dwc_otg_10:55:270
233072650,2sleep30-21swapper/311:30:553
12896996550,7cyclictest101ktimersoftd/011:55:420
1289699652,51cyclictest0-21swapper/009:00:270
12896996518,27cyclictest0-21swapper/010:25:410
12896996510,34cyclictest0-21swapper/009:35:460
1289699641,44cyclictest0-21swapper/008:20:570
1289699636,25cyclictest0-21swapper/011:50:450
28332620,2sleep00-21swapper/010:15:270
1289699621,48cyclictest0-21swapper/009:15:280
1289699612,48cyclictest0-21swapper/008:00:460
1289699611,48cyclictest0-21swapper/007:35:450
1289699601,42cyclictest14487-21tr12:05:390
1289699594,37cyclictest0-21swapper/007:10:410
1289699593,5cyclictest0-21swapper/010:35:450
1289699593,42cyclictest0-21swapper/012:10:550
1289699581,40cyclictest19477-21grep09:51:000
1289699576,38cyclictest0-21swapper/010:55:560
1289699576,38cyclictest0-21swapper/010:55:560
12896995732,4cyclictest5005-21cpuspeed09:30:440
12896995725,6cyclictest8050irq/56-dwc_otg_09:25:390
1289699572,41cyclictest0-21swapper/007:55:270
12896995722,6cyclictest4296-21wget08:15:440
1289699571,39cyclictest15513-21fschecks_time11:20:460
1289699562,3cyclictest0-21swapper/011:15:410
1289699542,27cyclictest26963-21unixbench_multi11:35:580
12896995415,5cyclictest0-21swapper/012:15:410
12896995415,5cyclictest0-21swapper/012:15:410
12898995319,3cyclictest7850irq/56-dwc_otg11:25:432
12896995337,4cyclictest101ktimersoftd/008:10:560
1289699532,9cyclictest18719-21latency_hist08:55:280
1289699532,37cyclictest0-21swapper/007:45:490
1289699532,37cyclictest0-21swapper/007:45:490
12896995322,3cyclictest0-21swapper/009:22:150
1289699521,39cyclictest1300-21wc08:05:540
1289699521,39cyclictest1300-21wc08:05:540
12896995127,5cyclictest7950irq/56-dwc_otg_11:35:270
1289699512,5cyclictest4333-21unixbench_singl12:35:550
1289699512,28cyclictest0-21swapper/012:25:550
1289699511,39cyclictest11190-21rm12:00:420
12896995024,6cyclictest7950irq/56-dwc_otg_09:40:550
1289699502,36cyclictest15477-21ps07:15:510
1289699492,33cyclictest19917-21sed11:25:580
1289699492,26cyclictest3360-21fschecks_time10:15:450
1289699492,25cyclictest20704-21aten_rbpower_cu10:40:430
1289699491,5cyclictest24549-21/usr/sbin/munin12:20:530
12896994913,9cyclictest0-21swapper/011:44:280
12896994913,9cyclictest0-21swapper/011:44:280
1289699491,12cyclictest19978-21ls07:30:270
1289799483,34cyclictest2109750ssh12:15:411
1289799483,34cyclictest2109750ssh12:15:411
1289699484,9cyclictest0-21swapper/008:40:400
12896994841,4cyclictest8050irq/56-dwc_otg_11:50:270
1289699482,30cyclictest13-21rcuc/010:05:410
12896994817,6cyclictest8050irq/56-dwc_otg_09:05:280
1289699481,29cyclictest0-21swapper/009:05:570
124342483,14sleep00-21swapper/007:08:050
1289899477,25cyclictest0-21swapper/208:35:422
1289899477,25cyclictest0-21swapper/208:35:422
1289699478,6cyclictest17652-21unixbench_multi07:21:000
1289699478,25cyclictest0-21swapper/009:55:430
1289699478,25cyclictest0-21swapper/009:55:430
1289699476,6cyclictest15937-21tail09:45:570
1289699471,26cyclictest7881-21acpi08:25:380
12896994712,32cyclictest101ktimersoftd/011:00:460
1289699471,22cyclictest0-21swapper/010:45:490
1289999456,25cyclictest0-21swapper/308:35:423
1289999456,25cyclictest0-21swapper/308:35:423
1289899456,24cyclictest4051-21ssh11:50:332
1289799452,18cyclictest0-21swapper/110:00:401
1289699452,31cyclictest0-21swapper/008:30:430
1289699451,5cyclictest0-21swapper/008:45:400
1289899447,25cyclictest0-21swapper/208:00:422
1289899447,24cyclictest0-21swapper/210:40:432
1289899444,4cyclictest0-21swapper/208:55:392
1289899442,4cyclictest8050irq/56-dwc_otg_10:15:382
1289699449,22cyclictest0-21swapper/011:05:310
1289699441,17cyclictest25591-21fschecks_time10:00:440
1289999437,25cyclictest0-21swapper/308:20:413
1289999433,8cyclictest0-21swapper/309:40:393
1289899434,9cyclictest7950irq/56-dwc_otg_10:45:422
1289699437,24cyclictest0-21swapper/007:30:410
12896994316,10cyclictest14264-21ntpq10:30:530
1289999427,22cyclictest0-21swapper/309:35:423
1289999416,25cyclictest0-21swapper/308:50:333
1289899417,28cyclictest0-21swapper/208:30:432
1289899417,25cyclictest0-21swapper/208:20:412
12898994113,20cyclictest16631-21ssh12:08:512
1289799417,22cyclictest0-21swapper/108:55:421
12897994114,23cyclictest0-21swapper/107:20:431
1289699414,22cyclictest0-21swapper/007:58:270
44812400,1sleep30-21swapper/308:15:473
1289999407,24cyclictest0-21swapper/308:25:413
12898994019,12cyclictest27735-21/usr/sbin/munin10:50:422
1289799407,22cyclictest0-21swapper/111:05:411
1289799407,22cyclictest0-21swapper/107:30:411
1289799407,20cyclictest0-21swapper/107:40:421
12897994012,15cyclictest13374-21wget08:40:441
12896994014,13cyclictest0-21swapper/008:35:570
12896994014,13cyclictest0-21swapper/008:35:570
1289999397,24cyclictest0-21swapper/311:45:333
1289999396,20cyclictest0-21swapper/309:05:443
1289999391,25cyclictest16491-21ls07:20:273
1289899398,20cyclictest0-21swapper/210:10:412
1289899397,19cyclictest0-21swapper/210:20:452
1289899394,33cyclictest7950irq/56-dwc_otg_08:15:322
12897993917,5cyclictest32226-21aten_rbpower_po10:10:421
12897993917,4cyclictest20809-21sh09:54:101
12897993915,5cyclictest13207-21latency_hist10:30:281
12897993915,4cyclictest21207-21latency10:40:511
12897993913,4cyclictest0-21swapper/111:00:481
1289999383,24cyclictest0-21swapper/311:36:343
12899993814,5cyclictest7850irq/56-dwc_otg10:05:273
1289899387,24cyclictest0-21swapper/208:50:332
1289899385,5cyclictest7950irq/56-dwc_otg_07:51:022
1289899381,29cyclictest2142-21ls08:10:272
1289899381,29cyclictest2142-21ls08:10:272
1289899381,25cyclictest16892-21ls10:35:282
12898993812,4cyclictest1868-21grep11:45:572
1289799385,21cyclictest7850irq/56-dwc_otg09:05:441
1289799381,19cyclictest0-21swapper/108:45:501
12897993811,21cyclictest11-21rcu_preempt09:25:321
12897993810,4cyclictest0-21swapper/110:34:491
123022380,7sleep21289899cyclictest11:15:522
1289999377,22cyclictest7950irq/56-dwc_otg_09:50:273
1289999371,26cyclictest14196-21ls08:40:563
1289899373,16cyclictest1121-21ssh09:25:272
1289899372,5cyclictest7950irq/56-dwc_otg_07:25:382
1289899371,6cyclictest8050irq/56-dwc_otg_09:35:412
1289799376,7cyclictest8050irq/56-dwc_otg_11:20:391
1289799375,22cyclictest0-21swapper/110:50:461
12897993717,5cyclictest978050wget08:30:441
12897993715,13cyclictest18658-21apt-get11:25:401
1289999364,24cyclictest7950irq/56-dwc_otg_07:45:483
1289999364,24cyclictest7950irq/56-dwc_otg_07:45:483
1289999363,26cyclictest0-21swapper/307:42:573
12899993615,6cyclictest28496-21fschecks_time12:25:473
1289899366,7cyclictest7850irq/56-dwc_otg09:05:492
1289899364,8cyclictest6598-21grep11:05:572
1289899364,18cyclictest8050irq/56-dwc_otg_09:15:562
1289899362,5cyclictest28504-21ssh12:25:472
12898993622,4cyclictest15195-21if_wlan009:45:472
1289899361,15cyclictest0-21swapper/210:05:462
1289799367,22cyclictest7850irq/56-dwc_otg12:00:491
1289799364,28cyclictest8050irq/56-dwc_otg_12:35:391
1289799362,5cyclictest0-21swapper/107:55:521
12897993612,18cyclictest0-21swapper/111:11:011
1289799360,17cyclictest21961-21ssh09:55:461
1289799360,17cyclictest21961-21ssh09:55:461
47342351,4sleep30-21swapper/311:04:473
1289999356,19cyclictest7950irq/56-dwc_otg_12:25:113
1289999355,22cyclictest7950irq/56-dwc_otg_11:25:473
1289999354,4cyclictest0-21swapper/312:00:403
12899993518,5cyclictest20223-21aten_rbpower_po07:30:423
1289899359,5cyclictest31064-21/usr/sbin/munin10:55:502
1289899359,5cyclictest31064-21/usr/sbin/munin10:55:502
1289899352,5cyclictest8050irq/56-dwc_otg_12:10:402
1289899351,5cyclictest26353-21latency11:35:502
1289899351,30cyclictest7815-21ls09:35:272
1289799356,22cyclictest0-21swapper/107:10:411
1289799356,21cyclictest0-21swapper/112:10:401
1289799351,8cyclictest31493-21tune2fs08:00:461
12897993516,11cyclictest8079-21ssh11:55:461
1289799351,25cyclictest24684-21cut09:10:481
314292340,4sleep330608-21/usr/sbin/munin09:20:513
187502340,1sleep20-21swapper/210:36:552
1289999348,5cyclictest7850irq/56-dwc_otg11:20:543
1289999342,21cyclictest0-21swapper/312:35:423
12899993420,11cyclictest8050irq/56-dwc_otg_09:30:183
12899993413,5cyclictest7850irq/56-dwc_otg07:10:413
1289999341,22cyclictest7950irq/56-dwc_otg_07:30:273
1289999341,22cyclictest0-21swapper/311:10:283
1289899347,15cyclictest0-21swapper/207:35:412
1289899343,24cyclictest8050irq/56-dwc_otg_12:00:382
1289899342,5cyclictest1140-21snmpd12:00:062
12898993415,5cyclictest18231-21/usr/sbin/munin09:50:532
1289899341,25cyclictest15465-21ls08:45:472
1289899341,20cyclictest14967-21cut07:15:442
1289899341,15cyclictest7950irq/56-dwc_otg_11:05:272
1289799349,15cyclictest15470-21ntpq09:45:511
1289799345,5cyclictest25866-21ssh11:35:441
12897993417,9cyclictest25336-21ntp_states12:20:541
12897993414,4cyclictest4872-21ssh09:30:431
1289799341,26cyclictest26583-21ls07:45:541
1289799341,26cyclictest26583-21ls07:45:541
1289799341,23cyclictest0-21swapper/112:08:381
1289799341,23cyclictest0-21swapper/109:35:421
1289799341,23cyclictest0-21swapper/107:15:411
12897993412,10cyclictest7378-21sh11:55:271
1289799341,17cyclictest7950irq/56-dwc_otg_11:35:271
133202331,4sleep20-21swapper/208:40:422
1289999334,4cyclictest5539-21ssh11:51:183
1289999333,22cyclictest4750irq/17-3f00b88010:50:523
12899993317,10cyclictest27139-21ssh09:15:273
12899993316,6cyclictest7950irq/56-dwc_otg_10:30:493
12899993312,13cyclictest7850irq/56-dwc_otg10:24:473
1289999331,16cyclictest7950irq/56-dwc_otg_11:15:273
1289899336,5cyclictest7850irq/56-dwc_otg11:45:272
1289899336,5cyclictest7850irq/56-dwc_otg11:45:272
1289899331,13cyclictest7950irq/56-dwc_otg_08:00:272
1289799336,17cyclictest7850irq/56-dwc_otg12:30:431
1289799332,17cyclictest8050irq/56-dwc_otg_11:15:501
1289799332,10cyclictest0-21swapper/109:15:551
1289799331,23cyclictest13178-21ls08:40:281
1289799331,23cyclictest13178-21ls08:40:281
12897993311,10cyclictest0-21swapper/110:55:531
12897993311,10cyclictest0-21swapper/110:55:531
1289999323,22cyclictest7950irq/56-dwc_otg_12:30:553
12899993225,4cyclictest7850irq/56-dwc_otg12:15:413
12899993225,4cyclictest7850irq/56-dwc_otg12:15:413
1289999322,15cyclictest5171-21fschecks_time09:30:473
1289899326,16cyclictest0-21swapper/209:25:332
1289899325,13cyclictest7950irq/56-dwc_otg_12:20:442
1289899323,14cyclictest7950irq/56-dwc_otg_08:30:272
1289899321,6cyclictest2195350ssh09:55:462
1289899321,6cyclictest2195350ssh09:55:462
12898993216,5cyclictest864050wget11:10:422
12898993216,5cyclictest22964-21sh12:17:202
12898993216,5cyclictest22964-21sh12:17:202
1289899321,19cyclictest14520-21ls07:15:272
1289899321,13cyclictest7850irq/56-dwc_otg12:35:392
1289799328,15cyclictest7850irq/56-dwc_otg11:40:441
1289799328,15cyclictest7850irq/56-dwc_otg11:40:441
1289799327,13cyclictest7755-21latency_hist08:25:281
1289799327,12cyclictest3008-21apt-get10:15:381
1289799325,17cyclictest7850irq/56-dwc_otg09:44:391
1289799323,21cyclictest7850irq/56-dwc_otg12:27:091
1289799323,21cyclictest0-21swapper/109:01:561
1289799321,5cyclictest7950irq/56-dwc_otg_08:15:441
12897993211,4cyclictest0-21swapper/110:22:251
1289799321,14cyclictest0-21swapper/108:50:541
1289999315,14cyclictest8050irq/56-dwc_otg_08:15:273
1289999314,19cyclictest7950irq/56-dwc_otg_07:40:273
12899993112,5cyclictest10182-21latency08:30:503
1289899316,5cyclictest7850irq/56-dwc_otg09:00:562
1289899315,5cyclictest8050irq/56-dwc_otg_10:26:102
1289899313,5cyclictest32457-21latency12:30:482
1289899311,5cyclictest7850irq/56-dwc_otg07:45:412
1289899311,5cyclictest7850irq/56-dwc_otg07:45:412
1289899311,4cyclictest7850irq/56-dwc_otg09:10:482
1289799317,5cyclictest8050irq/56-dwc_otg_07:40:271
1289799316,15cyclictest7850irq/56-dwc_otg10:35:581
1289799313,17cyclictest28759-21egrep10:05:441
1289999309,10cyclictest7950irq/56-dwc_otg_10:10:453
1289999307,4cyclictest0-21swapper/312:13:093
1289999305,4cyclictest7850irq/56-dwc_otg10:55:563
1289999305,4cyclictest7850irq/56-dwc_otg10:55:563
1289999303,17cyclictest26993-21ssh10:49:313
1289999302,7cyclictest14777-21ssh11:20:193
1289999302,6cyclictest7950irq/56-dwc_otg_10:40:413
1289999302,13cyclictest8050irq/56-dwc_otg_09:00:453
12899993015,6cyclictest19794-21ssh10:39:133
12899993014,6cyclictest7950irq/56-dwc_otg_07:52:433
12899993014,5cyclictest7950irq/56-dwc_otg_07:55:443
1289999301,14cyclictest31465-21fschecks_time08:00:463
12899993010,4cyclictest0-21swapper/312:05:593
1289899302,22cyclictest0-21swapper/211:23:132
12898993014,8cyclictest14091-21ssh09:44:452
1289799309,9cyclictest0-21swapper/110:49:091
1289799309,11cyclictest1303-21wc08:05:541
1289799309,11cyclictest1303-21wc08:05:541
1289799302,7cyclictest7950irq/56-dwc_otg_08:10:561
125982303,21sleep30-21swapper/307:10:063
1289999299,4cyclictest0-21swapper/311:58:013
1289999295,15cyclictest0-21swapper/308:05:483
1289999295,15cyclictest0-21swapper/308:05:483
1289999292,19cyclictest7950irq/56-dwc_otg_08:55:483
12899992915,5cyclictest24691-21ssh09:59:533
12899992915,5cyclictest24691-21ssh09:59:533
1289999291,21cyclictest0-21swapper/308:45:433
1289899295,13cyclictest8050irq/56-dwc_otg_07:40:542
1289899291,15cyclictest0-21swapper/210:01:362
1289799294,9cyclictest8050irq/56-dwc_otg_07:50:591
1289999282,14cyclictest8050irq/56-dwc_otg_11:40:483
1289999282,14cyclictest8050irq/56-dwc_otg_11:40:483
1289899285,4cyclictest7850irq/56-dwc_otg08:10:452
1289899283,5cyclictest8050irq/56-dwc_otg_07:20:572
1289799281,4cyclictest7850irq/56-dwc_otg08:25:391
1289799281,20cyclictest0-21swapper/107:25:471
1289899271,13cyclictest8050irq/56-dwc_otg_07:30:572
117822277,11sleep10-21swapper/107:05:551
110842255,13sleep211081-21apt-get07:05:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional