You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 09:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri Nov 07, 2025 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1581825490,5sleep1221ktimersoftd/119:05:401
1631522570,3sleep20-21swapper/219:05:492
176439912028,67cyclictest17642-21cyclictest19:10:310
204982960,2sleep31764699cyclictest20:45:513
71792890,5sleep31764699cyclictest22:48:543
17644998129,28cyclictest7850irq/56-dwc_otg19:25:431
1764499775,15cyclictest0-21swapper/119:10:311
17643997660,10cyclictest101ktimersoftd/000:19:130
17643997625,26cyclictest0-21swapper/023:35:430
17643997625,26cyclictest0-21swapper/023:35:430
17643997611,49cyclictest22040-21grep23:50:580
1764399716,45cyclictest0-21swapper/021:30:420
1764399696,43cyclictest0-21swapper/020:30:450
1764399696,43cyclictest0-21swapper/020:30:450
17643996928,5cyclictest0-21swapper/022:15:420
17643996928,5cyclictest0-21swapper/022:15:420
1764399691,55cyclictest0-21swapper/020:55:280
218382680,1sleep20-21swapper/223:08:342
218382680,1sleep20-21swapper/223:08:342
1764399685,51cyclictest32249-21cut19:50:430
1764399684,53cyclictest25911-21unixbench_multi23:55:580
1764399684,53cyclictest0-21swapper/020:20:570
1764399673,41cyclictest0-21swapper/021:50:420
17643996712,49cyclictest101ktimersoftd/020:25:550
17643996712,44cyclictest30731-21fschecks_time19:45:460
1764499667,13cyclictest0-21swapper/123:40:411
1764499663,8cyclictest0-21swapper/100:10:441
1764399664,52cyclictest0-21swapper/021:42:540
1764499656,12cyclictest8050irq/56-dwc_otg_00:35:431
1764399656,29cyclictest0-21swapper/019:30:450
17643996511,42cyclictest0-21swapper/022:35:420
1764499643,14cyclictest7850irq/56-dwc_otg22:55:481
1764399647,41cyclictest6246-21munin-plugin-st22:05:280
1764399647,24cyclictest16733-21ssh23:45:280
1764399643,49cyclictest0-21swapper/021:20:550
1764399643,44cyclictest2937-21sed23:25:580
1764399642,3cyclictest0-21swapper/022:45:420
17643996414,39cyclictest30797-21fschecks_count23:20:470
17643996410,34cyclictest20668-21unixbench_multi19:15:550
17643996410,34cyclictest20668-21unixbench_multi19:15:550
168782640,2sleep10-21swapper/100:28:121
177442630,2sleep30-21swapper/319:10:433
1764399634,47cyclictest0-21swapper/023:00:460
1764399626,39cyclictest0-21swapper/020:05:460
17643996247,8cyclictest8050irq/56-dwc_otg_22:10:400
1764399623,48cyclictest0-21swapper/021:48:220
1764399621,34cyclictest16291-21aten_rbpower_en20:35:450
17643996114,37cyclictest1111-21runrttasks22:50:440
1764399604,41cyclictest811-21aten_rbpower_en22:40:420
1764399603,40cyclictest23589-21ssh23:10:490
1764399602,42cyclictest12230-21fschecks_count22:55:470
17643996010,38cyclictest12565-21runrttasks00:21:430
1764399595,46cyclictest28327-21rm23:16:310
1764399594,45cyclictest0-21swapper/023:45:450
1764399587,41cyclictest0-21swapper/000:35:540
1764399581,33cyclictest0-21swapper/022:30:400
17643995743,9cyclictest8050irq/56-dwc_otg_00:30:450
17643995741,5cyclictest15713-21ssh21:35:370
17643995740,10cyclictest0-21swapper/023:09:130
17643995740,10cyclictest0-21swapper/023:09:130
17643995736,11cyclictest7950irq/56-dwc_otg_19:25:280
1764399571,8cyclictest101ktimersoftd/000:10:280
17643995636,5cyclictest24341-21unixbench_singl19:26:000
17643995630,6cyclictest0-21swapper/020:20:280
17643995627,18cyclictest0-21swapper/021:15:250
1764399562,41cyclictest4562-21grep20:00:570
1764399557,32cyclictest4298-21tr00:10:530
1764399554,34cyclictest2916-21runrttasks19:56:290
17643995535,5cyclictest8050irq/56-dwc_otg_21:00:280
17643995522,14cyclictest28779-21aten_rbpower_vo00:00:480
28542540,2sleep10-21swapper/100:09:411
1764499542,14cyclictest0-21swapper/121:05:401
17644995411,9cyclictest8050irq/56-dwc_otg_23:55:421
17643995437,11cyclictest0-21swapper/023:34:160
17643995433,8cyclictest8050irq/56-dwc_otg_21:55:400
1764399541,28cyclictest15744-21ssh00:25:590
170232543,12sleep00-21swapper/019:06:040
1764399537,23cyclictest406-21aten_rbpower_po21:15:450
17643995326,6cyclictest7132-21grep20:10:420
1764699524,5cyclictest7950irq/56-dwc_otg_22:55:413
17643995233,12cyclictest24077-21ssh22:27:460
17643995229,8cyclictest0-21swapper/021:25:410
17643995221,7cyclictest28371-21modprobe19:40:280
17643995110,38cyclictest101ktimersoftd/022:05:310
1764499502,8cyclictest0-21swapper/100:15:461
17644995019,6cyclictest14182-21latency_hist20:30:291
17644995010,27cyclictest675-21rm22:40:341
1764399502,5cyclictest0-21swapper/020:40:440
17643995012,6cyclictest18202-21ssh22:20:420
1764499492,15cyclictest25253-21taskset21:00:421
1764699485,11cyclictest0-21swapper/319:25:433
1764499486,8cyclictest0-21swapper/123:50:421
1764499477,27cyclictest26810-21ssh23:15:431
1764499477,26cyclictest0-21swapper/120:20:431
17643994722,5cyclictest8050irq/56-dwc_otg_21:05:280
17643994712,15cyclictest20887-21grep20:45:570
17643994710,19cyclictest29716-21tail19:40:560
199212460,5sleep11764499cyclictest00:31:591
17646994626,5cyclictest2370-21latency_hist22:00:293
1764599466,5cyclictest0-21swapper/222:10:402
17645994616,11cyclictest31-21ksoftirqd/200:10:492
1764499467,25cyclictest0-21swapper/119:35:431
1764499466,6cyclictest8050irq/56-dwc_otg_23:10:421
1764499466,25cyclictest0-21swapper/122:45:441
1764499465,26cyclictest14340-21ssh22:15:421
1764499465,26cyclictest14340-21ssh22:15:421
1764699457,24cyclictest0-21swapper/320:35:443
17645994514,16cyclictest0-21swapper/219:10:312
1764499456,25cyclictest19392-21runrttasks23:05:421
1764499456,25cyclictest19392-21runrttasks23:05:421
1764699447,7cyclictest7950irq/56-dwc_otg_23:55:413
1764699446,24cyclictest28965-21ssh21:10:423
1764699441,6cyclictest0-21swapper/320:20:413
1764599447,23cyclictest0-21swapper/222:25:422
1764499447,25cyclictest0-21swapper/120:35:341
1764499447,24cyclictest0-21swapper/122:50:421
1764499442,22cyclictest0-21swapper/120:40:471
17644994417,6cyclictest22030-21aten_rbpower_po22:25:421
1764399443,7cyclictest0-21swapper/021:05:560
1764699435,6cyclictest8050irq/56-dwc_otg_22:40:403
1764599438,23cyclictest0-21swapper/221:10:422
1764599437,23cyclictest0-21swapper/220:35:442
17645994322,6cyclictest31-21ksoftirqd/223:50:542
1764499434,24cyclictest0-21swapper/122:35:441
1764699429,6cyclictest7850irq/56-dwc_otg22:20:473
1764499428,22cyclictest0-21swapper/119:40:421
1764499427,25cyclictest0-21swapper/122:30:421
1764499422,9cyclictest7950irq/56-dwc_otg_21:55:391
1764499421,9cyclictest0-21swapper/121:35:401
1764599417,24cyclictest0-21swapper/220:15:452
17645994119,4cyclictest29-21rcuc/220:50:522
17645994112,5cyclictest31-21ksoftirqd/200:10:282
17644994119,5cyclictest29458-21/usr/sbin/munin00:00:581
47182400,1sleep30-21swapper/323:29:413
1764699407,24cyclictest0-21swapper/321:45:463
17646994014,9cyclictest0-21swapper/322:50:573
1764599408,12cyclictest29-21rcuc/200:35:542
17645994024,9cyclictest13227-21wget23:40:442
17645994019,5cyclictest1491-21latency_hist19:55:292
17645994012,5cyclictest7850irq/56-dwc_otg23:15:482
1764499406,21cyclictest15650-21ssh23:00:411
1764499404,14cyclictest7950irq/56-dwc_otg_20:10:571
1764499403,6cyclictest7950irq/56-dwc_otg_23:45:421
1764699397,22cyclictest0-21swapper/320:15:453
17644993912,13cyclictest8050irq/56-dwc_otg_21:25:281
1764699387,22cyclictest0-21swapper/322:25:423
1764699386,7cyclictest7950irq/56-dwc_otg_19:15:463
1764699386,7cyclictest7950irq/56-dwc_otg_19:15:463
1764699381,6cyclictest7850irq/56-dwc_otg00:21:003
1764699381,25cyclictest755-21ls19:50:573
1764699381,23cyclictest18015-21ls22:20:293
1764699381,23cyclictest18015-21ls22:20:293
17646993810,19cyclictest0-21swapper/323:39:593
17646993810,19cyclictest0-21swapper/323:39:593
1764599388,18cyclictest11-21rcu_preempt23:20:412
1764599386,12cyclictest29-21rcuc/219:15:492
1764599386,12cyclictest29-21rcuc/219:15:492
17645993818,6cyclictest31-21ksoftirqd/220:25:512
17645993815,10cyclictest9254-21/usr/sbin/munin23:35:412
17645993815,10cyclictest9254-21/usr/sbin/munin23:35:412
17645993812,12cyclictest301ktimersoftd/221:40:312
1764599381,11cyclictest0-21swapper/200:30:182
1764499385,13cyclictest7950irq/56-dwc_otg_21:50:511
1764499381,7cyclictest8050irq/56-dwc_otg_21:45:431
17644993812,17cyclictest7330-21sh22:05:521
17646993722,7cyclictest381ktimersoftd/321:30:423
1764599378,4cyclictest0-21swapper/221:50:562
1764599377,11cyclictest24458-21sh21:45:592
1764599376,15cyclictest31-21ksoftirqd/222:35:472
1764599374,25cyclictest17642-21cyclictest23:00:402
1764599372,26cyclictest0-21swapper/221:55:412
17645993713,14cyclictest31-21ksoftirqd/223:25:492
1764599371,25cyclictest18365-21grep20:40:502
17645993712,17cyclictest29-21rcuc/220:20:412
1764499376,7cyclictest10975-21ssh00:20:441
1764499373,10cyclictest0-21swapper/119:50:411
1764499371,18cyclictest21142-21rm21:41:241
17646993612,14cyclictest25681-21taskset23:13:483
1764599365,18cyclictest31-21ksoftirqd/222:33:162
17645993617,5cyclictest28456-21seq21:08:382
17645993612,7cyclictest0-21swapper/200:00:432
17644993611,12cyclictest3212-21rm22:00:501
1764699356,6cyclictest7850irq/56-dwc_otg00:05:563
1764699351,19cyclictest0-21swapper/300:31:013
1764599359,5cyclictest31-21ksoftirqd/222:50:572
1764599355,20cyclictest7850irq/56-dwc_otg19:23:052
1764599354,19cyclictest7850irq/56-dwc_otg21:31:062
1764599353,5cyclictest0-21swapper/220:10:402
1764599353,19cyclictest7850irq/56-dwc_otg22:00:372
1764599352,6cyclictest0-21swapper/220:00:582
17645993518,10cyclictest2447-21sh22:41:492
1764599351,7cyclictest140-21systemd-journal19:40:282
17645993515,7cyclictest7950irq/56-dwc_otg_00:24:542
17645993512,11cyclictest301ktimersoftd/221:15:442
1764499359,13cyclictest8050irq/56-dwc_otg_21:30:281
1764499354,18cyclictest8050irq/56-dwc_otg_23:30:341
1764499352,2cyclictest0-21swapper/122:20:411
1764699347,6cyclictest30497-21wget19:45:433
1764699346,4cyclictest7950irq/56-dwc_otg_22:40:293
1764699346,19cyclictest8050irq/56-dwc_otg_00:16:003
1764699346,17cyclictest0-21swapper/320:55:433
1764699345,23cyclictest28549-21ssh00:00:353
17646993418,4cyclictest39-21ksoftirqd/300:35:473
17646993413,4cyclictest5038-21cron20:05:283
1764699341,24cyclictest7950irq/56-dwc_otg_21:05:563
1764699341,24cyclictest27971-21missed_timers21:50:513
17646993410,5cyclictest4766-21unixbench_multi00:10:593
1764599343,22cyclictest301ktimersoftd/222:55:412
1764599342,9cyclictest7950irq/56-dwc_otg_23:45:472
17645993418,4cyclictest4782-21cat21:20:492
17645993415,10cyclictest301ktimersoftd/223:12:242
1764499349,13cyclictest7850irq/56-dwc_otg19:15:301
1764499349,13cyclictest7850irq/56-dwc_otg19:15:301
1764499347,5cyclictest7950irq/56-dwc_otg_20:50:491
17644993421,11cyclictest367650ssh21:20:071
1764499341,20cyclictest2218-21ssh23:25:491
1764699339,7cyclictest0-21swapper/322:05:503
17646993317,4cyclictest30223-21latency_hist23:20:283
1764699331,6cyclictest8050irq/56-dwc_otg_20:30:453
1764699331,6cyclictest8050irq/56-dwc_otg_20:30:453
1764699331,6cyclictest7850irq/56-dwc_otg23:50:523
17646993314,4cyclictest1111-21runrttasks21:38:533
1764699331,3cyclictest0-21swapper/321:15:433
1764699331,22cyclictest20037-21missed_timers23:05:513
1764699331,22cyclictest20037-21missed_timers23:05:513
1764699331,15cyclictest0-21swapper/323:40:333
1764599335,15cyclictest14340-21ssh22:15:422
1764599335,15cyclictest14340-21ssh22:15:422
1764599332,6cyclictest4561-21wget22:45:442
1764499337,16cyclictest0-21swapper/122:10:421
1764499335,19cyclictest19822-21/usr/sbin/munin20:45:511
17644993317,8cyclictest6600-21resolvconf20:07:271
1764499331,18cyclictest15427-21munin-plugin-st21:35:281
118092330,2sleep30-21swapper/322:11:063
1764699326,5cyclictest7950irq/56-dwc_otg_00:25:513
1764699323,6cyclictest7850irq/56-dwc_otg21:20:473
1764699321,3cyclictest0-21swapper/322:33:503
1764599329,12cyclictest29-21rcuc/219:58:532
1764599327,11cyclictest7950irq/56-dwc_otg_20:07:172
1764599323,14cyclictest8050irq/56-dwc_otg_21:40:292
17645993218,4cyclictest29-21rcuc/221:04:392
17645993214,9cyclictest30825-21iwlist19:45:482
17645993210,5cyclictest0-21swapper/200:35:292
1764499328,4cyclictest0-21swapper/123:35:371
1764499328,4cyclictest0-21swapper/123:35:371
1764499327,15cyclictest0-21swapper/123:20:431
1764499327,15cyclictest0-21swapper/119:20:431
1764499324,5cyclictest21-21rcuc/121:10:471
1764499322,8cyclictest23-21ksoftirqd/119:30:571
1764499321,10cyclictest9192-21fschecks_time20:15:491
170262323,23sleep30-21swapper/319:06:073
1764699314,18cyclictest0-21swapper/323:05:293
17646993121,7cyclictest26809-21idleruntime-cro21:05:283
17646993114,5cyclictest21844-21interrupts19:20:493
17646993113,11cyclictest14259-21cat20:30:293
1764699311,2cyclictest0-21swapper/319:30:433
1764699311,26cyclictest8964-21ls23:35:293
1764699311,19cyclictest0-21swapper/319:40:403
1764599315,5cyclictest31-21ksoftirqd/223:31:312
1764599313,11cyclictest301ktimersoftd/200:19:152
17645993124,5cyclictest31-21ksoftirqd/221:25:422
1764599311,7cyclictest25887-21ssh23:55:582
1764599311,6cyclictest10114-21munin-run22:10:282
1764599311,15cyclictest31-21ksoftirqd/220:34:092
1764599311,15cyclictest31-21ksoftirqd/220:34:092
17645993110,4cyclictest29017-21if_eth019:40:472
1764499317,22cyclictest0-21swapper/120:00:421
1764499311,4cyclictest7850irq/56-dwc_otg19:55:471
1764499311,15cyclictest8050irq/56-dwc_otg_20:35:281
1764499311,15cyclictest8050irq/56-dwc_otg_20:35:281
1764599305,14cyclictest7950irq/56-dwc_otg_20:45:572
17645993023,5cyclictest31-21ksoftirqd/222:20:492
1764599301,8cyclictest25034-21aten_rbpower_cu19:30:452
1764499307,11cyclictest8050irq/56-dwc_otg_20:55:461
1764499301,17cyclictest31422-21timerwakeupswit19:45:561
1764699297,10cyclictest8050irq/56-dwc_otg_23:45:503
1764699297,10cyclictest7850irq/56-dwc_otg21:25:423
1764699296,5cyclictest0-21swapper/321:40:583
1764699295,12cyclictest8050irq/56-dwc_otg_23:21:403
1764699292,5cyclictest8050irq/56-dwc_otg_20:40:553
1764699291,10cyclictest4960-21rm22:02:563
1764599291,21cyclictest0-21swapper/220:59:092
1764599287,10cyclictest8050irq/56-dwc_otg_19:25:592
1764699273,5cyclictest8050irq/56-dwc_otg_19:56:003
1764699273,16cyclictest5576-21tune2fs20:05:503
1764699273,10cyclictest8050irq/56-dwc_otg_20:50:593
1764699272,8cyclictest0-21swapper/319:35:553
17646992720,5cyclictest8661-21cron20:15:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional