You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 09:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Wed Nov 26, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4890992825,2cyclictest34-21ksoftirqd/322:00:003
488599280,1cyclictest16823-21latency_hist23:59:592
4885992718,2cyclictest313-21systemd-journal21:59:592
488599250,24cyclictest2340-21idleruntime-cro23:30:002
4877992523,1cyclictest24550-21idleruntime-cro20:55:000
488299240,23cyclictest4758-21awk23:35:001
487799240,1cyclictest13130-21turbostat22:45:000
487799240,0cyclictest0-21swapper/020:17:290
4890992321,1cyclictest24548-21cron20:55:003
489099231,21cyclictest14554-21apt-get20:35:003
489099230,22cyclictest20223-21apt-get20:47:293
489099230,22cyclictest16804-21tr00:00:003
489099230,22cyclictest15273-21sh19:29:593
489099230,22cyclictest0-21swapper/300:32:323
4885992321,1cyclictest15737-21apt-get19:32:282
4885992319,0cyclictest2151-21apt-get00:37:292
488599230,23cyclictest0-21swapper/220:57:302
4877992322,0cyclictest20793-21apt-get23:02:280
487799230,22cyclictest24906-21awk19:50:000
487799230,22cyclictest16825-21dump-pmu-power00:00:000
487799230,21cyclictest313-21systemd-journal23:00:010
487799230,1cyclictest313-21systemd-journal21:00:000
4890992222,0cyclictest4948-21apt-get20:17:293
4890992222,0cyclictest0-21swapper/321:52:313
4890992220,1cyclictest24909-21grep19:50:003
4890992220,1cyclictest20350-21sendmail-msp23:00:013
489099220,21cyclictest2091-21latency_hist20:10:003
489099220,21cyclictest0-21swapper/322:40:033
489099220,1cyclictest7862-21cron22:35:003
489099220,1cyclictest31730-21munin-run00:30:013
489099220,1cyclictest16259-21perf21:45:003
4885992220,2cyclictest0-21swapper/219:10:002
4885992220,1cyclictest0-21swapper/221:50:002
488599220,21cyclictest8319-21apt-get22:37:312
488599220,21cyclictest0-21swapper/222:52:302
488599220,21cyclictest0-21swapper/219:17:292
4882992222,0cyclictest0-21swapper/122:47:281
4882992220,1cyclictest13824-21sendmail21:39:591
488299220,0cyclictest0-21swapper/119:42:301
4877992220,1cyclictest8468-21apt-get19:17:290
487799220,21cyclictest31774-21perf21:10:000
487799220,1cyclictest22556-21apt-get00:12:290
4890992121,0cyclictest0-21swapper/323:52:313
4890992120,1cyclictest25341-21apt-get19:52:283
4890992119,2cyclictest6576-21sh21:25:003
4890992119,1cyclictest31791-21grep21:10:003
4890992119,1cyclictest15546-21munin-run22:50:003
489099210,20cyclictest0-21swapper/320:22:283
489099210,1cyclictest20081-21latency_hist19:40:013
489099210,1cyclictest0-21swapper/300:09:593
489099210,0cyclictest0-21swapper/321:37:283
489099210,0cyclictest0-21swapper/320:37:273
4885992121,0cyclictest0-21swapper/223:37:322
4885992120,1cyclictest7762-21apt-get20:22:292
4885992120,1cyclictest26318-21apt-get22:07:322
488599210,20cyclictest12464-21apt-get23:52:302
488599210,20cyclictest0-21swapper/221:40:002
488599210,20cyclictest0-21swapper/221:07:292
488599210,20cyclictest0-21swapper/220:07:292
488599210,20cyclictest0-21swapper/200:02:292
488599210,1cyclictest1768-21dump-pmu-power21:15:012
488599210,0cyclictest0-21swapper/221:42:302
4882992121,0cyclictest30158-21apt-get20:02:301
4882992120,1cyclictest14863-21apt-get23:57:301
4882992119,1cyclictest4948-21apt-get20:17:291
4882992119,1cyclictest19106-21apt-get21:52:311
488299210,21cyclictest0-21swapper/120:57:291
488299210,20cyclictest0-21swapper/100:37:301
488299210,0cyclictest0-21swapper/123:47:301
4877992121,0cyclictest0-21swapper/019:42:290
4877992121,0cyclictest0-21swapper/000:22:290
4877992120,0cyclictest2788-21apt-get23:32:300
487799210,21cyclictest0-21swapper/022:50:000
487799210,21cyclictest0-21swapper/021:32:300
487799210,21cyclictest0-21swapper/020:35:000
487799210,20cyclictest0-21swapper/021:47:290
487799210,20cyclictest0-21swapper/000:32:310
487799210,1cyclictest23466-21awk22:00:000
487799210,0cyclictest0-21swapper/023:52:310
4890992020,0cyclictest0-21swapper/323:22:273
4890992020,0cyclictest0-21swapper/323:02:273
4890992020,0cyclictest0-21swapper/319:10:013
489099200,0cyclictest0-21swapper/323:27:293
489099200,0cyclictest0-21swapper/320:02:303
4885992020,0cyclictest7614-21apt-get23:42:302
4885992020,0cyclictest0-21swapper/223:32:302
4885992020,0cyclictest0-21swapper/222:17:302
4885992020,0cyclictest0-21swapper/221:02:282
4885992020,0cyclictest0-21swapper/220:42:322
4885992020,0cyclictest0-21swapper/220:37:272
4885992020,0cyclictest0-21swapper/200:27:292
4885992018,1cyclictest21071-21munin-run21:55:002
488599200,0cyclictest7040-21apt-get21:27:292
488599200,0cyclictest32165-21apt-get00:32:322
488599200,0cyclictest20160-21apt-get00:07:292
488599200,0cyclictest0-21swapper/220:02:292
488599200,0cyclictest0-21swapper/219:42:302
4882992020,0cyclictest0-21swapper/123:52:311
4882992020,0cyclictest0-21swapper/123:37:321
4882992020,0cyclictest0-21swapper/122:32:291
4882992020,0cyclictest0-21swapper/121:47:281
4882992020,0cyclictest0-21swapper/121:32:301
4882992020,0cyclictest0-21swapper/121:27:291
4882992020,0cyclictest0-21swapper/120:35:011
4882992020,0cyclictest0-21swapper/119:12:291
4882992020,0cyclictest0-21swapper/100:25:001
4882992019,1cyclictest2218-21apt-get21:17:301
4882992019,1cyclictest0-21swapper/120:27:281
4882992018,1cyclictest14582-21awk20:35:001
4882992018,1cyclictest0-21swapper/119:35:011
4882992018,1cyclictest0-21swapper/100:10:001
488299200,20cyclictest0-21swapper/120:07:301
488299200,0cyclictest18133-21apt-get19:37:291
488299200,0cyclictest0-21swapper/123:27:291
488299200,0cyclictest0-21swapper/100:12:301
4877992020,0cyclictest27434-21apt-get21:02:300
4877992020,0cyclictest17764-21apt-get00:02:300
4877992020,0cyclictest0-21swapper/022:52:300
4877992020,0cyclictest0-21swapper/022:37:310
4877992020,0cyclictest0-21swapper/021:27:280
4877992020,0cyclictest0-21swapper/019:57:300
4877992020,0cyclictest0-21swapper/019:37:290
4877992020,0cyclictest0-21swapper/019:27:290
4877992020,0cyclictest0-21swapper/019:22:290
4877992019,0cyclictest0-21swapper/023:42:290
4877992018,1cyclictest4096-21systemctl00:39:590
4877992018,1cyclictest15737-21apt-get19:32:280
4890991919,0cyclictest0-21swapper/321:32:303
4890991919,0cyclictest0-21swapper/321:12:283
4890991919,0cyclictest0-21swapper/319:42:303
4890991919,0cyclictest0-21swapper/319:17:283
4890991919,0cyclictest0-21swapper/300:00:013
4890991918,1cyclictest0-21swapper/300:22:293
4885991919,0cyclictest9492-21apt-get21:32:302
4885991919,0cyclictest23192-21apt-get23:07:292
4885991919,0cyclictest10939-21apt-get19:22:292
4885991919,0cyclictest0-21swapper/223:47:302
4885991919,0cyclictest0-21swapper/222:32:292
4885991918,1cyclictest10964-21apt-get22:40:032
4885991918,0cyclictest22556-21apt-get00:12:302
4885991917,1cyclictest0-21swapper/222:15:002
4882991919,0cyclictest6893-21apt-get20:20:021
4882991919,0cyclictest31108-21apt-get22:17:301
4882991919,0cyclictest0-21swapper/122:22:301
4882991919,0cyclictest0-21swapper/120:47:291
4882991919,0cyclictest0-21swapper/119:17:291
4882991919,0cyclictest0-21swapper/100:32:311
4882991918,1cyclictest27984-21apt-get23:17:281
4882991918,1cyclictest25588-21apt-get23:12:311
4882991918,1cyclictest17831-21apt-get20:42:321
4882991917,1cyclictest21499-21apt-get21:57:281
4877991919,0cyclictest0-21swapper/023:37:320
4877991919,0cyclictest0-21swapper/023:07:290
4877991919,0cyclictest0-21swapper/022:07:320
4877991919,0cyclictest0-21swapper/020:47:300
4877991918,0cyclictest0-21swapper/019:12:290
487799190,18cyclictest7543-21apt-get20:20:020
4890991818,0cyclictest0-21swapper/321:47:293
4890991817,0cyclictest0-21swapper/319:57:303
489099180,0cyclictest0-21swapper/320:12:283
4885991818,0cyclictest22618-21apt-get20:52:292
4885991818,0cyclictest0-21swapper/222:57:282
4885991818,0cyclictest0-21swapper/222:27:292
4885991818,0cyclictest0-21swapper/219:37:292
4885991818,0cyclictest0-21swapper/200:22:292
4885991817,1cyclictest20190-21/usr/sbin/munin20:47:292
4882991818,0cyclictest0-21swapper/122:52:311
4882991818,0cyclictest0-21swapper/122:27:291
4882991818,0cyclictest0-21swapper/121:07:301
4882991818,0cyclictest0-21swapper/119:47:251
4882991818,0cyclictest0-21swapper/119:22:291
4882991818,0cyclictest0-21swapper/100:27:301
4882991817,1cyclictest23919-21apt-get22:02:301
4882991817,0cyclictest0-21swapper/121:02:291
4877991818,0cyclictest24955-21apt-get00:17:290
4877991818,0cyclictest0-21swapper/022:22:300
4877991818,0cyclictest0-21swapper/000:27:300
4877991817,1cyclictest0-21swapper/023:12:310
4877991816,1cyclictest593-21dbus-daemon00:10:000
4877991816,1cyclictest0-21swapper/020:12:280
4890991717,0cyclictest0-21swapper/322:52:303
4890991717,0cyclictest0-21swapper/322:37:313
4890991717,0cyclictest0-21swapper/321:27:293
4890991717,0cyclictest0-21swapper/321:02:303
4890991717,0cyclictest0-21swapper/319:22:283
4890991715,1cyclictest4098-21awk00:40:003
4885991717,0cyclictest0-21swapper/223:12:312
4885991717,0cyclictest0-21swapper/220:27:282
4885991717,0cyclictest0-21swapper/219:52:282
4882991717,0cyclictest30400-21apt-get23:22:271
4882991717,0cyclictest0-21swapper/122:37:301
4882991717,0cyclictest0-21swapper/100:17:291
4882991717,0cyclictest0-21swapper/100:02:301
4882991716,1cyclictest0-21swapper/122:57:281
4882991715,1cyclictest3164-21tail20:12:311
4877991717,0cyclictest0-21swapper/021:42:300
4877991716,1cyclictest28709-21apt-get22:12:300
4877991715,1cyclictest19777-21latency_hist20:45:000
4877991715,1cyclictest0-21swapper/019:55:000
487799170,1cyclictest0-21swapper/023:17:340
4890991616,0cyclictest0-21swapper/323:42:293
4890991616,0cyclictest0-21swapper/323:37:323
4890991616,0cyclictest0-21swapper/323:32:303
4890991616,0cyclictest0-21swapper/323:07:283
4890991616,0cyclictest0-21swapper/322:07:323
4890991616,0cyclictest0-21swapper/319:32:283
4890991615,0cyclictest0-21swapper/300:17:283
489099160,15cyclictest24301-21expr22:02:313
489099160,15cyclictest0-21swapper/322:25:003
4885991616,0cyclictest0-21swapper/223:02:292
4882991616,0cyclictest0-21swapper/123:42:291
4882991616,0cyclictest0-21swapper/122:42:321
4877991616,0cyclictest32555-21apt-get20:07:300
4877991616,0cyclictest0-21swapper/021:37:280
4877991614,1cyclictest7872-21awk22:35:000
487799160,2cyclictest19460-21cstates21:52:320
4890991513,1cyclictest0-21swapper/300:12:383
489099150,14cyclictest0-21swapper/323:17:283
489099150,14cyclictest0-21swapper/320:44:523
4885991515,0cyclictest4634-21apt-get21:22:302
4885991513,1cyclictest6912-21perf20:20:002
4885991513,1cyclictest15508-21cron22:49:592
488599150,0cyclictest0-21swapper/222:02:292
4882991513,1cyclictest6597-21turbostat21:25:001
4882991513,1cyclictest0-21swapper/121:12:301
488299150,14cyclictest0-21swapper/122:05:341
4877991515,0cyclictest0-21swapper/022:02:300
4877991515,0cyclictest0-21swapper/021:17:290
4877991513,1cyclictest4634-21apt-get21:22:300
487799150,14cyclictest0-21swapper/022:20:000
4890991414,0cyclictest0-21swapper/322:28:073
489099140,14cyclictest0-21swapper/323:10:053
489099140,14cyclictest0-21swapper/322:11:053
489099140,14cyclictest0-21swapper/321:19:103
489099140,14cyclictest0-21swapper/320:25:213
489099140,0cyclictest0-21swapper/322:17:303
4885991414,0cyclictest0-21swapper/221:15:042
488599140,9cyclictest25007-21/usr/sbin/munin00:17:302
488599140,14cyclictest0-21swapper/223:19:472
488599140,14cyclictest0-21swapper/222:23:522
488599140,14cyclictest0-21swapper/220:31:272
488599140,14cyclictest0-21swapper/220:14:372
488599140,14cyclictest0-21swapper/219:59:432
488599140,14cyclictest0-21swapper/219:45:022
488599140,14cyclictest0-21swapper/219:28:332
488599140,0cyclictest0-21swapper/223:22:472
488299140,14cyclictest0-21swapper/123:01:121
488299140,14cyclictest0-21swapper/119:50:281
488299140,0cyclictest0-21swapper/123:07:531
488299140,0cyclictest0-21swapper/122:10:231
488299140,0cyclictest0-21swapper/121:41:201
488299140,0cyclictest0-21swapper/120:50:151
4877991414,0cyclictest0-21swapper/020:03:480
487799140,4cyclictest0-21swapper/020:27:280
487799140,14cyclictest0-21swapper/023:46:010
487799140,13cyclictest0-21swapper/023:22:280
487799140,0cyclictest0-21swapper/023:25:290
487799140,0cyclictest0-21swapper/022:25:500
487799140,0cyclictest0-21swapper/020:30:180
4890991313,0cyclictest0-21swapper/320:56:523
489099130,0cyclictest0-21swapper/323:48:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional