You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 13:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Thu Nov 27, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1232499274,22cyclictest13-21ksoftirqd/022:35:000
1232899250,24cyclictest0-21swapper/122:02:331
1232899250,23cyclictest313-21systemd-journal23:59:591
1233499240,23cyclictest31513-21perf21:00:003
1233499240,23cyclictest11803-21ls21:30:003
12331992321,1cyclictest21549-21dump-pmu-power22:55:002
12331992320,2cyclictest6697-21perf22:25:002
1233199230,22cyclictest13966-21cron23:45:002
12328992320,2cyclictest11583-21munin-run20:14:591
12328992317,5cyclictest30949-21sendmail-msp00:20:001
12328992316,5cyclictest313-21systemd-journal21:40:011
1232899230,22cyclictest31481-21sh22:09:591
1232899230,22cyclictest26076-21munin-plugin-st20:45:001
1232899230,0cyclictest0-21swapper/123:07:351
12324992321,1cyclictest28761-21perf23:09:590
1232499230,22cyclictest26714-21lockfile-remove21:59:590
12334992220,1cyclictest29606-21apt-get23:12:323
1233499220,22cyclictest34-21ksoftirqd/321:10:223
1233499220,21cyclictest6691-21latency_hist22:25:003
1233499220,21cyclictest0-21swapper/322:20:003
1233499220,21cyclictest0-21swapper/319:47:333
1233499220,1cyclictest14433-21apt-get23:47:333
1233199220,21cyclictest4831-21apt-get20:02:332
1233199220,21cyclictest31400-21apt-get00:22:382
1233199220,21cyclictest28906-21apt-get20:52:322
1233199220,21cyclictest2159-21perf21:10:002
1233199220,21cyclictest19154-21wc22:50:002
1233199220,21cyclictest0-21swapper/222:00:002
12328992220,1cyclictest1543-21turbostat.cron23:20:011
1232899220,21cyclictest9940-21apt-get22:32:351
1232899220,21cyclictest24426-21apt-get23:02:371
1232899220,21cyclictest0-21swapper/121:12:301
1232899220,21cyclictest0-21swapper/120:27:351
1232899220,21cyclictest0-21swapper/119:47:331
1232899220,1cyclictest26138-21apt00:10:001
1232899220,1cyclictest21790-21apt-get00:02:341
12324992221,0cyclictest15159-21apt-get19:17:340
12324992220,1cyclictest11583-21basename23:40:000
1232499220,21cyclictest8144-21perf00:40:000
1232499220,21cyclictest24118-21apt-get20:42:350
1232499220,21cyclictest13983-21latency_hist20:20:000
1232499220,21cyclictest0-21swapper/023:42:330
1232499220,21cyclictest0-21swapper/023:32:340
1232499220,21cyclictest0-21swapper/021:37:340
1232499220,21cyclictest0-21swapper/021:32:340
1232499220,1cyclictest30938-21perf00:20:000
1232499220,1cyclictest23744-21munin-run00:05:010
12334992120,0cyclictest9378-21apt-get23:35:013
12334992120,0cyclictest0-21swapper/319:57:333
12334992119,1cyclictest0-21swapper/322:02:333
1233499210,21cyclictest0-21swapper/319:32:343
1233499210,20cyclictest14761-21apt-get22:42:343
1233499210,20cyclictest0-21swapper/322:27:333
1233499210,20cyclictest0-21swapper/319:12:333
1233499210,1cyclictest0-21swapper/322:15:003
1233499210,0cyclictest9625-21apt-get20:12:333
1233499210,0cyclictest0-21swapper/322:37:343
1233499210,0cyclictest0-21swapper/321:52:333
1233499210,0cyclictest0-21swapper/321:22:333
1233499210,0cyclictest0-21swapper/300:22:373
12331992121,0cyclictest6172-21apt-get00:37:342
12331992121,0cyclictest28976-21apt-get00:17:342
12331992121,0cyclictest2414-21apt-get19:57:332
12331992121,0cyclictest0-21swapper/223:35:012
12331992121,0cyclictest0-21swapper/222:27:322
12331992120,1cyclictest0-21swapper/200:07:342
12331992119,1cyclictest26577-21apt-get00:12:312
12331992119,1cyclictest19286-21apt-get23:57:372
12331992119,1cyclictest17632-21apt-get19:22:352
12331992119,1cyclictest0-21swapper/223:27:342
1233199210,21cyclictest0-21swapper/221:32:332
1233199210,21cyclictest0-21swapper/200:35:002
1233199210,20cyclictest22329-21apt-get21:52:332
1233199210,20cyclictest0-21swapper/200:02:352
1233199210,0cyclictest6816-21apt-get23:32:342
1233199210,0cyclictest12344-21apt-get22:37:352
1233199210,0cyclictest0-21swapper/223:07:352
12328992120,1cyclictest28768-21apt-get23:10:011
12328992120,1cyclictest0-21swapper/100:27:341
1232899210,2cyclictest2147-21dump-pmu-power21:10:001
1232899210,21cyclictest0-21swapper/123:22:351
1232899210,21cyclictest0-21swapper/119:42:331
1232899210,20cyclictest0-21swapper/120:32:351
1232899210,20cyclictest0-21swapper/119:22:341
1232899210,1cyclictest0-21swapper/123:35:001
1232899210,0cyclictest14761-21apt-get22:42:351
1232899210,0cyclictest12269-21apt-get21:32:341
1232899210,0cyclictest0-21swapper/100:12:321
12324992120,1cyclictest26515-21apt-get20:47:350
12324992120,1cyclictest0-21swapper/000:07:340
12324992120,0cyclictest0-21swapper/019:27:330
12324992119,1cyclictest2020-21apt-get23:22:350
1232499210,20cyclictest27147-21apt-get22:02:330
1232499210,20cyclictest0-21swapper/019:37:320
1232499210,0cyclictest0-21swapper/020:37:340
12334992020,0cyclictest0-21swapper/320:02:333
12334992020,0cyclictest0-21swapper/319:37:323
12334992020,0cyclictest0-21swapper/300:27:333
12334992019,1cyclictest3776-21apt-get00:32:343
12334992019,1cyclictest26697-21munin-plugin-st22:00:003
12334992019,1cyclictest0-21swapper/322:07:333
12334992018,1cyclictest14008-21munin-run20:20:003
12334992018,1cyclictest0-21swapper/320:35:003
12334992017,2cyclictest313-21systemd-journal21:05:003
1233499200,20cyclictest0-21swapper/320:52:333
1233499200,20cyclictest0-21swapper/320:37:343
1233499200,0cyclictest26819-21apt-get23:07:353
12331992020,0cyclictest5004-21apt-get21:17:342
12331992020,0cyclictest20025-21apt-get19:27:322
12331992020,0cyclictest0-21swapper/223:47:332
12331992018,1cyclictest24737-21cron19:34:592
1233199200,20cyclictest0-21swapper/221:12:302
1233199200,19cyclictest19933-21apt-get21:47:332
1233199200,0cyclictest0-21swapper/221:37:332
1233199200,0cyclictest0-21swapper/220:32:352
1233199200,0cyclictest0-21swapper/220:27:352
12328992020,0cyclictest0-21swapper/122:57:341
12328992020,0cyclictest0-21swapper/121:57:331
12328992020,0cyclictest0-21swapper/119:27:331
12328992020,0cyclictest0-21swapper/119:12:331
12328992020,0cyclictest0-21swapper/100:37:341
12328992019,1cyclictest25217-21apt-get19:37:331
12328992019,1cyclictest1469-21cron21:05:001
12328992019,0cyclictest0-21swapper/122:22:341
12328992019,0cyclictest0-21swapper/120:17:321
12328992018,1cyclictest1936-21cron19:55:001
12328992016,3cyclictest9606-21apt-get23:37:331
12328992014,5cyclictest593-21dbus-daemon20:05:001
1232899200,0cyclictest0-21swapper/122:17:341
1232899200,0cyclictest0-21swapper/122:12:331
1232899200,0cyclictest0-21swapper/121:42:341
12324992020,0cyclictest7141-21apt-get22:27:330
12324992020,0cyclictest0-21swapper/023:52:340
12324992020,0cyclictest0-21swapper/023:12:320
12324992020,0cyclictest0-21swapper/021:52:330
12324992020,0cyclictest0-21swapper/020:27:360
12324992019,1cyclictest4415-21apt-get23:27:350
12324992019,1cyclictest30026-21apt-get19:47:330
12324992019,1cyclictest14314-21perf22:40:000
1232499200,0cyclictest0-21swapper/020:07:350
1232499200,0cyclictest0-21swapper/000:32:350
12334991919,0cyclictest0-21swapper/323:32:333
12334991919,0cyclictest0-21swapper/321:37:343
12334991919,0cyclictest0-21swapper/320:42:343
12334991919,0cyclictest0-21swapper/320:27:363
12334991918,1cyclictest3958-21latency_hist23:25:003
1233499190,18cyclictest0-21swapper/323:42:343
12331991919,0cyclictest0-21swapper/222:42:352
12331991919,0cyclictest0-21swapper/219:37:332
12331991917,1cyclictest31986-21munin-run19:50:002
1233199190,18cyclictest11807-21turbostat21:29:592
12328991919,0cyclictest0-21swapper/122:47:331
12328991919,0cyclictest0-21swapper/120:52:331
12328991919,0cyclictest0-21swapper/119:30:011
12328991919,0cyclictest0-21swapper/100:32:351
12328991919,0cyclictest0-21swapper/100:22:371
12328991918,1cyclictest9356-21sh21:25:001
12328991917,1cyclictest24291-21munin-run21:55:001
12328991917,1cyclictest21878-21perf21:50:001
1232899190,1cyclictest0-21swapper/123:27:381
12324991919,0cyclictest0-21swapper/021:47:330
12324991919,0cyclictest0-21swapper/021:17:340
12324991919,0cyclictest0-21swapper/020:12:330
12324991919,0cyclictest0-21swapper/019:22:350
12324991919,0cyclictest0-21swapper/000:12:320
12324991918,1cyclictest16388-21perf23:49:590
12334991818,0cyclictest7230-21apt-get20:07:343
12334991818,0cyclictest19482-21perf21:44:593
12334991818,0cyclictest0-21swapper/323:57:373
12334991818,0cyclictest0-21swapper/320:22:343
12334991817,0cyclictest5806-21irqstats21:17:383
12334991817,0cyclictest0-21swapper/322:57:353
12334991816,1cyclictest21450irq/143-nvme0q400:20:003
12331991818,0cyclictest16829-21apt-get23:52:342
12331991818,0cyclictest0-21swapper/223:22:342
12331991818,0cyclictest0-21swapper/223:02:372
12331991818,0cyclictest0-21swapper/222:32:352
12331991818,0cyclictest0-21swapper/220:42:352
12331991818,0cyclictest0-21swapper/219:52:362
12331991818,0cyclictest0-21swapper/219:17:342
12331991815,2cyclictest16392-21awk20:24:592
1233199180,1cyclictest11577-21perf20:15:002
1233199180,18cyclictest0-21swapper/220:47:352
12328991818,0cyclictest9867-21apt-get21:27:331
12328991818,0cyclictest0-21swapper/123:52:341
12328991818,0cyclictest0-21swapper/122:37:341
12328991817,1cyclictest17175-21basename19:20:001
12324991818,0cyclictest31939-21apt-get22:12:330
12324991818,0cyclictest22003-21apt-get22:57:340
12324991818,0cyclictest0-21swapper/021:02:290
12324991818,0cyclictest0-21swapper/020:32:360
12334991717,0cyclictest0-21swapper/323:27:343
12334991717,0cyclictest0-21swapper/323:02:373
12334991717,0cyclictest0-21swapper/321:47:323
12334991717,0cyclictest0-21swapper/319:22:353
12334991717,0cyclictest0-21swapper/300:12:323
12334991716,1cyclictest18845-21munin-run23:55:003
12331991717,0cyclictest21698-21apt-get20:37:332
12331991717,0cyclictest0-21swapper/223:12:332
12331991717,0cyclictest0-21swapper/222:57:352
12331991717,0cyclictest0-21swapper/222:02:332
12331991715,1cyclictest31531-21sendmail-msp21:00:002
1233199170,12cyclictest7421-21apt-get21:22:332
12328991717,0cyclictest0-21swapper/121:17:331
12328991717,0cyclictest0-21swapper/120:22:341
12328991717,0cyclictest0-21swapper/119:57:331
1232899170,0cyclictest0-21swapper/120:47:361
12324991717,0cyclictest0-21swapper/020:22:340
12324991717,0cyclictest0-21swapper/019:32:340
12334991616,0cyclictest0-21swapper/322:30:023
12334991614,2cyclictest0-21swapper/322:47:333
12334991614,1cyclictest2123-21cron21:10:003
12331991616,0cyclictest12023-21apt-get20:17:332
12331991614,2cyclictest19486-21sleep21:45:002
12328991616,0cyclictest0-21swapper/122:52:341
1232899160,0cyclictest0-21swapper/122:27:151
12324991616,0cyclictest0-21swapper/000:27:340
12324991615,1cyclictest2599-21apt-get21:12:300
12324991614,1cyclictest0-21swapper/022:17:390
12324991614,1cyclictest0-21swapper/021:22:370
12324991613,2cyclictest593-21dbus-daemon22:45:000
1232499160,15cyclictest0-21swapper/023:17:390
12334991515,0cyclictest0-21swapper/321:32:343
12334991515,0cyclictest0-21swapper/319:52:363
12334991513,1cyclictest0-21swapper/300:37:393
12331991515,0cyclictest0-21swapper/222:17:342
12331991515,0cyclictest0-21swapper/219:42:332
12331991514,1cyclictest1368-21apt-get00:27:342
1233199150,1cyclictest7585-21expr20:07:362
1233199150,14cyclictest0-21swapper/222:07:352
12324991515,0cyclictest0-21swapper/021:09:400
12324991515,0cyclictest0-21swapper/000:22:380
12324991514,1cyclictest0-21swapper/022:52:340
12324991512,2cyclictest14690-21idleruntime-cro19:15:000
12334991414,0cyclictest0-21swapper/322:52:343
1233499140,14cyclictest0-21swapper/323:16:393
1233499140,14cyclictest0-21swapper/320:45:133
1233499140,14cyclictest0-21swapper/319:25:453
1233499140,14cyclictest0-21swapper/300:03:573
1233499140,0cyclictest0-21swapper/319:40:013
1233499140,0cyclictest0-21swapper/319:15:533
1233199140,14cyclictest0-21swapper/223:16:532
1233199140,14cyclictest0-21swapper/219:14:372
1233199140,0cyclictest0-21swapper/222:11:582
1233199140,0cyclictest0-21swapper/221:02:132
1232899140,14cyclictest0-21swapper/123:46:031
1232899140,14cyclictest0-21swapper/123:40:551
1232899140,0cyclictest0-21swapper/120:57:151
12324991414,0cyclictest0-21swapper/022:20:340
12324991414,0cyclictest0-21swapper/022:07:410
12324991414,0cyclictest0-21swapper/021:40:180
12324991414,0cyclictest0-21swapper/020:51:550
1232499140,14cyclictest0-21swapper/023:03:460
1232499140,14cyclictest0-21swapper/021:26:410
1232499140,14cyclictest0-21swapper/020:59:150
1232499140,14cyclictest0-21swapper/020:03:210
1232499140,14cyclictest0-21swapper/019:53:490
1232499140,0cyclictest0-21swapper/022:46:360
1232499140,0cyclictest0-21swapper/019:40:590
12334991313,0cyclictest0-21swapper/300:05:483
12324991313,0cyclictest0-21swapper/019:56:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional