You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 23:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot2.osadl.org (updated Sat Jul 19, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326682297262,22sleep30-21swapper/33
324142297262,22sleep20-21swapper/22
325592276239,23sleep10-21swapper/11
326702254219,22sleep00-21swapper/00
39199253250,2cyclictest0-21swapper/33
37899212207,3cyclictest7606-21apt-get1
37599145139,4cyclictest0-21swapper/00
384999895,2cyclictest0-21swapper/22
120372550,2sleep20-21swapper/22
232172420,1sleep00-21swapper/00
37599350,33cyclictest9558-21kworker/0:10
38499342,9cyclictest18646-21apt-get2
384993415,4cyclictest7361-21sh2
216062340,1sleep10-21swapper/11
59282330,0sleep00-21swapper/00
391993227,3cyclictest341rcuc/33
38499323,8cyclictest14437-21apt-get2
38499303,7cyclictest31610-21apt-get2
324622290,1sleep132464-21grep1
39199280,6cyclictest35-21ksoftirqd/33
38499281,10cyclictest5178-21munin-run2
39199253,8cyclictest35-21ksoftirqd/33
38499252,7cyclictest12784-21apt-get2
39199248,8cyclictest111rcu_preempt3
391992419,3cyclictest19258-21head3
39199241,5cyclictest341rcuc/33
38499241,5cyclictest10333-21gdbus2
39199237,8cyclictest35-21ksoftirqd/33
39199237,7cyclictest35-21ksoftirqd/33
39199232,7cyclictest35-21ksoftirqd/33
39199232,6cyclictest35-21ksoftirqd/33
391992320,2cyclictest341rcuc/33
391992319,3cyclictest341rcuc/33
391992319,3cyclictest341rcuc/33
391992312,3cyclictest111rcu_preempt3
39199230,8cyclictest35-21ksoftirqd/33
39199230,6cyclictest0-21swapper/33
38499234,17cyclictest23082-21latency_hist2
38499233,17cyclictest29688-21cat2
384992316,4cyclictest2365-21sh2
384992315,5cyclictest21448-21cron2
384992312,2cyclictest1759-21/usr/sbin/munin2
38499231,19cyclictest27153-21munin-run2
39199229,2cyclictest111rcu_preempt3
39199226,8cyclictest35-21ksoftirqd/33
39199222,7cyclictest35-21ksoftirqd/33
39199222,6cyclictest369-21cyclictest3
391992219,2cyclictest341rcuc/33
391992219,2cyclictest341rcuc/33
391992217,3cyclictest341rcuc/33
391992217,3cyclictest1125-21turbostat.cron3
391992212,2cyclictest111rcu_preempt3
391992211,3cyclictest111rcu_preempt3
391992211,2cyclictest111rcu_preempt3
39199220,7cyclictest35-21ksoftirqd/33
38499223,17cyclictest9257-21latency_hist2
384992217,3cyclictest11257-21grep2
378992215,4cyclictest21739-21df_inode1
378992215,4cyclictest2168-21seq1
378992215,4cyclictest2082-21systemd-journal1
378992214,5cyclictest9123-21apt-get1
37899220,20cyclictest0-21swapper/11
39199219,3cyclictest111rcu_preempt3
39199219,2cyclictest111rcu_preempt3
39199213,5cyclictest0-21swapper/33
39199213,5cyclictest0-21swapper/33
39199213,5cyclictest0-21swapper/33
39199213,5cyclictest0-21swapper/33
39199213,5cyclictest0-21swapper/33
39199213,5cyclictest0-21swapper/33
39199213,4cyclictest0-21swapper/33
39199212,6cyclictest0-21swapper/33
39199212,6cyclictest0-21swapper/33
39199212,6cyclictest0-21swapper/33
39199212,5cyclictest0-21swapper/33
39199212,5cyclictest0-21swapper/33
39199212,5cyclictest0-21swapper/33
391992118,2cyclictest341rcuc/33
391992118,2cyclictest341rcuc/33
391992118,2cyclictest341rcuc/33
39199211,7cyclictest35-21ksoftirqd/33
391992117,3cyclictest341rcuc/33
391992117,3cyclictest341rcuc/33
391992115,4cyclictest3497-21aten2_r9power_p3
391992114,4cyclictest6831-21sendmail3
39199210,7cyclictest35-21ksoftirqd/33
38499219,9cyclictest16449-21/usr/sbin/munin2
38499214,15cyclictest17117-21cat2
38499213,16cyclictest2866-21cat2
38499213,16cyclictest11389-21cat2
38499212,17cyclictest27253-21cat2
38499212,16cyclictest31722-21wc2
38499212,16cyclictest12051-21grep2
384992119,1cyclictest0-21swapper/22
384992116,3cyclictest4999-21munin-run2
384992116,3cyclictest27540-21cron2
384992115,4cyclictest19631-21cstates2
38499211,3cyclictest13781-1kworker/2:1H2
38499211,17cyclictest505-21dbus-daemon2
38499211,17cyclictest2082-21systemd-journal2
38499211,17cyclictest18637-21munin-run2
38499210,18cyclictest5595-21ls2
38499210,18cyclictest31688-21latency_hist2
37899213,15cyclictest13298-21head1
378992110,7cyclictest7492-21apt-get1
375992116,3cyclictest14319-21missed_timers0
21332210,1sleep10-21swapper/11
39199209,3cyclictest111rcu_preempt3
39199207,5cyclictest111rcu_preempt3
39199206,6cyclictest35-21ksoftirqd/33
39199203,5cyclictest35-21ksoftirqd/33
39199203,5cyclictest35-21ksoftirqd/33
39199203,5cyclictest35-21ksoftirqd/33
39199202,6cyclictest35-21ksoftirqd/33
39199202,5cyclictest0-21swapper/33
391992014,3cyclictest29388-21gltestperf3
391992014,3cyclictest24226-21ntp_offset3
391992013,4cyclictest2082-21systemd-journal3
38499209,6cyclictest14833-21apt-get2
38499202,16cyclictest2438-21cat2
384992017,2cyclictest0-21swapper/22
384992015,3cyclictest4065-21sshd2
384992014,4cyclictest29705-21cut2
384992012,5cyclictest7552-21ntp_kernel_pll_2
38499201,17cyclictest4781-21latency_hist2
38499201,17cyclictest13781-1kworker/2:1H2
38499201,17cyclictest10887-21cron2
38499201,16cyclictest1120-21munin-run2
384992011,5cyclictest0-21swapper/22
38499200,18cyclictest19132-21cstates2
378992016,2cyclictest18802-21apt-get1
378992015,3cyclictest4198-21ps1
378992015,3cyclictest26612-21seq1
378992015,3cyclictest26612-21seq1
378992015,3cyclictest16449-21/usr/sbin/munin1
378992015,3cyclictest15532-21sort1
378992014,4cyclictest23844-21cat1
378992014,4cyclictest23844-21cat1
378992014,4cyclictest18816-21cat1
378992014,4cyclictest10207-21seq1
378992012,5cyclictest505-21dbus-daemon1
375992016,2cyclictest19744-21aten2_r9power_e0
375992015,3cyclictest5833-21cat0
375992015,3cyclictest2216-21nfs4_client0
375992014,4cyclictest9-21ksoftirqd/00
375992014,4cyclictest2940-21date0
375992014,4cyclictest28864-21seq0
375992014,4cyclictest10377-21diskstats0
375992013,4cyclictest31108-21cron0
37599200,5cyclictest0-21swapper/00
254352200,1sleep30-21swapper/33
254352200,1sleep30-21swapper/33
39199198,3cyclictest111rcu_preempt3
391991910,2cyclictest111rcu_preempt3
38499193,8cyclictest13179-21apt-get2
384991916,2cyclictest0-21swapper/22
384991914,3cyclictest31139-21perf2
384991914,3cyclictest26812-21cat2
384991914,3cyclictest26812-21cat2
384991914,3cyclictest24628-21sed2
384991914,3cyclictest2218-21ntp_kernel_err2
384991914,3cyclictest19087-21sendmail-mta2
384991913,4cyclictest30293-21basename2
384991913,3cyclictest505-21dbus-daemon2
38499191,16cyclictest25506-21cat2
38499190,16cyclictest5173-21head2
38499190,16cyclictest13781-1kworker/2:1H2
37899193,14cyclictest21066-21ls1
378991915,2cyclictest5568-21/usr/sbin/munin1
378991915,2cyclictest25374-21/usr/sbin/munin1
378991914,3cyclictest7957-21ntpq1
378991914,3cyclictest7485-21seq1
378991914,3cyclictest7485-21seq1
378991914,3cyclictest5915-21sh1
378991914,3cyclictest411-21wc1
378991914,3cyclictest3908-21cut1
378991914,3cyclictest28928-21/usr/sbin/munin1
378991914,3cyclictest26431-21sed1
378991914,3cyclictest21694-21/usr/sbin/munin1
378991914,3cyclictest17411-21dump-pmu-power1
378991914,3cyclictest15875-21seq1
378991914,3cyclictest13726-21seq1
378991914,3cyclictest12433-21ntpq1
378991913,4cyclictest747-21idleruntime-cro1
378991913,4cyclictest30208-21ntpq1
378991913,4cyclictest29922-21awk1
378991913,4cyclictest20523-21cut1
378991912,4cyclictest17844-21cat1
37899190,17cyclictest0-21swapper/11
37599192,15cyclictest28310-21gltestperf0
375991916,2cyclictest0-21swapper/00
375991915,2cyclictest25439-21aten2_r9power_e0
375991914,3cyclictest7586-21grep0
375991914,3cyclictest6770-21aten2_r9power_p0
375991914,3cyclictest32615-21entropy0
375991914,3cyclictest30610-21sed0
375991914,3cyclictest29489-21idleruntime-cro0
375991914,3cyclictest18784-21/usr/sbin/munin0
375991914,3cyclictest15481-21sort0
375991914,3cyclictest12629-21uname0
375991914,3cyclictest10980-21ls0
375991914,3cyclictest10739-21ls0
375991913,4cyclictest7684-21cat0
375991913,4cyclictest7684-21cat0
375991913,4cyclictest32580-21cat0
375991913,4cyclictest24876-21seq0
375991913,4cyclictest24876-21seq0
375991913,4cyclictest23752-21expr0
375991913,4cyclictest23752-21expr0
375991912,5cyclictest2684-21awk0
287492190,4sleep128752-21pmu-power1
39199184,6cyclictest35-21ksoftirqd/33
38499182,4cyclictest17664-21apt-get2
384991813,3cyclictest6262-21entropy2
384991813,3cyclictest6262-21entropy2
384991813,3cyclictest23146-21sed2
384991811,4cyclictest6804-21cron2
38499181,14cyclictest15340-21latency_hist2
378991816,1cyclictest0-21swapper/11
378991816,1cyclictest0-21swapper/11
378991815,2cyclictest27379-21grep1
378991814,2cyclictest9933-21/usr/sbin/munin1
378991814,2cyclictest9499-21/usr/sbin/munin1
378991814,2cyclictest27704-21/usr/sbin/munin1
378991814,2cyclictest25761-21/usr/sbin/munin1
378991814,2cyclictest15216-21/usr/sbin/munin1
378991813,3cyclictest885-21seq1
378991813,3cyclictest6683-21/usr/sbin/munin1
378991813,3cyclictest5895-21sed1
378991813,3cyclictest4514-21cat1
378991813,3cyclictest3003-21expr1
378991813,3cyclictest24142-21expr1
378991813,3cyclictest23405-21expr1
378991813,3cyclictest20803-21/usr/sbin/munin1
378991813,3cyclictest20003-21needreboot1
378991813,3cyclictest16623-21ls1
378991812,4cyclictest13507-21cat1
37599182,10cyclictest665-21runrttasks0
375991816,1cyclictest0-21swapper/00
375991816,1cyclictest0-21swapper/00
375991815,2cyclictest18653-21perf0
375991814,2cyclictest3004-21/usr/sbin/munin0
375991814,2cyclictest29362-21df_inode0
375991814,2cyclictest17830-21if_enp2s00
375991813,3cyclictest9141-21cut0
375991813,3cyclictest801-21cat0
375991813,3cyclictest640-21aten2.4-expect0
375991813,3cyclictest31515-21idleruntime-cro0
375991813,3cyclictest25524-21sh0
375991813,3cyclictest23502-21idleruntime-cro0
375991813,3cyclictest22734-21seq0
375991813,3cyclictest18540-21ntp_states0
375991813,3cyclictest17811-21grep0
375991813,3cyclictest16838-21expr0
375991813,3cyclictest15148-21cat0
375991813,3cyclictest14813-21expr0
375991813,3cyclictest13415-21irqstats0
375991813,3cyclictest11143-21/usr/sbin/munin0
375991812,4cyclictest4292-21grep0
375991812,4cyclictest32337-21awk0
375991812,4cyclictest17270-21seq0
37599180,16cyclictest0-21swapper/00
384991715,1cyclictest0-21swapper/22
384991715,1cyclictest0-21swapper/22
38499171,1cyclictest0-21swapper/22
38499170,16cyclictest0-21swapper/22
378991713,2cyclictest2533-21munin-node1
378991712,3cyclictest665-21runrttasks1
378991712,3cyclictest665-21runrttasks1
378991711,3cyclictest11269-21cut1
378991710,4cyclictest23634-21cpuspeed_turbos1
375991713,2cyclictest4626-21/usr/sbin/munin0
375991712,3cyclictest9089-21expr0
375991712,3cyclictest7823-21cstates0
375991712,3cyclictest665-21runrttasks0
375991712,3cyclictest6392-21which0
375991712,3cyclictest303-21dump-pmu-power0
375991712,3cyclictest14822-21acpi0
37599170,15cyclictest0-21swapper/00
375991611,3cyclictest19283-21/usr/sbin/munin0
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional