You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 22:22
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot2.osadl.org (updated Tue Sep 16, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
143122307270,24sleep00-21swapper/00
143122307270,24sleep00-21swapper/00
126802302244,46sleep20-21swapper/22
126802302244,46sleep20-21swapper/22
143652288250,25sleep30-21swapper/33
143652288250,25sleep30-21swapper/33
1462599263259,2cyclictest0-21swapper/33
141852262223,26sleep10-21swapper/11
141852262223,26sleep10-21swapper/11
1462299205201,3cyclictest0-21swapper/22
1462099136130,3cyclictest22303-21strings1
1461599128124,3cyclictest0-21swapper/00
279442570,2sleep30-21swapper/33
206242530,2sleep30-21swapper/33
1462599328,21cyclictest29926-21basename3
14615993111,3cyclictest9-21ksoftirqd/00
14615992911,4cyclictest9-21ksoftirqd/00
14615992911,4cyclictest9-21ksoftirqd/00
1461599289,8cyclictest9-21ksoftirqd/00
14615992812,3cyclictest9-21ksoftirqd/00
201142270,2sleep20-21swapper/22
1462299270,25cyclictest5333-21kworker/2:02
1462099274,11cyclictest21-21ksoftirqd/11
1462099271,6cyclictest21-21ksoftirqd/11
1461599277,3cyclictest9-21ksoftirqd/00
1461599276,5cyclictest9-21ksoftirqd/00
14615992715,2cyclictest9-21ksoftirqd/00
14615992712,3cyclictest9-21ksoftirqd/00
1461599271,19cyclictest0-21swapper/00
14620992622,3cyclictest201rcuc/11
1462099261,10cyclictest21-21ksoftirqd/11
1461599266,5cyclictest9-21ksoftirqd/00
1461599266,4cyclictest9-21ksoftirqd/00
1461599262,1cyclictest111rcu_preempt0
14615992612,3cyclictest9-21ksoftirqd/00
14615992611,6cyclictest9-21ksoftirqd/00
14622992519,4cyclictest23094-21sshd2
1462099253,6cyclictest21-21ksoftirqd/11
14620992522,2cyclictest201rcuc/11
14620992520,3cyclictest201rcuc/11
1462099251,6cyclictest21-21ksoftirqd/11
1462099251,2cyclictest111rcu_preempt1
1462099251,11cyclictest21-21ksoftirqd/11
1462099250,8cyclictest21-21ksoftirqd/11
1461599258,4cyclictest9-21ksoftirqd/00
1461599258,3cyclictest9-21ksoftirqd/00
14615992511,4cyclictest9-21ksoftirqd/00
14615992510,3cyclictest9-21ksoftirqd/00
14615992510,3cyclictest111rcu_preempt0
14622992418,4cyclictest26660-21irqstats2
14622992412,9cyclictest24142-21sh2
1462099249,2cyclictest111rcu_preempt1
1462099249,2cyclictest111rcu_preempt1
1462099248,3cyclictest111rcu_preempt1
1462099243,6cyclictest11593-21apt-get1
14620992420,3cyclictest201rcuc/11
14620992420,2cyclictest201rcuc/11
14620992420,2cyclictest201rcuc/11
14620992419,3cyclictest201rcuc/11
14620992419,3cyclictest201rcuc/11
1462099240,6cyclictest21-21ksoftirqd/11
1461599247,5cyclictest9-21ksoftirqd/00
1461599245,9cyclictest13000-1kworker/0:2H0
1461599245,8cyclictest9-21ksoftirqd/00
1461599242,2cyclictest111rcu_preempt0
14615992413,9cyclictest3415-21seq0
1462299231,19cyclictest29843-21munin-run2
1462299231,19cyclictest2167-21perf2
1462099239,2cyclictest111rcu_preempt1
1462099238,3cyclictest111rcu_preempt1
1462099238,3cyclictest111rcu_preempt1
1462099238,3cyclictest111rcu_preempt1
1462099238,2cyclictest111rcu_preempt1
1462099237,3cyclictest111rcu_preempt1
1462099232,5cyclictest21-21ksoftirqd/11
14620992320,2cyclictest201rcuc/11
14620992320,2cyclictest201rcuc/11
14620992319,2cyclictest201rcuc/11
14620992319,2cyclictest201rcuc/11
14620992314,6cyclictest505-21dbus-daemon1
14620992310,3cyclictest111rcu_preempt1
1462099230,8cyclictest21-21ksoftirqd/11
1462099230,7cyclictest21-21ksoftirqd/11
1461599239,3cyclictest111rcu_preempt0
1461599235,5cyclictest9-21ksoftirqd/00
1461599233,6cyclictest9-21ksoftirqd/00
256392220,4sleep0101rcuc/00
1462299229,10cyclictest10333-21gdbus2
1462299223,17cyclictest28651-21cat2
14622992213,7cyclictest22418-21latency_hist2
1462299221,18cyclictest9116-21apt-get2
1462099228,2cyclictest111rcu_preempt1
1462099227,3cyclictest111rcu_preempt1
1462099227,3cyclictest111rcu_preempt1
1462099226,3cyclictest111rcu_preempt1
1462099222,5cyclictest0-21swapper/11
1462099222,2cyclictest21-21ksoftirqd/11
1462099222,2cyclictest21-21ksoftirqd/11
1462099221,5cyclictest0-21swapper/11
1462099221,5cyclictest0-21swapper/11
1462099220,6cyclictest21-21ksoftirqd/11
1462099220,6cyclictest21-21ksoftirqd/11
1461599229,2cyclictest111rcu_preempt0
1461599228,4cyclictest9-21ksoftirqd/00
1461599228,3cyclictest9-21ksoftirqd/00
1461599226,4cyclictest9-21ksoftirqd/00
1461599225,4cyclictest9-21ksoftirqd/00
1461599222,2cyclictest9-21ksoftirqd/00
14615992215,5cyclictest24718-21aten2_r9power_c0
36852210,2sleep20-21swapper/22
235532210,2sleep3341rcuc/33
14625992116,3cyclictest22543-21ntp_offset3
14625992115,4cyclictest21097-21fw_packets3
14625992114,4cyclictest19215-21sendmail3
14625992113,4cyclictest2082-21systemd-journal3
1462299219,10cyclictest5446-21latency_hist2
1462299213,16cyclictest11351-21cat2
1462299212,17cyclictest7270-21latency_hist2
14622992116,3cyclictest24684-21acpi2
14622992115,3cyclictest5139-21munin-run2
1462299211,4cyclictest17043-21perf2
1462299211,18cyclictest17568-21latency_hist2
14622992110,7cyclictest27772-21apt-get2
1462299210,3cyclictest16755-21uname2
1462299210,3cyclictest12245-21perf2
1462299210,18cyclictest15424-21munin-run2
1462099213,6cyclictest21-21ksoftirqd/11
14620992117,3cyclictest201rcuc/11
1462099211,5cyclictest0-21swapper/11
1462099211,5cyclictest0-21swapper/11
1462099211,5cyclictest0-21swapper/11
1462099211,5cyclictest0-21swapper/11
1462099211,5cyclictest0-21swapper/11
14620992112,5cyclictest505-21dbus-daemon1
1462099210,7cyclictest21-21ksoftirqd/11
1462099210,7cyclictest21-21ksoftirqd/11
1462099210,2cyclictest111rcu_preempt1
1461599218,10cyclictest20823-21switchtime0
1461599216,3cyclictest9-21ksoftirqd/00
1461599215,7cyclictest111rcu_preempt0
1461599214,5cyclictest9-21ksoftirqd/00
1461599213,5cyclictest9-21ksoftirqd/00
1461599212,9cyclictest9-21ksoftirqd/00
14615992116,3cyclictest4918-21head0
14615992114,4cyclictest29988-21date0
1461599211,17cyclictest15032-21cut0
14615992110,2cyclictest111rcu_preempt0
1462599202,15cyclictest28077-21cut3
14625992015,3cyclictest32371-21interrupts3
14625992015,3cyclictest29668-21open_files3
14625992015,3cyclictest27418-21uptime3
14625992014,4cyclictest20386-21dump-pmu-power3
14625992014,4cyclictest18052-21fw_packets3
14625992014,3cyclictest1473-21basename3
14625992013,5cyclictest505-21dbus-daemon3
14625992013,4cyclictest2082-21systemd-journal3
1462299202,5cyclictest14539-21apt-get2
14622992015,3cyclictest8063-21irqstats2
14622992015,3cyclictest32572-21sh2
14622992015,3cyclictest32479-21kernelversion2
14622992015,3cyclictest18241-21interrupts2
14622992015,3cyclictest1807-21aten2_r9power_v2
14622992015,3cyclictest17487-21open_inodes2
1462299201,3cyclictest27845-1kworker/2:1H2
1462299201,17cyclictest8584-21awk2
1462299201,17cyclictest21536-21cat2
1462299201,16cyclictest3085-21munin-run2
1462299200,3cyclictest17725-21aten2.4-expect2
1462299200,3cyclictest12945-1kworker/2:0H2
1462299200,3cyclictest12543-21grep2
1462099209,2cyclictest111rcu_preempt1
1462099203,6cyclictest21-21ksoftirqd/11
1462099201,6cyclictest0-21swapper/11
1462099201,5cyclictest0-21swapper/11
14620992015,3cyclictest201rcuc/11
1462099200,6cyclictest21-21ksoftirqd/11
1461599209,3cyclictest111rcu_preempt0
1461599205,5cyclictest9-21ksoftirqd/00
1461599202,1cyclictest111rcu_preempt0
14615992015,3cyclictest9-21ksoftirqd/00
14615992015,3cyclictest7498-21seq0
14615992015,3cyclictest11851-21cut0
14615992014,4cyclictest13985-21expr0
14615992011,7cyclictest91ksoftirqd/00
14615992011,7cyclictest11255-21perf0
1462599194,8cyclictest358-21apt-get3
14625991914,3cyclictest5162-21idleruntime-cro3
14625991914,3cyclictest5070-21awk3
14625991914,3cyclictest24879-21ntp_states3
14625991914,3cyclictest20287-21interrupts3
14625991914,3cyclictest14969-21df3
14625991914,3cyclictest14587-21sed3
14625991914,3cyclictest12902-21uptime3
14625991914,3cyclictest12445-21ntp_kernel_pll_3
14625991914,3cyclictest10100-21cut3
14625991913,4cyclictest8853-21cat3
14625991913,4cyclictest31827-21find3
14625991913,4cyclictest24464-21idleruntime-cro3
14622991915,2cyclictest18758-21sendmail_mailst2
14622991914,3cyclictest7974-21fw_packets2
14622991914,3cyclictest4669-21pluginstate2
14622991914,3cyclictest26633-21sh2
14622991914,3cyclictest24552-21sendmail_mailst2
14622991914,3cyclictest20511-21missed_timers2
14622991913,3cyclictest32543-21sh2
1462299190,17cyclictest6241-21munin-run2
1462299190,17cyclictest31864-21munin-run2
1462099197,3cyclictest111rcu_preempt1
1462099191,6cyclictest0-21swapper/11
1461599198,2cyclictest111rcu_preempt0
1461599197,3cyclictest9-21ksoftirqd/00
1461599197,3cyclictest111rcu_preempt0
14615991915,2cyclictest2393-21/usr/sbin/munin0
1461599191,4cyclictest111rcu_preempt0
14615991914,3cyclictest9-21ksoftirqd/00
14615991914,3cyclictest20648-21find0
14615991914,3cyclictest17752-21hostname0
14615991913,4cyclictest8207-21seq0
14615991913,4cyclictest3173-21seq0
14625991816,1cyclictest0-21swapper/33
14625991816,1cyclictest0-21swapper/33
14625991816,1cyclictest0-21swapper/33
14625991813,3cyclictest28058-21unixbench_singl3
14625991813,3cyclictest26216-21cat3
14625991813,3cyclictest25546-21ntp_kernel_pll_3
14625991813,3cyclictest1997-21expr3
14625991813,3cyclictest19746-21sh3
14625991813,3cyclictest15837-21run-parts3
14625991813,3cyclictest14525-21ntp_states3
14625991813,3cyclictest12448-21sed3
14625991813,3cyclictest10182-21cat3
14625991813,3cyclictest10182-21cat3
14625991812,4cyclictest665-21runrttasks3
14625991812,4cyclictest5780-21cat3
14625991812,4cyclictest31987-21cat3
14625991812,4cyclictest18689-21basename3
14625991812,4cyclictest14087-21cat3
14625991812,3cyclictest2919-21cut3
1462299182,4cyclictest31734-21apt-get2
1462299181,5cyclictest23452-21apt-get2
14622991813,3cyclictest9978-21sh2
14622991813,3cyclictest9978-21sh2
14622991813,3cyclictest30697-21uname2
14622991813,3cyclictest28281-21if_err_enp2s02
14622991813,3cyclictest23408-21load2
14622991813,3cyclictest22080-21expr2
14622991813,3cyclictest18023-21cat2
14622991812,4cyclictest19175-21cat2
14622991812,3cyclictest13922-21expr2
1462299180,16cyclictest0-21swapper/22
1462299180,16cyclictest0-21swapper/22
14615991813,3cyclictest27361-21seq0
45742170,2sleep30-21swapper/33
1462599173,8cyclictest6756-21apt-get3
1462599173,8cyclictest15588-21apt-get3
1462599172,8cyclictest11145-21apt-get3
14625991715,1cyclictest0-21swapper/33
14625991712,3cyclictest7855-21seq3
14625991712,3cyclictest7719-21cut3
14625991712,3cyclictest6459-21expr3
14625991712,3cyclictest5538-21aten2.4-expect3
14625991712,3cyclictest3698-21cpuspeed_turbos3
14625991712,3cyclictest25740-21find3
14625991712,3cyclictest22038-21cstates3
14625991712,3cyclictest16694-21aten2_r9power_v3
14625991712,3cyclictest16322-21irqstats3
14625991712,3cyclictest15213-21/usr/sbin/munin3
14622991713,2cyclictest28847-21aten2_r9power_p2
14622991712,3cyclictest29758-21cut2
14622991712,3cyclictest27409-21sendmail2
14622991712,3cyclictest16030-21forks2
14622991712,3cyclictest11936-21diskstats2
14622991711,4cyclictest8403-21grep2
1462299170,16cyclictest0-21swapper/22
14625991614,1cyclictest0-21swapper/33
14625991610,3cyclictest8842-21apt-key3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional