You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 16:55
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack9slot2.osadl.org (updated Sun Nov 23, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
310602282252,17sleep00-21swapper/00
3135599278271,4cyclictest0-21swapper/00
309952258199,46sleep30-21swapper/33
310662248210,25sleep10-21swapper/11
3137499236230,4cyclictest14343-21apt-get3
309842232196,24sleep20-21swapper/22
3136099216208,5cyclictest13798-21apt-get1
3136599170165,3cyclictest0-21swapper/22
28272630,4sleep02823-21apt0
250992470,5sleep10-21swapper/11
250992470,5sleep10-21swapper/11
165482440,1sleep00-21swapper/00
286432420,2sleep30-21swapper/33
14042420,1sleep10-21swapper/11
153572390,1sleep00-21swapper/00
221192360,2sleep20-21swapper/22
3135599340,9cyclictest11308-21sed0
3135599340,9cyclictest11308-21sed0
18092340,1sleep20-21swapper/22
3136599330,31cyclictest7910-21kworker/2:12
3135599330,9cyclictest23949-21cat0
3135599330,10cyclictest6378-21sed0
3135599330,10cyclictest2491-21sed0
3135599310,9cyclictest25137-21latency_hist0
3135599310,16cyclictest11599-21grep0
3135599310,16cyclictest0-21swapper/00
3135599310,15cyclictest13633-21ntpq0
3135599290,9cyclictest9684-21latency_hist0
3135599290,10cyclictest2376-21hostname0
3135599280,9cyclictest29004-21cat0
3135599280,8cyclictest841-21latency_hist0
3135599270,10cyclictest30931-21tail0
257212270,2sleep30-21swapper/33
31374992622,2cyclictest341rcuc/33
3135599260,9cyclictest6856-21readlink0
3135599260,11cyclictest0-21swapper/00
3135599260,10cyclictest0-21swapper/00
3135599250,10cyclictest30074-21kernelversion0
3135599250,10cyclictest0-21swapper/00
3135599250,10cyclictest0-21swapper/00
3135599250,10cyclictest0-21swapper/00
220062250,2sleep23136599cyclictest2
31374992413,3cyclictest35-21ksoftirqd/33
3135599240,9cyclictest5467-21cstates0
3135599240,9cyclictest25879-21unixbench_multi0
3135599240,9cyclictest25879-21unixbench_multi0
3135599240,9cyclictest22688-21idleruntime-cro0
3135599240,9cyclictest15049-21wc0
3135599240,9cyclictest0-21swapper/00
98332230,1sleep10-21swapper/11
3136599232,15cyclictest28635-21apt-get2
3135599230,9cyclictest0-21swapper/00
3135599230,8cyclictest30267-21date0
23802230,3sleep12377-21grep1
31374992218,3cyclictest35-21ksoftirqd/33
31374992218,2cyclictest35-21ksoftirqd/33
3136599228,3cyclictest8824-21/usr/sbin/munin2
3136599227,13cyclictest12619-21date2
3136599223,18cyclictest28780-21awk2
3136599223,17cyclictest28517-21cat2
31365992216,3cyclictest22572-21aten2_r9power_e2
3136599221,18cyclictest6918-21perf2
31355992215,4cyclictest3565-21wc0
31355992215,4cyclictest10876-21munin-run0
31355992214,6cyclictest32198-21seq0
3135599220,9cyclictest20257-21plymouth0
31374992116,3cyclictest27396-21sendmail_mailst3
31374992116,3cyclictest24073-21threads3
31374992116,3cyclictest24073-21threads3
31374992115,4cyclictest25369-21df3
31374992114,5cyclictest28366-21apt3
31374992114,4cyclictest8135-21swap3
31374992113,5cyclictest35-21ksoftirqd/33
31374992110,9cyclictest2082-21systemd-journal3
3136599212,17cyclictest864-21latency_hist2
31365992116,3cyclictest3758-21uptime2
31365992116,3cyclictest12913-21munin-run2
3136599211,4cyclictest28116-21cron2
3136599211,18cyclictest505-21dbus-daemon2
3136599211,17cyclictest25088-21latency_hist2
3136599211,17cyclictest12565-1kworker/2:0H2
3136599210,19cyclictest25322-21kworker/2:02
31360992115,4cyclictest2904-21ls1
31360992115,3cyclictest22067-21sh1
31360992114,4cyclictest15221-21sh1
31360992114,4cyclictest1220-21cat1
31355992112,6cyclictest0-21swapper/00
3135599210,8cyclictest0-21swapper/00
3135599210,8cyclictest0-21swapper/00
3135599210,8cyclictest0-21swapper/00
3135599210,8cyclictest0-21swapper/00
3135599210,8cyclictest0-21swapper/00
114642210,3sleep3341rcuc/33
31374992015,3cyclictest29926-21open_files3
31374992015,3cyclictest22199-21open_files3
31374992015,3cyclictest20582-21aten2_r9power_p3
31374992014,4cyclictest16713-21wc3
31374992011,3cyclictest35-21ksoftirqd/33
3137499200,10cyclictest35-21ksoftirqd/33
3136599208,3cyclictest20635-21sendmail2
3136599207,3cyclictest6188-21expr2
31365992016,2cyclictest18560-21pmu-power2
31365992015,3cyclictest969-21df2
31365992015,3cyclictest665-21runrttasks2
31365992015,3cyclictest627-21acpi2
31365992014,4cyclictest2822-21ls2
31365992014,4cyclictest16223-21cat2
31365992014,3cyclictest32373-21sh2
31365992014,3cyclictest25604-21mailstats2
31365992014,3cyclictest25604-21mailstats2
3136599201,3cyclictest2082-21systemd-journal2
3136599201,2cyclictest22330-1kworker/2:0H2
3136599201,17cyclictest6150-21latency_hist2
3136599200,3cyclictest4898-21sleep2
3136599200,17cyclictest4676-21apt-key2
3136599200,17cyclictest12362-21ls2
31360992015,3cyclictest26966-21/usr/sbin/munin1
31360992015,3cyclictest21778-21ntp_states1
31360992015,3cyclictest14049-21sh1
31360992014,4cyclictest8289-21seq1
31360992014,4cyclictest505-21dbus-daemon1
31360992014,4cyclictest505-21dbus-daemon1
31360992014,4cyclictest31035-21/usr/sbin/munin1
31360992014,4cyclictest28141-21idleruntime-cro1
31360992014,4cyclictest20499-21seq1
31360992014,4cyclictest20254-21pluginstate1
31360992014,4cyclictest17097-21seq1
31360992014,3cyclictest21262-21missed_timers1
31360992014,3cyclictest17326-21sh1
31360992013,5cyclictest5752-21grep1
31360992012,5cyclictest21-21ksoftirqd/11
31355992015,3cyclictest19732-21tail0
31355992014,4cyclictest5738-21cat0
31355992014,4cyclictest5545-21seq0
31355992014,4cyclictest27246-21sed0
31355992013,4cyclictest22683-21awk0
3135599201,16cyclictest7804-21missed_timers0
31355992010,7cyclictest19515-21cut0
169522200,4sleep20-21swapper/22
3137499199,2cyclictest111rcu_preempt3
3137499198,2cyclictest111rcu_preempt3
31374991915,2cyclictest1289-21kernelversion3
31374991914,3cyclictest5872-21memory3
31374991914,3cyclictest35-21ksoftirqd/33
31374991914,3cyclictest32344-21unixbench-2d3
31374991914,3cyclictest2895-21ls3
31374991914,3cyclictest23858-21awk3
31374991914,3cyclictest22770-21missed_timers3
31374991914,3cyclictest21037-21dump-pmu-power3
31374991914,3cyclictest19288-21entropy3
31374991914,3cyclictest18738-21switchtime3
31374991914,3cyclictest17969-21missed_timers3
31374991914,3cyclictest15537-21expr3
31374991914,3cyclictest1506-21sed3
31374991914,3cyclictest14917-21unixbench_singl3
31374991914,3cyclictest12731-21timerwakeupswit3
31374991913,3cyclictest4168-21expr3
31374991912,4cyclictest505-21dbus-daemon3
3137499191,1cyclictest5810-21apt-get3
3137499191,15cyclictest8643-21sort3
3137499190,18cyclictest27875-21kworker/3:23
3137499190,16cyclictest2207-21head3
3136599192,12cyclictest7455-21apt-get2
31365991916,2cyclictest0-21swapper/22
31365991914,3cyclictest9607-21latency_hist2
31365991914,3cyclictest6483-21ls2
31365991914,3cyclictest29911-21ntp_states2
31365991914,3cyclictest26495-21find2
31365991914,3cyclictest19271-21uname2
31365991913,4cyclictest4834-21systemctl2
3136599191,17cyclictest0-21swapper/22
3136599190,3cyclictest32470-1kworker/2:1H2
3136599190,3cyclictest12565-1kworker/2:0H2
3136599190,16cyclictest17648-21df_inode2
3136099191,5cyclictest21-21ksoftirqd/11
31360991915,3cyclictest21-21ksoftirqd/11
31360991915,2cyclictest6450-21sshd1
31360991915,2cyclictest28362-21/usr/sbin/munin1
31360991915,2cyclictest25337-21/usr/sbin/munin1
31360991915,2cyclictest21288-21/usr/sbin/munin1
31360991914,3cyclictest5848-21aten2_r9power_e1
31360991914,3cyclictest30959-21cat1
31360991914,3cyclictest30303-21ntpq1
31360991914,3cyclictest2801-21ls1
31360991914,3cyclictest27360-21/usr/sbin/munin1
31360991914,3cyclictest24008-21/usr/sbin/munin1
31360991914,3cyclictest21-21ksoftirqd/11
31360991914,3cyclictest18116-21seq1
31360991914,3cyclictest15548-21cat1
31360991914,3cyclictest12008-21seq1
31360991914,3cyclictest10707-21/usr/sbin/munin1
31360991914,3cyclictest10364-21ntp_states1
31360991914,3cyclictest10364-21ntp_states1
31360991913,4cyclictest7436-21/usr/sbin/munin1
31360991913,4cyclictest31124-21expr1
31360991913,4cyclictest24350-21date1
3136099191,15cyclictest6402-21grep1
31355991916,2cyclictest0-21swapper/00
31355991913,3cyclictest31698-21interrupts0
31355991912,4cyclictest19774-21aten2.4-expect0
31355991911,5cyclictest24784-21cat0
3135599190,18cyclictest0-21swapper/00
3137499183,8cyclictest1050-21apt-get3
31374991814,2cyclictest1-21systemd3
31374991813,3cyclictest9128-21forks3
31374991813,3cyclictest4961-21date3
31374991813,3cyclictest19717-21cat3
31374991813,3cyclictest13539-21cat3
31374991813,3cyclictest12092-21sh3
31374991812,4cyclictest9530-21fw_packets3
31365991816,1cyclictest0-21swapper/22
31365991816,1cyclictest0-21swapper/22
31365991816,1cyclictest0-21swapper/22
31365991814,2cyclictest9306-21cat2
31365991814,2cyclictest13645-21ntp_states2
31365991813,3cyclictest9475-21/usr/sbin/munin2
31365991813,3cyclictest9475-21/usr/sbin/munin2
31365991813,3cyclictest29513-21cut2
31365991813,3cyclictest28039-21cut2
31365991813,3cyclictest25475-21expr2
31365991813,3cyclictest24244-21cpuspeed_turbos2
31365991812,4cyclictest18183-21unixbench_singl2
31365991812,3cyclictest24078-21timerandwakeup2
31365991812,3cyclictest24078-21timerandwakeup2
3136599181,16cyclictest15345-21awk2
3136599181,15cyclictest19912-21grep2
31360991814,2cyclictest4678-21aten2_r9power_e1
31360991813,3cyclictest656-21hostname1
31360991813,3cyclictest4111-21ls1
31360991813,3cyclictest31165-21cut1
31360991813,3cyclictest17919-21switchtime1
31360991813,3cyclictest17861-21cat1
31360991813,3cyclictest13791-21sh1
31360991813,3cyclictest12757-21/usr/sbin/munin1
31360991813,3cyclictest10178-21cat1
31360991812,4cyclictest8802-21pmu-power1
31360991812,4cyclictest23699-21expr1
31360991812,4cyclictest11786-21sendmail-msp1
31360991812,3cyclictest11611-21sed1
31360991810,5cyclictest10333-21gdbus1
31355991816,1cyclictest0-21swapper/00
31355991813,3cyclictest4529-21seq0
31355991813,3cyclictest32746-21/usr/sbin/munin0
31355991813,3cyclictest31549-21apt-get0
31355991813,3cyclictest17615-21/usr/sbin/munin0
31355991813,3cyclictest10707-21/usr/sbin/munin0
31355991812,4cyclictest24244-21sed0
31355991812,4cyclictest24244-21sed0
31355991811,5cyclictest17874-21seq0
31355991810,5cyclictest29905-21sh0
3137499174,4cyclictest665-21runrttasks3
3137499174,4cyclictest665-21runrttasks3
3137499172,9cyclictest16837-21apt-get3
31374991713,2cyclictest4898-21apt-config3
31374991712,3cyclictest30447-21cut3
31374991712,3cyclictest24663-21aten2_r9power_v3
31374991712,3cyclictest24663-21aten2_r9power_v3
31374991712,3cyclictest20160-21expr3
31374991712,3cyclictest17829-21uptime3
31374991712,3cyclictest14039-21cat3
31374991711,4cyclictest505-21dbus-daemon3
31374991711,4cyclictest21036-21telnet3
31365991712,3cyclictest7039-21ls2
31365991712,3cyclictest16680-21cat2
31365991712,3cyclictest14025-21ntp_kernel_pll_2
31360991713,3cyclictest21-21ksoftirqd/11
31360991713,2cyclictest31446-21/usr/sbin/munin1
31360991712,3cyclictest32191-21irqstats1
31360991712,3cyclictest19968-21cat1
31360991711,3cyclictest26611-21aten2.4-expect1
31360991710,4cyclictest28944-21grep1
3135599172,9cyclictest12029-21apt-get0
31355991712,3cyclictest22978-21gltestperf0
31355991711,4cyclictest18398-21seq0
31355991710,5cyclictest13555-21/usr/sbin/munin0
3137499169,2cyclictest111rcu_preempt3
3137499161,6cyclictest4372-21acpi3
31374991611,3cyclictest665-21runrttasks3
3136099169,4cyclictest13840-21grep1
31374991511,2cyclictest7714-21plymouth3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional