You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-24 - 15:38
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Fri Oct 24, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4142306269,25sleep10-21swapper/11
327662299260,25sleep20-21swapper/22
3752298261,24sleep30-21swapper/33
5392264181,52sleep00-21swapper/00
73699241235,4cyclictest7361-21perf1
73999132127,3cyclictest0-21swapper/22
73199120114,4cyclictest7367-21basename0
74499105100,3cyclictest7349-21munin-run3
22972530,4sleep273999cyclictest2
295962450,1sleep10-21swapper/11
295682450,2sleep10-21swapper/11
38052440,0sleep00-21swapper/00
73999384,20cyclictest24150-21unixbench_multi2
744993713,4cyclictest35-21ksoftirqd/33
744993713,4cyclictest35-21ksoftirqd/33
73999352,17cyclictest0-21swapper/22
73999349,9cyclictest3793-21idleruntime-cro2
73999343,9cyclictest18492-21sed2
73199333,3cyclictest9-21ksoftirqd/00
731993329,2cyclictest101rcuc/00
731993329,2cyclictest101rcuc/00
73999327,9cyclictest1723-21cron2
73999327,9cyclictest1723-21cron2
73999322,14cyclictest12859-21tr2
731993212,4cyclictest9-21ksoftirqd/00
73999316,9cyclictest26590-21dump-pmu-power2
73999314,9cyclictest18845-21grep2
73999313,10cyclictest23878-21sed2
73999313,10cyclictest14027-21sed2
73999313,10cyclictest12547-21sed2
73999312,10cyclictest17738-21mailstats2
73999303,10cyclictest27638-21sed2
73999295,9cyclictest13926-21cron2
73999294,9cyclictest32184-21sed2
73999294,8cyclictest31673-21sed2
73999291,15cyclictest0-21swapper/22
731992924,3cyclictest101rcuc/00
731992924,3cyclictest101rcuc/00
74499288,4cyclictest35-21ksoftirqd/33
73999286,19cyclictest2692-21munin-setwarn2
73999286,19cyclictest2692-21munin-setwarn2
73999283,10cyclictest21481-21ntpq2
73999280,18cyclictest0-21swapper/22
731992815,3cyclictest9-21ksoftirqd/00
74499276,8cyclictest35-21ksoftirqd/33
73999275,20cyclictest13545-21perf2
73999273,10cyclictest16243-21apt-get2
73999272,9cyclictest28801-21cat2
73999271,9cyclictest28905-21if_enp3s02
731992715,2cyclictest111rcu_preempt0
731992713,3cyclictest111rcu_preempt0
74499266,9cyclictest35-21ksoftirqd/33
73999264,7cyclictest21675-21cut2
73999262,9cyclictest21239-21grep2
73199269,6cyclictest9-21ksoftirqd/00
73199269,4cyclictest9-21ksoftirqd/00
73199264,6cyclictest9-21ksoftirqd/00
731992614,3cyclictest9-21ksoftirqd/00
731992613,3cyclictest9-21ksoftirqd/00
230562260,2sleep30-21swapper/33
74499255,10cyclictest35-21ksoftirqd/33
744992515,3cyclictest35-21ksoftirqd/33
73999255,17cyclictest5451-21sort2
73999252,8cyclictest11039-21ntp_states2
73999251,12cyclictest2082-21systemd-journal2
731992521,2cyclictest101rcuc/00
731992521,2cyclictest101rcuc/00
731992520,3cyclictest10717-21turbostat.cron0
731992513,3cyclictest9-21ksoftirqd/00
731992510,3cyclictest111rcu_preempt0
73999244,5cyclictest0-21swapper/22
73999243,19cyclictest2442-21cat2
739992421,2cyclictest271rcuc/22
73999241,9cyclictest20192-21missed_timers2
73999240,17cyclictest28-21ksoftirqd/22
73699240,5cyclictest10971-21avahi-daemon1
73199249,3cyclictest9-21ksoftirqd/00
73199243,6cyclictest9-21ksoftirqd/00
731992419,3cyclictest9-21ksoftirqd/00
731992416,3cyclictest9-21ksoftirqd/00
74499239,2cyclictest111rcu_preempt3
74499239,2cyclictest111rcu_preempt3
74499238,6cyclictest111rcu_preempt3
744992313,3cyclictest35-21ksoftirqd/33
744992313,3cyclictest35-21ksoftirqd/33
744992310,10cyclictest35-21ksoftirqd/33
74499230,8cyclictest35-21ksoftirqd/33
73999233,18cyclictest23374-21cat2
73999233,17cyclictest5012-21latency_hist2
73999231,9cyclictest27781-21chrt2
73999231,4cyclictest28-21ksoftirqd/22
73699231,8cyclictest19158-21munin-run1
73199231,5cyclictest9-21ksoftirqd/00
731992315,2cyclictest9-21ksoftirqd/00
731992312,3cyclictest9-21ksoftirqd/00
731992312,3cyclictest9-21ksoftirqd/00
731992311,3cyclictest9-21ksoftirqd/00
731992311,2cyclictest111rcu_preempt0
73199231,11cyclictest9-21ksoftirqd/00
731992310,3cyclictest111rcu_preempt0
731992310,3cyclictest111rcu_preempt0
73199230,7cyclictest9-21ksoftirqd/00
73199230,6cyclictest9-21ksoftirqd/00
74499228,5cyclictest35-21ksoftirqd/33
744992217,3cyclictest35-21ksoftirqd/33
744992217,3cyclictest35-21ksoftirqd/33
744992216,3cyclictest22506-21cron3
744992214,5cyclictest7734-21grep3
744992210,3cyclictest35-21ksoftirqd/33
73999225,8cyclictest28-21ksoftirqd/22
73999224,15cyclictest5066-21gltestperf2
73999223,17cyclictest28529-21cat2
73999220,4cyclictest415-1kworker/2:0H2
73199229,3cyclictest9-21ksoftirqd/00
73199229,3cyclictest9-21ksoftirqd/00
73199228,4cyclictest9-21ksoftirqd/00
73199227,3cyclictest111rcu_preempt0
73199226,7cyclictest9-21ksoftirqd/00
73199223,7cyclictest9-21ksoftirqd/00
73199221,9cyclictest9-21ksoftirqd/00
73199221,7cyclictest9-21ksoftirqd/00
731992211,2cyclictest111rcu_preempt0
731992210,3cyclictest9-21ksoftirqd/00
731992210,2cyclictest111rcu_preempt0
73199220,8cyclictest9-21ksoftirqd/00
74499219,3cyclictest35-21ksoftirqd/33
74499219,2cyclictest111rcu_preempt3
74499218,5cyclictest111rcu_preempt3
74499218,4cyclictest35-21ksoftirqd/33
74499216,2cyclictest111rcu_preempt3
74499213,5cyclictest35-21ksoftirqd/33
74499213,2cyclictest111rcu_preempt3
744992116,3cyclictest26087-21if_enp3s03
744992115,4cyclictest20821-21users3
744992110,3cyclictest111rcu_preempt3
73999215,8cyclictest28-21ksoftirqd/22
739992113,4cyclictest30509-21systemctl2
73999211,2cyclictest32550-21perf2
73999211,2cyclictest32550-21perf2
739992111,2cyclictest111rcu_preempt2
736992116,3cyclictest20189-21/usr/sbin/munin1
736992115,4cyclictest9737-21seq1
736992115,4cyclictest28468-21seq1
736992114,4cyclictest30677-21sendmail-msp1
736992114,4cyclictest26257-21sed1
736992114,4cyclictest11174-21cat1
73699210,18cyclictest26722-21cut1
73199217,4cyclictest9-21ksoftirqd/00
73199217,4cyclictest9-21ksoftirqd/00
73199216,4cyclictest111rcu_preempt0
73199216,3cyclictest9-21ksoftirqd/00
73199216,3cyclictest111rcu_preempt0
731992116,3cyclictest9-21ksoftirqd/00
731992114,4cyclictest19451-21grep0
731992111,2cyclictest9-21ksoftirqd/00
731992110,3cyclictest9-21ksoftirqd/00
73199210,6cyclictest9-21ksoftirqd/00
73199210,11cyclictest9-21ksoftirqd/00
74499208,3cyclictest35-21ksoftirqd/33
74499208,3cyclictest111rcu_preempt3
74499207,6cyclictest111rcu_preempt3
74499207,6cyclictest111rcu_preempt3
74499204,2cyclictest111rcu_preempt3
744992015,3cyclictest35-21ksoftirqd/33
744992015,3cyclictest35-21ksoftirqd/33
744992015,3cyclictest35-21ksoftirqd/33
744992015,3cyclictest1710-21timerwakeupswit3
744992015,2cyclictest8514-21basename3
744992014,4cyclictest3527-21cat3
744992014,3cyclictest9764-21cpu3
744992013,4cyclictest6850-21expr3
74499200,10cyclictest35-21ksoftirqd/33
73999203,10cyclictest2534-21chrt2
73999202,6cyclictest28-21ksoftirqd/22
73999202,6cyclictest28-21ksoftirqd/22
73999202,5cyclictest28-21ksoftirqd/22
739992017,2cyclictest271rcuc/22
739992017,2cyclictest0-21swapper/22
73999201,6cyclictest28-21ksoftirqd/22
73999201,6cyclictest28-21ksoftirqd/22
73999201,5cyclictest0-21swapper/22
73999201,17cyclictest5290-21cron2
736992015,3cyclictest3227-21/usr/sbin/munin1
736992015,3cyclictest27526-21/usr/sbin/munin1
736992015,3cyclictest17564-21seq1
736992014,4cyclictest9326-21expr1
736992014,4cyclictest871-21seq1
736992014,4cyclictest2744-21expr1
736992014,4cyclictest19827-21expr1
736992014,4cyclictest19738-21seq1
736992014,3cyclictest21131-21idleruntime1
736992014,3cyclictest12933-21seq1
736992013,4cyclictest30979-21cat1
736992013,4cyclictest28527-21sed1
736992013,4cyclictest23825-21cat1
73199206,3cyclictest111rcu_preempt0
73199203,9cyclictest9-21ksoftirqd/00
731992016,3cyclictest9-21ksoftirqd/00
731992015,3cyclictest29167-21gltestperf0
73199200,9cyclictest9-21ksoftirqd/00
73199200,9cyclictest9-21ksoftirqd/00
73199200,6cyclictest9-21ksoftirqd/00
74499198,3cyclictest35-21ksoftirqd/33
74499196,2cyclictest111rcu_preempt3
744991914,3cyclictest35-21ksoftirqd/33
744991914,3cyclictest30849-21aten2_r9power_p3
744991914,3cyclictest30849-21aten2_r9power_p3
744991914,3cyclictest25092-21df_inode3
744991914,3cyclictest16217-21ntp_kernel_pll_3
744991914,3cyclictest11342-21hostname3
744991913,4cyclictest19204-21rt-features3
744991913,4cyclictest15051-21nfs4_client3
744991913,3cyclictest20196-21acpi3
744991912,5cyclictest4385-21cat3
73999192,5cyclictest28-21ksoftirqd/22
73999192,15cyclictest20091-21sed2
73999191,6cyclictest28-21ksoftirqd/22
73999191,6cyclictest28-21ksoftirqd/22
73999191,6cyclictest28-21ksoftirqd/22
73999191,5cyclictest28-21ksoftirqd/22
73699196,4cyclictest21-21ksoftirqd/11
736991915,2cyclictest466-21apt1
736991915,2cyclictest4543-21/usr/sbin/munin1
736991915,2cyclictest3506-21/usr/sbin/munin1
736991915,2cyclictest18163-21/usr/sbin/munin1
736991915,2cyclictest15340-21/usr/sbin/munin1
736991915,2cyclictest14638-21/usr/sbin/munin1
736991914,3cyclictest6442-21seq1
736991914,3cyclictest27098-21sed1
736991914,3cyclictest23789-21threads1
736991914,3cyclictest22283-21ntp_kernel_err1
736991914,3cyclictest18075-21sed1
736991913,4cyclictest7568-21seq1
736991913,4cyclictest505-21dbus-daemon1
736991913,4cyclictest26208-21sed1
736991913,4cyclictest24653-21/usr/sbin/munin1
736991913,4cyclictest14585-21cat1
736991913,4cyclictest13783-21head1
736991912,4cyclictest29295-21cat1
736991912,4cyclictest2779-21sort1
731991914,3cyclictest12731-21aten2.4-expect0
73199190,9cyclictest9-21ksoftirqd/00
74499183,4cyclictest35-21ksoftirqd/33
74499183,2cyclictest35-21ksoftirqd/33
74499182,2cyclictest111rcu_preempt3
744991814,3cyclictest35-21ksoftirqd/33
744991813,3cyclictest3531-21cut3
744991813,3cyclictest35-21ksoftirqd/33
744991813,3cyclictest2853-21cstates3
744991813,3cyclictest19040-21wc3
744991813,3cyclictest10732-21if_enp2s03
73999181,5cyclictest28-21ksoftirqd/22
73999181,5cyclictest0-21swapper/22
736991814,2cyclictest10043-21df1
736991813,3cyclictest9974-21apt-config1
736991813,3cyclictest30775-21aten2_r9power_c1
736991813,3cyclictest30775-21aten2_r9power_c1
736991813,3cyclictest23764-21seq1
736991813,3cyclictest22876-21sh1
736991813,3cyclictest2113-21hostname1
736991813,3cyclictest12530-21ls1
736991813,3cyclictest10112-21/usr/sbin/munin1
736991812,4cyclictest31202-21expr1
736991812,4cyclictest21913-21cat1
736991811,4cyclictest16777-21cut1
736991811,4cyclictest1612-21cat1
736991811,4cyclictest1612-21cat1
731991811,5cyclictest19063-21expr0
74499172,2cyclictest111rcu_preempt3
744991712,3cyclictest10969-21expr3
73699179,5cyclictest505-21dbus-daemon1
736991715,1cyclictest0-21swapper/11
736991712,3cyclictest8485-21ntpq1
736991712,3cyclictest32377-21/usr/sbin/munin1
736991712,3cyclictest32377-21/usr/sbin/munin1
736991712,3cyclictest1556-21cat1
736991712,3cyclictest12370-21latency1
736991711,4cyclictest20532-21cut1
73699170,16cyclictest0-21swapper/11
73699160,1cyclictest0-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional