You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 03:21
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Sat Nov 22, 2025 00:43:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25492297224,62sleep10-21swapper/11
24462263189,24sleep00-21swapper/00
284399260254,4cyclictest0-21swapper/00
24232259205,41sleep20-21swapper/22
24812252181,26sleep30-21swapper/33
285199182175,4cyclictest32134-21basename1
285599136129,4cyclictest32086-21munin-run2
286099113108,3cyclictest32132-21sendmail-msp3
279592510,1sleep10-21swapper/11
2851994610,21cyclictest25144-21seq1
2851994312,16cyclictest665-21runrttasks1
285599352,15cyclictest111rcu_preempt2
285199318,21cyclictest24339-21cron1
285199317,21cyclictest11878-21idleruntime-cro1
2851993115,13cyclictest4347-21ntp_offset1
2851993114,15cyclictest23905-21seq1
2851993111,18cyclictest1613-21cut1
285199310,2cyclictest7684-21aten2_r9power_c1
285199308,20cyclictest23606-21expr1
285199302,23cyclictest13130-21apt-get1
2851993014,14cyclictest8049-21/usr/sbin/munin1
2851993014,14cyclictest4161-21needreboot1
2851993014,14cyclictest4161-21needreboot1
2851993013,15cyclictest14182-21diskstats1
2851993013,14cyclictest7043-21seq1
2851993013,14cyclictest6844-21seq1
2851993013,14cyclictest11088-21seq1
285199300,3cyclictest7410-21apt-get1
285199300,2cyclictest207922sleep11
2855992917,3cyclictest28-21ksoftirqd/22
285199297,20cyclictest32500-21cat1
285199297,19cyclictest16494-21expr1
285199297,19cyclictest1585-21sed1
285199297,19cyclictest1165-21cron1
285199292,22cyclictest9704-21apt-get1
285199292,22cyclictest25340-21apt-get1
285199291,25cyclictest7674-21wc1
2851992910,17cyclictest3319-21sed1
285199290,4cyclictest26067-21chrt1
285199290,2cyclictest4128-21apt-get1
2855992813,5cyclictest28-21ksoftirqd/22
285199289,16cyclictest6955-21cpuspeed_turbos1
285199288,18cyclictest20202-21ls1
285199287,19cyclictest0-21swapper/11
285199287,18cyclictest30590-21grep1
285199287,18cyclictest27064-21cat1
285199287,18cyclictest26232-21gltestperf1
285199285,20cyclictest20753-21tail1
285199283,20cyclictest19630-21apt-get1
2851992813,13cyclictest2982-21/usr/sbin/munin1
285199281,22cyclictest13528-21apt-get1
285199280,3cyclictest16724-21seq1
285199280,3cyclictest12369-21ntp_states1
2855992710,14cyclictest1857-21idleruntime2
285599270,2cyclictest111rcu_preempt2
285199278,16cyclictest0-21swapper/11
285199277,18cyclictest22766-21grep1
285199277,17cyclictest9639-21sh1
285199276,19cyclictest12536-21df1
285199272,21cyclictest30716-21apt-get1
2851992713,12cyclictest665-21runrttasks1
2851992710,14cyclictest17174-21seq1
285199270,3cyclictest29357-21taskset1
285199267,17cyclictest3217-21taskset1
285199266,18cyclictest29183-21cut1
285199266,17cyclictest7890-21latency_hist1
285199266,17cyclictest7890-21latency_hist1
285199264,2cyclictest0-21swapper/11
285199264,2cyclictest0-21swapper/11
285199264,19cyclictest665-21runrttasks1
285199263,18cyclictest15826-21apt-get1
285199262,20cyclictest14149-21apt-get1
285199261,4cyclictest23653-21munin-run1
285199261,4cyclictest23653-21munin-run1
2851992610,14cyclictest5018-21pmu-power1
285599259,7cyclictest28-21ksoftirqd/22
285599251,20cyclictest9446-21munin-run2
285599251,20cyclictest1190-21perf2
285599251,19cyclictest15928-21munin-run2
285199256,16cyclictest4134-21cpu1
285199256,16cyclictest17247-21telnet1
285199253,5cyclictest0-21swapper/11
285199251,7cyclictest0-21swapper/11
295402240,2sleep30-21swapper/33
2860992410,4cyclictest35-21ksoftirqd/33
285599245,17cyclictest17791-21cat2
285599243,17cyclictest0-21swapper/22
2855992419,3cyclictest28-21ksoftirqd/22
2855992418,3cyclictest1680-21cron2
285599241,4cyclictest5760-21munin-run2
285199241,4cyclictest11286-21grep1
285199240,2cyclictest29138-21apt-get1
285599235,15cyclictest21630-21munin-run2
285599232,2cyclictest111rcu_preempt2
2855992316,4cyclictest31598-21turbostat2
2855992315,5cyclictest7385-21cron2
285599231,19cyclictest0-21swapper/22
2855992311,10cyclictest14893-21latency_hist2
285199231,4cyclictest18290-21tail1
2860992216,4cyclictest8767-21apt-get3
285599228,12cyclictest0-21swapper/22
285599223,17cyclictest1033-21latency_hist2
285599223,16cyclictest10539-21latency_hist2
285599223,13cyclictest17215-21apt-get2
285599222,18cyclictest19510-21cat2
285599222,18cyclictest19510-21cat2
2855992216,3cyclictest21398-21munin-run2
285599221,18cyclictest22138-21apt-config2
285599221,18cyclictest10333-21gdbus2
285599221,17cyclictest25106-21sendmail-msp2
285599220,4cyclictest7175-21idleruntime-cro2
285599213,16cyclictest32486-21latency_hist2
285599212,17cyclictest6452-21latency_hist2
2855992116,3cyclictest1444-21apt-get2
2855992115,4cyclictest2082-21systemd-journal2
285599211,3cyclictest11876-21sendmail2
285599211,2cyclictest281ksoftirqd/22
285599211,17cyclictest29874-21apt-config2
2855992111,2cyclictest28-21ksoftirqd/22
2855992110,3cyclictest28-21ksoftirqd/22
2843992116,3cyclictest17583-21latency_hist0
2843992115,4cyclictest9281-21grep0
2843992115,3cyclictest15573-21sendmail0
286099202,16cyclictest5293-21ls3
2860992018,1cyclictest0-21swapper/33
2860992015,3cyclictest9201-21perf3
2860992015,3cyclictest750-21forks3
2860992015,3cyclictest26417-21switchtime3
2860992015,3cyclictest2186-21memory3
2860992015,3cyclictest20365-21memory3
2860992015,3cyclictest17104-21open_inodes3
2860992015,3cyclictest16950-21uptime3
2860992015,3cyclictest11920-21cpuspeed3
2860992015,3cyclictest1153-21cron3
2860992014,4cyclictest7982-21fw_packets3
2860992014,4cyclictest28100-21memory3
2860992014,4cyclictest25517-21uname3
2860992014,4cyclictest12247-21seq3
286099201,2cyclictest111rcu_preempt3
286099200,18cyclictest11474-21latency_hist3
285599209,10cyclictest28-21ksoftirqd/22
285599202,6cyclictest3360-21apt-get2
285599202,6cyclictest3360-21apt-get2
285599202,16cyclictest9330-21latency_hist2
2855992016,2cyclictest17877-21aten2_r9power_e2
2855992015,3cyclictest28425-21latency_hist2
2855992014,3cyclictest31530-21wc2
285599201,3cyclictest26360-1kworker/2:0H2
285599201,17cyclictest26401-21latency_hist2
285599201,17cyclictest26360-1kworker/2:0H2
285599201,16cyclictest22386-21gltestperf2
285599201,16cyclictest22386-21gltestperf2
285599200,17cyclictest27532-21cron2
2843992015,3cyclictest9-21ksoftirqd/00
2843992015,3cyclictest7045-21rt-features0
2843992015,3cyclictest7045-21rt-features0
2843992015,3cyclictest28721-21pmu-power0
2843992015,3cyclictest10428-21proc_pri0
2843992014,4cyclictest9685-21/usr/sbin/munin0
2843992014,4cyclictest26643-21/usr/sbin/munin0
2843992014,4cyclictest22895-21expr0
2843992014,4cyclictest16959-21awk0
284399200,18cyclictest21780-21wc0
2860991915,2cyclictest8282-21if_err_enp2s03
2860991915,2cyclictest4570-21apt-config3
2860991915,2cyclictest2982-21/usr/sbin/munin3
2860991915,2cyclictest16794-21proc_pri3
2860991914,3cyclictest31755-21cat3
2860991914,3cyclictest26609-21kernelversion3
2860991914,3cyclictest22368-21switchtime3
2860991914,3cyclictest21693-21aten2_r9power_e3
2860991914,3cyclictest20243-21acpi3
2860991914,3cyclictest1934-21if_err_enp3s03
2860991914,3cyclictest19138-21pmu-power3
2860991914,3cyclictest15181-21aten2.4-expect3
2860991914,3cyclictest14835-21switchtime3
2860991914,3cyclictest10419-21grep3
2860991913,4cyclictest4370-21expr3
2860991913,4cyclictest24323-21expr3
2860991912,5cyclictest35-21ksoftirqd/33
2855991915,2cyclictest30548-21cat2
2855991914,3cyclictest8171-21sed2
2855991914,3cyclictest28-21ksoftirqd/22
2855991914,3cyclictest28-21ksoftirqd/22
285599191,3cyclictest3730-21munin-run2
285599191,3cyclictest25324-1kworker/2:2H2
285599191,2cyclictest0-21swapper/22
285599190,3cyclictest26360-1kworker/2:0H2
285599190,17cyclictest8486-21munin-run2
284399193,14cyclictest12276-21sort0
284399192,15cyclictest2227-21sort0
2843991915,2cyclictest9-21ksoftirqd/00
2843991915,2cyclictest9-21ksoftirqd/00
2843991915,2cyclictest20580-21cstates0
2843991915,2cyclictest10778-21/usr/sbin/munin0
2843991914,3cyclictest9191-21seq0
2843991914,3cyclictest8039-21seq0
2843991914,3cyclictest6761-21sed0
2843991914,3cyclictest665-21runrttasks0
2843991914,3cyclictest32159-21tr0
2843991914,3cyclictest30147-21expr0
2843991914,3cyclictest25701-21expr0
2843991914,3cyclictest23974-21sed0
2843991914,3cyclictest23227-21seq0
2843991914,3cyclictest13716-21sed0
2843991914,3cyclictest12221-21sh0
2843991913,4cyclictest8510-21sendmail0
2843991913,4cyclictest26328-21seq0
2843991913,4cyclictest23654-21date0
2843991913,4cyclictest20514-21awk0
2843991913,4cyclictest19429-21cat0
2843991913,4cyclictest18792-21seq0
2843991913,4cyclictest13970-21seq0
2843991912,5cyclictest14900-21sh0
2843991912,4cyclictest505-21dbus-daemon0
2843991912,4cyclictest505-21dbus-daemon0
2843991912,4cyclictest31122-21awk0
2843991912,4cyclictest26016-21cut0
284399190,16cyclictest25876-21aten2_r9power_v0
286099189,2cyclictest111rcu_preempt3
2860991814,3cyclictest35-21ksoftirqd/33
2860991814,2cyclictest6716-21apt3
2860991814,2cyclictest2687-21missed_timers3
2860991813,3cyclictest8284-21cpuspeed_turbos3
2860991813,3cyclictest32741-21expr3
2860991813,3cyclictest17985-21expr3
2860991813,3cyclictest17985-21expr3
2860991813,3cyclictest14177-21aten2.4-expect3
2860991813,3cyclictest10644-21cat3
2860991812,4cyclictest6758-21cat3
2860991812,4cyclictest5758-21expr3
2860991812,4cyclictest505-21dbus-daemon3
2860991812,4cyclictest29803-21cat3
2860991812,4cyclictest27665-21cut3
2860991812,4cyclictest25622-21ls3
2860991812,3cyclictest13662-21cpu3
286099181,16cyclictest0-21swapper/33
286099181,16cyclictest0-21swapper/33
286099180,17cyclictest0-21swapper/33
285599187,3cyclictest111rcu_preempt2
2855991814,2cyclictest9449-21/usr/sbin/munin2
2855991813,3cyclictest26520-21if_enp2s02
2855991813,3cyclictest13028-21open_inodes2
2855991813,3cyclictest10110-21expr2
2855991812,4cyclictest3734-21tr2
284399183,12cyclictest22196-21sed0
2843991814,2cyclictest15541-21/usr/sbin/munin0
2843991813,3cyclictest899-21ps0
2843991813,3cyclictest7211-21seq0
2843991813,3cyclictest665-21runrttasks0
2843991813,3cyclictest4869-21ntp_states0
2843991813,3cyclictest3411-21expr0
2843991813,3cyclictest32749-21sendmail_mailst0
2843991813,3cyclictest27752-21expr0
2843991813,3cyclictest2318-21expr0
2843991813,3cyclictest19375-21grep0
2843991813,3cyclictest19370-21latency_hist0
2843991813,3cyclictest19370-21latency_hist0
2843991813,3cyclictest1313-21cat0
2843991812,4cyclictest6054-21grep0
2843991812,4cyclictest27965-21seq0
2843991812,4cyclictest2082-21systemd-journal0
2843991812,3cyclictest29566-21ntpq0
2843991811,4cyclictest17377-21cat0
286099173,9cyclictest21887-21apt-get3
286099173,9cyclictest21887-21apt-get3
2860991715,1cyclictest0-21swapper/33
2860991715,1cyclictest0-21swapper/33
286099171,2cyclictest111rcu_preempt3
2860991712,3cyclictest31109-21expr3
2860991712,3cyclictest20256-21expr3
2860991712,3cyclictest18049-21ntp_kernel_err3
2860991712,3cyclictest12699-21if_enp3s03
2860991711,4cyclictest35-21ksoftirqd/33
2860991711,4cyclictest29816-21cut3
285599172,4cyclictest21782-21gltestperf2
2843991715,1cyclictest0-21swapper/00
2843991713,2cyclictest2320-21/usr/sbin/munin0
2843991712,3cyclictest665-21runrttasks0
2843991712,3cyclictest665-21runrttasks0
2843991712,3cyclictest665-21runrttasks0
2843991712,3cyclictest1461-21timerandwakeup0
2843991710,4cyclictest10177-21cut0
2860991611,3cyclictest27093-21entropy3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional