You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 22:21
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack9slot2.osadl.org (updated Thu Nov 20, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291442295258,25sleep20-21swapper/22
291252289254,22sleep30-21swapper/33
290612266202,52sleep10-21swapper/11
284942264182,17sleep00-21swapper/00
2950599248244,2cyclictest0-21swapper/22
2949999214210,2cyclictest0-21swapper/00
2950199183177,4cyclictest29667-21strings1
2951299161158,2cyclictest0-21swapper/33
845821020,1sleep00-21swapper/00
2262521010,4sleep32951299cyclictest3
31152480,1sleep00-21swapper/00
319342340,2sleep20-21swapper/22
307382340,3sleep09-21ksoftirqd/00
29512993414,3cyclictest28873-21apt-config3
108042340,1sleep20-21swapper/22
29512993211,4cyclictest3271-21cat3
29512993114,2cyclictest29340-21cron3
29512993113,3cyclictest15210-21ls3
29512993112,3cyclictest3039-21cat3
29499993123,5cyclictest9-21ksoftirqd/00
29512993014,3cyclictest32233-21proc_pri3
29512993011,3cyclictest22730-21cat3
107822300,4sleep010773-21awk0
29512992911,3cyclictest30245-21cat3
29512992911,2cyclictest7724-21cut3
2951299283,9cyclictest8954-21apt-get3
29512992810,2cyclictest9907-21sendmail3
2951299271,7cyclictest10451-21taskset3
29512992711,3cyclictest17952-21timerwakeupswit3
2951299269,3cyclictest1330-21open_inodes3
29512992612,3cyclictest665-21runrttasks3
29512992612,2cyclictest14717-21latency3
29512992610,2cyclictest29008-21cmp3
29505992610,14cyclictest28-21ksoftirqd/22
103642260,4sleep010358-21systemctl0
2951299251,9cyclictest5992-21apt-get3
29512992511,2cyclictest22828-21aten2_r9power_v3
29512992510,2cyclictest25458-21latency_hist3
2950599255,18cyclictest0-21swapper/22
2951299241,6cyclictest24826-21apt-get3
29512992411,2cyclictest665-21runrttasks3
2950599243,19cyclictest0-21swapper/22
29505992418,3cyclictest23283-21head2
2951299234,4cyclictest0-21swapper/33
2950599234,17cyclictest24544-21cat2
2950599231,19cyclictest505-21dbus-daemon2
2950599231,19cyclictest20379-21latency_hist2
2949999232,14cyclictest23511-21chrt0
29499992318,3cyclictest20792-21cut0
29499992314,5cyclictest31704-21idleruntime-cro0
29512992218,2cyclictest25993-21basename3
2951299221,5cyclictest20151-21apt-get3
2950599229,11cyclictest2082-21systemd-journal2
2950599223,17cyclictest7365-21latency_hist2
2950599223,17cyclictest6662-21latency_hist2
2950599223,17cyclictest12894-21cat2
29505992216,4cyclictest1276-21users2
29505992215,5cyclictest10333-21gdbus2
2950599221,4cyclictest2082-21systemd-journal2
2950599221,4cyclictest1-21systemd2
2950599221,18cyclictest2239-21tr2
2950599220,19cyclictest5725-21munin-run2
2950599220,19cyclictest18524-21sendmail-msp2
29501992215,5cyclictest20854-21ls1
2949999225,15cyclictest15909-21cut0
2949999222,18cyclictest12799-21turbostat.cron0
29499992215,4cyclictest24767-21aten2.4-expect0
29499992215,4cyclictest22149-21grep0
29499992213,5cyclictest505-21dbus-daemon0
29499992213,5cyclictest2082-21systemd-journal0
29499992210,3cyclictest9-21ksoftirqd/00
29512992114,4cyclictest30614-21pluginstate3
2950599212,17cyclictest10332-21latency_hist2
29505992116,3cyclictest12567-21ls2
29505992115,4cyclictest27398-21if_err_enp3s02
29505992115,4cyclictest17832-21sh2
29505992115,4cyclictest10333-21gdbus2
2950599211,4cyclictest3286-21head2
29505992113,5cyclictest26617-21idleruntime-cro2
2950599210,3cyclictest7812-21latency_hist2
2950599210,18cyclictest19858-21cron2
29501992116,3cyclictest5852-21proc_pri1
29501992116,3cyclictest22864-21uptime1
29501992115,4cyclictest30797-21seq1
29501992114,4cyclictest4344-21cut1
29501992113,5cyclictest13666-21cat1
2950199211,16cyclictest28275-21/usr/sbin/munin1
2950199210,19cyclictest0-21swapper/11
29499992115,3cyclictest6281-21turbostat.cron0
29499992114,5cyclictest26017-21gltestperf0
29499992114,4cyclictest6541-21ls0
29499992114,4cyclictest13309-21needreboot0
281162210,2sleep20-21swapper/22
29512992015,3cyclictest956-21swap3
29512992015,3cyclictest5591-21/usr/sbin/munin3
29512992015,3cyclictest1263-21fw_packets3
29512992014,4cyclictest26432-21pmu-power3
29512992013,4cyclictest12674-21aten2.4-expect3
2950599202,2cyclictest26538-21cpuspeed_turbos2
29505992015,3cyclictest30178-21tr2
29505992015,3cyclictest20183-21sh2
29505992015,3cyclictest16852-21missed_timers2
29505992014,4cyclictest3977-21ls2
29505992014,4cyclictest20696-21sed2
2950599201,3cyclictest9902-21idleruntime-cro2
2950599201,3cyclictest25983-21perf2
2950599201,2cyclictest5376-1kworker/2:3H2
2950599201,17cyclictest29343-21latency_hist2
2950599201,17cyclictest10757-21munin-run2
2950599201,17cyclictest0-21swapper/22
2950599201,16cyclictest4647-21apt2
2950599200,3cyclictest12247-21munin-run2
2950599200,18cyclictest9613-21cmp2
2950599200,18cyclictest30823-1kworker/2:0H2
29501992015,3cyclictest4418-21seq1
29501992015,3cyclictest2738-21needreboot1
29501992015,3cyclictest24808-21/usr/sbin/munin1
29501992015,3cyclictest20635-21gltestperf1
29501992014,4cyclictest7959-21date1
29501992014,4cyclictest32329-21expr1
29501992014,4cyclictest26880-21apt1
29501992014,4cyclictest26185-21cut1
29501992014,4cyclictest22711-21cat1
29501992014,4cyclictest17513-21seq1
29501992014,4cyclictest15751-21seq1
29501992013,4cyclictest4416-21grep1
29501992013,4cyclictest31943-21ls1
29501992013,4cyclictest31775-21cat1
29501992013,4cyclictest31266-21df1
29501992013,4cyclictest30105-21expr1
29501992013,4cyclictest22657-21expr1
29501992013,4cyclictest15104-21cat1
2950199200,3cyclictest21-21ksoftirqd/11
29499992015,3cyclictest17491-21ls0
29499992015,3cyclictest16743-21aten2_r9power_e0
29499992014,4cyclictest8309-21dump-pmu-power0
29499992014,4cyclictest32080-21sh0
29499992014,4cyclictest27610-21grep0
29499992014,3cyclictest29361-21sleep0
2949999201,15cyclictest505-21dbus-daemon0
29512991915,2cyclictest18287-21ntp_states3
29512991914,3cyclictest7910-21processes3
29512991914,3cyclictest665-21runrttasks3
29512991914,3cyclictest3333-21idleruntime-cro3
29512991914,3cyclictest3249-21ls3
29512991914,3cyclictest24720-21/usr/sbin/munin3
29512991914,3cyclictest21473-21ntpq3
29512991914,3cyclictest2082-21systemd-journal3
29512991914,3cyclictest15626-21cpuspeed_turbos3
29512991914,3cyclictest10736-21load3
29512991913,4cyclictest6727-21cat3
29512991913,4cyclictest31715-21seq3
29512991912,4cyclictest35-21ksoftirqd/33
29512991912,4cyclictest11758-21memory3
29505991916,2cyclictest0-21swapper/22
29505991915,2cyclictest3465-21processes2
29505991914,3cyclictest30584-21ntp_states2
29505991914,3cyclictest28258-21acpi2
29505991914,3cyclictest2082-21systemd-journal2
29505991913,4cyclictest21141-21expr2
29505991913,3cyclictest31444-21cpuspeed2
29505991912,4cyclictest7066-21grep2
29505991912,4cyclictest5412-21awk2
2950599191,17cyclictest0-21swapper/22
2950599191,16cyclictest665-21runrttasks2
2950599191,16cyclictest14489-21latency_hist2
2950599190,17cyclictest8304-21munin-run2
2950599190,17cyclictest11754-21sendmail2
2950199192,15cyclictest2265-21wc1
29501991915,2cyclictest1913-21apt1
29501991915,2cyclictest1504-21/usr/sbin/munin1
29501991914,3cyclictest9208-21seq1
29501991914,3cyclictest8992-21expr1
29501991914,3cyclictest8001-21cracklib-runtim1
29501991914,3cyclictest5642-21sh1
29501991914,3cyclictest31655-21munin-node1
29501991914,3cyclictest30286-21/usr/sbin/munin1
29501991914,3cyclictest27565-21/usr/sbin/munin1
29501991914,3cyclictest26954-21seq1
29501991914,3cyclictest25884-21wc1
29501991914,3cyclictest19897-21threads1
29501991914,3cyclictest12207-21seq1
29501991913,4cyclictest665-21runrttasks1
29501991913,4cyclictest23854-21cat1
29501991913,4cyclictest19796-21ls1
29501991913,4cyclictest17132-21cat1
29501991913,4cyclictest15220-21basename1
29501991913,4cyclictest13091-21cat1
29501991913,4cyclictest12033-21seq1
29501991913,4cyclictest11404-21expr1
29501991913,3cyclictest10807-21needreboot1
29499991915,2cyclictest23317-21timerandwakeup0
29499991914,3cyclictest7616-21/usr/sbin/munin0
29499991914,3cyclictest3567-21/usr/sbin/munin0
29499991914,3cyclictest31926-21/usr/sbin/munin0
29499991914,3cyclictest25070-21ntp_states0
29499991914,3cyclictest22700-21/usr/sbin/munin0
29499991913,3cyclictest2408-21sed0
29499991912,4cyclictest29711-21gltestperf0
2949999191,13cyclictest415-21apt-get0
2951299183,12cyclictest6719-21sort3
29512991814,2cyclictest14298-21proc_pri3
29512991814,2cyclictest0-21swapper/33
29512991813,3cyclictest4939-21entropy3
29512991813,3cyclictest4693-21load3
29512991813,3cyclictest31551-21sed3
29512991813,3cyclictest29201-21seq3
29512991813,3cyclictest2340-21cat3
29512991813,3cyclictest12824-21needreboot3
29512991812,4cyclictest26505-21expr3
29512991812,4cyclictest19488-21ntpq3
2951299180,16cyclictest0-21swapper/33
2950599187,2cyclictest309032aten2_r9power_e2
29505991815,2cyclictest0-21swapper/22
29505991812,4cyclictest21784-21expr2
2950199183,9cyclictest32336-21apt-get1
29501991813,3cyclictest8886-21latency_hist1
29501991813,3cyclictest32295-21gzip1
29501991813,3cyclictest20759-21/usr/sbin/munin1
29501991813,3cyclictest19633-21processes1
29501991813,3cyclictest16687-21/usr/sbin/munin1
29501991813,3cyclictest12882-21ntpq1
29501991812,4cyclictest9330-21expr1
29501991812,4cyclictest803-21/usr/sbin/munin1
29501991812,3cyclictest4850-21expr1
2950199181,15cyclictest28019-21basename1
29499991816,1cyclictest0-21swapper/00
29499991815,2cyclictest0-21swapper/00
29499991814,2cyclictest18596-21/usr/sbin/munin0
29499991813,3cyclictest9-21ksoftirqd/00
29499991813,3cyclictest665-21runrttasks0
29499991813,3cyclictest665-21runrttasks0
29499991813,3cyclictest6435-21grep0
29499991813,3cyclictest501-21interrupts0
29499991813,3cyclictest31399-21basename0
29499991813,3cyclictest30193-21sed0
29499991813,3cyclictest19440-21/usr/sbin/munin0
29499991813,3cyclictest14657-21/usr/sbin/munin0
29499991813,3cyclictest13078-21cat0
29499991812,4cyclictest27350-21grep0
29499991812,4cyclictest22850-21cut0
29499991812,4cyclictest1208-21awk0
29499991811,4cyclictest4415-21grep0
29499991811,4cyclictest31371-21aten2_r9power_v0
238262180,7sleep121-21ksoftirqd/11
29512991713,2cyclictest16652-21aten2.4-expect3
29512991713,2cyclictest10945-21cstates3
29512991712,3cyclictest16920-21ntp_kernel_err3
2950599179,5cyclictest19458-21apt-get2
2950599170,16cyclictest0-21swapper/22
29501991713,2cyclictest18083-21/usr/sbin/munin1
29501991713,2cyclictest13266-21interrupts1
29499991712,3cyclictest665-21runrttasks0
29499991712,3cyclictest665-21runrttasks0
29499991712,3cyclictest20699-21aten2.4-expect0
29499991712,3cyclictest18200-21aten2_r9power_v0
29499991712,3cyclictest17465-21cut0
29499991711,4cyclictest665-21runrttasks0
29499991711,4cyclictest26598-21cat0
29499991711,4cyclictest2076-21cat0
29512991612,2cyclictest20795-21unixbench_singl3
29512991611,3cyclictest23359-21basename3
29512991610,4cyclictest9139-21basename3
2951299160,15cyclictest0-21swapper/33
29501991612,2cyclictest3567-21/usr/sbin/munin1
29499991614,1cyclictest0-21swapper/00
29499991610,4cyclictest18769-21cut0
29499991610,4cyclictest1-21systemd0
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional