You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 00:15
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Tue Nov 18, 2025 12:43:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
281832328208,107sleep10-21swapper/11
264752285248,24sleep00-21swapper/00
2865199272262,7cyclictest9839-21awk1
2864799260250,5cyclictest9832-21sendmail-mta0
284442252180,25sleep20-21swapper/22
284802244209,22sleep30-21swapper/33
2865599187184,2cyclictest11250irq/128-xhci_hc2
2866099182177,3cyclictest9792-21munin-run3
274462800,2sleep00-21swapper/00
274462800,2sleep00-21swapper/00
227902620,4sleep12865199cyclictest1
29962540,2sleep10-21swapper/11
146502480,2sleep00-21swapper/00
165962440,4sleep10-21swapper/11
315652410,1sleep30-21swapper/33
154272380,1sleep30-21swapper/33
105742380,1sleep10-21swapper/11
2865199346,14cyclictest25671-21taskset1
2865199336,11cyclictest12583-21ntp_states1
2864799323,6cyclictest9-21ksoftirqd/00
2864799320,7cyclictest9-21ksoftirqd/00
2865199304,10cyclictest10130-21taskset1
2865199303,10cyclictest30325-21sed1
2865199301,18cyclictest0-21swapper/11
2865199301,18cyclictest0-21swapper/11
99112290,2sleep00-21swapper/00
2865199293,10cyclictest5063-21date1
2865199293,10cyclictest2977-21sed1
2865199293,10cyclictest2977-21sed1
2865199292,16cyclictest0-21swapper/11
2865199284,9cyclictest9632-21cat1
2865199284,10cyclictest10504-21grep1
2865199281,14cyclictest0-21swapper/11
2865199281,10cyclictest0-21swapper/11
2865199274,9cyclictest11105-21date1
2865199274,10cyclictest19465-21taskset1
2865199273,10cyclictest2091-21chrt1
2865199271,9cyclictest0-21swapper/11
2865199270,16cyclictest0-21swapper/11
2865199264,8cyclictest13168-21latency_hist1
2865199263,10cyclictest31822-21chrt1
2865199262,8cyclictest0-21swapper/11
2865199262,6cyclictest32332-21apt-get1
2865199261,9cyclictest0-21swapper/11
2865199261,8cyclictest0-21swapper/11
2865199261,15cyclictest0-21swapper/11
2865199261,15cyclictest0-21swapper/11
2865199261,15cyclictest0-21swapper/11
2865199260,9cyclictest0-21swapper/11
2865199260,16cyclictest0-21swapper/11
2865199260,16cyclictest0-21swapper/11
2864799269,9cyclictest9-21ksoftirqd/00
2864799266,11cyclictest9-21ksoftirqd/00
28647992623,2cyclictest101rcuc/00
2864799260,13cyclictest319572hostname0
28660992516,5cyclictest6088-21apt-get3
2865199252,9cyclictest665-21runrttasks1
2865199252,9cyclictest3998-21telnet1
2865199251,9cyclictest0-21swapper/11
2865199251,9cyclictest0-21swapper/11
2865199250,9cyclictest0-21swapper/11
2865199250,15cyclictest0-21swapper/11
2864799256,9cyclictest9-21ksoftirqd/00
2864799255,8cyclictest9-21ksoftirqd/00
2864799254,5cyclictest9-21ksoftirqd/00
2864799253,7cyclictest9-21ksoftirqd/00
28647992510,4cyclictest9-21ksoftirqd/00
126272250,4sleep0101rcuc/00
2865599246,16cyclictest24431-21idleruntime-cro2
2865599241,20cyclictest17191-21sendmail-msp2
2865199243,7cyclictest29899-21/usr/sbin/munin1
2865199242,9cyclictest0-21swapper/11
2865199241,9cyclictest17008-21seq1
2865199241,9cyclictest0-21swapper/11
2865199241,8cyclictest0-21swapper/11
2865199241,8cyclictest0-21swapper/11
2865199240,9cyclictest0-21swapper/11
2865199240,9cyclictest0-21swapper/11
2865199240,8cyclictest0-21swapper/11
2864799247,6cyclictest9-21ksoftirqd/00
2864799246,8cyclictest9-21ksoftirqd/00
2864799243,6cyclictest9-21ksoftirqd/00
28647992421,2cyclictest101rcuc/00
2864799241,6cyclictest9-21ksoftirqd/00
2864799240,6cyclictest9-21ksoftirqd/00
2865599231,5cyclictest762-21munin-run2
2865599231,4cyclictest16302-21munin-run2
2865599231,4cyclictest10333-21gdbus2
2865599231,15cyclictest0-21swapper/22
2865199232,8cyclictest0-21swapper/11
2865199231,9cyclictest0-21swapper/11
28651992311,6cyclictest18224-21apt-get1
2865199230,9cyclictest0-21swapper/11
2865199230,9cyclictest0-21swapper/11
2865199230,9cyclictest0-21swapper/11
2865199230,8cyclictest0-21swapper/11
2864799236,9cyclictest9-21ksoftirqd/00
2864799236,9cyclictest9-21ksoftirqd/00
2864799236,9cyclictest9-21ksoftirqd/00
2864799236,9cyclictest9-21ksoftirqd/00
2864799236,9cyclictest9-21ksoftirqd/00
2864799235,9cyclictest9-21ksoftirqd/00
2864799235,10cyclictest9-21ksoftirqd/00
2864799234,10cyclictest9-21ksoftirqd/00
2864799233,7cyclictest9-21ksoftirqd/00
2864799233,6cyclictest9-21ksoftirqd/00
28647992320,2cyclictest101rcuc/00
28647992320,2cyclictest101rcuc/00
2864799231,6cyclictest101rcuc/00
2864799231,5cyclictest9-21ksoftirqd/00
2864799230,7cyclictest9-21ksoftirqd/00
2865599223,17cyclictest9133-21cat2
2865599223,17cyclictest0-21swapper/22
2865599221,3cyclictest4862-21perf2
2865599221,18cyclictest23540-21munin-run2
2865199229,8cyclictest7130-21apt-get1
2865199221,9cyclictest0-21swapper/11
2865199221,8cyclictest0-21swapper/11
28651992216,4cyclictest20376-21idleruntime-cro1
2865199220,9cyclictest0-21swapper/11
2865199220,10cyclictest0-21swapper/11
2864799226,8cyclictest9-21ksoftirqd/00
2864799226,8cyclictest9-21ksoftirqd/00
2864799225,9cyclictest9-21ksoftirqd/00
2864799225,9cyclictest9-21ksoftirqd/00
2864799225,9cyclictest9-21ksoftirqd/00
2864799225,9cyclictest9-21ksoftirqd/00
2864799225,9cyclictest9-21ksoftirqd/00
2864799225,6cyclictest9-21ksoftirqd/00
2864799223,8cyclictest9-21ksoftirqd/00
2864799223,7cyclictest9-21ksoftirqd/00
2864799223,5cyclictest9-21ksoftirqd/00
28647992219,2cyclictest101rcuc/00
28647992211,3cyclictest9-21ksoftirqd/00
2864799220,10cyclictest9-21ksoftirqd/00
28660992116,3cyclictest30535-21sh3
28660992116,3cyclictest17435-21ntp_states3
28660992116,3cyclictest12739-21irqstats3
28660992115,4cyclictest8071-21forks3
28660992115,4cyclictest505-21dbus-daemon3
28660992115,3cyclictest20083-21cpuspeed_turbos3
2865599213,16cyclictest6869-21latency_hist2
2865599213,16cyclictest28-21ksoftirqd/22
2865599212,17cyclictest28012-21latency_hist2
28655992116,3cyclictest25870-21sed2
28655992116,3cyclictest25870-21sed2
28655992116,3cyclictest22962-21sed2
28655992116,3cyclictest18596-21sed2
28655992116,3cyclictest14522-21ls2
28655992115,4cyclictest31027-21sed2
28655992115,3cyclictest26604-21sed2
2865599211,3cyclictest21987-1kworker/2:2H2
2865599211,3cyclictest10333-21gdbus2
2865599211,18cyclictest2082-21systemd-journal2
2865599211,18cyclictest2082-21systemd-journal2
2865599210,5cyclictest7364-21cron2
2865599210,4cyclictest21271-21procmail2
2865599210,3cyclictest23694-1kworker/2:1H2
2865599210,3cyclictest22022-21munin-run2
2865599210,2cyclictest111rcu_preempt2
2865199210,8cyclictest0-21swapper/11
2864799215,8cyclictest4849-21turbostat.cron0
2864799214,9cyclictest9-21ksoftirqd/00
2864799214,9cyclictest9-21ksoftirqd/00
2864799214,9cyclictest9-21ksoftirqd/00
2864799214,9cyclictest9-21ksoftirqd/00
2864799214,9cyclictest9-21ksoftirqd/00
2864799214,9cyclictest9-21ksoftirqd/00
2864799214,9cyclictest9-21ksoftirqd/00
2864799213,9cyclictest9-21ksoftirqd/00
2864799213,9cyclictest9-21ksoftirqd/00
28647992118,2cyclictest101rcuc/00
2864799210,7cyclictest9-21ksoftirqd/00
2864799210,7cyclictest9-21ksoftirqd/00
174782210,3sleep20-21swapper/22
28660992015,3cyclictest9605-21sed3
28660992015,3cyclictest6081-21gltestperf3
28660992015,3cyclictest4859-21open_inodes3
28660992015,3cyclictest4000-21diskstats3
28660992015,3cyclictest32096-21unixbench_multi3
28660992015,3cyclictest31625-21cpuspeed3
28660992015,3cyclictest31543-21find3
28660992015,3cyclictest30307-21date3
28660992015,3cyclictest2446-21vmstat3
28660992015,3cyclictest2446-21vmstat3
28660992015,3cyclictest18961-21missed_timers3
28660992015,3cyclictest1640-21unixbench-2d3
28660992015,3cyclictest10551-21head3
28660992014,4cyclictest25990-21seq3
28660992014,4cyclictest2082-21systemd-journal3
28660992014,4cyclictest15923-21basename3
28660992013,4cyclictest19999-21users3
2865599205,13cyclictest2082-21systemd-journal2
2865599202,2cyclictest0-21swapper/22
28655992018,1cyclictest0-21swapper/22
28655992015,4cyclictest2171-21idleruntime-cro2
28655992014,4cyclictest6134-21grep2
28655992014,4cyclictest13127-21sed2
28655992014,4cyclictest12610-21cut2
28655992014,3cyclictest31298-21grep2
2865599201,2cyclictest21987-1kworker/2:2H2
2865599201,2cyclictest0-21swapper/22
2865599201,17cyclictest15917-21latency_hist2
2865199201,8cyclictest32214-21taskset1
28651992013,4cyclictest5938-21awk1
2864799204,8cyclictest9-21ksoftirqd/00
2864799201,6cyclictest9-21ksoftirqd/00
28647992016,2cyclictest101rcuc/00
28647992014,4cyclictest31224-21diskstats0
28660991915,2cyclictest4385-21unixbench_multi3
28660991915,2cyclictest22877-21cpuspeed_turbos3
28660991914,3cyclictest27873-21seq3
28660991914,3cyclictest24192-21ls3
28660991914,3cyclictest20668-21entropy3
28660991914,3cyclictest16681-21pluginstate3
28660991914,3cyclictest12508-21cat3
28660991914,3cyclictest10007-21seq3
28660991913,4cyclictest7341-21ls3
28660991913,4cyclictest30271-21seq3
28660991913,4cyclictest28513-21cat3
28660991913,4cyclictest1924-21expr3
28660991913,4cyclictest16057-21date3
28660991913,4cyclictest10848-21seq3
28660991913,3cyclictest524-21sendmail_mailst3
28660991912,4cyclictest17654-21cat3
28660991912,4cyclictest14528-21cat3
28660991912,4cyclictest11953-21cpuspeed_turbos3
28660991910,5cyclictest4058-21apt-get3
2866099190,7cyclictest604-21sendmail-mta3
2865599193,5cyclictest27233-21tail2
28655991915,2cyclictest1261-21munin-run2
28655991914,3cyclictest5284-21seq2
28655991914,3cyclictest28121-21idleruntime-cro2
28655991914,3cyclictest28121-21idleruntime-cro2
28655991914,3cyclictest1739-21cron2
28655991914,3cyclictest13409-21awk2
28655991914,3cyclictest11837-21grep2
28655991914,3cyclictest11356-21sed2
2865599190,5cyclictest11052-21cron2
2865599190,3cyclictest30767-21munin-node2
2865599190,3cyclictest23694-1kworker/2:1H2
2865599190,3cyclictest21987-1kworker/2:2H2
2865599190,17cyclictest3335-21idleruntime-cro2
2865599190,17cyclictest0-21swapper/22
2865599190,17cyclictest0-21swapper/22
2865599190,16cyclictest32163-21latency_hist2
2865199191,5cyclictest0-21swapper/11
28660991813,3cyclictest532-21if_enp2s03
28660991813,3cyclictest5301-21cat3
28660991813,3cyclictest497-21interrupts3
28660991813,3cyclictest30034-21cut3
28660991813,3cyclictest29604-21seq3
28660991813,3cyclictest27354-21interrupts3
28660991813,3cyclictest26724-21cat3
28660991813,3cyclictest26724-21cat3
28660991813,3cyclictest26353-21expr3
28660991813,3cyclictest24075-21apt-config3
28660991813,3cyclictest19431-21/usr/sbin/munin3
28660991813,3cyclictest19093-21unixbench-2d3
28660991813,3cyclictest16538-21expr3
28660991813,3cyclictest14577-21ntp_kernel_err3
28660991813,3cyclictest13167-21latency3
28660991813,3cyclictest10887-21cat3
28660991811,4cyclictest2082-21systemd-journal3
28655991816,1cyclictest0-21swapper/22
28655991816,1cyclictest0-21swapper/22
28655991816,1cyclictest0-21swapper/22
28655991816,1cyclictest0-21swapper/22
28655991816,1cyclictest0-21swapper/22
28655991814,2cyclictest3797-21latency_hist2
28655991814,2cyclictest19107-21awk2
28655991813,3cyclictest18117-21sed2
2865199180,6cyclictest0-21swapper/11
28660991712,3cyclictest6286-21expr3
28660991712,3cyclictest24735-21hostname3
28660991712,3cyclictest24735-21hostname3
28660991712,3cyclictest21867-21basename3
28660991711,4cyclictest23048-21cat3
2865599170,16cyclictest0-21swapper/22
2865599170,15cyclictest0-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional