You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 23:55
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot2.osadl.org (updated Wed Dec 03, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87592341273,35sleep30-21swapper/33
89902309270,26sleep00-21swapper/00
9208992840,1cyclictest0-21swapper/22
87922257218,26sleep10-21swapper/11
90322239188,38sleep20-21swapper/22
921099218214,2cyclictest0-21swapper/33
2668421680,5sleep1920399cyclictest1
3123721450,6sleep20-21swapper/22
920399106103,2cyclictest0-21swapper/11
174192650,16sleep121-21ksoftirqd/11
269532540,1sleep00-21swapper/00
245382530,2sleep024541-21sed0
176622440,2sleep30-21swapper/33
250392410,1sleep20-21swapper/22
9210993512,20cyclictest2082-21systemd-journal3
921099329,20cyclictest27954-21cat3
9210993116,13cyclictest13434-21uptime3
9210993116,13cyclictest11718-21ntp_states3
9210993115,13cyclictest30994-21ntp_states3
921099310,5cyclictest18886-21grep3
921099310,4cyclictest3113-21memory3
921099310,3cyclictest1450-21df3
921099309,18cyclictest11321-21cut3
921099308,19cyclictest29263-21basename3
9210993016,12cyclictest13329-21proc_pri3
9210993011,17cyclictest22336-21expr3
9210993011,16cyclictest20530-21cat3
9210993010,17cyclictest18438-21expr3
920899300,28cyclictest24225-21kworker/2:02
921099298,18cyclictest992-21uname3
921099296,20cyclictest505-21dbus-daemon3
921099293,21cyclictest1743-21apt-get3
9210992913,13cyclictest20142-21ntp_states3
9210992912,15cyclictest21567-21seq3
921099290,3cyclictest11938-21df3
921099290,2cyclictest0-21swapper/33
921099289,16cyclictest5473-21swap3
921099288,17cyclictest9528-21unixbench-2d3
921099288,17cyclictest24234-21seq3
921099287,18cyclictest27778-21cat3
921099287,18cyclictest13349-21sed3
921099287,18cyclictest1183-21ntpq3
921099287,18cyclictest1183-21ntpq3
921099279,16cyclictest26386-21rt-features3
921099278,16cyclictest27206-21missed_timers3
921099278,16cyclictest25496-21head3
921099277,17cyclictest2082-21systemd-journal3
921099277,17cyclictest11257-21cron3
921099276,5cyclictest26570-21gltestperf3
921099271,6cyclictest14396-21apt-get3
921099269,15cyclictest7430-21/usr/sbin/munin3
921099269,15cyclictest31878-21cron3
921099267,17cyclictest23859-21grep3
921099267,16cyclictest28707-21if_err_enp3s03
921099261,4cyclictest6863-21munin-run3
9210992612,12cyclictest12369-21forks3
9210992611,13cyclictest665-21runrttasks3
9210992610,14cyclictest31775-21cpuspeed_turbos3
9210992610,14cyclictest31775-21cpuspeed_turbos3
921099257,16cyclictest24621-21unixbench-2d3
921099256,3cyclictest15616-21aten2_r9power_c3
921099251,4cyclictest30511-21apt-get3
921099251,4cyclictest16105-21apt-get3
9203992515,6cyclictest505-21dbus-daemon1
921099247,15cyclictest5010-21ntp_kernel_err3
921099247,14cyclictest15464-21vmstat3
921099246,15cyclictest17684-21aten2_r9power_e3
921099246,15cyclictest15127-21timerandwakeup3
921099243,3cyclictest8902-21/usr/sbin/munin3
921099242,6cyclictest13554-21apt-get3
921099242,4cyclictest21289-21apt-get3
921099241,6cyclictest28489-21apt-get3
921099241,4cyclictest31198-21diskstats3
921099241,3cyclictest14907-21apt-get3
9198992419,3cyclictest16189-21sendmail0
9198992411,3cyclictest16134-21gltestperf0
921099236,3cyclictest24495-21expr3
921099232,5cyclictest505-21dbus-daemon3
921099231,5cyclictest16882-21ntp_offset3
921099231,4cyclictest6220-21diskstats3
921099231,4cyclictest31178-21munin-run3
921099230,4cyclictest8721-21ntp_states3
921099230,4cyclictest3944-21ntp_states3
920899234,17cyclictest8055-21cat2
920899234,17cyclictest2107-21latency_hist2
920899234,17cyclictest2107-21latency_hist2
920399233,3cyclictest251052munin-plugin-st1
9203992317,4cyclictest15715-21cron1
921099221,3cyclictest6444-21/usr/sbin/munin3
920899223,17cyclictest15844-21latency_hist2
9208992217,3cyclictest23058-21munin-plugin-st2
9208992216,4cyclictest28-21ksoftirqd/22
920899221,3cyclictest4085-21sendmail-mta2
9203992214,5cyclictest11279-21sh1
9203992213,5cyclictest23896-21sendmail1
9198992215,4cyclictest14269-21cut0
9208992116,3cyclictest21428-21sh2
9208992116,3cyclictest15723-21munin-run2
9208992115,4cyclictest11085-21seq2
9208992115,3cyclictest32277-21/usr/sbin/munin2
920899211,18cyclictest31883-21munin-run2
920899211,18cyclictest17389-21idleruntime-cro2
920899211,17cyclictest4141-21latency_hist2
920899211,17cyclictest23330-21grep2
920899211,17cyclictest15524-21cat2
920899211,17cyclictest12284-21grep2
920899211,17cyclictest10273-21sendmail2
920899211,16cyclictest0-21swapper/22
920899210,4cyclictest15668-1kworker/2:1H2
9203992115,3cyclictest15352-21sendmail1
9203992114,4cyclictest1286-21timerandwakeup1
920399211,17cyclictest32555-21basename1
9198992116,3cyclictest13729-21sh0
9198992115,4cyclictest9993-21if_err_enp3s00
9198992115,4cyclictest4530-21seq0
9198992115,4cyclictest22768-21seq0
9198992115,4cyclictest22358-21cron0
920899202,16cyclictest16679-21latency_hist2
9208992016,2cyclictest8310-21cat2
9208992016,2cyclictest25470-21aten2_r9power_v2
9208992015,3cyclictest22706-21interrupts2
9208992015,3cyclictest19960-21sed2
9208992015,3cyclictest19597-21sed2
920899201,3cyclictest21966-1kworker/2:2H2
9208992013,4cyclictest292-21jbd2/mmcblk0p2-2
9208992012,5cyclictest27796-21cat2
920899201,17cyclictest15668-1kworker/2:1H2
920899201,17cyclictest0-21swapper/22
9208992010,5cyclictest32169-21apt-get2
920899200,3cyclictest28813-1kworker/2:0H2
920899200,3cyclictest25833-21awk2
920899200,3cyclictest21966-1kworker/2:2H2
920899200,18cyclictest3508-21cron2
920399202,12cyclictest7453-21chrt1
9203992015,3cyclictest9829-21sed1
9203992015,3cyclictest21035-21grep1
9203992014,4cyclictest23196-21sed1
9203992014,4cyclictest10777-21awk1
9203992014,3cyclictest14644-21idleruntime-cro1
9203992014,3cyclictest12404-21aten2.4-expect1
9203992013,5cyclictest30536-21sendmail-msp1
9203992013,5cyclictest1009-21grep1
9203992013,4cyclictest3929-21idleruntime-cro1
9203992013,4cyclictest22620-21grep1
9203992012,6cyclictest2082-21systemd-journal1
9203992012,5cyclictest505-21dbus-daemon1
9203992012,5cyclictest3965-21sort1
9203992012,5cyclictest30270-21ls1
9203992012,4cyclictest25218-21timerandwakeup1
919899203,8cyclictest9-21ksoftirqd/00
919899203,8cyclictest0-21swapper/00
919899202,16cyclictest603-21grep0
919899202,16cyclictest603-21grep0
9198992015,3cyclictest10593-21gpgconf0
9198992014,4cyclictest21190-21seq0
9198992014,4cyclictest15554-21sed0
9198992014,3cyclictest12634-21switchtime0
9198992013,4cyclictest26183-21wc0
9198992013,4cyclictest25308-21cat0
252222200,2sleep00-21swapper/00
920899198,9cyclictest0-21swapper/22
9208991915,2cyclictest14620-21latency_hist2
9208991914,3cyclictest9472-21/usr/sbin/munin2
9208991914,3cyclictest4271-21cut2
9208991914,3cyclictest29977-21cat2
9208991914,3cyclictest20750-21sed2
9208991914,3cyclictest18159-21sed2
9208991913,4cyclictest9106-21cat2
9208991913,4cyclictest7566-21sh2
9208991912,4cyclictest3998-21pmu-power2
920899191,15cyclictest11978-21grep2
920899190,3cyclictest793-21perf2
920899190,3cyclictest793-21perf2
920899190,3cyclictest21966-1kworker/2:2H2
920899190,17cyclictest19397-21latency_hist2
920899190,17cyclictest0-21swapper/22
9203991915,2cyclictest8228-21/usr/sbin/munin1
9203991915,2cyclictest6665-21ntp_states1
9203991915,2cyclictest1084-21aten2_r9power_e1
9203991914,3cyclictest5299-21sed1
9203991914,3cyclictest2471-21sh1
9203991914,3cyclictest20629-21cat1
9203991914,3cyclictest19308-21sh1
9203991914,3cyclictest16918-21/usr/sbin/munin1
9203991914,3cyclictest1105-21sed1
9203991914,3cyclictest1105-21sed1
9203991913,4cyclictest7324-21ntp_states1
9203991913,4cyclictest6172-21/usr/sbin/munin1
9203991913,4cyclictest31877-21grep1
9203991913,4cyclictest16034-21wc1
9203991913,4cyclictest13955-21sed1
9203991913,4cyclictest11603-21cut1
9203991912,4cyclictest665-21runrttasks1
9203991912,4cyclictest505-21dbus-daemon1
9203991912,4cyclictest24413-21cron1
920399191,15cyclictest29868-21ls1
919899197,10cyclictest12605-21expr0
919899192,15cyclictest13134-21ls0
919899191,5cyclictest0-21swapper/00
9198991915,2cyclictest28471-21/usr/sbin/munin0
9198991915,2cyclictest18436-21apt0
9198991914,3cyclictest8218-21seq0
9198991914,3cyclictest6906-21cat0
9198991914,3cyclictest6606-21seq0
9198991914,3cyclictest30930-21seq0
9198991914,3cyclictest1974-21cat0
9198991914,3cyclictest16539-21seq0
9198991913,4cyclictest7576-21cat0
9198991913,4cyclictest6168-21expr0
9198991913,4cyclictest5297-21seq0
9198991913,4cyclictest26488-21diskstats0
9198991913,4cyclictest24058-21date0
9198991913,4cyclictest19580-21seq0
9198991913,4cyclictest17454-21seq0
9198991913,4cyclictest11020-21expr0
919899191,16cyclictest17227-21cpuspeed_turbos0
919899190,16cyclictest20081-21basename0
42362190,2sleep20-21swapper/22
9208991815,2cyclictest0-21swapper/22
9208991814,2cyclictest19778-21cat2
9208991813,3cyclictest29126-21kernelversion2
9208991813,3cyclictest1795-21cat2
9208991813,3cyclictest14774-21expr2
9208991813,3cyclictest13706-21ls2
9208991812,4cyclictest16992-21ps2
920899180,3cyclictest10083-21latency_hist2
920399189,7cyclictest309212chrt1
920399188,6cyclictest26453-21idleruntime-cro1
920399187,7cyclictest27391-21apt-get1
9203991814,2cyclictest30100-21/usr/sbin/munin1
9203991814,2cyclictest10317-21/usr/sbin/munin1
9203991813,3cyclictest18011-21/usr/sbin/munin1
9203991813,3cyclictest10392-21sed1
9203991813,3cyclictest10153-21sed1
9203991812,4cyclictest28672-21sed1
9203991812,4cyclictest19247-21/usr/sbin/munin1
9203991812,3cyclictest27190-21sed1
9203991811,4cyclictest31641-21cpuspeed1
9203991811,4cyclictest31641-21cpuspeed1
9203991810,4cyclictest26035-21latency_hist1
9203991810,4cyclictest0-21swapper/11
9198991814,2cyclictest22626-21/usr/sbin/munin0
9198991813,3cyclictest9616-21cat0
9198991813,3cyclictest7782-21cat0
9198991813,3cyclictest769-21seq0
9198991813,3cyclictest769-21seq0
9198991813,3cyclictest29300-21timerwakeupswit0
9198991813,3cyclictest27741-21expr0
9198991813,3cyclictest2659-21seq0
9198991813,3cyclictest18207-21cat0
9198991813,3cyclictest1686-21cut0
9198991813,3cyclictest14416-21grep0
9198991813,3cyclictest14303-21cat0
9198991813,3cyclictest1021-21pmu-power0
9198991812,4cyclictest31244-21cat0
9198991812,4cyclictest27057-21ls0
9198991812,4cyclictest20582-21grep0
9198991812,4cyclictest11009-21grep0
919899181,1cyclictest0-21swapper/00
919899181,15cyclictest25694-21ls0
9198991811,4cyclictest4107-21latency_hist0
9208991715,1cyclictest0-21swapper/22
9208991712,3cyclictest12912-21basename2
9208991712,3cyclictest0-21swapper/22
920399179,5cyclictest3538-21grep1
9203991715,1cyclictest0-21swapper/11
9203991714,2cyclictest0-21swapper/11
9203991713,2cyclictest14378-21/usr/sbin/munin1
9203991711,4cyclictest16880-21grep1
9203991711,4cyclictest14974-21aten2_r9power_p1
9198991712,3cyclictest4202-21aten2.4-expect0
9198991712,3cyclictest4192-21sed0
9198991712,3cyclictest31392-21cat0
9198991712,3cyclictest29286-21if_enp2s00
9198991712,3cyclictest21999-21hostname0
919899171,14cyclictest31286-21cat0
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional