You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-08 - 22:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack9slot2 (updated Fri Dec 08, 2023 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1362999288280,6cyclictest0-21swapper/11
117852288220,58sleep20-21swapper/22
133772281213,36sleep00-21swapper/00
134732233195,25sleep30-21swapper/33
133132230193,25sleep10-21swapper/11
1363499225220,3cyclictest15853-21sed2
1363999201194,4cyclictest0-21swapper/33
1248821400,3sleep01362499cyclictest0
414121200,2sleep21363499cyclictest2
131342610,2sleep00-21swapper/00
61772560,1sleep00-21swapper/00
53922560,2sleep00-21swapper/00
216622400,1sleep20-21swapper/22
231202330,1sleep10-21swapper/11
1363499291,25cyclictest16436-21sendmail2
132912270,3sleep01362499cyclictest0
185192260,2sleep00-21swapper/00
1363499262,18cyclictest32488-21chrt2
13634992518,4cyclictest9720-21munin-run2
131932250,1sleep10-21swapper/11
1363499243,19cyclictest18168-21munin-run2
13634992416,4cyclictest1726-21perf2
1363499241,5cyclictest29865-21apt-get2
13634992410,3cyclictest28-21ksoftirqd/22
1362499241,9cyclictest111rcu_preempt0
13624992410,11cyclictest2082-21systemd-journal0
18702230,2sleep00-21swapper/00
1363499234,17cyclictest32374-21cat2
1363499231,19cyclictest23840-21munin-run2
1363499231,19cyclictest15919-21turbostat2
13629992314,5cyclictest20190-21apt-get1
13624992316,4cyclictest505-21dbus-daemon0
13639992216,4cyclictest20273-21cut3
13639992216,3cyclictest14766-21aten2_r9power_v3
13639992215,4cyclictest20601-21df3
1363499227,13cyclictest5715-21latency_hist2
1363499222,18cyclictest13406-1kworker/2:0H2
13634992215,4cyclictest505-21dbus-daemon2
13634992212,6cyclictest665-21runrttasks2
1363499221,18cyclictest12373-21munin-run2
1362999227,3cyclictest21-21ksoftirqd/11
13639992116,3cyclictest9724-21sh3
13639992116,3cyclictest8860-21missed_timers3
13639992116,3cyclictest14147-21/usr/sbin/munin3
13639992115,4cyclictest810-21ls3
13639992115,4cyclictest16115-21df_inode3
1363499212,16cyclictest13406-1kworker/2:0H2
1363499211,8cyclictest246382grep2
1363499211,8cyclictest246382grep2
13634992116,3cyclictest22518-21aten2_r9power_p2
13634992115,4cyclictest29332-21sed2
13634992115,4cyclictest1829-21munin-run2
13634992115,3cyclictest17850-21perf2
13634992114,4cyclictest30298-21sort2
13634992114,4cyclictest11313-21grep2
1363499211,3cyclictest25342-1kworker/2:0H2
1363499211,3cyclictest25342-1kworker/2:0H2
1363499211,18cyclictest9895-21basename2
1363499211,18cyclictest3856-21munin-run2
1363499211,17cyclictest28285-21cron2
1363499211,17cyclictest25342-1kworker/2:0H2
1363499210,5cyclictest13941-21latency_hist2
1363499210,18cyclictest12072-21apt-config2
13629992116,3cyclictest25706-21seq1
13629992115,4cyclictest9646-21seq1
13629992115,4cyclictest31999-21seq1
13629992115,4cyclictest29004-21fw_packets1
13629992115,4cyclictest19569-21seq1
13629992114,4cyclictest19858-21seq1
1362999211,17cyclictest101292sleep11
1362499219,10cyclictest2729-21if_err_enp2s00
13624992116,3cyclictest9-21ksoftirqd/00
13624992112,6cyclictest505-21dbus-daemon0
246352200,3sleep024641-21gltestperf0
246352200,3sleep024641-21gltestperf0
13639992015,3cyclictest5170-21switchtime3
13639992015,3cyclictest444-21diskstats3
13639992015,3cyclictest3373-21timerwakeupswit3
13639992015,3cyclictest31605-21timerandwakeup3
13639992015,3cyclictest30962-21ls3
13639992015,3cyclictest29005-21gltestperf3
13639992015,3cyclictest26554-21diskstats3
13639992015,3cyclictest12303-21apt-key3
13639992015,3cyclictest12066-21/usr/sbin/munin3
13639992015,3cyclictest11022-21ntp_states3
13639992014,4cyclictest853-21sh3
13639992014,4cyclictest8529-21df3
13639992014,4cyclictest16975-21uname3
13639992013,4cyclictest3869-21grep3
1363999201,17cyclictest11812-21cron3
13634992015,3cyclictest9859-21sendmail2
13634992015,3cyclictest7768-21date2
13634992015,3cyclictest6107-21/usr/sbin/munin2
13634992014,3cyclictest29505-21head2
1363499201,3cyclictest22114-21sendmail-mta2
1363499201,17cyclictest5848-21latency_hist2
1363499201,17cyclictest25885-21sendmail-mta2
1363499201,17cyclictest21837-21latency_hist2
1363499201,16cyclictest14592-21gltestperf2
1363499201,16cyclictest12437-21/usr/sbin/munin2
13629992016,2cyclictest20901-21df1
13629992015,3cyclictest30406-21/usr/sbin/munin1
13629992015,3cyclictest24424-21/usr/sbin/munin1
13629992015,3cyclictest17587-21find1
13629992015,3cyclictest15745-21munin-node1
13629992014,4cyclictest8526-21cat1
13629992014,4cyclictest5557-21seq1
13629992014,4cyclictest28450-21cat1
13629992014,4cyclictest26931-21needreboot1
13629992014,4cyclictest23987-21cat1
13629992014,4cyclictest17148-21sed1
13629992014,4cyclictest15535-21sh1
13629992013,4cyclictest7149-21timerandwakeup1
13629992013,4cyclictest584-21basename1
13624992016,2cyclictest9-21ksoftirqd/00
13624992015,3cyclictest3943-21grep0
13624992015,3cyclictest28532-21/usr/sbin/munin0
13624992014,4cyclictest6543-21gltestperf0
13624992013,4cyclictest30277-21cron0
1363999197,3cyclictest228792expr3
13639991915,2cyclictest25499-21processes3
13639991915,2cyclictest16453-21/usr/sbin/munin3
13639991915,2cyclictest14639-21idleruntime3
13639991914,3cyclictest6482-21ls3
13639991914,3cyclictest5135-21timerandwakeup3
13639991914,3cyclictest29125-21interrupts3
13639991914,3cyclictest24928-21gltestperf3
13639991914,3cyclictest20468-21/usr/sbin/munin3
13639991914,3cyclictest14518-21aten2_r9power_e3
13639991913,4cyclictest4123-21apt3
13639991913,4cyclictest23069-21cat3
13639991913,4cyclictest19841-21cron3
13639991913,4cyclictest13057-21sh3
1363499196,11cyclictest14381-21cron2
1363499192,15cyclictest13919-21cat2
13634991916,2cyclictest0-21swapper/22
13634991915,2cyclictest30285-21cut2
1363499191,4cyclictest3851-21cron2
13634991914,3cyclictest10850-21sed2
13634991914,3cyclictest10021-21sed2
13634991913,4cyclictest18150-21seq2
1363499191,16cyclictest25342-1kworker/2:0H2
1363499190,17cyclictest5994-21cat2
1362999198,9cyclictest0-21swapper/11
13629991915,2cyclictest22226-21aten2_r9power_v1
13629991915,2cyclictest2061-21apt1
13629991915,2cyclictest1975-21/usr/sbin/munin1
13629991914,3cyclictest8511-21ls1
13629991914,3cyclictest4205-21cut1
13629991914,3cyclictest32531-21/usr/sbin/munin1
13629991914,3cyclictest32390-21seq1
13629991914,3cyclictest30470-21/usr/sbin/munin1
13629991914,3cyclictest30061-21seq1
13629991914,3cyclictest26892-21expr1
13629991914,3cyclictest16393-21date1
13629991914,3cyclictest10991-21seq1
13629991914,3cyclictest10823-21sed1
13629991914,3cyclictest10465-21snmpd1
13629991914,3cyclictest10059-21/usr/sbin/munin1
13629991913,4cyclictest9516-21seq1
13629991913,4cyclictest6112-21cat1
13629991913,4cyclictest28940-21expr1
13629991913,4cyclictest2082-21systemd-journal1
13629991913,4cyclictest14200-21sh1
13624991916,2cyclictest0-21swapper/00
1362499191,5cyclictest0-21swapper/00
13624991915,2cyclictest9997-21/usr/sbin/munin0
13624991915,2cyclictest26549-21/usr/sbin/munin0
13624991915,2cyclictest26484-21/usr/sbin/munin0
13624991915,2cyclictest16453-21/usr/sbin/munin0
13624991915,2cyclictest13811-21/usr/sbin/munin0
13624991915,2cyclictest10121-21/usr/sbin/munin0
13624991915,2cyclictest10059-21/usr/sbin/munin0
13624991914,3cyclictest9-21ksoftirqd/00
13624991914,3cyclictest8107-21acpi0
13624991914,3cyclictest19570-21sed0
13624991914,3cyclictest17829-21latency_hist0
13624991913,4cyclictest29471-21awk0
13624991913,4cyclictest23943-21grep0
13624991913,4cyclictest18987-21sed0
13624991912,4cyclictest9729-21sendmail0
1362499191,17cyclictest0-21swapper/00
13624991910,5cyclictest20116-21apt-get0
13639991816,1cyclictest0-21swapper/33
13639991814,2cyclictest6056-21apt-get3
13639991814,2cyclictest26541-21aten2_r9power_e3
13639991813,3cyclictest970-21ntpq3
13639991813,3cyclictest27307-21missed_timers3
13639991813,3cyclictest26600-21expr3
13639991813,3cyclictest25924-21idleruntime-cro3
13639991813,3cyclictest25924-21idleruntime-cro3
13639991813,3cyclictest2082-21systemd-journal3
13639991813,3cyclictest19443-21grep3
13639991813,3cyclictest12660-21sed3
13639991812,4cyclictest8468-21df3
13639991812,4cyclictest6330-21cat3
13639991812,4cyclictest31916-21awk3
13639991812,4cyclictest20204-21hostname3
13639991812,3cyclictest16085-21acpi3
1363499184,8cyclictest26130-21apt-get2
13634991813,3cyclictest30931-21fw_packets2
13634991813,3cyclictest27061-21ntpq2
13634991813,3cyclictest22999-21sed2
13634991812,4cyclictest2115-21cut2
13634991811,5cyclictest19526-21rt-features2
1363499180,3cyclictest25342-1kworker/2:0H2
13629991814,2cyclictest26176-21/usr/sbin/munin1
13629991814,2cyclictest12128-21/usr/sbin/munin1
13629991813,3cyclictest6030-21seq1
13629991813,3cyclictest5016-21proc_pri1
13629991813,3cyclictest4456-21expr1
13629991813,3cyclictest4023-21cut1
13629991813,3cyclictest28423-21cat1
13629991813,3cyclictest26286-21date1
13629991813,3cyclictest22013-21seq1
13629991813,3cyclictest2133-21http1
13629991813,3cyclictest19929-21latency_hist1
13629991813,3cyclictest14202-21seq1
13629991812,4cyclictest1984-21date1
13629991812,4cyclictest14571-21expr1
1362499182,13cyclictest8880-21grep0
13624991816,1cyclictest0-21swapper/00
13624991816,1cyclictest0-21swapper/00
13624991816,1cyclictest0-21swapper/00
13624991813,3cyclictest665-21runrttasks0
13624991813,3cyclictest4233-21wc0
13624991813,3cyclictest32450-21sed0
13624991813,3cyclictest31062-21cat0
13624991813,3cyclictest27047-21ntp_kernel_pll_0
13624991813,3cyclictest26354-21sed0
13624991813,3cyclictest24093-21sort0
13624991813,3cyclictest10465-21snmpd0
13639991715,1cyclictest0-21swapper/33
13639991713,3cyclictest22167-21/usr/sbin/munin3
13639991713,2cyclictest22162-21aten2_r9power_e3
13639991712,3cyclictest3900-21cat3
13639991712,3cyclictest31094-21idleruntime3
13639991712,3cyclictest29196-21cat3
13639991712,3cyclictest25179-21cat3
13639991712,3cyclictest20894-21fw_packets3
13639991711,4cyclictest10760-21load3
1363999170,16cyclictest0-21swapper/33
13634991712,3cyclictest23906-21sh2
13634991712,3cyclictest19155-21sed2
13634991711,4cyclictest7224-21cat2
1363499170,15cyclictest0-21swapper/22
13629991713,2cyclictest665-21runrttasks1
13629991711,4cyclictest23039-21cat1
13629991711,4cyclictest18472-21seq1
13624991715,1cyclictest0-21swapper/00
13624991715,1cyclictest0-21swapper/00
13624991713,2cyclictest665-21runrttasks0
13624991713,2cyclictest22468-21/usr/sbin/munin0
13624991712,3cyclictest665-21runrttasks0
13624991712,3cyclictest665-21runrttasks0
13624991712,3cyclictest665-21runrttasks0
13624991712,3cyclictest10576-21sh0
1362499171,1cyclictest0-21swapper/00
13624991711,4cyclictest9-21ksoftirqd/00
13624991711,4cyclictest13195-21cat0
1362499171,13cyclictest2518-21sed0
13624991710,4cyclictest2082-21systemd-journal0
13639991611,3cyclictest4167-21cut3
13639991611,3cyclictest19360-21ntp_offset3
1363499161,14cyclictest19560-21/usr/sbin/munin2
13629991612,2cyclictest24237-21aten2.4-expect1
13629991612,2cyclictest24237-21aten2.4-expect1
13629991610,4cyclictest12317-21seq1
13624991612,2cyclictest26462-21cstates0
13624991611,3cyclictest6606-21grep0
13624991610,3cyclictest20558-21cut0
13639991510,3cyclictest12411-21latency_hist3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional