You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 07:33
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack9slot2.osadl.org (updated Sun Apr 28, 2024 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305402265206,47sleep10-21swapper/11
306332262190,25sleep00-21swapper/00
305062260203,44sleep20-21swapper/22
304562238170,25sleep30-21swapper/33
3097099220214,3cyclictest9052-21awk0
3097099220214,3cyclictest9052-21awk0
3098099207202,3cyclictest11250irq/128-xhci_hc2
3098099207202,3cyclictest11250irq/128-xhci_hc2
1685821770,2sleep10-21swapper/11
3263521660,4sleep03097099cyclictest0
2562521500,3sleep23098099cyclictest2
1718221480,3sleep30-21swapper/33
3098599141136,3cyclictest9028-21munin-run3
3098599141136,3cyclictest9028-21munin-run3
750021350,2sleep03097099cyclictest0
3097599135128,4cyclictest0-21swapper/11
3097599135128,4cyclictest0-21swapper/11
212232580,3sleep00-21swapper/00
30980995012,15cyclictest32623-21cron2
3097099502,19cyclictest28820-21apt-get0
30970995014,15cyclictest6436-21latency_hist0
96622490,1sleep19667-21grep1
30980994814,8cyclictest8467-21idleruntime-cro2
291802480,1sleep20-21swapper/22
30980994711,16cyclictest0-21swapper/22
274812470,2sleep10-21swapper/11
30980994511,9cyclictest0-21swapper/22
30980994312,9cyclictest20686-21cat2
30970994312,17cyclictest665-21runrttasks0
30980994215,8cyclictest22592-21dpkg2
30980994210,15cyclictest0-21swapper/22
288892420,3sleep00-21swapper/00
30980994116,14cyclictest6941-21cpuspeed2
30980994115,7cyclictest12290-21latency_hist2
30980994115,15cyclictest31324-21cat2
30980994115,13cyclictest28861-21sed2
30980994114,8cyclictest24413-21apt-key2
30980994114,10cyclictest5755-21gltestperf2
30980994114,10cyclictest5004-21cpuspeed_turbos2
30980994112,17cyclictest0-21swapper/22
30980994016,10cyclictest3073-21ls2
30980994015,15cyclictest28636-21sort2
30980994015,14cyclictest29995-21sed2
30980994015,14cyclictest28073-21sed2
30980994014,9cyclictest19665-21sed2
30980994014,10cyclictest24872-21expr2
30980993916,8cyclictest21671-21ntp_states2
30980993916,7cyclictest15433-21ntp_states2
30980993915,9cyclictest3558-21cpuspeed_turbos2
30980993915,9cyclictest13638-21cut2
30980993915,13cyclictest17581-21sed2
30980993915,10cyclictest4750-21/usr/sbin/munin2
30980993912,9cyclictest0-21swapper/22
30980993910,7cyclictest0-21swapper/22
155532390,1sleep30-21swapper/33
30980993816,8cyclictest23855-21switchtime2
30980993816,8cyclictest11678-21awk2
30980993816,8cyclictest10719-21apt-get2
30980993715,8cyclictest30622-21munin-run2
30980993715,8cyclictest26781-21/usr/sbin/munin2
30980993715,8cyclictest10175-21awk2
30980993715,10cyclictest1520-21needreboot2
30980993714,8cyclictest23915-21timerandwakeup2
30980993713,9cyclictest21984-21timerwakeupswit2
30970993710,14cyclictest665-21runrttasks0
3098099369,19cyclictest0-21swapper/22
30980993615,8cyclictest25377-21if_enp2s02
30980993615,8cyclictest12812-21aten2_r9power_c2
30980993615,7cyclictest13665-21latency_hist2
30980993610,9cyclictest0-21swapper/22
30980993514,8cyclictest426-21/usr/sbin/munin2
30980993514,8cyclictest20783-21cpuspeed_turbos2
30980993512,9cyclictest12532-21/usr/sbin/munin2
30980993414,7cyclictest9415-21cat2
30980993414,7cyclictest5564-21ps2
30980993413,7cyclictest18807-21cat2
3097099341,3cyclictest28929-21/usr/sbin/munin0
174692340,1sleep00-21swapper/00
30980993314,7cyclictest20575-21taskset2
30980993313,7cyclictest665-21runrttasks2
3098099329,8cyclictest0-21swapper/22
3098099329,8cyclictest0-21swapper/22
30980993217,12cyclictest30910-21/usr/sbin/munin2
30980993112,7cyclictest24804-21apt-get2
30980993112,7cyclictest24804-21apt-get2
3097599310,18cyclictest30962ls1
30980993017,10cyclictest15405-21/usr/sbin/munin2
30980993015,12cyclictest17380-21if_enp2s02
30970993014,14cyclictest27115-21cat0
30970993014,13cyclictest2533-21grep0
30980992915,11cyclictest16049-21grep2
30980992914,12cyclictest2082-21systemd-journal2
3097099299,16cyclictest4532-21apt-get0
30970992915,12cyclictest32542-21sh0
30970992914,12cyclictest25872-21processes0
30970992913,14cyclictest2887-21df0
30970992913,13cyclictest2082-21systemd-journal0
30970992912,14cyclictest4069-21wc0
30970992911,15cyclictest24816-21awk0
3097099282,23cyclictest31443-21grep0
3097099282,20cyclictest14721-21apt-get0
30970992812,13cyclictest9286-21apt-get0
30970992812,13cyclictest22521-21latency_hist0
30970992811,14cyclictest10988-21awk0
3097099280,3cyclictest23961-21vmstat0
30980992718,6cyclictest18911-21/usr/sbin/munin2
30980992713,12cyclictest8255-21cron2
30980992713,12cyclictest14287-21munin-run2
3097099277,17cyclictest15195-21df0
3097099277,17cyclictest12547-21interrupts0
3097099276,19cyclictest10333-21gdbus0
3097099275,19cyclictest31370-21basename0
3097099272,22cyclictest505-21dbus-daemon0
3097099272,22cyclictest25137-21cstates0
3097099272,22cyclictest25137-21cstates0
30970992713,11cyclictest665-21runrttasks0
30970992711,13cyclictest1947-21unixbench-2d0
30970992710,14cyclictest20564-21cat0
3097099270,4cyclictest14298-21grep0
3097099270,3cyclictest3899-21users0
3097099270,3cyclictest2702-21apt-get0
3097099270,3cyclictest18923-21cstates0
3097099270,2cyclictest10719-21apt-get0
30980992618,5cyclictest2684-21/usr/sbin/munin2
30980992617,6cyclictest31042-21ls2
3097099269,14cyclictest0-21swapper/00
3097099268,16cyclictest495-21sed0
3097099268,15cyclictest27773-21head0
3097099267,16cyclictest15351-21sed0
3097099265,18cyclictest505-21dbus-daemon0
3097099264,19cyclictest28897-21hostname0
3097099262,20cyclictest0-21swapper/00
3097099262,18cyclictest0-21swapper/00
3097099261,20cyclictest26918-21apt-get0
3097099260,3cyclictest25821-21timerwakeupswit0
3097099260,3cyclictest22669-21aten2_r9power_v0
3097099260,2cyclictest28695-21snmpd0
3097099260,2cyclictest0-21swapper/00
30975992521,3cyclictest21-21ksoftirqd/11
3097099259,13cyclictest26901-21aten2_r9power_v0
3097099254,18cyclictest670-21cat0
3097099252,20cyclictest505-21dbus-daemon0
3097099251,20cyclictest505-21dbus-daemon0
3097099250,22cyclictest2082-21systemd-journal0
3097099249,12cyclictest20561-21apt-get0
3097099245,16cyclictest28695-21snmpd0
3097099245,16cyclictest0-21swapper/00
3097099243,18cyclictest18326-21awk0
3097099241,20cyclictest9017-21cpuspeed_turbos0
3097099241,20cyclictest9017-21cpuspeed_turbos0
167092240,3sleep33098599cyclictest3
30985992316,5cyclictest505-21dbus-daemon3
3097099233,17cyclictest821-21cpuspeed_turbos0
70552220,2sleep10-21swapper/11
3098599227,3cyclictest174672open_inodes3
30985992216,4cyclictest4963-21diskstats3
30975992216,4cyclictest14539-21http1
3097099221,4cyclictest19163-21cat0
3097099221,16cyclictest0-21swapper/00
30985992116,3cyclictest32639-21idleruntime-cro3
30985992115,4cyclictest14534-21apt-get3
30985992115,3cyclictest2082-21systemd-journal3
30985992114,4cyclictest7497-21memory3
30985992114,4cyclictest27735-21ntp_kernel_pll_3
30985992114,4cyclictest13916-21users3
30985992114,4cyclictest13205-21df3
30975992115,4cyclictest22343-21seq1
30975992112,6cyclictest505-21dbus-daemon1
3098599209,9cyclictest29469-21gltestperf3
30985992015,3cyclictest25094-21cut3
30985992015,3cyclictest25094-21cut3
30985992015,3cyclictest20312-21sh3
30985992015,3cyclictest15336-21needreboot3
30985992015,3cyclictest12903-21cpuspeed3
30985992014,4cyclictest304-21cut3
30985992014,4cyclictest27775-21ntp_kernel_pll_3
30985992014,3cyclictest11405-21if_err_enp3s03
30985992013,5cyclictest8734-21cat3
30985992013,4cyclictest29453-21awk3
30985992013,4cyclictest19333-21wc3
30985992012,5cyclictest22934-21ls3
30975992015,3cyclictest27374-21head1
30975992015,3cyclictest21-21ksoftirqd/11
30975992014,4cyclictest32560-21latency_hist1
30975992014,4cyclictest21-21ksoftirqd/11
30975992014,4cyclictest18755-21cat1
30975992013,4cyclictest10333-21gdbus1
3098599199,6cyclictest2463-21apt-get3
30985991914,3cyclictest9708-21processes3
30985991914,3cyclictest9708-21processes3
30985991914,3cyclictest6692-21apt-get3
30985991914,3cyclictest6015-21load3
30985991914,3cyclictest480-21sh3
30985991914,3cyclictest396-21sed3
30985991914,3cyclictest21674-21cat3
30985991914,3cyclictest19022-21cpuspeed3
30985991914,3cyclictest11684-21ls3
30985991913,4cyclictest6483-21date3
30985991913,4cyclictest505-21dbus-daemon3
30985991913,4cyclictest3901-21vmstat3
30985991913,4cyclictest28676-21cat3
30985991913,4cyclictest2570-21cat3
30985991913,4cyclictest21710-21cut3
30985991913,4cyclictest18449-21seq3
30985991913,3cyclictest28815-21aten2_r9power_c3
3098599190,4cyclictest14308-21awk3
30975991915,2cyclictest8702-21/usr/sbin/munin1
30975991915,2cyclictest8702-21/usr/sbin/munin1
30975991915,2cyclictest29596-21kernelversion1
30975991915,2cyclictest22570-21/usr/sbin/munin1
30975991914,3cyclictest8769-21/usr/sbin/munin1
30975991914,3cyclictest646-21munin-node1
30975991914,3cyclictest4750-21/usr/sbin/munin1
30975991914,3cyclictest30535-21seq1
30975991914,3cyclictest24785-21/usr/sbin/munin1
30975991914,3cyclictest24785-21/usr/sbin/munin1
30975991914,3cyclictest19428-21awk1
30975991914,3cyclictest16679-21cat1
30975991914,3cyclictest11523-21seq1
30975991914,3cyclictest11434-21open_files1
30975991914,3cyclictest10145-21ntpq1
30975991913,4cyclictest578-21readlink1
30975991913,4cyclictest13068-21expr1
30975991913,3cyclictest21-21ksoftirqd/11
30975991912,4cyclictest12397-21seq1
30985991813,3cyclictest9686-21expr3
30985991813,3cyclictest7017-21expr3
30985991813,3cyclictest650-21unixbench-2d3
30985991813,3cyclictest32146-21rt-features3
30985991813,3cyclictest31026-21ls3
30985991813,3cyclictest29284-21expr3
30985991813,3cyclictest25250-21cat3
30985991813,3cyclictest23208-21cat3
30985991813,3cyclictest20811-21cpuspeed_turbos3
30985991813,3cyclictest18756-21cat3
30985991813,3cyclictest11844-21timerandwakeup3
30985991812,4cyclictest25841-21cat3
30985991812,4cyclictest25016-21cat3
30985991812,4cyclictest2363-21cut3
3097599187,6cyclictest16579-21apt-get1
3097599183,9cyclictest31117-21apt-get1
30975991814,2cyclictest30769-21/usr/sbin/munin1
30975991814,2cyclictest21-21ksoftirqd/11
30975991814,2cyclictest16552-21/usr/sbin/munin1
30975991813,3cyclictest6510-21/usr/sbin/munin1
30975991813,3cyclictest5973-21sed1
30975991813,3cyclictest5109-21cat1
30975991813,3cyclictest28996-21aten2_r9power_v1
30975991813,3cyclictest25790-21cat1
30975991813,3cyclictest21167-21cat1
30975991813,3cyclictest18986-21diskstats1
30975991813,3cyclictest12775-21/usr/sbin/munin1
30975991813,3cyclictest12262-21ls1
30975991813,3cyclictest10727-21aten2_r9power_c1
30975991812,4cyclictest505-21dbus-daemon1
30975991812,4cyclictest24057-21grep1
30975991812,4cyclictest21-21ksoftirqd/11
3097599181,16cyclictest0-21swapper/11
30975991811,4cyclictest21126-21cat1
30975991811,4cyclictest16061-21unixbench_multi1
3097599180,17cyclictest0-21swapper/11
30985991713,2cyclictest18775-21/usr/sbin/munin3
30985991712,3cyclictest3921-21interrupts3
30985991712,3cyclictest27358-21cat3
30985991712,3cyclictest24523-21munin-plugin-st3
30985991712,3cyclictest10671-21cat3
30985991711,4cyclictest22551-21idleruntime-cro3
3097599172,9cyclictest4706-21apt-get1
30975991712,3cyclictest868-21cstates1
30975991712,3cyclictest7033-21ls1
30975991712,3cyclictest665-21runrttasks1
30975991712,3cyclictest31649-21cat1
30975991712,3cyclictest31011-21aten2_r9power_e1
30975991712,3cyclictest19476-21cut1
30975991712,3cyclictest15823-21grep1
30975991712,3cyclictest13188-21expr1
30975991711,4cyclictest27085-21cat1
3098599169,5cyclictest4848-21cut3
30985991611,3cyclictest11985-21aten2.4-expect3
3097599160,5cyclictest0-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional