You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-24 - 16:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack9slot2 (updated Sat Feb 24, 2024 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51032276184,24sleep30-21swapper/33
51352261202,46sleep00-21swapper/00
51512259193,35sleep10-21swapper/11
52242243166,43sleep20-21swapper/22
550499211203,4cyclictest7933-21diskstats1
551299169164,3cyclictest0-21swapper/33
2377521650,3sleep1550499cyclictest1
550799148143,3cyclictest7954-21latency_hist2
549999107104,2cyclictest0-21swapper/00
129302540,10sleep228-21ksoftirqd/22
550499479,22cyclictest22884-21seq1
5504994611,25cyclictest665-21runrttasks1
5504994611,25cyclictest665-21runrttasks1
69222450,3sleep09-21ksoftirqd/00
550499447,20cyclictest4984-21expr1
5504993515,17cyclictest6542-21date1
550499330,4cyclictest20090-21grep1
550499328,21cyclictest19039-21latency_hist1
5504993216,14cyclictest14107-21cat1
5504993210,19cyclictest2082-21systemd-journal1
550499318,20cyclictest11733-21cat1
550499318,20cyclictest11733-21cat1
550499312,26cyclictest9289-21sed1
5504993116,13cyclictest14999-21sed1
5504993115,14cyclictest21590-21ntp_states1
5504993111,18cyclictest26498-21systemctl1
5504993111,17cyclictest18791-21cut1
550499310,3cyclictest7462-21grep1
550799309,3cyclictest28-21ksoftirqd/22
550499309,18cyclictest15750-21wc1
550499308,19cyclictest29419-21cat1
5504993015,13cyclictest7578-21ps1
5504993015,13cyclictest11600-21proc_pri1
550499299,18cyclictest27602-21expr1
550499299,17cyclictest5443-21expr1
550499299,15cyclictest25009-21apt-get1
550499297,20cyclictest12555-21cut1
550499297,20cyclictest12555-21cut1
550499293,21cyclictest12688-21apt-get1
5504992910,17cyclictest27856-21needreboot1
5504992910,17cyclictest27856-21needreboot1
5504992910,17cyclictest1507-21cut1
5504992910,16cyclictest23566-21needreboot1
550499290,4cyclictest23189-21/usr/sbin/munin1
550499290,3cyclictest31990-21df1
550499290,3cyclictest15516-21ntp_states1
550499289,18cyclictest8063-21unixbench-2d1
550499289,17cyclictest21821-21cat1
550499289,17cyclictest17086-21expr1
550499289,16cyclictest8731-21cpu1
550499289,16cyclictest42292chrt1
550499287,19cyclictest9815-21sed1
550499287,19cyclictest31882-21awk1
5504992815,11cyclictest17179-21/usr/sbin/munin1
550499281,3cyclictest3820-21ps1
5504992810,16cyclictest12971-21expr1
549999280,2cyclictest0-21swapper/00
551299279,4cyclictest35-21ksoftirqd/33
551299279,4cyclictest35-21ksoftirqd/33
5512992715,3cyclictest35-21ksoftirqd/33
5512992714,4cyclictest35-21ksoftirqd/33
551299270,6cyclictest35-21ksoftirqd/33
551299270,6cyclictest35-21ksoftirqd/33
550499277,17cyclictest9476-21sh1
550499277,17cyclictest9475-21grep1
550499271,5cyclictest19218-21apt-get1
5504992715,7cyclictest31082-21apt-get1
550499271,21cyclictest29579-21apt-get1
550499271,21cyclictest15246-21apt-get1
550499270,5cyclictest1605-21grep1
550499270,3cyclictest31963-21ls1
550499270,3cyclictest27453-21cstates1
550799269,14cyclictest4372-21cron2
550799261,2cyclictest111rcu_preempt2
550499269,15cyclictest1102-21/usr/sbin/munin1
550499266,17cyclictest17859-21timerwakeupswit1
5504992611,13cyclictest28984-21sshd1
5504992610,13cyclictest3158-21apt-get1
550499260,23cyclictest11654-21grep1
551299257,7cyclictest35-21ksoftirqd/33
551299251,2cyclictest35-21ksoftirqd/33
550499256,16cyclictest26105-21sort1
550499255,17cyclictest29074-21aten2_r9power_e1
550499251,5cyclictest15192-21apt-get1
5512992419,3cyclictest341rcuc/33
5512992414,3cyclictest35-21ksoftirqd/33
551299241,2cyclictest111rcu_preempt3
5512992411,4cyclictest35-21ksoftirqd/33
5512992411,4cyclictest35-21ksoftirqd/33
551299240,8cyclictest35-21ksoftirqd/33
550799248,3cyclictest28-21ksoftirqd/22
550499242,4cyclictest22705-21apt-get1
550499241,3cyclictest0-21swapper/11
551299234,2cyclictest111rcu_preempt3
551299233,2cyclictest111rcu_preempt3
5512992318,3cyclictest35-21ksoftirqd/33
5512992318,3cyclictest35-21ksoftirqd/33
5512992318,3cyclictest35-21ksoftirqd/33
5512992318,3cyclictest35-21ksoftirqd/33
5512992315,2cyclictest35-21ksoftirqd/33
550799238,7cyclictest28-21ksoftirqd/22
550799232,18cyclictest6519-21cron2
550799231,3cyclictest28-21ksoftirqd/22
550799231,2cyclictest0-21swapper/22
5507992312,3cyclictest28-21ksoftirqd/22
550799231,19cyclictest14686-21perf2
5507992310,11cyclictest28-21ksoftirqd/22
550799230,14cyclictest111rcu_preempt2
550499231,6cyclictest2786-21cpuspeed1
550499231,4cyclictest6530-21sendmail1
550499231,4cyclictest29948-21ntp_states1
550499231,4cyclictest20110-21ntpq1
550499230,4cyclictest1832-21grep1
549999234,18cyclictest0-21swapper/00
549999234,18cyclictest0-21swapper/00
5512992214,4cyclictest2082-21systemd-journal3
550799223,17cyclictest4946-21cat2
5507992216,4cyclictest13609-21df_inode2
550799221,3cyclictest10328-21munin-run2
550799221,2cyclictest111rcu_preempt2
550799221,18cyclictest29851-21nfs4_client2
550799221,18cyclictest2497-21munin-run2
550799220,6cyclictest26765-21plymouth2
549999229,3cyclictest118982grep0
5499992215,4cyclictest10919-21dump-pmu-power0
551299216,4cyclictest35-21ksoftirqd/33
551299213,3cyclictest35-21ksoftirqd/33
551299212,2cyclictest111rcu_preempt3
5512992117,3cyclictest35-21ksoftirqd/33
5512992116,3cyclictest35-21ksoftirqd/33
5512992116,3cyclictest341rcuc/33
5512992116,3cyclictest341rcuc/33
5512992116,3cyclictest341rcuc/33
5512992116,3cyclictest11969-21unixbench-2d3
5512992116,3cyclictest11969-21unixbench-2d3
5512992115,4cyclictest5737-21sed3
5512992115,3cyclictest9875-21timerwakeupswit3
5512992111,3cyclictest35-21ksoftirqd/33
550799217,8cyclictest11908-21unixbench_multi2
550799217,2cyclictest111rcu_preempt2
550799212,16cyclictest0-21swapper/22
5507992117,2cyclictest24627-21users2
5507992116,3cyclictest18190-21pmu-power2
550799211,4cyclictest10333-21gdbus2
550799211,4cyclictest10333-21gdbus2
5507992114,4cyclictest31638-21diskstats2
5507992114,4cyclictest20427-21cron2
550799211,3cyclictest31305-21munin-run2
550799211,3cyclictest20958-21munin-run2
550799211,2cyclictest111rcu_preempt2
550799211,18cyclictest30800-21munin-run2
550799211,18cyclictest16622-21users2
550799210,18cyclictest5013-21diskstats2
549999213,15cyclictest22438-21apt0
5499992115,4cyclictest30955-21idleruntime-cro0
5499992115,4cyclictest19023-21idleruntime-cro0
5499992113,5cyclictest9-21ksoftirqd/00
5499992113,5cyclictest505-21dbus-daemon0
551299209,3cyclictest111rcu_preempt3
551299209,2cyclictest111rcu_preempt3
551299208,2cyclictest111rcu_preempt3
551299207,2cyclictest111rcu_preempt3
551299206,4cyclictest35-21ksoftirqd/33
551299203,9cyclictest6603-21apt-get3
5512992015,3cyclictest35-21ksoftirqd/33
5512992014,3cyclictest2861-21cut3
551299201,3cyclictest35-21ksoftirqd/33
5512992012,5cyclictest10945-21basename3
551299200,7cyclictest35-21ksoftirqd/33
551299200,4cyclictest35-21ksoftirqd/33
550799209,3cyclictest28-21ksoftirqd/22
550799209,3cyclictest28-21ksoftirqd/22
550799208,3cyclictest28-21ksoftirqd/22
550799205,13cyclictest111rcu_preempt2
550799202,2cyclictest111rcu_preempt2
550799202,17cyclictest28971-21awk2
5507992016,2cyclictest27907-21ntp_states2
5507992016,2cyclictest27907-21ntp_states2
5507992015,3cyclictest27967-21open_files2
5507992015,3cyclictest26407-21nfs4_client2
5507992015,3cyclictest14856-21cpuspeed_turbos2
5507992015,3cyclictest13526-21ntpq2
5507992013,4cyclictest32429-21ntp_kernel_pll_2
5507992013,4cyclictest27952-21cat2
550799201,16cyclictest22150-21ntp_states2
550799200,5cyclictest28-21ksoftirqd/22
550799200,3cyclictest16959-21sleep2
550799200,18cyclictest8332-21munin-run2
550799200,18cyclictest32747-21munin-run2
550799200,18cyclictest2303-21munin-run2
5499992015,3cyclictest9-21ksoftirqd/00
5499992015,3cyclictest19319-21aten2_r9power_e0
5499992015,3cyclictest10465-21snmpd0
5499992014,4cyclictest26923-21latency_hist0
5499992014,4cyclictest26923-21latency_hist0
5499992013,4cyclictest29333-21basename0
5499992011,7cyclictest10971-21avahi-daemon0
91852190,4sleep00-21swapper/00
551299199,2cyclictest111rcu_preempt3
551299198,3cyclictest111rcu_preempt3
551299198,3cyclictest111rcu_preempt3
551299198,2cyclictest111rcu_preempt3
551299197,3cyclictest111rcu_preempt3
551299196,3cyclictest35-21ksoftirqd/33
551299194,7cyclictest20737-21/usr/sbin/munin3
551299191,6cyclictest35-21ksoftirqd/33
5512991915,2cyclictest17045-21sed3
551299191,2cyclictest0-21swapper/33
5512991910,2cyclictest111rcu_preempt3
551299190,10cyclictest35-21ksoftirqd/33
551299190,10cyclictest35-21ksoftirqd/33
550799195,12cyclictest0-21swapper/22
5507991914,3cyclictest23945-21load2
5507991914,3cyclictest23377-21latency2
5507991910,2cyclictest111rcu_preempt2
550799190,2cyclictest31195-21/usr/sbin/munin2
5499991917,1cyclictest0-21swapper/00
5499991915,2cyclictest4832-21aten2_r9power_e0
5499991915,2cyclictest4569-21/usr/sbin/munin0
5499991914,3cyclictest27059-21/usr/sbin/munin0
5499991914,3cyclictest23571-21expr0
5499991914,3cyclictest15612-21sh0
5499991914,3cyclictest11179-21/usr/sbin/munin0
5499991913,4cyclictest30810-21date0
5499991913,4cyclictest26952-21ls0
5499991913,4cyclictest26219-21kernelversion0
5499991913,4cyclictest24040-21ls0
5499991913,4cyclictest20021-21grep0
5499991912,4cyclictest3409-21needreboot0
5499991912,4cyclictest15342-21latency0
5499991911,5cyclictest15692-21forks0
5499991911,4cyclictest0-21swapper/00
551299189,2cyclictest35-21ksoftirqd/33
551299189,2cyclictest111rcu_preempt3
551299189,2cyclictest111rcu_preempt3
551299181,5cyclictest35-21ksoftirqd/33
5512991813,3cyclictest967-21ls3
5512991813,3cyclictest30050-21head3
5512991813,3cyclictest29225-21http3
551299181,2cyclictest111rcu_preempt3
551299180,5cyclictest35-21ksoftirqd/33
550799184,3cyclictest15003-21grep2
5507991814,2cyclictest23000-21sendmail-mta2
5507991814,2cyclictest12563-21uptime2
5507991814,2cyclictest10717-21apt-config2
5507991814,2cyclictest10717-21apt-config2
5507991813,3cyclictest8513-21aten2_r9power_p2
5507991812,4cyclictest7887-21sed2
5499991816,1cyclictest0-21swapper/00
5499991816,1cyclictest0-21swapper/00
5499991816,1cyclictest0-21swapper/00
5499991814,2cyclictest18693-21/usr/sbin/munin0
5499991814,2cyclictest12669-21/usr/sbin/munin0
5499991813,3cyclictest8378-21cat0
5499991813,3cyclictest7171-21expr0
5499991813,3cyclictest7039-21expr0
5499991813,3cyclictest665-21runrttasks0
5499991813,3cyclictest31005-21/usr/sbin/munin0
5499991813,3cyclictest3069-21expr0
5499991813,3cyclictest2777-21/usr/sbin/munin0
5499991813,3cyclictest24591-21wc0
5499991813,3cyclictest23197-21cut0
5499991813,3cyclictest23131-21seq0
5499991813,3cyclictest14407-21cut0
5499991813,3cyclictest12670-21pluginstate0
5499991813,3cyclictest12670-21pluginstate0
5499991813,3cyclictest12433-21ls0
5499991813,3cyclictest12433-21ls0
5499991812,4cyclictest32520-21grep0
5499991812,4cyclictest28012-21cat0
5499991812,3cyclictest15392-21latency0
5499991812,3cyclictest14054-21ntp_offset0
549999180,16cyclictest0-21swapper/00
5507991712,3cyclictest11144-21fw_packets2
5507991712,3cyclictest11144-21fw_packets2
5499991715,1cyclictest0-21swapper/00
5499991713,2cyclictest21842-21interrupts0
5499991713,2cyclictest20737-21/usr/sbin/munin0
5499991712,3cyclictest29560-21/usr/sbin/munin0
5499991712,3cyclictest26013-21sed0
549999170,16cyclictest0-21swapper/00
549999170,16cyclictest0-21swapper/00
5499991611,3cyclictest923-21latency_hist0
549999160,1cyclictest0-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional