You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-08 - 17:59
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Wed May 08, 2024 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302152362286,41sleep30-21swapper/33
303092307270,24sleep20-21swapper/22
303122302245,44sleep00-21swapper/00
302282302266,23sleep10-21swapper/11
3065099203198,3cyclictest633-21sed0
3066199150146,2cyclictest626-21if_enp2s02
3066499149145,2cyclictest0-21swapper/33
1462821400,2sleep03065099cyclictest0
3065699110106,2cyclictest0-21swapper/11
138202530,4sleep03065099cyclictest0
24032470,1sleep30-21swapper/33
129742460,4sleep012977-21gltestperf0
306862440,3sleep20-21swapper/22
232892350,4sleep1201rcuc/11
30650993413,3cyclictest26726-21expr0
30664993314,4cyclictest35-21ksoftirqd/33
30664993313,4cyclictest35-21ksoftirqd/33
30661993312,4cyclictest28-21ksoftirqd/22
30650993211,3cyclictest20002-21cut0
38872300,2sleep00-21swapper/00
3065099309,4cyclictest31409-21grep0
30650993014,2cyclictest16573-21/usr/sbin/munin0
30650993012,3cyclictest22349-21apt0
30650993011,3cyclictest3389-21sed0
30650993011,3cyclictest19230-21ls0
30664992913,3cyclictest35-21ksoftirqd/33
3065099299,4cyclictest9788-21sh0
30650992911,3cyclictest31222-21cut0
3065099289,3cyclictest16467-21expr0
3065099285,4cyclictest0-21swapper/00
3065099282,11cyclictest23853-21grep0
30650992817,6cyclictest10066-21apt-get0
3066499278,7cyclictest35-21ksoftirqd/33
30661992710,4cyclictest28-21ksoftirqd/22
30661992710,4cyclictest28-21ksoftirqd/22
3065099278,4cyclictest21988-21latency_hist0
30650992711,3cyclictest19850-21unixbench-2d0
30650992711,2cyclictest4502-21aten2_r9power_c0
30650992710,3cyclictest8586-21aten2_r9power_p0
30650992710,3cyclictest14157-21aten2_r9power_p0
3066199261,6cyclictest10290-21munin-run2
3065099262,6cyclictest24341-21apt-get0
3065099261,8cyclictest3911-21apt-get0
30650992612,3cyclictest665-21runrttasks0
30650992612,3cyclictest665-21runrttasks0
30650992611,3cyclictest665-21runrttasks0
30650992610,3cyclictest4234-21latency_hist0
30664992510,4cyclictest35-21ksoftirqd/33
30661992514,9cyclictest6770-21cpuspeed2
3065099253,7cyclictest10586-21apt-get0
30650992514,1cyclictest0-21swapper/00
49792240,2sleep10-21swapper/11
3066499249,3cyclictest35-21ksoftirqd/33
3066499248,3cyclictest111rcu_preempt3
3066199249,3cyclictest28-21ksoftirqd/22
3066199246,10cyclictest28-21ksoftirqd/22
3066199244,17cyclictest28095-21latency_hist2
3066199241,2cyclictest111rcu_preempt2
30661992411,2cyclictest111rcu_preempt2
30661992410,4cyclictest28-21ksoftirqd/22
30650992415,5cyclictest20537-21apt-get0
30650992415,5cyclictest20537-21apt-get0
30650992411,2cyclictest665-21runrttasks0
3066499239,3cyclictest35-21ksoftirqd/33
30661992312,9cyclictest271rcuc/22
3066199231,19cyclictest18506-21sed2
3066199231,15cyclictest24526-21apt-get2
30661992310,2cyclictest111rcu_preempt2
30650992310,3cyclictest665-21runrttasks0
30664992217,3cyclictest341rcuc/33
30664992217,3cyclictest341rcuc/33
30664992216,4cyclictest11636-21sed3
3066499220,20cyclictest0-21swapper/33
3066199227,3cyclictest111rcu_preempt2
3066199224,16cyclictest20416-21cat2
3066199223,2cyclictest111rcu_preempt2
3066199222,14cyclictest12077-21apt-get2
30661992217,3cyclictest271rcuc/22
30661992216,3cyclictest13468-21apt2
30661992215,4cyclictest3421-21awk2
30661992215,4cyclictest21009-21df2
30661992215,4cyclictest20469-21date2
3066199221,18cyclictest3987-21munin-run2
3066199220,19cyclictest17231-21irqstats2
3065099220,15cyclictest0-21swapper/00
233532220,3sleep223355-21sed2
3066499213,3cyclictest111rcu_preempt3
30664992115,3cyclictest32765-21kernelversion3
30664992114,4cyclictest13605-21wc3
3066199218,3cyclictest111rcu_preempt2
3066199218,2cyclictest111rcu_preempt2
3066199213,3cyclictest111rcu_preempt2
3066199213,3cyclictest111rcu_preempt2
3066199212,17cyclictest30766-21cat2
30661992116,3cyclictest5392-21ntp_kernel_err2
30661992115,3cyclictest16923-21ntp_kernel_err2
3066199211,4cyclictest111rcu_preempt2
3066199210,3cyclictest11227-21irqstats2
30656992114,5cyclictest14382-21cut1
3065699211,18cyclictest2221-21munin-run1
30650992116,3cyclictest32093-21/usr/sbin/munin0
30650992114,4cyclictest505-21dbus-daemon0
252402210,1sleep00-21swapper/00
208292210,2sleep30-21swapper/33
3066499209,2cyclictest111rcu_preempt3
3066499209,2cyclictest111rcu_preempt3
3066499205,2cyclictest111rcu_preempt3
3066499203,7cyclictest111rcu_preempt3
30664992017,2cyclictest35-21ksoftirqd/33
30664992016,2cyclictest35-21ksoftirqd/33
30664992016,2cyclictest341rcuc/33
30664992015,3cyclictest4046-21expr3
30664992015,3cyclictest35-21ksoftirqd/33
3066499201,4cyclictest111rcu_preempt3
30664992014,4cyclictest20517-21/usr/sbin/munin3
30664992014,4cyclictest20517-21/usr/sbin/munin3
3066199209,3cyclictest111rcu_preempt2
3066199209,3cyclictest111rcu_preempt2
3066199209,2cyclictest111rcu_preempt2
3066199208,3cyclictest111rcu_preempt2
3066199208,2cyclictest111rcu_preempt2
3066199207,2cyclictest111rcu_preempt2
3066199207,2cyclictest111rcu_preempt2
30661992015,3cyclictest7369-21ntp_offset2
30661992014,4cyclictest26019-21cut2
30661992014,4cyclictest26019-21cut2
30661992014,4cyclictest20972-21pluginstate2
30661992014,4cyclictest1832-21grep2
3066199201,17cyclictest5251-1kworker/2:0H2
3066199201,17cyclictest5251-1kworker/2:0H2
3065699206,10cyclictest111rcu_preempt1
30656992015,3cyclictest8499-21aten2_r9power_c1
30656992015,3cyclictest26917-21needreboot1
30656992015,3cyclictest25188-21ntpq1
30656992015,3cyclictest14586-21http1
30656992014,4cyclictest9653-21sed1
30656992014,4cyclictest8531-21/usr/sbin/munin1
30656992014,4cyclictest6646-21which1
30656992014,4cyclictest4741-21cat1
30656992014,4cyclictest24512-21df_inode1
30656992014,4cyclictest24512-21df_inode1
30656992014,4cyclictest23414-21sh1
30656992014,4cyclictest23140-21awk1
30656992014,4cyclictest17814-21sendmail-msp1
30656992014,3cyclictest4476-21/usr/sbin/munin1
30656992013,5cyclictest11681-21seq1
30650992015,3cyclictest14563-21/usr/sbin/munin0
30650992012,6cyclictest22382-21tr0
3066499199,3cyclictest111rcu_preempt3
3066499195,3cyclictest111rcu_preempt3
3066499195,2cyclictest111rcu_preempt3
3066499194,2cyclictest111rcu_preempt3
3066499193,3cyclictest111rcu_preempt3
3066499193,2cyclictest111rcu_preempt3
3066499192,3cyclictest111rcu_preempt3
3066499192,3cyclictest111rcu_preempt3
3066499192,3cyclictest111rcu_preempt3
3066499192,3cyclictest111rcu_preempt3
3066499192,3cyclictest111rcu_preempt3
3066499192,2cyclictest111rcu_preempt3
30664991917,1cyclictest0-21swapper/33
30664991914,3cyclictest8259-21seq3
30664991914,3cyclictest30765-21expr3
30664991914,3cyclictest25441-21sed3
30664991914,3cyclictest22873-21expr3
30664991914,3cyclictest17894-21cut3
3066499191,3cyclictest35-21ksoftirqd/33
3066499191,3cyclictest111rcu_preempt3
3066499191,3cyclictest111rcu_preempt3
30664991913,4cyclictest12420-21date3
30664991912,4cyclictest5733-21aten2_r9power_e3
30664991910,2cyclictest35-21ksoftirqd/33
3066499190,3cyclictest2572-21ntp_states3
3066199197,3cyclictest111rcu_preempt2
3066199192,2cyclictest28-21ksoftirqd/22
30661991915,2cyclictest5954-21aten2_r9power_e2
30661991915,2cyclictest271rcuc/22
30661991914,3cyclictest6539-21ntp_kernel_pll_2
30661991914,3cyclictest27298-21memory2
30661991914,3cyclictest13187-21if_err_enp2s02
3066199191,3cyclictest111rcu_preempt2
30661991913,4cyclictest19162-21threads2
30661991913,4cyclictest18940-21cut2
30661991913,3cyclictest11218-21cat2
3066199190,3cyclictest808-21cstates2
3066199190,2cyclictest15622-21apt-config2
30656991915,2cyclictest9687-21/usr/sbin/munin1
30656991915,2cyclictest6478-21/usr/sbin/munin1
30656991915,2cyclictest20576-21/usr/sbin/munin1
30656991915,2cyclictest10505-21/usr/sbin/munin1
30656991915,2cyclictest0-21swapper/11
30656991914,3cyclictest4197-21cut1
30656991914,3cyclictest32530-21/usr/sbin/munin1
30656991914,3cyclictest29333-21sed1
30656991914,3cyclictest20089-21aten2_r9power_e1
30656991914,3cyclictest17614-21seq1
30656991914,3cyclictest1671-21/usr/sbin/munin1
30656991913,4cyclictest24197-21ls1
30656991913,4cyclictest24197-21ls1
30656991913,4cyclictest19311-21grep1
30656991913,4cyclictest18066-21date1
30656991913,4cyclictest10951-21expr1
30656991913,3cyclictest30529-21diskstats1
30656991912,4cyclictest27207-21ntpq1
3065699190,6cyclictest19409-21ntpq1
3065699190,4cyclictest0-21swapper/11
3065699190,4cyclictest0-21swapper/11
30650991914,3cyclictest3881-21sed0
30650991914,3cyclictest30772-21expr0
30650991914,3cyclictest30402-21grep0
30650991914,3cyclictest22755-21expr0
30650991914,3cyclictest22755-21expr0
30650991914,3cyclictest11628-21cat0
30650991913,4cyclictest7666-21cron0
3066499186,2cyclictest111rcu_preempt3
3066499186,2cyclictest111rcu_preempt3
3066499183,3cyclictest111rcu_preempt3
3066499182,3cyclictest111rcu_preempt3
3066499181,4cyclictest111rcu_preempt3
30664991814,2cyclictest29743-21cut3
3066499181,3cyclictest111rcu_preempt3
3066499181,3cyclictest111rcu_preempt3
3066499181,3cyclictest111rcu_preempt3
3066499181,3cyclictest111rcu_preempt3
30664991813,3cyclictest30961-21expr3
30664991813,3cyclictest25538-21kernelversion3
30664991813,3cyclictest12447-21expr3
30664991812,4cyclictest32094-21sed3
3066199186,3cyclictest28-21ksoftirqd/22
30661991814,2cyclictest24423-21aten2_r9power_p2
30661991813,3cyclictest3893-21/usr/sbin/munin2
30661991812,3cyclictest4565-21aten2_r9power_p2
3065699182,13cyclictest17061-21grep1
30656991816,1cyclictest0-21swapper/11
30656991816,1cyclictest0-21swapper/11
30656991816,1cyclictest0-21swapper/11
30656991816,1cyclictest0-21swapper/11
30656991814,2cyclictest32732-21aten2.4-expect1
30656991814,2cyclictest28894-21/usr/sbin/munin1
30656991814,2cyclictest24507-21/usr/sbin/munin1
30656991814,2cyclictest20517-21/usr/sbin/munin1
30656991814,2cyclictest20517-21/usr/sbin/munin1
30656991814,2cyclictest12818-21missed_timers1
30656991813,3cyclictest7973-21seq1
30656991813,3cyclictest7887-21/usr/sbin/munin1
30656991813,3cyclictest32093-21/usr/sbin/munin1
30656991813,3cyclictest31639-21switchtime1
30656991813,3cyclictest29538-21sed1
30656991813,3cyclictest14741-21cut1
30656991813,3cyclictest14497-21/usr/sbin/munin1
30650991816,1cyclictest0-21swapper/00
30650991816,1cyclictest0-21swapper/00
30650991814,2cyclictest1256-21/usr/sbin/munin0
30650991813,3cyclictest673-21cat0
30650991813,3cyclictest665-21runrttasks0
30650991813,3cyclictest28695-21snmpd0
30650991813,3cyclictest24064-21/usr/sbin/munin0
30650991813,3cyclictest24064-21/usr/sbin/munin0
30650991813,3cyclictest18222-21aten2_r9power_v0
30650991813,3cyclictest17315-21cat0
30650991813,3cyclictest12135-21sh0
30650991813,3cyclictest10281-21cat0
30650991812,4cyclictest698-21cat0
30650991812,4cyclictest4952-21cat0
3065699173,8cyclictest24896-21apt-get1
30656991715,1cyclictest0-21swapper/11
30656991713,2cyclictest2489-21/usr/sbin/munin1
30656991712,3cyclictest7178-21grep1
30656991712,3cyclictest665-21runrttasks1
30656991712,3cyclictest31159-21cat1
30656991712,3cyclictest21008-21grep1
30656991712,3cyclictest14332-21sh1
3065099173,8cyclictest28350-21apt-get0
3065099172,10cyclictest26549-21apt-get0
30650991715,1cyclictest0-21swapper/00
30650991712,3cyclictest665-21runrttasks0
30650991712,3cyclictest665-21runrttasks0
30650991712,3cyclictest0-21swapper/00
30650991711,4cyclictest12416-21seq0
30656991610,4cyclictest26519-21date1
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional