You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:59
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack9slot2.osadl.org (updated Sat Jul 12, 2025 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1961799281276,3cyclictest0-21swapper/11
193162268186,67sleep10-21swapper/11
191882256184,25sleep00-21swapper/00
192502236182,42sleep20-21swapper/22
193682231193,25sleep30-21swapper/33
1962899212207,3cyclictest0-21swapper/33
1962099208203,3cyclictest22444-21apt-get2
3155921720,3sleep31962899cyclictest3
982821710,2sleep01961499cyclictest0
1961499111107,2cyclictest0-21swapper/00
210742480,1sleep10-21swapper/11
262642470,2sleep30-21swapper/33
187892430,3sleep00-21swapper/00
95712420,2sleep00-21swapper/00
152392420,2sleep20-21swapper/22
62092400,2sleep10-21swapper/11
125642340,2sleep00-21swapper/00
1962899318,3cyclictest111rcu_preempt3
19620993026,2cyclictest271rcuc/22
1962099300,6cyclictest28-21ksoftirqd/22
211812290,9sleep335-21ksoftirqd/33
19628992926,2cyclictest341rcuc/33
1962099282,6cyclictest28-21ksoftirqd/22
1962099281,3cyclictest28-21ksoftirqd/22
1962099278,2cyclictest111rcu_preempt2
1962099277,8cyclictest28-21ksoftirqd/22
19620992724,2cyclictest271rcuc/22
19620992714,4cyclictest28-21ksoftirqd/22
19628992623,2cyclictest341rcuc/33
19628992623,2cyclictest341rcuc/33
1962899260,5cyclictest35-21ksoftirqd/33
1962099260,7cyclictest28-21ksoftirqd/22
19628992522,2cyclictest341rcuc/33
19628992522,2cyclictest341rcuc/33
1962099256,3cyclictest111rcu_preempt2
1962099252,7cyclictest28-21ksoftirqd/22
1962099251,16cyclictest0-21swapper/22
1962099250,6cyclictest28-21ksoftirqd/22
1962099250,5cyclictest28-21ksoftirqd/22
156702250,1sleep10-21swapper/11
1962899246,9cyclictest111rcu_preempt3
19628992421,2cyclictest341rcuc/33
19628992420,3cyclictest341rcuc/33
19628992420,2cyclictest341rcuc/33
1962099247,8cyclictest28-21ksoftirqd/22
1962099244,2cyclictest111rcu_preempt2
1962099244,12cyclictest111rcu_preempt2
19614992418,3cyclictest8314-21apt-key0
1961499240,12cyclictest236022missed_timers0
1962899239,2cyclictest111rcu_preempt3
19628992320,2cyclictest341rcuc/33
19628992320,2cyclictest341rcuc/33
1962099239,3cyclictest28-21ksoftirqd/22
1962099239,12cyclictest32407-21cut2
1962099236,8cyclictest28-21ksoftirqd/22
1962099236,3cyclictest28-21ksoftirqd/22
1962099235,8cyclictest28-21ksoftirqd/22
1962099234,10cyclictest28-21ksoftirqd/22
19620992310,11cyclictest19892-21cat2
1962099230,4cyclictest25504-21grep2
157692230,2sleep228-21ksoftirqd/22
1962899228,3cyclictest111rcu_preempt3
1962899227,3cyclictest111rcu_preempt3
1962899227,2cyclictest111rcu_preempt3
1962899226,3cyclictest111rcu_preempt3
19628992219,2cyclictest341rcuc/33
19628992217,3cyclictest4677-21cron3
1962899221,6cyclictest35-21ksoftirqd/33
1962899221,6cyclictest341rcuc/33
19628992215,4cyclictest7133-21cron3
19628992215,4cyclictest7133-21cron3
19628992210,2cyclictest111rcu_preempt3
1962099223,2cyclictest111rcu_preempt2
19620992217,3cyclictest29110-21munin-run2
19620992215,4cyclictest5880-21aten2_r9power_v2
19620992214,5cyclictest10333-21gdbus2
1962099221,3cyclictest27106-21apt2
1962099221,19cyclictest10333-21gdbus2
1962099220,7cyclictest28-21ksoftirqd/22
1962099220,7cyclictest28-21ksoftirqd/22
19617992218,2cyclictest13714-21apt-get1
19617992217,3cyclictest24479-21cron1
19614992215,4cyclictest665-21runrttasks0
1962899219,2cyclictest111rcu_preempt3
1962899217,3cyclictest111rcu_preempt3
1962899217,2cyclictest111rcu_preempt3
1962899217,2cyclictest111rcu_preempt3
19628992117,3cyclictest35-21ksoftirqd/33
19628992116,3cyclictest6858-21wc3
19628992114,4cyclictest20957-21grep3
1962899211,2cyclictest111rcu_preempt3
19628992110,2cyclictest111rcu_preempt3
1962099219,3cyclictest111rcu_preempt2
1962099219,10cyclictest30174-21cat2
1962099217,3cyclictest111rcu_preempt2
1962099215,8cyclictest111rcu_preempt2
1962099215,4cyclictest111rcu_preempt2
1962099212,7cyclictest28-21ksoftirqd/22
1962099212,17cyclictest15407-21cat2
19620992116,3cyclictest30351-21users2
1962099211,3cyclictest7125-21munin-run2
1962099211,3cyclictest7125-21munin-run2
1962099211,3cyclictest25437-1kworker/2:1H2
1962099210,8cyclictest28-21ksoftirqd/22
1962099210,6cyclictest28-21ksoftirqd/22
1962099210,3cyclictest26279-21perf2
19617992115,4cyclictest32227-21grep1
19617992115,4cyclictest29069-21grep1
19617992115,4cyclictest29018-21seq1
19617992115,4cyclictest24754-21seq1
1961499211,9cyclictest20776-21cups-browsed0
19614992116,3cyclictest1080-21vmstat0
19614992112,6cyclictest3999-21idleruntime-cro0
1962899208,3cyclictest111rcu_preempt3
1962899207,3cyclictest111rcu_preempt3
1962899207,3cyclictest111rcu_preempt3
1962899207,3cyclictest111rcu_preempt3
1962899207,3cyclictest111rcu_preempt3
1962899207,2cyclictest111rcu_preempt3
1962899207,2cyclictest111rcu_preempt3
1962899207,2cyclictest111rcu_preempt3
1962899206,3cyclictest111rcu_preempt3
1962899205,3cyclictest111rcu_preempt3
1962899201,7cyclictest35-21ksoftirqd/33
19628992016,3cyclictest341rcuc/33
19628992014,3cyclictest16504-21awk3
1962099209,3cyclictest111rcu_preempt2
1962099205,3cyclictest111rcu_preempt2
1962099204,3cyclictest111rcu_preempt2
1962099204,3cyclictest111rcu_preempt2
19620992015,3cyclictest12624-21sed2
19620992014,3cyclictest2168-21grep2
19620992014,3cyclictest12830-21munin-run2
19620992013,4cyclictest9462-21diskstats2
19620992011,3cyclictest111rcu_preempt2
1962099200,5cyclictest14466-21perf2
19617992016,2cyclictest15617-21aten2_r9power_p1
19617992015,3cyclictest810-21sed1
19617992015,3cyclictest639-21users1
19617992015,3cyclictest21369-21seq1
19617992015,3cyclictest19633-21sh1
19617992015,3cyclictest14695-21seq1
19617992014,4cyclictest7991-21cat1
19617992014,4cyclictest7327-21expr1
19617992014,4cyclictest5229-21cat1
19617992014,4cyclictest32007-21seq1
19617992014,4cyclictest2989-21seq1
19617992014,4cyclictest2875-21seq1
19617992014,4cyclictest12667-21find1
19617992013,4cyclictest29354-21cat1
19617992013,4cyclictest21071-21cat1
19614992016,2cyclictest23260-21/usr/sbin/munin0
19614992015,3cyclictest27464-21cut0
19614992015,3cyclictest25721-21/usr/sbin/munin0
19614992015,3cyclictest19799-21/usr/sbin/munin0
19614992014,4cyclictest2150-21ntp_kernel_err0
19614992013,5cyclictest1037-21idleruntime-cro0
19614992013,4cyclictest5818-21sh0
19614992013,4cyclictest30333-21grep0
19614992013,4cyclictest1225-21mailstats0
19614992011,5cyclictest505-21dbus-daemon0
1961499200,19cyclictest0-21swapper/00
1962899197,2cyclictest111rcu_preempt3
1962899196,3cyclictest111rcu_preempt3
1962899196,3cyclictest111rcu_preempt3
1962899196,3cyclictest111rcu_preempt3
1962899196,2cyclictest111rcu_preempt3
1962899195,3cyclictest111rcu_preempt3
1962899194,3cyclictest111rcu_preempt3
1962899193,3cyclictest111rcu_preempt3
1962899191,6cyclictest35-21ksoftirqd/33
19628991915,3cyclictest341rcuc/33
19628991914,3cyclictest8008-21munin-node3
19628991914,3cyclictest452-21tr3
19628991914,3cyclictest27729-21seq3
19628991914,3cyclictest17996-21expr3
19628991913,4cyclictest29427-21ntp_states3
19628991913,4cyclictest16846-21cut3
1962099199,2cyclictest111rcu_preempt2
1962099195,3cyclictest111rcu_preempt2
1962099195,3cyclictest111rcu_preempt2
1962099194,3cyclictest111rcu_preempt2
1962099190,7cyclictest28-21ksoftirqd/22
1962099190,7cyclictest28-21ksoftirqd/22
1962099190,6cyclictest4945-21apt-get2
19617991914,3cyclictest8702-21awk1
19617991914,3cyclictest8594-21/usr/sbin/munin1
19617991914,3cyclictest816-21awk1
19617991914,3cyclictest6238-21/usr/sbin/munin1
19617991914,3cyclictest27435-21seq1
19617991914,3cyclictest2619-21seq1
19617991914,3cyclictest24726-21/usr/sbin/munin1
19617991914,3cyclictest23667-21/usr/sbin/munin1
19617991914,3cyclictest22946-21aten2_r9power_p1
19617991914,3cyclictest21237-21/usr/sbin/munin1
19617991914,3cyclictest18621-21sh1
19617991914,3cyclictest17480-21ls1
19617991914,3cyclictest16557-21expr1
19617991913,4cyclictest6245-21awk1
19617991913,4cyclictest6245-21awk1
19617991913,4cyclictest26264-21grep1
19617991913,4cyclictest15582-21ntp_states1
19617991913,4cyclictest14111-21cat1
19617991913,4cyclictest10476-21seq1
19617991913,3cyclictest31576-21basename1
1961499198,7cyclictest22458-21munin-run0
19614991916,2cyclictest0-21swapper/00
19614991914,3cyclictest9018-21/usr/sbin/munin0
19614991914,3cyclictest3847-21sed0
19614991914,3cyclictest26526-21latency_hist0
19614991913,4cyclictest24612-21sed0
19614991913,4cyclictest17812-21which0
19614991913,4cyclictest10204-21cat0
19614991913,3cyclictest20030-21cat0
1961499191,2cyclictest0-21swapper/00
1961799189,5cyclictest19622-21apt-get1
19617991816,1cyclictest0-21swapper/11
19617991814,2cyclictest5761-21/usr/sbin/munin1
19617991814,2cyclictest21650-21/usr/sbin/munin1
19617991814,2cyclictest17570-21/usr/sbin/munin1
19617991813,3cyclictest4776-21idleruntime1
19617991813,3cyclictest29527-21munin-plugin-st1
19617991813,3cyclictest2899-21/usr/sbin/munin1
19617991813,3cyclictest26933-21apt-get1
19617991813,3cyclictest2588-21expr1
19617991813,3cyclictest25474-21if_enp2s01
19617991813,3cyclictest18772-21/usr/sbin/munin1
19617991813,3cyclictest17752-21if_enp2s01
19617991813,3cyclictest12590-21seq1
1961499183,9cyclictest26933-21apt-get0
1961499183,12cyclictest32656-21head0
1961499182,9cyclictest19630-21taskset0
19614991816,1cyclictest0-21swapper/00
19614991816,1cyclictest0-21swapper/00
19614991816,1cyclictest0-21swapper/00
19614991816,1cyclictest0-21swapper/00
19614991816,1cyclictest0-21swapper/00
19614991814,2cyclictest5343-21/usr/sbin/munin0
19614991814,2cyclictest5343-21/usr/sbin/munin0
19614991814,2cyclictest3727-21/usr/sbin/munin0
19614991814,2cyclictest29770-21/usr/sbin/munin0
19614991814,2cyclictest26239-21grep0
19614991814,2cyclictest24480-21/usr/sbin/munin0
19614991814,2cyclictest1701-21/usr/sbin/munin0
19614991813,3cyclictest665-21runrttasks0
19614991813,3cyclictest26217-21sed0
19614991812,4cyclictest11666-21cron0
19614991812,4cyclictest11207-21cat0
19614991812,4cyclictest11087-21cut0
19614991812,3cyclictest29055-21gltestperf0
19617991713,2cyclictest9874-21/usr/sbin/munin1
19617991713,2cyclictest10631-21/usr/sbin/munin1
19617991712,3cyclictest27004-21gltestperf1
19617991712,3cyclictest10485-21gzip1
19617991711,4cyclictest12517-21cat1
1961799170,16cyclictest0-21swapper/11
1961499179,4cyclictest13509-21apt-get0
19614991715,1cyclictest0-21swapper/00
19614991712,3cyclictest665-21runrttasks0
19614991712,3cyclictest665-21runrttasks0
19614991712,3cyclictest665-21runrttasks0
19614991712,3cyclictest665-21runrttasks0
19614991712,3cyclictest28741-21cut0
19614991712,3cyclictest27296-21cat0
19614991712,3cyclictest2659-21dump-pmu-power0
19614991712,3cyclictest21294-21aten2_r9power_e0
19614991712,3cyclictest20971-21seq0
1961499170,16cyclictest0-21swapper/00
19617991611,3cyclictest4615-21ntp_states1
19617991610,3cyclictest24607-21aten2_r9power_v1
1961499162,8cyclictest13096-21apt-get0
19614991614,1cyclictest0-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional