You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 13:08
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Tue Sep 16, 2025 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
970699276272,2cyclictest0-21swapper/11
970999273270,2cyclictest0-21swapper/22
93372256198,45sleep00-21swapper/00
95422247176,35sleep10-21swapper/11
92882245180,46sleep20-21swapper/22
93552242169,25sleep30-21swapper/33
970499190185,3cyclictest15407-21strings0
971499170166,2cyclictest0-21swapper/33
2819121590,1sleep20-21swapper/22
1143221200,2sleep0970499cyclictest0
26772550,3sleep10-21swapper/11
30902470,1sleep00-21swapper/00
970699400,38cyclictest5534-21kworker/1:01
970999295,10cyclictest28-21ksoftirqd/22
970699284,3cyclictest21-21ksoftirqd/11
970699283,5cyclictest21-21ksoftirqd/11
93642280,2sleep20-21swapper/22
93642280,2sleep20-21swapper/22
970999276,3cyclictest28-21ksoftirqd/22
970699279,3cyclictest21-21ksoftirqd/11
9706992713,3cyclictest21-21ksoftirqd/11
970499270,4cyclictest31542-21cron0
971499267,18cyclictest0-21swapper/33
9714992619,4cyclictest14592-21cut3
970999263,17cyclictest1333-21cat2
9709992612,3cyclictest49852cpuspeed2
9706992610,3cyclictest21-21ksoftirqd/11
970699260,8cyclictest21-21ksoftirqd/11
9709992519,4cyclictest491-40mandb2
970999251,6cyclictest28-21ksoftirqd/22
9709992514,10cyclictest28-21ksoftirqd/22
970999251,20cyclictest22193-21munin-run2
9709992511,3cyclictest28-21ksoftirqd/22
9709992510,3cyclictest28-21ksoftirqd/22
970699258,6cyclictest21-21ksoftirqd/11
970699253,8cyclictest111rcu_preempt1
970699252,7cyclictest21-21ksoftirqd/11
9706992510,2cyclictest21-21ksoftirqd/11
970699250,8cyclictest21-21ksoftirqd/11
174912250,2sleep10-21swapper/11
970999249,3cyclictest111rcu_preempt2
970999242,3cyclictest28-21ksoftirqd/22
9709992420,2cyclictest271rcuc/22
9709992415,3cyclictest28-21ksoftirqd/22
9709992410,3cyclictest28-21ksoftirqd/22
970999240,6cyclictest28-21ksoftirqd/22
970999240,6cyclictest28-21ksoftirqd/22
970999240,5cyclictest28-21ksoftirqd/22
970699242,3cyclictest21-21ksoftirqd/11
970699241,4cyclictest21-21ksoftirqd/11
49822240,4sleep1201rcuc/11
237072240,2sleep20-21swapper/22
201272240,3sleep335-21ksoftirqd/33
970999239,3cyclictest111rcu_preempt2
970999239,3cyclictest111rcu_preempt2
970999239,3cyclictest111rcu_preempt2
970999233,7cyclictest28-21ksoftirqd/22
970999233,17cyclictest4618-21latency_hist2
9709992311,2cyclictest111rcu_preempt2
970999231,10cyclictest26105-21seq2
970999231,10cyclictest26105-21seq2
9709992310,3cyclictest111rcu_preempt2
9709992310,3cyclictest111rcu_preempt2
9706992320,2cyclictest4229-21kworker/1:21
9706992319,2cyclictest201rcuc/11
9706992319,2cyclictest201rcuc/11
970699231,5cyclictest0-21swapper/11
970699231,4cyclictest21-21ksoftirqd/11
9706992311,3cyclictest21-21ksoftirqd/11
971499229,10cyclictest27512-21sendmail3
970999229,3cyclictest28-21ksoftirqd/22
970999229,3cyclictest28-21ksoftirqd/22
970999229,3cyclictest111rcu_preempt2
970999229,2cyclictest111rcu_preempt2
970999228,3cyclictest111rcu_preempt2
970999227,3cyclictest111rcu_preempt2
970999227,3cyclictest111rcu_preempt2
970999227,3cyclictest111rcu_preempt2
970999227,3cyclictest111rcu_preempt2
970999223,6cyclictest28-21ksoftirqd/22
970999223,6cyclictest28-21ksoftirqd/22
970999223,17cyclictest9989-21latency_hist2
9709992215,4cyclictest31342-21ntpq2
970999221,18cyclictest30921-21latency_hist2
9709992210,5cyclictest111rcu_preempt2
9709992210,2cyclictest111rcu_preempt2
970999220,6cyclictest28-21ksoftirqd/22
970999220,6cyclictest28-21ksoftirqd/22
970699225,8cyclictest111rcu_preempt1
9706992219,2cyclictest201rcuc/11
9706992217,3cyclictest201rcuc/11
9706992214,5cyclictest505-21dbus-daemon1
9706992212,7cyclictest1456-21grep1
9706992210,3cyclictest21-21ksoftirqd/11
9706992210,3cyclictest21-21ksoftirqd/11
9704992217,3cyclictest20654-21/usr/sbin/munin0
9714992116,3cyclictest21508-21ntp_kernel_err3
9714992116,3cyclictest14312-21grep3
971499211,5cyclictest0-21swapper/33
970999219,2cyclictest111rcu_preempt2
970999218,3cyclictest111rcu_preempt2
970999217,3cyclictest111rcu_preempt2
970999216,3cyclictest111rcu_preempt2
970999212,5cyclictest0-21swapper/22
970999212,17cyclictest463-21awk2
970999211,6cyclictest0-21swapper/22
970999211,6cyclictest0-21swapper/22
9709992116,3cyclictest29767-21/usr/sbin/munin2
9709992116,3cyclictest0-21swapper/22
9709992115,3cyclictest28186-21diskstats2
9709992115,3cyclictest28186-21diskstats2
9709992114,4cyclictest11888-21sed2
9709992114,4cyclictest11888-21sed2
9709992110,2cyclictest111rcu_preempt2
970699219,3cyclictest21-21ksoftirqd/11
970699215,8cyclictest111rcu_preempt1
970699214,9cyclictest111rcu_preempt1
970699214,8cyclictest111rcu_preempt1
970699214,8cyclictest111rcu_preempt1
970699214,8cyclictest111rcu_preempt1
970699212,7cyclictest111rcu_preempt1
970699212,7cyclictest111rcu_preempt1
970699212,3cyclictest111rcu_preempt1
9706992116,3cyclictest201rcuc/11
9706992115,4cyclictest8159-21expr1
9706992115,4cyclictest4633-21awk1
9706992115,4cyclictest17853-21sed1
9706992110,3cyclictest21-21ksoftirqd/11
970699210,6cyclictest21-21ksoftirqd/11
970499212,16cyclictest23152-21dpkg0
970499212,14cyclictest25732-21apt-get0
9704992115,4cyclictest4207-21seq0
9704992114,4cyclictest18399-21idleruntime-cro0
9704992114,4cyclictest1314-21cat0
9704992113,5cyclictest14295-21acpi0
11762210,3sleep121-21ksoftirqd/11
971499203,15cyclictest17604-21cut3
971499201,6cyclictest7923-21munin-run3
9714992015,3cyclictest6958-21cpu3
9714992015,3cyclictest4982-21timerwakeupswit3
9714992015,3cyclictest29134-21/usr/sbin/munin3
9714992015,3cyclictest2313-21open_inodes3
9714992015,3cyclictest22505-21timerandwakeup3
9714992015,3cyclictest14035-21seq3
9714992014,4cyclictest4729-21cat3
9714992014,4cyclictest31991-21ntpq3
9714992014,4cyclictest18340-21sed3
9714992014,4cyclictest14243-21cut3
9714992014,3cyclictest2082-21systemd-journal3
9714992014,3cyclictest2082-21systemd-journal3
970999209,2cyclictest111rcu_preempt2
970999202,6cyclictest28-21ksoftirqd/22
970999200,7cyclictest28-21ksoftirqd/22
970999200,6cyclictest28-21ksoftirqd/22
970699208,3cyclictest21-21ksoftirqd/11
970699206,2cyclictest111rcu_preempt1
9706992016,3cyclictest201rcuc/11
9706992016,2cyclictest201rcuc/11
9706992015,3cyclictest7250-21seq1
9706992013,4cyclictest23383-21grep1
9706992013,4cyclictest2082-21systemd-journal1
9704992017,2cyclictest0-21swapper/00
9704992015,3cyclictest31127-21kernelversion0
9704992015,3cyclictest28440-21/usr/sbin/munin0
9704992014,4cyclictest8344-21seq0
9704992014,4cyclictest6559-21cat0
9704992014,4cyclictest3731-21/usr/sbin/munin0
9704992014,4cyclictest30235-21expr0
9704992014,4cyclictest2742-21cat0
9704992014,4cyclictest23919-21cat0
9704992014,4cyclictest1971-21sed0
9704992014,4cyclictest17052-21sed0
9704992014,4cyclictest15753-21seq0
9704992014,4cyclictest14045-21cron0
9704992014,4cyclictest10673-21seq0
9704992014,4cyclictest10673-21seq0
9704992013,5cyclictest505-21dbus-daemon0
9704992013,4cyclictest2082-21systemd-journal0
9704992013,4cyclictest2082-21systemd-journal0
9714991915,2cyclictest9136-21/usr/sbin/munin3
9714991915,2cyclictest17155-21ntp_offset3
9714991914,3cyclictest9919-21proc_pri3
9714991914,3cyclictest9919-21proc_pri3
9714991914,3cyclictest32014-21ntp_kernel_err3
9714991914,3cyclictest30654-21cat3
9714991914,3cyclictest3014-21cpu3
9714991914,3cyclictest29918-21nfs4_client3
9714991914,3cyclictest27630-21ntp_offset3
9714991914,3cyclictest22946-21expr3
9714991914,3cyclictest22730-21expr3
9714991914,3cyclictest22044-21seq3
9714991913,4cyclictest8411-21cat3
9714991913,4cyclictest28665-21pluginstate3
9714991913,4cyclictest28665-21pluginstate3
9714991913,4cyclictest25613-21ntpq3
9714991913,4cyclictest20532-21cat3
9714991913,4cyclictest19227-21cut3
9714991912,4cyclictest665-21runrttasks3
970699196,2cyclictest111rcu_preempt1
970699195,2cyclictest111rcu_preempt1
970699194,3cyclictest111rcu_preempt1
970699194,3cyclictest111rcu_preempt1
970699193,2cyclictest21-21ksoftirqd/11
970699191,5cyclictest21-21ksoftirqd/11
9706991914,3cyclictest2083-21/usr/sbin/munin1
9706991913,4cyclictest4171-21seq1
9706991913,4cyclictest24763-21expr1
9706991913,4cyclictest24763-21expr1
9706991913,4cyclictest11843-21seq1
970699190,4cyclictest27762-21apt-get1
970699190,4cyclictest27762-21apt-get1
9704991914,3cyclictest8474-21tr0
9704991914,3cyclictest4888-21aten2_r9power_c0
9704991914,3cyclictest28932-21sed0
9704991914,3cyclictest28932-21sed0
9704991914,3cyclictest26754-21/usr/sbin/munin0
9704991914,3cyclictest18288-21/usr/sbin/munin0
9704991914,3cyclictest13214-21/usr/sbin/munin0
9704991913,4cyclictest25590-21cat0
9704991913,4cyclictest2506-21idleruntime-cro0
9704991913,4cyclictest16359-21/usr/sbin/munin0
9704991913,3cyclictest31595-21sh0
9714991815,2cyclictest0-21swapper/33
9714991814,2cyclictest28440-21/usr/sbin/munin3
9714991814,2cyclictest23713-21apt3
9714991813,3cyclictest7538-21sendmail3
9714991813,3cyclictest5557-21cat3
9714991813,3cyclictest32740-21pluginstate3
9714991813,3cyclictest2888-21cpu3
9714991813,3cyclictest23503-21cat3
9714991813,3cyclictest20322-21/usr/sbin/munin3
9714991813,3cyclictest16967-21sed3
9714991813,3cyclictest16642-21awk3
9714991813,3cyclictest12654-21expr3
9714991813,3cyclictest1109-21rt-features3
9714991812,4cyclictest18830-21cpuspeed3
971499181,16cyclictest0-21swapper/33
9714991811,4cyclictest11471-21grep3
9714991811,4cyclictest11471-21grep3
971499180,16cyclictest10159-21apt-get3
971499180,16cyclictest0-21swapper/33
970699183,2cyclictest21-21ksoftirqd/11
970699181,4cyclictest16288-21apt-get1
970499188,8cyclictest13082-21aten2.4-expect0
9704991816,1cyclictest0-21swapper/00
9704991816,1cyclictest0-21swapper/00
9704991814,2cyclictest11842-21munin-node0
9704991813,3cyclictest8739-21sed0
9704991813,3cyclictest7411-21sed0
9704991813,3cyclictest665-21runrttasks0
9704991813,3cyclictest665-21runrttasks0
9704991813,3cyclictest665-21runrttasks0
9704991813,3cyclictest26009-21seq0
9704991813,3cyclictest26009-21seq0
9704991813,3cyclictest21038-21seq0
9704991813,3cyclictest12454-21expr0
9704991813,3cyclictest11285-21seq0
9704991813,3cyclictest11198-21gzip0
9704991812,4cyclictest4180-21cut0
9704991812,4cyclictest26929-21hostname0
9704991812,4cyclictest26503-21latency_hist0
9704991812,4cyclictest24196-21seq0
9704991812,4cyclictest18095-21cut0
9704991811,5cyclictest505-21dbus-daemon0
9704991811,4cyclictest505-21dbus-daemon0
9714991712,3cyclictest9080-21ntpq3
9714991712,3cyclictest8855-21cut3
9714991712,3cyclictest665-21runrttasks3
9714991712,3cyclictest5701-21cut3
9714991712,3cyclictest27260-21nfs4_client3
9714991712,3cyclictest17106-21gltestperf3
9714991712,3cyclictest12596-21expr3
9714991712,3cyclictest12189-21cat3
9706991712,3cyclictest32182-21cat1
9704991713,2cyclictest2818-21/usr/sbin/munin0
9704991712,3cyclictest9888-21/usr/sbin/munin0
9704991712,3cyclictest665-21runrttasks0
9704991712,3cyclictest665-21runrttasks0
9704991712,3cyclictest665-21runrttasks0
9704991712,3cyclictest29412-21expr0
9704991712,3cyclictest16503-21expr0
9704991711,3cyclictest17589-21grep0
9714991612,2cyclictest1137-21sh3
9714991611,3cyclictest30849-21expr3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional