You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-19 - 02:52
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Thu Apr 18, 2024 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
132132312275,24sleep00-21swapper/00
131602272190,68sleep20-21swapper/22
133002266192,26sleep10-21swapper/11
1360599263259,2cyclictest0-21swapper/22
131812253198,42sleep30-21swapper/33
1361199236229,5cyclictest20461-21idleruntime-cro3
1359999226221,3cyclictest20442-21perf0
1737121460,2sleep317374-21grep3
1360199117111,4cyclictest20444-21munin-run1
310562510,3sleep10-21swapper/11
22382510,5sleep22225-21dump-pmu-power2
97152450,4sleep10-21swapper/11
232922370,1sleep20-21swapper/22
146652370,1sleep30-21swapper/33
146652370,1sleep30-21swapper/33
93092350,1sleep20-21swapper/22
1360599319,19cyclictest0-21swapper/22
13599992619,4cyclictest10333-21gdbus0
13599992520,3cyclictest30551-21turbostat0
1361199242,20cyclictest0-21swapper/33
1360599244,17cyclictest22653-21cat2
1360599242,18cyclictest26458-21cron2
13605992418,4cyclictest14381-21cron2
13601992419,4cyclictest21-21ksoftirqd/11
13611992317,4cyclictest28305-21users3
1360599231,5cyclictest28677-21munin-run2
1360599231,5cyclictest12385-21munin-run2
13611992217,3cyclictest13590-21latency3
13611992216,4cyclictest9030-21df_inode3
13611992215,4cyclictest8935-21fw_packets3
1361199220,4cyclictest0-21swapper/33
1360599223,17cyclictest18614-21cat2
1360599223,17cyclictest18479-21cat2
13605992216,3cyclictest2088-21munin-run2
1360599221,5cyclictest30728-21munin-run2
1360599221,3cyclictest14656-21sendmail2
13599992214,5cyclictest2082-21systemd-journal0
13611992116,3cyclictest5133-21irqstats3
13611992116,3cyclictest26881-21aten2_r9power_p3
13611992115,4cyclictest18897-21cpuspeed3
13611992115,3cyclictest3006-21interrupts3
13611992114,5cyclictest6915-21df_inode3
13611992114,4cyclictest11467-21awk3
13611992114,4cyclictest10145-21sendmail-msp3
1360599219,3cyclictest17678-21missed_timers2
1360599219,11cyclictest15533-21basename2
1360599214,14cyclictest2082-21systemd-journal2
1360599212,6cyclictest30737-21apt-get2
1360599212,17cyclictest32623-21cat2
13605992115,4cyclictest11260-21ntpq2
13605992114,4cyclictest10231-21munin-run2
1360599211,3cyclictest26919-1kworker/2:2H2
1360599211,18cyclictest28542-21munin-run2
1360599211,17cyclictest10023-1kworker/2:1H2
1360199212,17cyclictest28667-21cron1
13601992114,4cyclictest29347-21df1
13599992116,3cyclictest9316-21strings0
13611992015,3cyclictest30903-21aten2_r9power_v3
13611992015,3cyclictest29797-21ntp_offset3
13611992015,3cyclictest24013-21rt-features3
13611992015,3cyclictest13052-21df3
13611992014,4cyclictest30779-21cut3
13611992014,4cyclictest23160-21diskstats3
13611992014,4cyclictest18024-21users3
13611992014,3cyclictest30542-21turbostat.cron3
13611992013,4cyclictest29450-21cat3
13611992013,4cyclictest27848-21cat3
13611992013,4cyclictest23723-21cat3
13611992013,4cyclictest13099-21cat3
13611992013,4cyclictest11039-21cat3
1361199201,17cyclictest4261-21apt3
1360599203,1cyclictest0-21swapper/22
1360599202,16cyclictest6350-21latency_hist2
1360599202,16cyclictest20415-21latency_hist2
1360599202,16cyclictest0-21swapper/22
13605992016,2cyclictest10333-21gdbus2
13605992014,4cyclictest4102-21latency_hist2
13605992014,4cyclictest18450-21basename2
13605992014,3cyclictest0-21swapper/22
13605992013,4cyclictest10333-21gdbus2
1360599201,17cyclictest26640-1kworker/2:0H2
1360599201,17cyclictest14304-21perf2
1360599201,17cyclictest0-21swapper/22
1360599201,15cyclictest1375-21cut2
1360599201,15cyclictest1375-21cut2
1360599200,3cyclictest26575-1kworker/2:2H2
13601992017,2cyclictest0-21swapper/11
13601992016,2cyclictest18725-21/usr/sbin/munin1
13601992015,3cyclictest16770-21aten2_r9power_v1
13601992015,3cyclictest10508-21/usr/sbin/munin1
13601992014,4cyclictest32642-21wc1
13601992014,4cyclictest311-21cut1
13601992014,4cyclictest2802-21cat1
13599992015,3cyclictest665-21runrttasks0
13599992015,3cyclictest29149-21sh0
13599992015,3cyclictest26782-21aten2_r9power_e0
13599992015,3cyclictest2298-21/usr/sbin/munin0
13599992015,3cyclictest12864-21/usr/sbin/munin0
13599992014,4cyclictest7377-21sh0
13599992014,4cyclictest18392-21cron0
13599992014,4cyclictest11731-21sed0
13599992012,5cyclictest20855-21hostname0
1359999200,5cyclictest0-21swapper/00
105472200,2sleep00-21swapper/00
13611991916,2cyclictest0-21swapper/33
13611991915,2cyclictest21604-21pmu-power3
13611991914,3cyclictest9291-21memory3
13611991914,3cyclictest7540-21timerwakeupswit3
13611991914,3cyclictest7540-21timerwakeupswit3
13611991914,3cyclictest528-21apt-get3
13611991914,3cyclictest528-21apt-get3
13611991914,3cyclictest32717-21find3
13611991914,3cyclictest29357-21expr3
13611991914,3cyclictest27396-21ls3
13611991914,3cyclictest25631-21pmu-power3
13611991914,3cyclictest20944-21cut3
13611991914,3cyclictest1610-21sed3
13611991914,3cyclictest16059-21sshd3
13611991914,3cyclictest1182-21ntpq3
13611991913,4cyclictest4340-21cut3
13611991913,4cyclictest26160-21expr3
13611991913,4cyclictest23729-21rt-features3
13611991913,4cyclictest17092-21cat3
13611991913,4cyclictest16895-21cat3
13611991913,4cyclictest10806-21df3
13611991913,3cyclictest3190-21sh3
13611991913,3cyclictest31411-21sh3
13611991912,5cyclictest4891-21tail3
13611991912,4cyclictest15641-21sed3
13605991916,2cyclictest0-21swapper/22
13605991916,2cyclictest0-21swapper/22
13605991915,2cyclictest14319-21cat2
13605991914,3cyclictest6712-21expr2
13605991914,3cyclictest6712-21expr2
13605991914,3cyclictest5375-21ntp_states2
13605991914,3cyclictest30577-21ls2
13605991914,3cyclictest2842-21ls2
13605991914,3cyclictest25608-21sed2
13605991914,3cyclictest10794-21cut2
13605991913,4cyclictest19840-21pluginstate2
13605991913,4cyclictest16307-21basename2
13605991913,4cyclictest16307-21basename2
1360599191,2cyclictest26919-1kworker/2:2H2
13605991912,4cyclictest30514-21latency_hist2
1360599191,17cyclictest26522-21latency_hist2
1360599191,16cyclictest18488-21sed2
13605991910,6cyclictest505-21dbus-daemon2
1360599190,4cyclictest10290-21cut2
1360599190,16cyclictest0-21swapper/22
13601991915,2cyclictest8590-21/usr/sbin/munin1
13601991915,2cyclictest64772chrt1
13601991915,2cyclictest30719-21/usr/sbin/munin1
13601991915,2cyclictest16890-21/usr/sbin/munin1
13601991914,3cyclictest8473-21/usr/sbin/munin1
13601991914,3cyclictest441-21/usr/sbin/munin1
13601991914,3cyclictest26981-21/usr/sbin/munin1
13601991914,3cyclictest24902-21/usr/sbin/munin1
13601991914,3cyclictest24854-21grep1
13601991914,3cyclictest2484-21/usr/sbin/munin1
13601991914,3cyclictest2305-21idleruntime-cro1
13601991914,3cyclictest2305-21idleruntime-cro1
13601991914,3cyclictest22628-21grep1
13601991914,3cyclictest16528-21/usr/sbin/munin1
13601991914,3cyclictest13895-21awk1
13601991913,4cyclictest7340-21sed1
13601991913,4cyclictest5283-21cat1
13601991913,4cyclictest28970-21/usr/sbin/munin1
13601991913,4cyclictest15700-21cat1
13601991913,4cyclictest12080-21seq1
13601991913,4cyclictest11155-21cat1
13601991913,3cyclictest14763-21cat1
13601991913,3cyclictest14763-21cat1
13601991912,4cyclictest22724-21aten2.4-expect1
13599991915,2cyclictest4478-21/usr/sbin/munin0
13599991914,3cyclictest9231-21sh0
13599991914,3cyclictest7750-21ls0
13599991914,3cyclictest4580-21cpuspeed_turbos0
13599991914,3cyclictest402-21cat0
13599991914,3cyclictest28916-21/usr/sbin/munin0
13599991914,3cyclictest19054-21cut0
13599991914,3cyclictest12123-21cron0
13599991914,3cyclictest10894-21expr0
13599991913,4cyclictest22434-21idleruntime-cro0
13599991913,4cyclictest18394-21date0
13599991913,4cyclictest15780-21munin-node0
13599991913,4cyclictest14081-21seq0
13599991912,4cyclictest23251-21cat0
1361199183,4cyclictest665-21runrttasks3
13611991816,1cyclictest0-21swapper/33
13611991813,3cyclictest8367-21ls3
13611991813,3cyclictest25087-21expr3
13611991812,4cyclictest25109-21cat3
13611991812,4cyclictest13742-21/usr/sbin/munin3
1361199180,16cyclictest0-21swapper/33
13605991816,1cyclictest0-21swapper/22
13605991814,2cyclictest24463-21latency_hist2
13605991813,3cyclictest26981-21/usr/sbin/munin2
1360599181,15cyclictest26798-21cat2
1360599180,16cyclictest0-21swapper/22
13601991816,1cyclictest0-21swapper/11
13601991816,1cyclictest0-21swapper/11
13601991814,2cyclictest22965-21/usr/sbin/munin1
13601991814,2cyclictest20903-21/usr/sbin/munin1
13601991814,2cyclictest14633-21/usr/sbin/munin1
13601991814,2cyclictest10740-21aten2.4-expect1
13601991813,3cyclictest8330-21/usr/sbin/munin1
13601991813,3cyclictest7458-21cat1
13601991813,3cyclictest7458-21cat1
13601991813,3cyclictest5467-21proc_pri1
13601991813,3cyclictest2758-21awk1
13601991813,3cyclictest27126-21expr1
13601991813,3cyclictest24673-21find1
13601991813,3cyclictest23607-21ntp_states1
13601991813,3cyclictest21797-21expr1
13601991813,3cyclictest17232-21sed1
13601991813,3cyclictest14012-21find1
13601991813,3cyclictest13502-21ntpq1
13601991813,3cyclictest1028-21memory1
13601991812,4cyclictest22561-21cat1
13601991812,3cyclictest19358-21diskstats1
13601991811,5cyclictest30083-21sed1
13599991815,2cyclictest28695-21snmpd0
13599991814,2cyclictest16704-21/usr/sbin/munin0
13599991814,2cyclictest14515-21/usr/sbin/munin0
13599991814,2cyclictest14515-21/usr/sbin/munin0
13599991813,3cyclictest665-21runrttasks0
13599991813,3cyclictest32487-21perf0
13599991813,3cyclictest28821-21date0
13599991813,3cyclictest24930-21cut0
13599991813,3cyclictest20673-21/usr/sbin/munin0
13599991813,3cyclictest19390-21kernelversion0
13599991813,3cyclictest1040-21sed0
13599991813,2cyclictest23796-21ntp_states0
13599991812,4cyclictest307-21cat0
13599991812,4cyclictest26836-21cut0
13599991812,4cyclictest17689-21cat0
13599991812,4cyclictest16169-21pluginstate0
13599991812,4cyclictest1020-21basename0
1359999181,16cyclictest0-21swapper/00
13599991811,4cyclictest9-21ksoftirqd/00
13599991811,4cyclictest2702-21cpu0
13599991810,5cyclictest16179-21cut0
1359999180,7cyclictest22661-21munin-run0
1359999180,16cyclictest3305-21pmu-power0
13611991712,3cyclictest665-21runrttasks3
13611991712,3cyclictest16906-21cat3
13611991711,4cyclictest21985-21grep3
13611991711,3cyclictest15227-21cstates3
13611991710,4cyclictest10962-21entropy3
13605991713,2cyclictest6661-21aten2_r9power_e2
13605991712,3cyclictest665-21runrttasks2
13605991712,3cyclictest15230-21ls2
13601991712,3cyclictest665-21runrttasks1
13601991712,3cyclictest665-21runrttasks1
13601991712,3cyclictest4917-21tr1
13601991712,3cyclictest3056-21cat1
13601991712,3cyclictest26461-21munin-plugin-st1
13601991712,3cyclictest17093-21df1
13601991711,4cyclictest12652-21grep1
1359999179,4cyclictest12389-21apt-get0
13599991714,2cyclictest0-21swapper/00
13599991714,2cyclictest0-21swapper/00
13599991713,2cyclictest5225-21latency0
13599991712,3cyclictest665-21runrttasks0
13599991712,3cyclictest665-21runrttasks0
13599991712,3cyclictest665-21runrttasks0
13599991712,3cyclictest6406-21latency_hist0
13599991712,3cyclictest23342-21ls0
13599991710,5cyclictest28276-21unixbench_multi0
1359999170,16cyclictest0-21swapper/00
13605991611,3cyclictest22702-21aten2_r9power_c2
13601991611,3cyclictest21-21ksoftirqd/11
13601991611,3cyclictest15202-21needreboot1
13599991614,1cyclictest0-21swapper/00
13599991612,2cyclictest665-21runrttasks0
13599991611,3cyclictest665-21runrttasks0
13599991611,3cyclictest19764-21switchtime0
13599991611,3cyclictest1373-21switchtime0
1361199158,4cyclictest19683-21awk3
1360199152,7cyclictest18685-21apt-get1
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional