You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 04:31

x86 Intel Atom E3950 @1600 MHz, Linux 5.6.19-rt12 (Profile)

Latency plot of system in rack #9, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack9slot2 (updated Mon Jun 27, 2022 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
223402249180,24sleep10-21swapper/11
223622243205,25sleep30-21swapper/33
223192241204,24sleep20-21swapper/22
208552240190,17sleep00-21swapper/00
1487821340,4sleep02271099cyclictest0
85402570,4sleep08541-21unixbench-2d0
128442570,2sleep20-21swapper/22
275292450,1sleep10-21swapper/11
213712430,1sleep30-21swapper/33
190292390,1sleep00-21swapper/00
49452330,3sleep20-21swapper/22
49452330,3sleep20-21swapper/22
280162270,1sleep20-21swapper/22
2271599270,25cyclictest11613-21kworker/1:01
256552250,3sleep121-21ksoftirqd/11
2271899257,16cyclictest5143-21latency_hist2
164702250,2sleep20-21swapper/22
175682240,1sleep10-21swapper/11
72472230,2sleep20-21swapper/22
40062230,3sleep228-21ksoftirqd/22
22715992317,4cyclictest7378-21/usr/sbin/munin1
98612220,2sleep00-21swapper/00
22723992216,4cyclictest9822-21sendmail3
22723992216,3cyclictest22176-21munin-plugin-st3
2271899222,17cyclictest10483-21cat2
22718992216,4cyclictest1995-21vmstat2
2271899221,4cyclictest4331-21munin-run2
2271899221,3cyclictest14541-21munin-run2
2271899221,19cyclictest8512-1kworker/2:3H2
22715992215,4cyclictest22497-21cat1
22710992216,4cyclictest21932-21sed0
2272399211,17cyclictest24021-21awk3
2271899213,16cyclictest27639-21latency_hist2
2271899213,15cyclictest17091-21cat2
2271899213,15cyclictest17091-21cat2
22718992116,3cyclictest6316-21dump-pmu-power2
22718992116,3cyclictest20652-21cpuspeed_turbos2
22718992116,3cyclictest1408-21aten2_r9power_c2
22718992116,3cyclictest1408-21aten2_r9power_c2
22718992114,5cyclictest21565-21seq2
2271899211,3cyclictest8512-1kworker/2:3H2
2271899211,3cyclictest20312-1kworker/2:1H2
22715992116,3cyclictest19189-21/usr/sbin/munin1
22715992115,4cyclictest8789-21sed1
22715992115,4cyclictest31588-21sh1
22715992115,4cyclictest11273-21/usr/sbin/munin1
22710992116,3cyclictest5187-21sed0
22710992116,3cyclictest5187-21sed0
22710992115,4cyclictest28120-21sed0
22710992115,4cyclictest26244-21sed0
22710992115,4cyclictest24294-21sed0
22710992115,4cyclictest21600-21awk0
22723992017,2cyclictest0-21swapper/33
22723992015,3cyclictest7728-21diskstats3
22723992015,3cyclictest29620-21dump-pmu-power3
22723992015,3cyclictest23973-21threads3
22723992015,3cyclictest22657-21tr3
22723992015,3cyclictest18055-21ntp_kernel_pll_3
22723992014,4cyclictest6330-21basename3
22723992014,4cyclictest13728-21ls3
22723992014,4cyclictest11335-21sed3
22723992014,3cyclictest28690-21basename3
22723992013,4cyclictest32400-21cat3
22723992011,5cyclictest505-21dbus-daemon3
22723992011,5cyclictest27645-21idleruntime-cro3
2271899202,17cyclictest28-21ksoftirqd/22
2271899202,16cyclictest698-21latency_hist2
22718992016,2cyclictest25856-21df_inode2
22718992015,3cyclictest9809-21processes2
22718992015,3cyclictest7856-21unixbench_singl2
22718992015,3cyclictest368-21cpuspeed_turbos2
22718992015,3cyclictest24623-21chrt2
22718992015,3cyclictest20214-21ls2
22718992015,3cyclictest20078-21open_inodes2
22718992015,3cyclictest18753-21/usr/sbin/munin2
22718992015,3cyclictest18574-21if_err_enp2s02
22718992014,4cyclictest7272-21df2
22718992014,4cyclictest26734-21hostname2
22718992014,4cyclictest26024-21awk2
22718992014,4cyclictest17900-21seq2
22718992014,4cyclictest13862-21cat2
22718992014,3cyclictest13376-21ntp_offset2
22718992012,5cyclictest24079-21seq2
2271899201,17cyclictest0-21swapper/22
22715992016,2cyclictest21-21ksoftirqd/11
22715992015,3cyclictest9306-21/usr/sbin/munin1
22715992015,3cyclictest925-21/usr/sbin/munin1
22715992015,3cyclictest9203-21expr1
22715992015,3cyclictest3241-21cat1
22715992015,3cyclictest31834-21/usr/sbin/munin1
22715992015,3cyclictest31713-21ls1
22715992015,3cyclictest25080-21expr1
22715992015,3cyclictest24492-21seq1
22715992015,3cyclictest21480-21expr1
22715992015,3cyclictest21479-21missed_timers1
22715992015,3cyclictest14491-21/usr/sbin/munin1
22715992015,3cyclictest14184-21seq1
22715992014,4cyclictest18753-21/usr/sbin/munin1
22715992014,4cyclictest11228-21cat1
2271099201,5cyclictest0-21swapper/00
22710992015,3cyclictest8745-21wc0
22710992015,3cyclictest7231-21sed0
22710992015,3cyclictest25886-21date0
22710992015,3cyclictest18440-21sed0
22710992015,3cyclictest12096-21sed0
22710992014,4cyclictest639-21snmpd0
22710992014,4cyclictest30178-21seq0
22710992014,4cyclictest12382-21timerandwakeup0
22710992013,5cyclictest31731-21wc0
2272399197,10cyclictest16225-21cron3
2272399192,15cyclictest32339-21cron3
2272399192,14cyclictest4826-21sort3
2272399192,14cyclictest4826-21sort3
2272399192,14cyclictest19675-21cut3
22723991915,3cyclictest35-21ksoftirqd/33
22723991914,3cyclictest9064-21cut3
22723991914,3cyclictest8724-21sed3
22723991914,3cyclictest7090-21dump-pmu-power3
22723991914,3cyclictest2751-21sed3
22723991914,3cyclictest27120-21/usr/sbin/munin3
22723991914,3cyclictest18575-21aten2_r9power_v3
22723991913,4cyclictest7825-21cron3
22723991913,4cyclictest665-21runrttasks3
22723991913,4cyclictest28772-21awk3
22723991913,4cyclictest24458-21basename3
22723991913,4cyclictest15247-21cut3
22723991913,4cyclictest15247-21cut3
2271899192,15cyclictest31134-21sed2
22718991914,3cyclictest5977-21kernelversion2
22718991914,3cyclictest5254-21expr2
22718991914,3cyclictest30453-21ls2
22718991914,3cyclictest30317-21unixbench-2d2
22718991914,3cyclictest27830-21cat2
22718991914,3cyclictest25582-21sshd2
22718991914,3cyclictest22669-21memory2
22718991914,3cyclictest22440-21cron2
22718991914,3cyclictest22068-21pluginstate2
22718991914,3cyclictest11805-21processes2
22718991913,4cyclictest9247-21cat2
22718991913,4cyclictest9111-21sed2
22718991913,4cyclictest31770-21cat2
22718991913,4cyclictest27562-21grep2
22718991913,4cyclictest23583-21open_files2
22718991913,4cyclictest13061-21sed2
22718991913,3cyclictest1150-21cat2
2271899191,2cyclictest0-21swapper/22
2271899191,16cyclictest32349-21munin-run2
2271899191,16cyclictest31186-21procmail2
2271899190,4cyclictest18483-21munin-run2
2271899190,17cyclictest15328-21awk2
22715991915,2cyclictest22698-21/usr/sbin/munin1
22715991915,2cyclictest10829-21/usr/sbin/munin1
22715991914,3cyclictest8519-21sed1
22715991914,3cyclictest4827-21cat1
22715991914,3cyclictest4827-21cat1
22715991914,3cyclictest4680-21hostname1
22715991914,3cyclictest4059-21seq1
22715991914,3cyclictest3757-21expr1
22715991914,3cyclictest29267-21ls1
22715991914,3cyclictest28645-21/usr/sbin/munin1
22715991914,3cyclictest2355-21cron1
22715991914,3cyclictest19954-21/usr/sbin/munin1
22715991914,3cyclictest18340-21seq1
22715991914,3cyclictest17968-21cut1
22715991914,3cyclictest16384-21seq1
22715991914,3cyclictest15601-21expr1
22715991914,3cyclictest15601-21expr1
22715991914,3cyclictest12939-21sh1
22715991914,3cyclictest10063-21/usr/sbin/munin1
22715991913,4cyclictest31039-21cat1
22715991913,4cyclictest17986-21/usr/sbin/munin1
22715991913,3cyclictest5348-21acpi1
22715991913,3cyclictest24305-21cat1
22715991912,5cyclictest12158-21cut1
22715991912,4cyclictest16589-21aten2_r9power_p1
2271599190,16cyclictest8028-21cat1
2271099193,6cyclictest111rcu_preempt0
22710991915,2cyclictest6547-21/usr/sbin/munin0
22710991914,4cyclictest9-21ksoftirqd/00
22710991914,4cyclictest9-21ksoftirqd/00
22710991914,3cyclictest8578-21/usr/sbin/munin0
22710991914,3cyclictest779-21sendmail-mta0
22710991914,3cyclictest665-21runrttasks0
22710991914,3cyclictest639-21snmpd0
22710991914,3cyclictest3281-21cat0
22710991914,3cyclictest3281-21cat0
22710991914,3cyclictest29089-21grep0
22710991914,3cyclictest25441-21/usr/sbin/munin0
22710991913,4cyclictest505-21dbus-daemon0
22710991913,4cyclictest30424-21cut0
22710991913,4cyclictest27737-21sed0
22710991913,4cyclictest2366-21seq0
22710991913,4cyclictest10069-21cat0
2272399187,7cyclictest505-21dbus-daemon3
22723991816,1cyclictest0-21swapper/33
22723991815,2cyclictest0-21swapper/33
22723991814,2cyclictest21932-21/usr/sbin/munin3
22723991813,3cyclictest9840-21idleruntime3
22723991813,3cyclictest665-21runrttasks3
22723991813,3cyclictest32228-21timerwakeupswit3
22723991813,3cyclictest31788-21sed3
22723991813,3cyclictest29088-21/usr/sbin/munin3
22723991813,3cyclictest26337-21sed3
22723991813,3cyclictest18376-21wc3
22723991812,4cyclictest322-21systemd-journal3
22723991812,3cyclictest31952-21if_enp3s03
22723991812,3cyclictest14793-21aten2_r9power_c3
22723991812,3cyclictest11370-21cut3
22718991814,2cyclictest2386-21cpuspeed_turbos2
22718991814,2cyclictest10063-21/usr/sbin/munin2
22718991813,3cyclictest11072-21cat2
22718991812,4cyclictest18687-21seq2
22715991814,2cyclictest27887-21/usr/sbin/munin1
22715991814,2cyclictest26672-21/usr/sbin/munin1
22715991814,2cyclictest2152-21/usr/sbin/munin1
22715991813,3cyclictest665-21runrttasks1
22715991813,3cyclictest665-21runrttasks1
22715991813,3cyclictest653-21expr1
22715991813,3cyclictest5213-21seq1
22715991813,3cyclictest27816-21expr1
22715991813,3cyclictest26196-21cat1
22715991813,3cyclictest25968-21hostname1
22715991813,3cyclictest25635-21expr1
22715991813,3cyclictest22409-21/usr/sbin/munin1
22715991813,3cyclictest21665-21sed1
22715991813,3cyclictest18454-21seq1
22715991813,3cyclictest14410-21expr1
22715991813,3cyclictest13654-21sh1
2271099182,14cyclictest10387-21sort0
22710991815,2cyclictest8834-21/usr/sbin/munin0
22710991814,3cyclictest25916-21/usr/sbin/munin0
22710991814,2cyclictest31584-21ps0
22710991814,2cyclictest27120-21/usr/sbin/munin0
22710991814,2cyclictest20439-21/usr/sbin/munin0
22710991814,2cyclictest19189-21/usr/sbin/munin0
22710991813,3cyclictest665-21runrttasks0
22710991813,3cyclictest23674-21ps0
22710991813,3cyclictest12997-21grep0
22710991813,3cyclictest1253-21cat0
22710991812,4cyclictest9-21ksoftirqd/00
22710991812,4cyclictest18951-21munin-run0
22710991812,4cyclictest16447-21grep0
22710991812,4cyclictest1212-21sed0
22710991812,3cyclictest6219-21grep0
22710991811,4cyclictest15220-21grep0
25592170,1sleep30-21swapper/33
25592170,1sleep30-21swapper/33
2272399173,11cyclictest13108-21cpuspeed_turbos3
22723991715,1cyclictest0-21swapper/33
22723991713,3cyclictest9848-21kworker/u8:23
22723991713,2cyclictest20717-21/usr/sbin/munin3
22723991713,2cyclictest13676-21forks3
22723991712,3cyclictest665-21runrttasks3
22723991711,4cyclictest19198-21grep3
22723991710,4cyclictest322-21systemd-journal3
2272399170,16cyclictest0-21swapper/33
22715991713,2cyclictest6097-21/usr/sbin/munin1
22715991712,3cyclictest29827-21diskstats1
22710991715,1cyclictest0-21swapper/00
22710991714,1cyclictest0-21swapper/00
22710991713,2cyclictest31385-21/usr/sbin/munin0
22710991713,2cyclictest24675-21/usr/sbin/munin0
22710991712,3cyclictest21932-21/usr/sbin/munin0
22710991712,3cyclictest19732-21processes0
22710991712,3cyclictest15467-21cut0
22710991712,3cyclictest13208-21sh0
22710991711,3cyclictest13879-21irqstats0
22710991710,4cyclictest23732-21ntpq0
2272399169,4cyclictest29511-21aten2_r9power_v3
2272399168,5cyclictest1136-21gltestperf3
22723991611,3cyclictest446-21cstates3
22723991611,3cyclictest28019-21uptime3
22723991610,4cyclictest20043-21cut3
22723991610,3cyclictest3408-21basename3
2272399160,1cyclictest0-21swapper/33
2272399160,1cyclictest0-21swapper/33
2271099165,8cyclictest665-21runrttasks0
2271099162,12cyclictest17677-21grep0
22710991611,3cyclictest24801-21munin-node0
22710991611,3cyclictest12372-21cstates0
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional