You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-02 - 17:55

x86 Intel Atom E3950 @1600 MHz, Linux 5.6.19-rt12 (Profile)

Latency plot of system in rack #9, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot2 (updated Fri Dec 02, 2022 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
316972320181,17sleep20-21swapper/22
9672312274,25sleep30-21swapper/33
138599286282,2cyclictest0-21swapper/11
139399277272,3cyclictest0-21swapper/33
9112256183,25sleep00-21swapper/00
139099240236,2cyclictest0-21swapper/22
10872233196,24sleep10-21swapper/11
2660421560,4sleep30-21swapper/33
138099141136,3cyclictest10718-21strings0
227132780,1sleep00-21swapper/00
47242440,2sleep20-21swapper/22
270772420,2sleep10-21swapper/11
1393993615,4cyclictest8797-21ntp_states3
1393993614,3cyclictest6732-21unixbench_multi3
1393993414,3cyclictest32078-21open_files3
1393993412,3cyclictest505-21dbus-daemon3
139399339,4cyclictest322-21systemd-journal3
1393993212,3cyclictest9903-21cat3
1393993212,2cyclictest3273-21latency_hist3
1393993211,4cyclictest17779-21cat3
139399321,12cyclictest20565-21sed3
1390992720,5cyclictest4512-21sendmail-mta2
157042250,5sleep10-21swapper/11
1393992517,4cyclictest5238-21idleruntime-cro3
28302240,5sleep32831-21sed3
139099242,9cyclictest20056-21cat2
1390992421,2cyclictest271rcuc/22
1390992420,2cyclictest271rcuc/22
1390992411,2cyclictest16348-21latency_hist2
139099240,6cyclictest928-21awk2
1390992318,3cyclictest5250-21grep2
139099231,11cyclictest28-21ksoftirqd/22
1393992216,4cyclictest5729-21grep3
1393992215,4cyclictest29469-21sh3
139099223,17cyclictest7840-21latency_hist2
139099222,18cyclictest23771-21cat2
139099222,17cyclictest0-21swapper/22
1390992219,2cyclictest271rcuc/22
139099221,7cyclictest28-21ksoftirqd/22
139099221,6cyclictest28-21ksoftirqd/22
1390992216,4cyclictest9807-21sh2
139099221,19cyclictest559-21gdbus2
139099220,6cyclictest24086-21taskset2
139099220,6cyclictest16979-21taskset2
138599221,18cyclictest14567-21fw_packets1
1380992211,9cyclictest31519-21cstates0
139399215,13cyclictest4455-21cron3
139099212,6cyclictest28204-21cron2
139099211,7cyclictest28-21ksoftirqd/22
139099211,6cyclictest28-21ksoftirqd/22
139099211,6cyclictest28-21ksoftirqd/22
1390992116,3cyclictest26626-21open_files2
139099211,5cyclictest0-21swapper/22
1390992115,4cyclictest30931-21seq2
1390992115,4cyclictest19620-21seq2
1390992114,4cyclictest8649-21date2
139099211,17cyclictest0-21swapper/22
1390992110,10cyclictest261832sleep22
139099210,7cyclictest28-21ksoftirqd/22
139099210,7cyclictest28-21ksoftirqd/22
139099210,7cyclictest28-21ksoftirqd/22
139099210,7cyclictest28-21ksoftirqd/22
139099210,7cyclictest28-21ksoftirqd/22
139099210,6cyclictest19400-21taskset2
1385992115,4cyclictest23291-21cat1
1385992114,5cyclictest28939-21grep1
138099218,11cyclictest32196-21perf0
1380992117,2cyclictest9-21ksoftirqd/00
1380992116,3cyclictest1223-21/usr/sbin/munin0
1380992115,4cyclictest744-21/usr/sbin/munin0
1393992015,3cyclictest647-21latency3
1393992015,3cyclictest4744-21kernelversion3
1393992015,3cyclictest30419-21sed3
1393992015,3cyclictest26149-21open_inodes3
1393992015,3cyclictest25540-21swap3
1393992015,3cyclictest2067-21unixbench_singl3
1393992015,3cyclictest18463-21kernelversion3
1393992015,3cyclictest16238-21cpu3
1393992015,3cyclictest13959-21cut3
1393992014,4cyclictest32390-21sed3
1393992013,4cyclictest31294-21grep3
1393992012,5cyclictest1250-21idleruntime-cro3
1393992012,5cyclictest1250-21idleruntime-cro3
139099202,5cyclictest271rcuc/22
1390992017,2cyclictest271rcuc/22
1390992017,2cyclictest271rcuc/22
139099201,6cyclictest28-21ksoftirqd/22
139099201,6cyclictest28-21ksoftirqd/22
139099201,6cyclictest28-21ksoftirqd/22
1390992015,3cyclictest634-21timerandwakeup2
1390992015,3cyclictest634-21timerandwakeup2
1390992015,3cyclictest31630-21gltestperf2
1390992015,3cyclictest24560-21ls2
1390992015,3cyclictest22996-21timerwakeupswit2
1390992015,3cyclictest20609-21aten2_r9power_e2
1390992014,4cyclictest20066-21find2
1390992014,4cyclictest15697-21cut2
139099201,3cyclictest24021-1kworker/2:2H2
139099201,3cyclictest24021-1kworker/2:2H2
139099200,7cyclictest28-21ksoftirqd/22
139099200,6cyclictest8995-21taskset2
139099200,6cyclictest6846-21taskset2
139099200,6cyclictest5007-21taskset2
139099200,6cyclictest28-21ksoftirqd/22
139099200,6cyclictest17282-21taskset2
139099200,6cyclictest16102-21taskset2
1385992015,3cyclictest7835-21cron1
1385992015,3cyclictest4022-21aten2_r9power_e1
1385992015,3cyclictest3454-21munin-node1
1385992015,3cyclictest31936-21/usr/sbin/munin1
1385992015,3cyclictest31936-21/usr/sbin/munin1
1385992015,3cyclictest31411-21seq1
1385992015,3cyclictest11388-21/usr/sbin/munin1
1385992014,4cyclictest9459-21needreboot1
1385992014,4cyclictest5917-21cat1
1385992014,4cyclictest27606-21seq1
1385992014,4cyclictest20910-21ntp_offset1
1385992014,4cyclictest16369-21gltestperf1
1385992013,4cyclictest18283-21grep1
1380992015,3cyclictest6608-21cut0
1380992015,3cyclictest5296-21sed0
1380992015,3cyclictest18323-21latency_hist0
1380992014,4cyclictest8094-21find0
1380992014,4cyclictest7701-21idleruntime-cro0
1380992014,4cyclictest27452-21cat0
1380992014,4cyclictest22106-21needreboot0
1380992014,4cyclictest12297-21cat0
1380992014,4cyclictest12297-21cat0
1380992013,5cyclictest10406-21idleruntime-cro0
1393991915,2cyclictest9927-21unixbench_multi3
1393991914,3cyclictest7788-21sed3
1393991914,3cyclictest2401-21entropy3
1393991914,3cyclictest23890-21users3
1393991914,3cyclictest22848-21sed3
1393991914,3cyclictest21409-21uptime3
1393991914,3cyclictest19439-21uptime3
1393991914,3cyclictest14687-21aten2_r9power_p3
1393991913,4cyclictest18441-21sed3
1393991913,4cyclictest13144-21cat3
1393991913,3cyclictest322-21systemd-journal3
1393991912,4cyclictest29725-21sendmail3
1393991912,4cyclictest25503-21sh3
139399191,15cyclictest6636-21cut3
139399190,17cyclictest322-21systemd-journal3
139099191,6cyclictest0-21swapper/22
1390991914,3cyclictest6382-21forks2
1390991914,3cyclictest26071-21grep2
139099190,7cyclictest28-21ksoftirqd/22
139099190,6cyclictest624-21taskset2
139099190,6cyclictest2900-21taskset2
139099190,6cyclictest28-21ksoftirqd/22
1385991915,2cyclictest29223-21/usr/sbin/munin1
1385991915,2cyclictest1542-21/usr/sbin/munin1
1385991915,2cyclictest10635-21/usr/sbin/munin1
1385991914,3cyclictest9333-21seq1
1385991914,3cyclictest7652-21ls1
1385991914,3cyclictest7003-21expr1
1385991914,3cyclictest32419-21/usr/sbin/munin1
1385991914,3cyclictest30472-21cut1
1385991914,3cyclictest23823-21ls1
1385991914,3cyclictest22857-21cat1
1385991914,3cyclictest22533-21/usr/sbin/munin1
1385991914,3cyclictest2188-21cat1
1385991914,3cyclictest19312-21/usr/sbin/munin1
1385991914,3cyclictest19085-21expr1
1385991914,3cyclictest18886-21needreboot1
1385991914,3cyclictest16113-21/usr/sbin/munin1
1385991914,3cyclictest14256-21sh1
1385991914,3cyclictest14256-21sh1
1385991914,3cyclictest1223-21/usr/sbin/munin1
1385991913,4cyclictest4544-21date1
1385991913,4cyclictest29835-21cut1
1385991913,4cyclictest29707-21/usr/sbin/munin1
1385991913,4cyclictest26320-21cat1
1385991913,4cyclictest16597-21/usr/sbin/munin1
1385991913,4cyclictest15343-21grep1
1385991912,5cyclictest22303-21idleruntime-cro1
1385991912,4cyclictest505-21dbus-daemon1
1380991915,3cyclictest9-21ksoftirqd/00
1380991914,3cyclictest7071-21expr0
1380991914,3cyclictest5103-21expr0
1380991914,3cyclictest31654-21sed0
1380991914,3cyclictest26314-21cat0
1380991914,3cyclictest25494-21latency_hist0
1380991914,3cyclictest19756-21cut0
1380991914,3cyclictest18566-21/usr/sbin/munin0
1380991913,4cyclictest772-21sed0
1380991913,4cyclictest772-21sed0
1380991913,4cyclictest29888-21cat0
1380991912,4cyclictest15705-21cstates0
1380991912,4cyclictest15454-21grep0
1380991911,5cyclictest26275-21cpuspeed_turbos0
270652180,2sleep30-21swapper/33
1393991813,3cyclictest505-21dbus-daemon3
1393991813,3cyclictest30856-21df_inode3
1393991813,3cyclictest28205-21cron3
1393991813,3cyclictest24776-21sed3
1393991813,3cyclictest17425-21ntp_kernel_err3
1393991813,3cyclictest14623-21sed3
1393991813,3cyclictest13383-21find3
1393991813,3cyclictest13383-21find3
1393991813,3cyclictest10212-21needreboot3
1393991812,4cyclictest20313-21perf3
1393991812,3cyclictest10576-21head3
1393991811,4cyclictest8262-21cut3
1393991810,5cyclictest20077-21cpuspeed_turbos3
1393991810,4cyclictest505-21dbus-daemon3
139399180,16cyclictest0-21swapper/33
139399180,16cyclictest0-21swapper/33
139099182,5cyclictest28-21ksoftirqd/22
139099181,5cyclictest0-21swapper/22
1385991814,3cyclictest1978-21/usr/sbin/munin1
138599181,3cyclictest0-21swapper/11
1385991813,3cyclictest8448-21expr1
1385991813,3cyclictest31191-21/usr/sbin/munin1
1385991813,3cyclictest28480-21/usr/sbin/munin1
1385991813,3cyclictest24987-21expr1
1385991813,3cyclictest17718-21expr1
1385991813,3cyclictest15123-21grep1
1385991813,3cyclictest10476-21df1
1385991812,4cyclictest4903-21cat1
1385991812,4cyclictest13342-21cat1
1385991810,5cyclictest4216-21missed_timers1
1380991816,1cyclictest0-21swapper/00
1380991815,2cyclictest0-21swapper/00
1380991815,2cyclictest0-21swapper/00
1380991813,3cyclictest665-21runrttasks0
1380991813,3cyclictest4531-21sed0
1380991813,3cyclictest32597-21cut0
1380991813,3cyclictest26249-21ls0
1380991813,3cyclictest21110-21basename0
1380991813,3cyclictest12635-21/usr/sbin/munin0
1380991813,3cyclictest11789-21expr0
1380991812,4cyclictest322-21systemd-journal0
138099181,16cyclictest0-21swapper/00
1380991811,4cyclictest30230-21cpuspeed_turbos0
1380991811,4cyclictest2973-21grep0
1380991811,4cyclictest0-21swapper/00
1380991810,4cyclictest322-21systemd-journal0
1393991715,1cyclictest0-21swapper/33
1393991714,2cyclictest0-21swapper/33
1393991712,3cyclictest27555-21cstates3
1393991712,3cyclictest16089-21load3
1393991712,3cyclictest13220-21grep3
1393991712,3cyclictest13220-21grep3
1385991715,1cyclictest0-21swapper/11
1385991714,2cyclictest25258-21/usr/sbin/munin1
1385991712,3cyclictest665-21runrttasks1
1385991712,3cyclictest5856-21expr1
1385991712,3cyclictest24052-21aten2_r9power_c1
1385991712,3cyclictest13187-21cat1
1385991712,3cyclictest11901-21/usr/sbin/munin1
1385991712,3cyclictest11901-21/usr/sbin/munin1
138599170,16cyclictest0-21swapper/11
138099178,7cyclictest10218-21needreboot0
138099178,7cyclictest0-21swapper/00
138099175,10cyclictest9-21ksoftirqd/00
1380991715,1cyclictest0-21swapper/00
1380991713,2cyclictest22057-21/usr/sbin/munin0
1380991713,2cyclictest10635-21/usr/sbin/munin0
1380991712,3cyclictest8779-21ntpq0
1380991712,3cyclictest665-21runrttasks0
1380991712,3cyclictest3710-21cron0
1380991712,3cyclictest29252-21seq0
1380991712,3cyclictest29048-21if_enp3s00
1380991712,3cyclictest24060-21cat0
1380991712,3cyclictest19826-21seq0
1380991712,3cyclictest13980-21aten2_r9power_v0
1380991712,3cyclictest12031-21/usr/sbin/munin0
138099171,15cyclictest0-21swapper/00
1380991711,4cyclictest665-21runrttasks0
1380991711,4cyclictest3233-21latency_hist0
1380991711,3cyclictest322-21systemd-journal0
1380991710,4cyclictest10154-21grep0
1393991611,3cyclictest12662-21aten2_r9power_c3
1385991614,1cyclictest0-21swapper/11
1385991611,3cyclictest381-21cat1
1380991614,1cyclictest0-21swapper/00
1380991610,4cyclictest18570-21gltestperf0
1393991510,3cyclictest30469-21missed_timers3
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional