You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-17 - 14:22

Intel(R) Atom(TM) Processor E3950 @ 1.60GHz, Linux 4.19.182-rt74 (Profile)

Latency plot of system in rack #d, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot6.osadl.org (updated Sun Jun 06, 2021 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17366993420,9cyclictest13227-21cat19:35:043
17363993227,3cyclictest9-21ksoftirqd/021:55:010
1736399321,15cyclictest0-21swapper/022:30:000
17366993124,4cyclictest8262-21grep20:57:003
17363993111,17cyclictest8561-21sudo21:00:010
1736399310,16cyclictest352-21systemd-journal19:55:010
17363993016,8cyclictest21416-21timerwakeupswit20:40:180
17366992917,7cyclictest32178-21if_enp2s0.720:20:193
17366992915,8cyclictest22022-21wc21:10:173
17366992915,8cyclictest22022-21wc21:10:173
17365992916,7cyclictest20187-21cat00:35:172
1736399298,18cyclictest629-21dbus-daemon00:25:010
1736399283,12cyclictest12439-21max_latency_gen19:35:010
17363992823,3cyclictest9-21ksoftirqd/000:30:010
17363992823,3cyclictest9-21ksoftirqd/000:00:010
1736399281,13cyclictest8067-21idleruntime-cro20:30:010
17366992715,6cyclictest5755-21cat21:25:173
1736399278,17cyclictest3017-21munin-run20:55:010
17366992612,11cyclictest11258-21if_enp3s021:30:163
1736399264,19cyclictest18651-21awk20:10:010
17363992620,4cyclictest9-21ksoftirqd/000:10:010
17363992615,9cyclictest9-21ksoftirqd/022:50:000
17366992518,4cyclictest26203-21sh22:15:003
1736399256,17cyclictest28321-21munin-run00:15:010
17363992522,2cyclictest9-21ksoftirqd/021:35:010
17363992520,3cyclictest9-21ksoftirqd/020:00:010
1736699247,10cyclictest26041-21timerwakeupswit19:45:183
17366992417,4cyclictest22449-21cat19:11:003
17366992416,4cyclictest31940-21sendmail22:20:013
17366992414,5cyclictest0-21swapper/321:55:053
17366992412,8cyclictest1962-21cat00:20:043
1736599241,20cyclictest11833-21grep00:00:012
17364992419,3cyclictest25-21ksoftirqd/123:15:011
17363992421,2cyclictest9-21ksoftirqd/021:05:000
1736699238,9cyclictest3173-21grep23:50:173
17366992316,4cyclictest9228-21grep19:30:173
17366992312,7cyclictest3280-21apt-get23:20:213
17366992312,7cyclictest15474-21apt-get21:35:063
17366992312,6cyclictest9564-21if_enp2s0.600:25:183
17365992317,4cyclictest30240-21sed21:15:382
17365992317,3cyclictest33-21ksoftirqd/221:30:012
17365992316,4cyclictest5033-21grep20:25:182
17364992317,3cyclictest16390-21cat20:35:201
17364992311,7cyclictest4337-21head19:55:171
1736399234,17cyclictest22367-21munin-run23:40:010
1736399231,3cyclictest121rcu_preempt21:20:190
17363992313,6cyclictest71150irq/126-enp1s0-21:15:190
17363992313,6cyclictest24971-21bc19:15:160
1736399230,3cyclictest19756-21sudo21:10:010
1736699229,8cyclictest16154-21apt-get22:05:063
1736699229,8cyclictest16154-21apt-get22:05:063
1736699222,13cyclictest7926-21sed22:55:183
1736699222,10cyclictest18060-21tr22:35:173
17366992215,4cyclictest25725-21awk19:15:193
17366992215,4cyclictest1877-21head21:50:233
1736699221,18cyclictest14662-21tc_qdisc_statis00:30:163
17366992210,8cyclictest31486-21apt-get21:20:073
1736599222,13cyclictest26174-21cut20:15:172
17365992217,3cyclictest33-21ksoftirqd/219:45:002
17365992215,4cyclictest24820-21cat23:10:192
17365992215,4cyclictest15427-21mailstats21:05:112
17365992212,6cyclictest16506-21cpuspeed_turbos21:35:152
1736499222,14cyclictest20418-21unixbench_multi19:40:171
17364992217,3cyclictest25-21ksoftirqd/100:40:001
17364992214,6cyclictest5357-21ls23:50:291
17363992217,3cyclictest9-21ksoftirqd/023:20:010
17363992217,3cyclictest9-21ksoftirqd/023:10:010
17363992217,3cyclictest24051-21ethtool_rx_queu20:15:010
17363992216,3cyclictest1237-21if_enp2s0.523:50:030
1736399221,18cyclictest29518-21find20:20:010
1736699213,11cyclictest27511-21timerwakeupswit21:15:183
1736699213,11cyclictest23949-21if_enp2s0.622:40:183
1736699212,6cyclictest14606-21ifqueues_enp2s021:05:043
1736699212,13cyclictest9140-21find23:55:203
1736699212,11cyclictest4290-21ethtool_rx_queu19:55:163
1736699212,11cyclictest24246-21unixbench_singl23:10:173
17366992116,3cyclictest41-21ksoftirqd/322:25:013
17366992116,3cyclictest41-21ksoftirqd/322:25:013
17366992114,5cyclictest572-21awk22:50:043
17366992114,4cyclictest1335-21max_latency_gen00:20:013
1736599213,15cyclictest26192-21cron22:15:002
17365992114,4cyclictest5324-21/usr/sbin/munin22:25:162
17365992114,4cyclictest30739-21needreboot19:50:152
17365992114,4cyclictest17050-21pluginstate23:05:052
1736499211,6cyclictest13607-21cpu_utilization23:00:191
17364992116,3cyclictest25-21ksoftirqd/121:00:011
17364992116,3cyclictest16735-21find21:05:191
17364992115,4cyclictest20517-21cpu20:10:151
17364992115,3cyclictest1232-21if_enp2s0.620:50:281
17364992114,5cyclictest2047-21grep00:20:041
17364992114,4cyclictest4296-21sed21:25:051
17364992111,6cyclictest352-21systemd-journal00:00:021
1736399216,9cyclictest10287-21unixbench_multi20:30:170
17363992116,3cyclictest9-21ksoftirqd/023:35:010
17363992116,3cyclictest9-21ksoftirqd/023:15:010
17363992116,3cyclictest9-21ksoftirqd/020:40:010
17363992116,3cyclictest9-21ksoftirqd/000:20:000
17363992115,4cyclictest9-21ksoftirqd/023:00:010
17363992115,4cyclictest15219-21fschecks_count00:00:250
17363992115,4cyclictest11328-21sh23:29:010
17363992115,3cyclictest27032-21max_latency_gen22:45:010
17363992110,7cyclictest20267-21apt-get19:40:170
1736699202,11cyclictest28145-21/usr/sbin/munin23:45:183
17366992015,3cyclictest27757-21timedrift00:10:423
17366992015,3cyclictest24908-21apt-get20:15:073
17366992015,3cyclictest20519-21missed_timers20:40:143
17366992015,3cyclictest15331-21ifqueues_enp2s023:30:303
17366992014,4cyclictest9338-21kernelversion22:25:273
17366992014,4cyclictest3491-21open62541latenc19:25:153
17366992014,4cyclictest32513-21grep20:50:193
17366992014,4cyclictest22366-21sh23:40:013
17366992014,4cyclictest16252-21http20:35:193
17366992014,4cyclictest12809-21tr20:30:333
17366992014,3cyclictest28524-21cpu_utilization21:45:223
17366992013,4cyclictest8874-21mailstats20:00:113
17366992013,4cyclictest30694-21df_inode23:15:343
17366992013,4cyclictest18567-21sudo20:10:003
17366992013,4cyclictest12291-21awk21:00:253
17366992012,5cyclictest629-21dbus-daemon21:45:003
17365992015,3cyclictest8136-21wc19:30:112
17365992015,3cyclictest7780-21http_loadtime21:55:222
17365992015,3cyclictest32196-21sendmail_mailst23:15:472
17365992015,3cyclictest19609-21irqstats22:05:412
17365992015,3cyclictest19609-21irqstats22:05:412
17365992015,3cyclictest13815-21/usr/sbin/munin20:35:172
17365992014,4cyclictest5198-21date22:25:012
17365992014,4cyclictest5198-21date22:25:012
17365992014,4cyclictest29316-21if_enp2s0.100:15:102
17365992014,4cyclictest26363-21cpu20:45:152
17365992014,4cyclictest24867-21users21:10:382
17365992014,4cyclictest24867-21users21:10:382
17365992014,4cyclictest23303-21cat21:40:282
17365992013,5cyclictest18080-21sed23:35:122
17365992013,5cyclictest16878-21max_latency_gen23:35:012
17365992013,4cyclictest5548-21sh22:55:002
17365992013,4cyclictest31885-21apt-get20:20:192
17365992012,5cyclictest5227-21grep20:55:182
17365992012,5cyclictest26264-21cut19:45:192
1736499209,6cyclictest26349-21idleruntime20:15:171
17364992015,3cyclictest29365-21cpu_utilization22:15:231
17364992015,3cyclictest25435-21/usr/sbin/munin21:15:111
17364992015,3cyclictest23258-21cstates21:40:281
17364992015,3cyclictest10395-21max_latency_gen22:29:011
17364992014,4cyclictest26408-21idleruntime19:15:251
17364992014,4cyclictest13755-21df22:30:251
17364992014,4cyclictest12385-21ifqueues_enp2s021:00:261
17364992014,3cyclictest22101-21http21:10:181
17364992014,3cyclictest22101-21http21:10:181
17364992014,3cyclictest13773-21threads23:30:181
17364992013,5cyclictest25240-21cat22:40:261
17364992013,4cyclictest30080-21munin-run20:50:011
17364992013,4cyclictest26223-21max_latency_gen22:15:001
17364992013,4cyclictest18788-21cat22:35:191
17364992013,4cyclictest12978-21cat19:35:021
17364992013,4cyclictest11549-21cat20:00:441
17364992010,6cyclictest71250irq/127-enp1s0-21:20:191
1736499200,18cyclictest1319-21cron00:20:001
1736399202,16cyclictest18238-21sendmail-msp19:40:010
1736399202,16cyclictest1621-21grep20:20:310
17363992015,3cyclictest9-21ksoftirqd/022:00:000
17363992014,4cyclictest1798-21/usr/sbin/munin19:25:160
17363992014,4cyclictest17360-21awk21:35:180
17363992014,3cyclictest9-21ksoftirqd/022:20:010
17363992013,5cyclictest16109-21cat22:35:010
17363992013,4cyclictest9-21ksoftirqd/022:25:010
17363992013,4cyclictest9-21ksoftirqd/022:25:010
17363992012,5cyclictest20815-21cpuspeed_turbos00:35:190
1736699198,7cyclictest629-21dbus-daemon19:20:073
1736699198,7cyclictest14553-21apt-get00:00:213
17366991914,3cyclictest3119-21cpu_utilization20:25:043
17366991913,4cyclictest23560-21sed23:40:123
17366991913,4cyclictest21097-21phc2sys23:05:373
17366991913,4cyclictest18448-21cat00:35:043
17366991913,4cyclictest11476-21forks22:30:083
17366991913,3cyclictest8311-21/usr/sbin/munin23:25:213
17366991913,3cyclictest20047-21cut00:05:203
17366991911,5cyclictest15269-21latency_hist22:05:013
17366991910,5cyclictest31365-21unixbench_singl19:50:183
1736599196,3cyclictest10697-21tail21:00:172
17365991914,3cyclictest985-21cpu_utilization00:29:402
17365991914,3cyclictest6261-21/usr/sbin/munin23:25:162
17365991914,3cyclictest23117-21/usr/sbin/munin00:10:192
17365991914,3cyclictest1918-21/usr/sbin/munin00:20:122
17365991914,3cyclictest16164-21unixbench_singl23:00:372
17365991914,3cyclictest1437-21ifqueues_enp2s020:50:302
17365991914,3cyclictest13714-21ps20:05:042
17365991914,3cyclictest12898-21sudo20:01:002
17365991913,4cyclictest424-21cut21:50:152
17365991913,4cyclictest32455-21grep19:20:282
17365991913,4cyclictest31045-21cat23:45:212
17365991913,4cyclictest27187-21sed22:15:052
17365991913,4cyclictest18464-21sed00:05:112
17365991913,3cyclictest7005-21sudo19:30:002
17365991913,3cyclictest31801-21mailstats22:45:342
17365991912,5cyclictest28760-1kworker/2:1H22:55:082
17365991912,5cyclictest17910-21apt22:35:162
17365991912,4cyclictest24061-21tc_qdisc_statis20:15:012
17365991912,4cyclictest21909-21cut22:40:042
17365991912,4cyclictest13467-21tr00:00:152
17365991911,5cyclictest25864-21/usr/sbin/munin21:45:202
17365991911,5cyclictest1-21systemd20:00:012
17364991914,3cyclictest30510-21if_enp2s0.319:50:131
17364991914,3cyclictest30455-21timerandwakeup20:20:071
17364991914,3cyclictest25-21ksoftirqd/119:10:011
17364991914,3cyclictest24675-21/usr/sbin/munin23:40:281
17364991914,3cyclictest16652-21awk22:05:121
17364991914,3cyclictest16652-21awk22:05:121
17364991914,3cyclictest13410-21/usr/sbin/munin20:05:061
17364991914,3cyclictest10124-21/usr/sbin/munin22:00:161
17364991913,4cyclictest28877-21if_enp2s023:15:131
17364991913,4cyclictest26239-21sh00:10:291
17364991913,4cyclictest20238-21cut23:35:281
17364991913,4cyclictest17156-21sed19:35:371
17364991913,4cyclictest12879-21userThreadLaten00:00:081
17364991913,3cyclictest5764-21munin-plugin-st22:55:011
17364991913,3cyclictest5588-21cpu_utilization20:25:311
17364991913,3cyclictest29239-21unixbench_multi22:45:171
17364991913,3cyclictest18437-21open_inodes23:05:161
17364991912,5cyclictest11391-21sed00:25:311
17364991912,4cyclictest7420-21cat23:25:131
17364991912,4cyclictest14576-21cron21:35:011
17364991911,5cyclictest18006-21cron00:35:011
17364991910,6cyclictest30311-21cut19:20:161
17364991910,6cyclictest17469-21cpuspeed_turbos21:35:181
1736399197,10cyclictest9-21ksoftirqd/023:45:140
17363991915,3cyclictest9-21ksoftirqd/020:50:010
17363991914,3cyclictest9-21ksoftirqd/021:45:000
17363991914,3cyclictest4093-21grep21:25:040
17363991914,3cyclictest28990-21http_loadtime21:45:260
17363991914,3cyclictest19630-21mpstat21:10:230
17363991914,3cyclictest19630-21mpstat21:10:230
17363991914,3cyclictest16254-21/usr/sbin/munin22:35:180
17363991914,3cyclictest0-21swapper/000:30:310
17363991913,3cyclictest26616-21apt-config22:15:010
17363991912,4cyclictest2588-21cat22:50:190
17363991910,6cyclictest0-21swapper/023:00:180
1736699182,13cyclictest27040-21tune2fs20:45:193
17366991812,4cyclictest28297-21sh22:45:123
17366991811,4cyclictest13264-21grep23:00:183
1736599187,9cyclictest11828-21cpu_utilization21:30:182
17365991813,3cyclictest3144-21sed23:50:172
17365991813,3cyclictest24280-21sudo20:42:002
17365991813,3cyclictest15269-21latency_hist22:05:012
17365991813,3cyclictest1487-21forks23:20:072
17365991813,3cyclictest12986-21/usr/sbin/munin19:35:152
17365991812,4cyclictest8515-21grep20:30:042
17365991812,4cyclictest31050-21grep21:20:042
17365991812,4cyclictest23928-21cat19:15:072
17365991812,4cyclictest12758-21ls22:30:182
17365991812,3cyclictest17191-21pmu-power00:30:302
17365991811,5cyclictest17862-21sed19:10:052
17364991813,3cyclictest726-21/usr/sbin/munin21:50:231
17364991813,3cyclictest7106-21/usr/sbin/munin22:55:351
17364991813,3cyclictest3794-21which19:25:171
17364991813,3cyclictest28915-21sudo19:47:011
17364991813,3cyclictest1362-21apt-get23:20:061
17364991813,3cyclictest13056-21timerwakeupswit20:30:361
17364991812,4cyclictest32413-21cut22:20:041
17364991812,4cyclictest32413-21cut22:20:041
17364991812,4cyclictest19754-21sed20:40:051
17364991812,3cyclictest32415-21bc23:45:311
17364991812,3cyclictest17657-21/usr/sbin/munin00:05:111
17364991811,4cyclictest20094-21awk19:10:281
1736399189,5cyclictest352-21systemd-journal19:50:000
17363991813,3cyclictest5044-21awk23:20:330
17363991813,3cyclictest17281-21open62541latenc22:05:150
17363991813,3cyclictest17281-21open62541latenc22:05:150
17363991812,4cyclictest30765-21basename19:20:180
17363991812,4cyclictest11913-21cat22:00:170
17363991812,3cyclictest9062-21kernelversion20:00:120
17363991811,4cyclictest27040-21grep23:40:360
17363991811,4cyclictest20333-21grep19:10:300
1736699179,5cyclictest20421-21grep19:40:173
17366991712,3cyclictest19555-21cat20:10:073
17365991710,4cyclictest23267-21ls23:40:072
1736499173,9cyclictest26428-21apt-get21:45:061
17364991712,3cyclictest6815-21awk21:55:181
17363991712,3cyclictest9-21ksoftirqd/019:10:010
1736699148,4cyclictest17421-21cat19:10:023
1736599126,3cyclictest321ktimersoftd/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional