You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-07-29 - 18:34

Intel(R) Atom(TM) CPU N455 @ 1.66GHz, Linux 5.4.34-rt21 (Profile)

Latency plot of system in rack #3, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot6 (updated Thu Jul 29, 2021 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2915821540,16sleep0202199cyclictest11:40:300
1800521060,20sleep1202599cyclictest07:55:241
202599928,25cyclictest11216-21apt-get07:35:171
166129021,32sleep00-21swapper/007:07:300
38028530,41sleep10-21swapper/107:05:131
202199856,22cyclictest19178-21apt-get09:35:160
2025998210,21cyclictest7452-21apt-get07:25:151
202199818,21cyclictest24183-21apt-get09:50:160
202199816,23cyclictest9876-21apt-get12:20:190
202199766,20cyclictest7882-21apt-get07:25:160
70222680,4sleep10-21swapper/110:35:181
205092640,21sleep1202599cyclictest11:15:181
205092640,21sleep1202599cyclictest11:15:171
45312620,6sleep0101rcuc/007:15:250
301552610,22sleep1202599cyclictest10:05:431
2025996144,12cyclictest21-21ksoftirqd/109:20:011
202599607,45cyclictest3418-21gltestperf12:00:251
2021995928,11cyclictest9-21ksoftirqd/010:05:050
2021995924,13cyclictest9-21ksoftirqd/012:05:030
2021995923,13cyclictest25945-21netstat11:30:250
60162580,20sleep1202599cyclictest07:20:181
2021995827,24cyclictest8557-21systemd-journal11:01:040
2021995823,14cyclictest11036-21ntpq10:45:420
2021995823,13cyclictest13598-21netstat07:40:360
2021995722,13cyclictest473-21df10:15:180
2025995627,24cyclictest488-21dbus-daemon12:00:021
2025995611,16cyclictest0-21swapper/111:10:141
2021995630,20cyclictest29166-21latency_hist10:05:010
2025995539,11cyclictest21-21ksoftirqd/109:30:001
2025995526,24cyclictest9724-21ls12:20:011
2021995524,25cyclictest8557-21systemd-journal07:23:280
2021995523,11cyclictest9-21ksoftirqd/011:11:380
151232553,44sleep115227-21awk09:20:381
2025995431,10cyclictest0-21swapper/110:45:251
2025995425,25cyclictest488-21dbus-daemon08:20:001
2025995424,25cyclictest488-21dbus-daemon11:40:011
2025995424,25cyclictest1820-21latency_hist10:20:021
2025995413,32cyclictest0-21swapper/112:35:161
202199549,39cyclictest0-21swapper/012:13:430
2021995423,13cyclictest9-21ksoftirqd/008:23:210
2021995422,14cyclictest5962-21netstat10:30:330
2021995421,15cyclictest13891-21open_inodes12:30:370
2021995421,13cyclictest9-21ksoftirqd/007:58:300
2021995418,10cyclictest11216-21apt-get07:35:180
2025995327,21cyclictest0-21swapper/108:40:021
2025995311,13cyclictest0-21swapper/110:40:241
202199535,21cyclictest7011-21apt-get10:35:150
2021995331,13cyclictest0-21swapper/008:35:150
2021995330,13cyclictest0-21swapper/007:30:160
2025995237,10cyclictest0-21swapper/111:33:031
2025995234,14cyclictest8557-21systemd-journal09:45:381
2025995230,14cyclictest5801-21sendmail-msp07:20:001
2025995211,15cyclictest0-21swapper/111:40:251
2021995232,14cyclictest24473-21cron08:14:590
2025995133,11cyclictest28692-21sshd10:01:391
2025995128,13cyclictest488-21dbus-daemon09:00:021
2025995128,11cyclictest0-21swapper/108:20:131
2025995123,15cyclictest0-21swapper/108:50:141
2025995120,26cyclictest15244-21latency_hist11:00:011
2021995134,11cyclictest8266-21awk12:15:170
2021995133,12cyclictest7513-21seq09:00:170
2021995132,12cyclictest8557-21systemd-journal09:32:550
2021995126,11cyclictest0-21swapper/009:40:180
2025995033,11cyclictest3682-21sshd08:49:391
2025995024,16cyclictest0-21swapper/109:05:161
2025995010,26cyclictest9554-21apt-get07:30:161
202199506,20cyclictest14872-21apt-get12:35:160
2021995033,11cyclictest8557-21systemd-journal08:35:000
2021995032,12cyclictest8557-21systemd-journal11:19:160
2021995032,12cyclictest8557-21systemd-journal11:19:150
2021995029,15cyclictest8557-21systemd-journal08:43:460
202199502,42cyclictest3501-21cron10:25:010
202599498,29cyclictest13007-21apt-get07:40:191
202599498,24cyclictest17027-21apt-get11:05:171
2025994934,11cyclictest21487-21tr09:40:311
2025994923,13cyclictest0-21swapper/108:10:201
2021994934,11cyclictest13490-21unixbench_multi09:15:370
202199493,40cyclictest8557-21systemd-journal09:45:450
2021994933,10cyclictest9-21ksoftirqd/008:20:010
2021994932,12cyclictest8557-21systemd-journal10:22:530
2021994931,12cyclictest8557-21systemd-journal08:45:410
2021994930,13cyclictest9333-21expr09:05:230
2021994928,15cyclictest488-21dbus-daemon07:55:000
2021994920,11cyclictest12150-21cstates10:50:240
2021994919,12cyclictest9-21ksoftirqd/008:53:550
2021994918,25cyclictest32030-21/usr/sbin/munin10:10:430
202599488,23cyclictest29662-21apt-get08:30:171
2025994832,9cyclictest20091-21tr08:00:271
2025994831,12cyclictest8557-21systemd-journal10:00:011
2025994831,12cyclictest8557-21systemd-journal08:25:281
2025994811,32cyclictest0-21swapper/109:00:251
202199489,12cyclictest0-21swapper/011:50:220
202199485,28cyclictest0-21swapper/009:21:210
202199484,35cyclictest1690-21df11:55:190
2021994829,13cyclictest15559-21timerwakeupswit07:45:380
2021994827,15cyclictest20488-21cat08:00:380
2021994826,16cyclictest6143-21cut08:55:440
202599477,23cyclictest6524-21apt-get12:10:141
202599476,36cyclictest19825-21tr09:35:351
202599475,35cyclictest10655-21python312:20:371
2025994730,12cyclictest5168-21sh10:30:011
2025994728,14cyclictest488-21dbus-daemon10:30:221
2025994712,10cyclictest0-21swapper/110:25:011
2025994711,29cyclictest0-21swapper/109:30:191
202199479,33cyclictest0-21swapper/009:25:220
202199476,22cyclictest0-21swapper/008:05:360
202199475,22cyclictest0-21swapper/009:56:000
2021994729,11cyclictest25052-21sshd11:28:380
2021994729,11cyclictest25052-21sshd11:28:370
2021994727,14cyclictest28231-21seq11:37:040
2021994727,14cyclictest17477-21sed11:05:270
2021994726,11cyclictest488-21dbus-daemon11:00:010
2021994723,12cyclictest0-21swapper/010:40:210
2021994717,24cyclictest12431-21timerandwakeup12:25:470
2021994712,10cyclictest0-21swapper/009:13:520
202599465,35cyclictest12183-21snmpd07:53:051
202599463,37cyclictest15502-21expr11:00:201
2025994629,11cyclictest32551-21grep11:50:281
2025994622,12cyclictest0-21swapper/111:20:191
2025994622,11cyclictest0-21swapper/112:25:261
2021994628,13cyclictest8557-21systemd-journal12:00:310
2021994628,12cyclictest22678-21sshd11:20:380
202599457,21cyclictest30629-21apt-get11:45:161
202599455,19cyclictest25115-21sort09:50:431
2025994531,10cyclictest24430-21processes11:25:321
2025994531,10cyclictest24430-21processes11:25:311
2025994530,10cyclictest21-21ksoftirqd/112:35:001
2025994529,11cyclictest15523-21sort07:45:391
2025994527,13cyclictest22870-21cat08:10:011
202599452,39cyclictest23266-21kworker/1:010:10:111
2025994522,13cyclictest520-21gdbus10:55:001
2021994527,13cyclictest32198-21sed11:50:000
2021994526,14cyclictest9-21ksoftirqd/007:10:160
202599443,35cyclictest12183-21snmpd08:42:331
202599439,25cyclictest4876-21apt-get12:05:221
202599437,23cyclictest10832-21apt-get09:10:171
2021994328,10cyclictest39450irq/29-enp3s008:25:310
202599429,23cyclictest2113-21apt-get07:10:151
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional