You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-19 - 09:41

Intel(R) Atom(TM) CPU N455 @ 1.66GHz, Linux 5.4.34-rt21 (Profile)

Latency plot of system in rack #3, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack3slot6 (updated Wed Jan 19, 2022 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2942921360,5sleep12272899cyclictest21:05:011
21775210840,49sleep10-21swapper/119:05:321
143812940,6sleep121-21ksoftirqd/123:30:441
2112428928,48sleep00-21swapper/019:05:200
22728998628,24cyclictest488-21dbus-daemon23:40:181
295362850,17sleep12272899cyclictest22:40:391
22720998454,16cyclictest520-21gdbus22:40:190
22720998451,22cyclictest9811-21sendmail-msp23:20:210
22720998357,12cyclictest0-21swapper/019:30:200
22720998348,23cyclictest488-21dbus-daemon23:40:170
22720998250,22cyclictest23170-21sendmail-msp00:00:190
206292810,5sleep10-21swapper/123:50:351
22728997939,10cyclictest21-21ksoftirqd/121:20:341
22728997939,10cyclictest21-21ksoftirqd/121:20:331
22720997848,18cyclictest520-21gdbus19:40:200
22720997848,16cyclictest0-21swapper/023:10:200
22720997848,15cyclictest0-21swapper/019:35:220
22720997750,16cyclictest0-21swapper/020:05:210
22720997749,15cyclictest9-21ksoftirqd/022:50:220
22720997747,16cyclictest0-21swapper/000:15:200
2272899764,13cyclictest9542-21python321:40:501
2272899762,11cyclictest3931-21ntp_states23:00:451
22728997612,27cyclictest111rcu_preempt19:20:541
2272099769,11cyclictest9-21ksoftirqd/023:00:490
22720997648,16cyclictest9-21ksoftirqd/021:30:200
22720997647,18cyclictest9-21ksoftirqd/023:25:180
22720997646,23cyclictest488-21dbus-daemon19:20:210
22728997535,22cyclictest21-21ksoftirqd/100:20:301
2272899753,14cyclictest7396-21sendmail-msp20:00:431
2272899752,12cyclictest9811-21sendmail-msp23:20:221
22720997545,18cyclictest0-21swapper/022:05:520
22720997447,15cyclictest0-21swapper/021:10:210
22720997446,15cyclictest0-21swapper/022:35:170
22720997444,17cyclictest0-21swapper/000:35:210
2272899733,14cyclictest2013-1kworker/1:0H20:20:461
2272899732,9cyclictest21003-21sendmail-msp20:41:321
22720997346,15cyclictest0-21swapper/023:50:170
22720997345,17cyclictest0-21swapper/022:15:200
22720997345,17cyclictest0-21swapper/000:05:240
22720997345,15cyclictest0-21swapper/020:10:190
22728997237,20cyclictest21-21ksoftirqd/122:00:241
22728997230,24cyclictest21-21ksoftirqd/119:41:161
22720997244,15cyclictest0-21swapper/022:25:190
22720997243,14cyclictest0-21swapper/022:10:180
22720997242,18cyclictest0-21swapper/021:50:220
22720997242,15cyclictest0-21swapper/019:55:180
22720997210,13cyclictest8580-21sendmail-msp21:41:130
2272899712,14cyclictest2052-1kworker/1:2H22:20:191
2272099717,11cyclictest9-21ksoftirqd/020:20:230
22720997145,14cyclictest0-21swapper/023:45:190
22720997143,15cyclictest0-21swapper/000:25:190
22720997140,17cyclictest9-21ksoftirqd/022:45:170
2272899702,11cyclictest15345-21latency_hist22:00:021
2272099706,10cyclictest9-21ksoftirqd/021:20:410
2272099706,10cyclictest9-21ksoftirqd/021:20:410
22720997043,16cyclictest0-21swapper/021:35:180
22720997038,18cyclictest0-21swapper/000:30:180
2272899693,11cyclictest23343-21apt-get00:00:221
2272099695,19cyclictest9-21ksoftirqd/022:20:040
22720996943,15cyclictest9-21ksoftirqd/020:35:210
22720996941,15cyclictest0-21swapper/020:55:170
2272099693,13cyclictest7781-21gltestperf20:00:320
22720996910,10cyclictest9-21ksoftirqd/000:20:200
2272899683,12cyclictest2013-1kworker/1:0H20:20:031
22720996836,22cyclictest488-21dbus-daemon21:00:190
22720996812,8cyclictest111rcu_preempt22:00:580
22728996737,12cyclictest21-21ksoftirqd/119:30:001
2272899672,13cyclictest23139-21apt00:00:001
2272899672,13cyclictest23139-21apt00:00:001
2272099677,12cyclictest21003-21sendmail-msp20:41:390
22720996739,16cyclictest0-21swapper/020:15:190
2272899663,11cyclictest28968-21latency_hist22:40:021
22728996538,11cyclictest21-21ksoftirqd/119:20:011
2272899653,13cyclictest28150-21awk00:15:001
2272899653,13cyclictest28150-21awk00:15:001
22720996538,15cyclictest0-21swapper/020:25:180
2272899649,16cyclictest111rcu_preempt23:30:011
2272899643,13cyclictest5702-21munin-plugin-st19:55:001
2272899643,13cyclictest5702-21munin-plugin-st19:54:591
22728996415,15cyclictest21-21ksoftirqd/121:40:011
22720996436,19cyclictest9-21ksoftirqd/023:15:200
22720996432,18cyclictest9-21ksoftirqd/019:25:190
22720996415,17cyclictest0-21swapper/023:09:550
22728996335,11cyclictest21-21ksoftirqd/119:40:011
2272899633,14cyclictest12368-21latency_hist20:15:001
2272899633,13cyclictest30991-21turbostat21:10:011
2272899633,13cyclictest25504-21munin-run22:30:011
2272899633,12cyclictest8126-21awk23:15:011
2272899633,12cyclictest27678-21sendmail-msp21:00:001
22728996310,14cyclictest111rcu_preempt00:20:011
2272899628,14cyclictest111rcu_preempt23:00:031
2272899624,10cyclictest9787-21apt23:19:591
22728996232,10cyclictest21-21ksoftirqd/100:40:011
2272899623,13cyclictest2646-21apt00:35:001
22720996235,15cyclictest0-21swapper/023:55:170
22720996235,15cyclictest0-21swapper/023:55:170
22720996234,17cyclictest5793-21systemd-journal21:20:010
22720996234,16cyclictest9-21ksoftirqd/019:50:200
22720996234,16cyclictest9-21ksoftirqd/019:50:190
22720996233,21cyclictest22711-21cyclictest19:10:220
22720996216,33cyclictest0-21swapper/020:30:200
204392620,5sleep1201rcuc/122:15:201
186462620,16sleep12272899cyclictest20:30:511
2272899619,33cyclictest0-21swapper/100:10:001
22728996128,13cyclictest21-21ksoftirqd/120:55:011
2272899612,11cyclictest21-21ksoftirqd/122:54:591
22720996133,20cyclictest22711-21cyclictest21:05:190
22720996131,21cyclictest22711-21cyclictest20:50:170
38672600,8sleep10-21swapper/121:25:221
2272899609,8cyclictest0-21swapper/121:10:351
2272899605,10cyclictest32164-21cron22:50:011
2272899604,11cyclictest24322-21idleruntime-cro20:50:001
2272899604,10cyclictest6888-21apt21:35:011
2272899603,14cyclictest31469-21idleruntime-cro19:35:001
2272899603,13cyclictest19055-21apt-get20:35:061
2272899603,13cyclictest13560-21munin-run21:55:001
2272899603,11cyclictest1876-21munin-run21:20:001
2272899602,13cyclictest17915-21python322:06:151
22720996032,16cyclictest0-21swapper/000:10:210
22720996032,16cyclictest0-21swapper/000:10:200
2272899593,12cyclictest4043-21apt19:50:001
2272899593,12cyclictest21-21ksoftirqd/120:00:021
2272899593,12cyclictest17381-21munin-run20:30:011
2272899592,12cyclictest10711-21ls20:10:001
22720995928,19cyclictest488-21dbus-daemon19:20:010
22720995914,33cyclictest0-21swapper/020:45:180
22720995910,9cyclictest111rcu_preempt22:30:320
22728995833,10cyclictest21-21ksoftirqd/121:50:011
2272899582,12cyclictest16476-21plymouth23:40:011
2272099589,9cyclictest111rcu_preempt23:00:030
22720995829,16cyclictest0-21swapper/021:45:160
22720995817,31cyclictest0-21swapper/023:35:190
2272899573,13cyclictest24361-21munin-node19:15:021
2272899573,12cyclictest6178-21kworker/1:123:09:541
2272899573,12cyclictest27146-21anacron22:34:551
22720995731,18cyclictest22711-21cyclictest23:30:190
2272899569,32cyclictest18949-21fw_packets22:10:261
22728995629,11cyclictest21-21ksoftirqd/123:46:321
22720995634,14cyclictest5793-21systemd-journal22:00:000
22720995533,13cyclictest9-21ksoftirqd/019:45:190
22720995510,9cyclictest111rcu_preempt21:25:230
2272899544,10cyclictest31850-21df_inode00:25:221
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional