You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-22 - 03:24

Intel(R) Atom(TM) CPU N455 @ 1.66GHz, Linux 5.4.34-rt21 (Profile)

Latency plot of system in rack #3, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot6 (updated Fri Oct 22, 2021 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20423210538,53sleep00-21swapper/019:08:380
63452880,23sleep02068099cyclictest23:15:000
2031228532,41sleep10-21swapper/119:07:251
20688997924,49cyclictest488-21dbus-daemon23:20:221
28142700,7sleep02816-21sendmail00:40:000
28142700,7sleep02816-21sendmail00:40:000
292912670,7sleep09-21ksoftirqd/022:45:240
20680996631,27cyclictest488-21dbus-daemon00:20:200
20680996341,12cyclictest488-21dbus-daemon00:00:100
174492635,10sleep10-21swapper/122:10:191
20688996227,12cyclictest21-21ksoftirqd/100:20:071
20688996126,13cyclictest21-21ksoftirqd/122:40:221
20680996123,31cyclictest0-21swapper/023:05:160
20688996030,25cyclictest488-21dbus-daemon21:20:031
20680996038,15cyclictest488-21dbus-daemon21:40:200
20680996038,15cyclictest488-21dbus-daemon21:40:190
20680996037,12cyclictest14964-21cpuspeed_turbos23:40:210
190942600,9sleep00-21swapper/022:15:190
60222595,9sleep10-21swapper/123:11:391
20688995929,24cyclictest13715-21cat22:00:041
20688995926,27cyclictest21391-40mandb00:00:101
20688995923,13cyclictest21-21ksoftirqd/121:40:141
20688995923,13cyclictest21-21ksoftirqd/121:40:131
20688995829,21cyclictest0-21swapper/123:00:051
20688995826,26cyclictest488-21dbus-daemon22:00:101
20688995826,26cyclictest488-21dbus-daemon21:00:011
20688995826,25cyclictest520-21gdbus20:20:201
20680995821,17cyclictest488-21dbus-daemon20:20:200
80642570,7sleep09-21ksoftirqd/023:20:010
20688995727,23cyclictest12183-21snmpd19:20:191
20688995725,25cyclictest488-21dbus-daemon19:40:201
20688995723,28cyclictest488-21dbus-daemon20:40:201
20680995732,14cyclictest27139-21sendmail-msp22:40:170
20680995727,12cyclictest488-21dbus-daemon19:20:130
277222562,46sleep127515-21/usr/sbin/munin21:05:251
20688995627,22cyclictest0-21swapper/123:40:101
20688995626,25cyclictest21391-40mandb00:00:071
20688995626,25cyclictest21391-40mandb00:00:061
20688995625,25cyclictest488-21dbus-daemon20:00:171
2068899562,50cyclictest29737-21kworker/1:200:35:121
2068899562,50cyclictest29737-21kworker/1:200:35:121
20688995622,12cyclictest0-21swapper/119:45:181
20688995620,14cyclictest21-21ksoftirqd/122:20:191
2068099564,44cyclictest488-21dbus-daemon20:40:110
20680995626,24cyclictest5793-21systemd-journal19:58:370
20680995626,24cyclictest5793-21systemd-journal19:58:360
20680995610,15cyclictest0-21swapper/000:10:170
303332555,9sleep10-21swapper/100:25:251
303332555,9sleep10-21swapper/100:25:241
256662552,10sleep125758-21cat22:35:171
2068899552,49cyclictest16765-21kworker/1:219:15:021
20688995523,13cyclictest520-21gdbus21:00:231
20680995536,14cyclictest488-21dbus-daemon23:20:190
20680995533,15cyclictest488-21dbus-daemon19:40:130
20680995532,15cyclictest520-21gdbus21:00:240
20680995529,13cyclictest0-21swapper/019:30:220
20680995528,11cyclictest9-21ksoftirqd/023:00:010
191652553,45sleep019264-21users23:50:360
323652543,43sleep032363-21sendmail-msp21:20:010
283352543,11sleep0488-21dbus-daemon00:20:010
20688995433,14cyclictest8081-21sed23:20:021
20688995420,13cyclictest21-21ksoftirqd/121:20:071
20680995434,14cyclictest488-21dbus-daemon22:20:170
20680995434,14cyclictest488-21dbus-daemon20:00:050
20680995430,15cyclictest1368-21sendmail-msp23:00:230
125352543,10sleep012608-21uname21:55:400
36752533,11sleep03247-21apt-get21:30:160
296262533,42sleep129660-21proc_pri22:45:341
20688995332,15cyclictest488-21dbus-daemon19:30:011
20688995324,9cyclictest157782chrt22:05:381
20688995324,12cyclictest5793-21systemd-journal19:58:321
20688995324,12cyclictest5793-21systemd-journal19:58:311
2068099534,43cyclictest488-21dbus-daemon23:40:020
20680995333,14cyclictest488-21dbus-daemon21:20:120
20680995331,12cyclictest9-21ksoftirqd/023:25:160
20680995329,13cyclictest0-21swapper/020:10:120
20680995324,24cyclictest488-21dbus-daemon22:00:070
20680995323,26cyclictest5793-21systemd-journal23:33:280
2068099526,9cyclictest0-21swapper/019:50:170
20680995234,12cyclictest488-21dbus-daemon22:40:020
20680995230,12cyclictest0-21swapper/021:10:190
162292520,5sleep00-21swapper/020:30:270
20688995133,12cyclictest22780-21if_enp2s020:50:261
20688995132,13cyclictest488-21dbus-daemon00:20:021
20688995132,12cyclictest7689-21seq20:05:301
2068899512,43cyclictest32078-21apt-get00:30:201
20688995124,21cyclictest0-21swapper/123:00:001
2068899512,40cyclictest3188-21apt-get23:05:151
20680995135,11cyclictest30492-21tr19:35:300
20680995132,14cyclictest5793-21systemd-journal20:57:320
20680995124,14cyclictest0-21swapper/000:25:180
20680995124,14cyclictest0-21swapper/000:25:180
167002510,6sleep112183-21snmpd20:30:371
9492503,40sleep0955-21df_inode19:45:220
20680995035,9cyclictest22928-21ntpq22:25:280
2068099503,41cyclictest11028-21ntp_states21:50:480
20680995033,12cyclictest488-21dbus-daemon19:19:590
20680995033,10cyclictest15755-21cpuspeed_turbos22:05:390
20680995030,11cyclictest488-21dbus-daemon21:40:030
20680995022,14cyclictest0-21swapper/020:25:140
20680995011,31cyclictest0-21swapper/022:10:150
20688994933,12cyclictest13012-21turbostat23:35:001
20688994931,13cyclictest488-21dbus-daemon21:40:021
20688994928,14cyclictest22494-21sed19:15:051
20688994923,16cyclictest20474-21latency_hist22:20:011
20688994922,21cyclictest0-21swapper/120:20:031
2068099497,37cyclictest0-21swapper/021:25:230
20680994932,12cyclictest9-21ksoftirqd/000:00:010
20680994932,12cyclictest9-21ksoftirqd/000:00:010
20680994928,16cyclictest11256-21/usr/sbin/munin20:15:280
20680994928,11cyclictest0-21swapper/020:05:220
20680994926,12cyclictest9-21ksoftirqd/020:35:290
241532487,18sleep121-21ksoftirqd/100:05:371
2068899487,23cyclictest8679-21apt-get21:45:151
20688994831,11cyclictest18454-21grep20:35:521
20688994830,12cyclictest31661-21grep22:50:421
20688994830,12cyclictest13316-21wc23:35:201
20688994827,16cyclictest30482-21awk19:35:291
20688994827,14cyclictest10016-21sed20:10:361
2068099485,38cyclictest0-21swapper/021:05:150
20680994827,11cyclictest0-21swapper/000:30:160
20680994812,30cyclictest0-21swapper/020:45:170
20688994732,9cyclictest21-21ksoftirqd/121:30:151
20688994727,14cyclictest28443-21head19:30:291
20688994726,15cyclictest12183-21snmpd23:46:571
20688994721,21cyclictest25451-21awk22:35:001
2068099476,36cyclictest0-21swapper/022:52:350
20680994732,10cyclictest21306-21needreboot19:10:270
20680994730,12cyclictest24094-21netstat00:05:330
177282474,6sleep00-21swapper/023:46:530
20688994628,13cyclictest12183-21snmpd20:45:411
20688994625,15cyclictest26667-21cut00:15:011
20680994630,11cyclictest5793-21systemd-journal22:31:540
20680994617,9cyclictest9163-21/usr/sbin/munin21:45:250
2068899457,31cyclictest0-21swapper/119:50:151
20688994530,9cyclictest22811-21missed_timers22:25:271
20688994526,13cyclictest18178-21seq23:50:031
20688994521,19cyclictest10623-21gltestperf21:50:391
20688994520,19cyclictest504-21in:imuxsock21:12:011
20688994515,23cyclictest0-21swapper/120:25:321
2068099458,31cyclictest0-21swapper/019:25:220
20680994530,10cyclictest23413-21unixbench-2d20:50:390
2068899444,18cyclictest3248-21munin-run21:30:011
20688994427,12cyclictest12183-21snmpd23:25:421
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:123
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
none17:13:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional