You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-29 - 04:20

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack2slot0.osadl.org (updated Wed Sep 29, 2021 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
309512780,0sleep30-21swapper/321:35:453
13802510,0sleep30-21swapper/322:25:073
255452480,0sleep00-21swapper/023:03:210
299592470,0sleep30-21swapper/320:05:243
212592460,0sleep10-21swapper/121:49:451
561123928,7sleep00-21swapper/019:09:160
83962360,0sleep20-21swapper/222:35:002
561223523,7sleep10-21swapper/119:09:171
54952309,7sleep20-21swapper/219:07:502
536023018,7sleep30-21swapper/319:06:073
218672200,0sleep1231rcuc/100:15:071
191382180,0sleep00-21swapper/023:34:090
212172170,0sleep20-21swapper/221:57:392
5782991615,0cyclictest3643-21lspci00:02:211
5782991514,0cyclictest31808-21lspci22:48:481
5782991514,0cyclictest20532-21lspci23:58:361
19242150,0sleep20-21swapper/222:25:152
579399130,11cyclictest16412-21bash21:24:243
578999120,1cyclictest0-21swapper/222:57:272
5782991211,0cyclictest17783-21lspci21:16:301
5782991210,1cyclictest0-21swapper/121:58:081
5793991110,0cyclictest0-21swapper/321:53:143
579399110,10cyclictest0-21swapper/322:11:143
579399110,10cyclictest0-21swapper/321:45:343
5789991110,0cyclictest0-21swapper/221:33:372
578999110,10cyclictest0-21swapper/223:48:072
578999110,10cyclictest0-21swapper/223:20:072
578999110,10cyclictest0-21swapper/222:51:062
578999110,10cyclictest0-21swapper/222:45:172
578999110,10cyclictest0-21swapper/200:33:472
578299119,1cyclictest26909-21sshd23:11:381
5782991110,0cyclictest0-21swapper/121:13:081
578299110,10cyclictest29099-21lspci22:55:581
578299110,10cyclictest28459-21bash00:08:381
578299110,10cyclictest25-21ksoftirqd/100:13:381
578299110,10cyclictest0-21swapper/122:43:471
578299110,10cyclictest0-21swapper/122:19:471
578299110,10cyclictest0-21swapper/122:09:281
578299110,10cyclictest0-21swapper/119:55:181
577699119,1cyclictest0-21swapper/022:06:170
577699113,4cyclictest17806-21munin-run23:50:000
577699110,10cyclictest0-21swapper/022:25:170
577699110,10cyclictest0-21swapper/021:26:470
579399109,0cyclictest0-21swapper/322:23:343
579399109,0cyclictest0-21swapper/300:38:153
579399109,0cyclictest0-21swapper/300:05:453
579399100,9cyclictest0-21swapper/320:02:533
579399100,9cyclictest0-21swapper/319:25:543
579399100,0cyclictest0-21swapper/323:10:243
579399100,0cyclictest0-21swapper/322:06:343
579399100,0cyclictest0-21swapper/321:28:343
579399100,0cyclictest0-21swapper/300:15:453
578999109,0cyclictest0-21swapper/223:41:072
578999109,0cyclictest0-21swapper/221:54:372
578999109,0cyclictest0-21swapper/221:39:172
578999109,0cyclictest0-21swapper/200:26:272
578999109,0cyclictest0-21swapper/200:22:472
578999109,0cyclictest0-21swapper/200:09:072
578999100,0cyclictest0-21swapper/222:39:272
578999100,0cyclictest0-21swapper/221:45:572
578299109,0cyclictest0-21swapper/123:35:581
578299109,0cyclictest0-21swapper/123:03:181
578299109,0cyclictest0-21swapper/100:35:381
578299100,9cyclictest0-21swapper/122:34:281
578299100,9cyclictest0-21swapper/121:25:481
577699109,0cyclictest0-21swapper/022:59:470
577699109,0cyclictest0-21swapper/019:10:270
577699109,0cyclictest0-21swapper/000:36:270
577699108,1cyclictest614-21dbus-daemon21:37:070
577699108,1cyclictest1-21systemd00:06:170
5776991010,0cyclictest24389-21kworker/0:121:48:570
577699100,0cyclictest0-21swapper/021:51:570
577699100,0cyclictest0-21swapper/021:23:070
160222100,0sleep30-21swapper/322:52:513
57939998,0cyclictest0-21swapper/323:45:353
57939998,0cyclictest0-21swapper/323:24:343
57939998,0cyclictest0-21swapper/323:19:243
57939998,0cyclictest0-21swapper/323:02:343
57939998,0cyclictest0-21swapper/322:42:343
57939998,0cyclictest0-21swapper/300:28:353
57939998,0cyclictest0-21swapper/300:01:053
57939997,1cyclictest0-21swapper/322:37:143
57939990,0cyclictest0-21swapper/319:57:143
57939990,0cyclictest0-21swapper/319:38:133
57899998,0cyclictest0-21swapper/223:25:172
57899998,0cyclictest0-21swapper/223:17:472
57899998,0cyclictest0-21swapper/223:05:472
57899998,0cyclictest0-21swapper/221:14:472
57899998,0cyclictest0-21swapper/200:39:472
57829998,0cyclictest0-21swapper/123:27:081
57829997,1cyclictest30523-21sshd21:35:381
57769998,0cyclictest0-21swapper/022:41:170
57769998,0cyclictest0-21swapper/022:24:470
57769998,0cyclictest0-21swapper/021:43:460
57769998,0cyclictest0-21swapper/020:15:160
57769996,2cyclictest1-21systemd00:34:370
57769991,4cyclictest10373-21perf19:20:000
57769990,0cyclictest0-21swapper/023:07:060
57769990,0cyclictest0-21swapper/020:59:570
57939987,0cyclictest0-21swapper/323:53:053
57939987,0cyclictest0-21swapper/323:40:553
57939987,0cyclictest0-21swapper/323:38:353
57939987,0cyclictest0-21swapper/323:27:243
57939987,0cyclictest0-21swapper/322:55:143
57939987,0cyclictest0-21swapper/322:15:543
57939987,0cyclictest0-21swapper/321:33:143
57939987,0cyclictest0-21swapper/300:10:453
57939985,2cyclictest598-21avahi-daemon21:13:393
57939980,7cyclictest0-21swapper/319:52:443
57899987,0cyclictest0-21swapper/223:53:172
57899987,0cyclictest0-21swapper/221:28:262
57899987,0cyclictest0-21swapper/221:24:172
57899986,1cyclictest598-21avahi-daemon00:02:002
57899985,2cyclictest598-21avahi-daemon22:44:062
57899985,2cyclictest598-21avahi-daemon22:18:312
57899985,2cyclictest598-21avahi-daemon22:00:372
57899985,2cyclictest598-21avahi-daemon21:41:332
57899985,2cyclictest598-21avahi-daemon00:12:222
57829988,0cyclictest970-21snmpd00:26:371
57829987,0cyclictest32296-21kworker/1:122:50:081
57829987,0cyclictest0-21swapper/123:47:381
57829987,0cyclictest0-21swapper/122:38:281
57829987,0cyclictest0-21swapper/121:41:481
57829985,2cyclictest598-21avahi-daemon23:30:501
57829985,2cyclictest598-21avahi-daemon22:11:321
57829981,1cyclictest131rcu_sched21:22:001
57769987,0cyclictest0-21swapper/023:15:470
57769987,0cyclictest0-21swapper/022:10:570
57769987,0cyclictest0-21swapper/021:31:270
57769987,0cyclictest0-21swapper/021:10:570
57769986,2cyclictest1312-21kworker/0:323:20:290
57769985,2cyclictest598-21avahi-daemon23:57:280
57769985,2cyclictest598-21avahi-daemon23:52:560
57769985,2cyclictest598-21avahi-daemon23:35:430
57769985,2cyclictest598-21avahi-daemon00:20:210
57769985,2cyclictest598-21avahi-daemon00:01:500
57769983,4cyclictest14605-21crond20:50:000
57769982,3cyclictest12402-21perf19:25:010
57939976,0cyclictest0-21swapper/321:09:043
57939975,1cyclictest598-21avahi-daemon23:56:063
57939975,1cyclictest598-21avahi-daemon23:32:103
57939975,1cyclictest598-21avahi-daemon23:06:293
57939975,1cyclictest598-21avahi-daemon22:31:293
57939975,1cyclictest598-21avahi-daemon22:00:003
57939975,1cyclictest598-21avahi-daemon00:33:463
57939974,2cyclictest598-21avahi-daemon22:46:313
57939974,2cyclictest598-21avahi-daemon21:40:153
57939974,2cyclictest598-21avahi-daemon21:18:253
57939974,2cyclictest598-21avahi-daemon00:22:143
57899975,1cyclictest6311-21systemd-cgroups23:30:402
57899975,1cyclictest598-21avahi-daemon23:55:512
57899975,1cyclictest598-21avahi-daemon23:35:092
57899975,1cyclictest598-21avahi-daemon23:11:512
57899975,1cyclictest598-21avahi-daemon22:20:142
57899975,1cyclictest598-21avahi-daemon22:11:262
57899975,1cyclictest598-21avahi-daemon22:05:022
57899975,1cyclictest598-21avahi-daemon21:15:202
57899974,2cyclictest598-21avahi-daemon23:03:322
57899974,2cyclictest598-21avahi-daemon00:19:472
57829975,1cyclictest598-21avahi-daemon23:20:081
57829975,1cyclictest598-21avahi-daemon22:25:191
57829975,1cyclictest598-21avahi-daemon22:20:031
57829975,1cyclictest598-21avahi-daemon00:30:401
57829975,1cyclictest598-21avahi-daemon00:20:431
57829975,1cyclictest25-21ksoftirqd/121:50:141
57829974,2cyclictest598-21avahi-daemon21:30:361
57829973,3cyclictest598-21avahi-daemon23:16:011
57829973,1cyclictest0-21swapper/122:00:381
57829971,1cyclictest633-21polkitd23:05:291
57829970,2cyclictest25-21ksoftirqd/123:50:201
57829970,1cyclictest131rcu_sched23:40:161
57829970,0cyclictest131rcu_sched19:25:011
57769976,0cyclictest0-21swapper/022:47:070
57769976,0cyclictest0-21swapper/019:40:170
57769975,1cyclictest633-21polkitd22:03:080
57769975,1cyclictest598-21avahi-daemon23:43:390
57769975,1cyclictest598-21avahi-daemon23:26:170
57769975,1cyclictest598-21avahi-daemon22:53:580
57769975,1cyclictest598-21avahi-daemon22:35:200
57769975,1cyclictest598-21avahi-daemon21:16:510
57769974,2cyclictest614-21dbus-daemon00:17:250
57769974,2cyclictest598-21avahi-daemon23:10:370
57769974,2cyclictest598-21avahi-daemon22:32:230
57769974,2cyclictest598-21avahi-daemon22:18:330
57769974,2cyclictest598-21avahi-daemon21:55:440
57769974,2cyclictest598-21avahi-daemon00:25:580
57769974,2cyclictest598-21avahi-daemon00:11:160
57939965,0cyclictest0-21swapper/319:19:543
57939963,2cyclictest970-21snmpd20:59:003
57939963,2cyclictest614-21dbus-daemon21:55:173
57939960,5cyclictest0-21swapper/320:24:043
57939960,5cyclictest0-21swapper/320:19:243
57939960,5cyclictest0-21swapper/319:49:343
57939960,5cyclictest0-21swapper/319:31:243
57939960,0cyclictest0-21swapper/320:30:443
57899964,1cyclictest643-21gdbus19:55:002
57899963,2cyclictest970-21snmpd19:27:352
57899963,2cyclictest970-21snmpd19:24:302
57899960,5cyclictest0-21swapper/221:08:572
57829964,1cyclictest25-21ksoftirqd/121:10:011
57829963,2cyclictest970-21snmpd20:50:441
57829963,2cyclictest970-21snmpd19:12:121
57829962,1cyclictest0-21swapper/120:33:531
57769964,1cyclictest614-21dbus-daemon21:05:010
57769964,1cyclictest614-21dbus-daemon20:39:550
57769963,2cyclictest970-21snmpd21:08:420
57769963,2cyclictest970-21snmpd19:55:010
57769963,2cyclictest970-21snmpd19:32:480
57769960,5cyclictest0-21swapper/020:43:260
57939954,0cyclictest0-21swapper/319:40:043
57939953,1cyclictest970-21snmpd21:02:263
57939953,1cyclictest970-21snmpd20:53:063
57939953,1cyclictest970-21snmpd20:45:393
57939953,1cyclictest970-21snmpd20:43:093
57939953,1cyclictest970-21snmpd20:35:283
57939953,1cyclictest970-21snmpd20:25:293
57939953,1cyclictest970-21snmpd20:14:203
57939953,1cyclictest970-21snmpd19:20:453
57939953,1cyclictest970-21snmpd19:10:023
57899954,0cyclictest0-21swapper/219:36:262
57899953,1cyclictest970-21snmpd20:55:542
57899953,1cyclictest970-21snmpd20:50:122
57899953,1cyclictest970-21snmpd20:40:332
57899953,1cyclictest970-21snmpd20:15:262
57899953,1cyclictest970-21snmpd20:07:232
57899953,1cyclictest970-21snmpd20:00:062
57899953,1cyclictest970-21snmpd19:50:072
57899953,1cyclictest970-21snmpd19:48:192
57899953,1cyclictest970-21snmpd19:34:132
57899953,1cyclictest7216-21cat20:30:222
57899953,1cyclictest633-21polkitd20:29:542
57899953,1cyclictest6114-21cpuspeed_turbos19:10:162
57899953,1cyclictest598-21avahi-daemon20:35:382
57899953,1cyclictest31893-21ps20:10:222
57899953,1cyclictest14609-21sadc20:50:002
57899952,2cyclictest970-21snmpd20:20:082
57899952,2cyclictest20431-21taskset21:01:352
57829953,1cyclictest970-21snmpd20:38:211
57829953,1cyclictest970-21snmpd20:00:131
57829953,1cyclictest970-21snmpd19:50:391
57829953,1cyclictest970-21snmpd19:31:331
57829953,1cyclictest970-21snmpd19:26:411
57829953,1cyclictest966-21runrttasks20:55:261
57829953,1cyclictest633-21polkitd21:04:561
57829953,1cyclictest4673-21/usr/sbin/munin20:25:141
57829953,1cyclictest25-21ksoftirqd/119:15:011
57829953,1cyclictest1270-21ls20:15:181
57829953,1cyclictest11247-21cat20:40:231
57829952,2cyclictest970-21snmpd20:46:211
57829952,2cyclictest970-21snmpd19:47:551
57829950,4cyclictest0-21swapper/120:20:381
57829950,1cyclictest28893-21/usr/sbin/munin20:05:251
57829950,1cyclictest131rcu_sched19:40:001
57829950,1cyclictest0-21swapper/119:35:181
57769953,1cyclictest970-21snmpd20:13:030
57769953,1cyclictest970-21snmpd20:06:140
57769953,1cyclictest970-21snmpd19:45:280
57769953,1cyclictest970-21snmpd19:37:010
57769953,1cyclictest970-21snmpd19:25:180
57769953,1cyclictest598-21avahi-daemon20:01:000
57769953,1cyclictest23275-21cat19:50:180
57769953,1cyclictest15305-21cat20:50:180
57769950,1cyclictest6495-21systemd-cgroups20:30:000
57769950,0cyclictest0-21swapper/020:30:270
57899942,1cyclictest9085-21cut19:15:202
57899942,1cyclictest18738-21cpu19:40:142
57829942,1cyclictest30988-21sort20:10:151
57769942,1cyclictest2918-21diskstats20:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional