You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2019-06-16 - 07:17

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.1-rt3 (Profile)

Latency plot of system in rack #2, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack2slot0.osadl.org (updated Sun Jun 16, 2019 00:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2612021210,0sleep00-21swapper/023:07:520
205212920,0sleep10-21swapper/122:00:311
206532830,2sleep01797899cyclictest22:26:030
244432740,0sleep10-21swapper/122:26:401
47432700,0sleep10-21swapper/121:26:441
230182570,0sleep00-21swapper/000:29:130
1754725745,7sleep20-21swapper/219:06:432
1780425543,7sleep00-21swapper/019:09:570
1768825443,7sleep10-21swapper/119:08:341
14782540,0sleep00-21swapper/022:38:270
326542520,0sleep20-21swapper/222:43:222
53472490,0sleep10-21swapper/122:54:271
309312480,0sleep10-21swapper/100:05:001
1769024735,7sleep30-21swapper/319:08:373
173802460,0sleep00-21swapper/023:52:300
185122390,0sleep20-21swapper/223:42:262
218942370,0sleep30-21swapper/322:21:093
190082370,0sleep30-21swapper/322:51:213
3032340,0sleep20-21swapper/200:10:142
123132260,0sleep30-21swapper/321:33:193
218912180,0sleep21-21systemd23:48:072
290112170,0sleep10-21swapper/122:32:321
211102170,0sleep021113-21bash00:13:360
135132170,0sleep20-21swapper/220:15:102
75812160,0sleep20-21swapper/200:16:262
116782160,0sleep30-21swapper/300:17:083
191162150,0sleep00-21swapper/022:51:210
17979991515,0cyclictest529-40kipmi023:32:141
17979991515,0cyclictest4775-21kworker/1:021:47:541
17979991515,0cyclictest4775-21kworker/1:021:47:541
1797999150,2cyclictest25-21ksoftirqd/123:26:041
23042140,0sleep12301-21sshd22:18:111
1797999144,1cyclictest25-21ksoftirqd/123:36:491
17979991414,0cyclictest9379-21kworker/1:023:59:391
17979991414,0cyclictest4775-21kworker/1:021:51:331
17979991414,0cyclictest24273-21kworker/1:122:47:211
17979991414,0cyclictest24215-21kworker/1:022:43:221
17979991414,0cyclictest11584-21kworker/1:122:08:531
164242140,1sleep316423-21sshd23:36:593
158902140,0sleep00-21swapper/021:13:050
124722140,0sleep30-21swapper/322:09:393
309902130,0sleep30-21swapper/322:27:453
1797999132,1cyclictest25-21ksoftirqd/122:10:371
17979991313,0cyclictest529-40kipmi019:55:061
17979991313,0cyclictest30664-21kworker/1:123:04:211
17979991312,0cyclictest1947-21kworker/1:021:31:041
100622130,0sleep3391rcuc/300:27:053
1799199120,11cyclictest0-21swapper/323:57:573
1799199120,11cyclictest0-21swapper/322:12:403
1798199120,1cyclictest0-21swapper/221:52:052
1798199120,1cyclictest0-21swapper/200:00:172
1798199120,11cyclictest0-21swapper/223:10:042
17979991212,0cyclictest529-40kipmi020:50:061
17979991212,0cyclictest457-21kworker/1:223:51:281
1797999121,1cyclictest25-21ksoftirqd/100:24:051
17978991210,1cyclictest0-21swapper/022:22:360
1797899120,11cyclictest645-21dbus-daemon23:03:560
1797899120,11cyclictest11712-21lspci21:54:010
318702110,0sleep00-21swapper/021:31:110
1799199119,1cyclictest0-21swapper/322:55:283
1799199116,4cyclictest4105-21gdbus22:43:003
1799199110,10cyclictest0-21swapper/323:34:233
1798199119,1cyclictest0-21swapper/223:19:162
1798199110,10cyclictest0-21swapper/223:07:002
1798199110,10cyclictest0-21swapper/222:14:512
1798199110,10cyclictest0-21swapper/221:18:392
1798199110,0cyclictest0-21swapper/200:32:542
1797999111,1cyclictest25-21ksoftirqd/100:28:351
17979991111,0cyclictest529-40kipmi019:45:061
17979991111,0cyclictest529-40kipmi019:15:061
17979991111,0cyclictest13049-21kworker/1:000:31:101
17979991111,0cyclictest11718-21kworker/1:100:05:211
17979991110,0cyclictest28107-21lspci21:20:091
17979991110,0cyclictest28107-21lspci21:20:081
1797999110,1cyclictest17419-21lspci21:39:141
1797999110,10cyclictest0-21swapper/123:40:581
1797999110,10cyclictest0-21swapper/123:18:211
1797999110,10cyclictest0-21swapper/121:56:211
1797999110,10cyclictest0-21swapper/121:18:521
17978991110,0cyclictest0-21swapper/023:27:300
1797899110,10cyclictest0-21swapper/000:21:450
1799199109,0cyclictest0-21swapper/323:14:513
1799199109,0cyclictest0-21swapper/322:33:213
1799199109,0cyclictest0-21swapper/321:47:493
1799199109,0cyclictest0-21swapper/321:47:483
1799199109,0cyclictest0-21swapper/321:18:263
1799199109,0cyclictest0-21swapper/321:10:033
1799199109,0cyclictest0-21swapper/300:37:393
1799199109,0cyclictest0-21swapper/300:21:223
1799199109,0cyclictest0-21swapper/300:11:123
1799199100,0cyclictest0-21swapper/322:19:283
1799199100,0cyclictest0-21swapper/321:37:483
1799199100,0cyclictest0-21swapper/321:25:523
1799199100,0cyclictest0-21swapper/300:01:003
1798199109,0cyclictest0-21swapper/223:33:392
1798199109,0cyclictest0-21swapper/223:22:212
1798199109,0cyclictest0-21swapper/222:45:422
1798199109,0cyclictest0-21swapper/222:31:392
1798199109,0cyclictest0-21swapper/222:15:492
1798199109,0cyclictest0-21swapper/221:33:012
1798199109,0cyclictest0-21swapper/221:20:462
1798199109,0cyclictest0-21swapper/221:20:452
1798199100,0cyclictest0-21swapper/222:25:392
1798199100,0cyclictest0-21swapper/221:44:012
1797999109,0cyclictest0-21swapper/121:10:301
1797999108,1cyclictest25-21ksoftirqd/121:40:041
1797999103,2cyclictest25-21ksoftirqd/100:10:401
1797999103,1cyclictest631-21polkitd22:55:261
1797999103,1cyclictest25-21ksoftirqd/100:38:011
1797999101,0cyclictest25-21ksoftirqd/123:06:081
17979991010,0cyclictest457-21kworker/1:223:20:171
17979991010,0cyclictest28591-21kworker/1:120:16:131
17979991010,0cyclictest20114-21kworker/1:019:20:451
17979991010,0cyclictest17238-21kworker/1:220:37:211
17979991010,0cyclictest16421-21kworker/1:222:24:381
1797999100,9cyclictest0-21swapper/123:12:141
1797999100,1cyclictest121rcu_preempt00:19:071
1797899109,0cyclictest0-21swapper/023:43:580
1797899109,0cyclictest0-21swapper/023:19:160
1797899109,0cyclictest0-21swapper/022:42:370
1797899109,0cyclictest0-21swapper/022:33:350
1797899109,0cyclictest0-21swapper/000:31:550
1797899109,0cyclictest0-21swapper/000:18:410
1797899100,0cyclictest0-21swapper/023:35:440
1797899100,0cyclictest0-21swapper/021:57:040
179919998,0cyclictest0-21swapper/323:54:533
179919998,0cyclictest0-21swapper/322:47:233
179919998,0cyclictest0-21swapper/321:51:513
179919998,0cyclictest0-21swapper/320:55:023
179919990,0cyclictest0-21swapper/323:26:093
179819998,0cyclictest0-21swapper/223:51:052
179819998,0cyclictest0-21swapper/223:03:552
179819998,0cyclictest0-21swapper/221:10:162
179819997,1cyclictest0-21swapper/222:09:022
179819995,3cyclictest4105-21gdbus23:57:312
179799999,0cyclictest28591-21kworker/1:120:11:221
179799999,0cyclictest24561-21kworker/1:020:59:531
179799999,0cyclictest20114-21kworker/1:019:31:291
179799999,0cyclictest17238-21kworker/1:220:42:131
179799999,0cyclictest14593-21kworker/1:020:20:011
179799998,0cyclictest17238-21kworker/1:220:33:341
179799998,0cyclictest14593-21kworker/1:020:28:221
179799997,1cyclictest25-21ksoftirqd/123:45:091
179799990,1cyclictest131rcu_sched22:36:111
179789998,0cyclictest0-21swapper/021:41:540
179789997,1cyclictest9-21ksoftirqd/023:31:140
179789997,1cyclictest9-21ksoftirqd/021:23:450
179789997,1cyclictest9-21ksoftirqd/021:23:450
179789997,1cyclictest658-21gdbus00:08:300
179789990,0cyclictest0-21swapper/020:43:370
179919987,0cyclictest0-21swapper/323:45:413
179919987,0cyclictest0-21swapper/323:08:433
179919987,0cyclictest0-21swapper/321:20:313
179919987,0cyclictest0-21swapper/321:20:313
179919987,0cyclictest0-21swapper/300:34:363
179819987,0cyclictest0-21swapper/223:36:422
179819987,0cyclictest0-21swapper/222:53:462
179819987,0cyclictest0-21swapper/200:29:502
179819986,1cyclictest4641-21lspci21:47:332
179819986,1cyclictest4641-21lspci21:47:322
179819985,2cyclictest4492-21gdbus23:29:042
179799988,0cyclictest529-40kipmi019:35:061
179799988,0cyclictest28591-21kworker/1:120:09:371
179799988,0cyclictest28591-21kworker/1:120:00:471
179799988,0cyclictest20114-21kworker/1:019:27:311
179799988,0cyclictest11675-21kworker/1:219:10:411
179789987,0cyclictest0-21swapper/022:57:480
179789987,0cyclictest0-21swapper/021:28:590
179789986,1cyclictest9-21ksoftirqd/023:59:560
179789985,2cyclictest658-21gdbus22:15:530
8986270,5sleep34492-21gdbus23:40:523
5889270,1sleep3656-21gdbus23:15:013
179919976,0cyclictest0-21swapper/319:10:143
179919975,1cyclictest29527-21grepconf.sh22:02:023
179919975,1cyclictest20519-21lspci23:22:153
179919974,2cyclictest658-21gdbus21:43:243
179919974,2cyclictest645-21dbus-daemon23:04:253
179919973,3cyclictest994-21in:imjournal21:56:213
179919972,2cyclictest645-21dbus-daemon00:08:583
179919970,6cyclictest0-21swapper/320:05:463
179819976,0cyclictest0-21swapper/222:02:152
179819976,0cyclictest0-21swapper/221:56:082
179819975,1cyclictest658-21gdbus21:36:582
179819975,1cyclictest645-21dbus-daemon22:24:222
179819975,1cyclictest460-21lspci22:38:182
179819975,1cyclictest33-21ksoftirqd/200:35:042
179819974,2cyclictest658-21gdbus21:25:342
179819974,2cyclictest19138-21sshd00:23:282
179819973,3cyclictest645-21dbus-daemon00:05:152
179819973,1cyclictest658-21gdbus22:56:592
179799977,0cyclictest28591-21kworker/1:119:53:521
179799977,0cyclictest28591-21kworker/1:119:41:331
179799977,0cyclictest27427-21kworker/1:121:08:241
179799977,0cyclictest24561-21kworker/1:021:01:171
179789976,0cyclictest0-21swapper/023:22:200
179789976,0cyclictest0-21swapper/021:36:540
179789976,0cyclictest0-21swapper/021:08:000
179789976,0cyclictest0-21swapper/020:36:210
179789975,1cyclictest9-21ksoftirqd/021:49:420
179789975,1cyclictest9-21ksoftirqd/021:49:410
179789975,1cyclictest9-21ksoftirqd/000:39:230
179789975,1cyclictest9-21ksoftirqd/000:04:370
179789975,1cyclictest656-21gdbus21:15:480
179789975,1cyclictest631-21polkitd22:48:460
179789975,1cyclictest631-21polkitd22:04:470
179789974,2cyclictest631-21polkitd23:47:440
179789973,1cyclictest9-21ksoftirqd/023:13:100
4997260,0sleep30-21swapper/319:55:043
179919964,1cyclictest645-21dbus-daemon19:20:003
179919962,3cyclictest14581-21sshd22:35:173
179799966,0cyclictest21926-21kworker/1:120:45:421
179789965,0cyclictest0-21swapper/021:04:470
179789965,0cyclictest0-21swapper/020:45:440
179789965,0cyclictest0-21swapper/020:21:490
179789965,0cyclictest0-21swapper/019:38:040
179789965,0cyclictest0-21swapper/019:11:060
179789964,1cyclictest658-21gdbus22:10:030
179789963,2cyclictest16838-21sshd22:05:090
179789960,0cyclictest0-21swapper/019:51:000
179919953,1cyclictest979-21snmpd21:01:543
179919953,1cyclictest979-21snmpd20:51:453
179919953,1cyclictest979-21snmpd20:46:363
179919953,1cyclictest979-21snmpd20:40:153
179919953,1cyclictest979-21snmpd20:31:083
179919953,1cyclictest979-21snmpd20:27:173
179919953,1cyclictest979-21snmpd20:01:263
179919953,1cyclictest979-21snmpd19:41:203
179919953,1cyclictest979-21snmpd19:36:073
179919953,1cyclictest979-21snmpd19:23:093
179919953,1cyclictest658-21gdbus20:15:003
179919953,1cyclictest658-21gdbus19:29:563
179919953,1cyclictest3012-21cat19:50:053
179919953,1cyclictest27256-21cat19:30:053
179919953,1cyclictest1390-21ntpdc19:45:093
179919953,1cyclictest11230-21cat20:10:073
179919950,4cyclictest18453-21sh20:35:073
179919950,0cyclictest0-21swapper/320:23:333
179819953,1cyclictest979-21snmpd21:05:422
179819953,1cyclictest979-21snmpd20:56:252
179819953,1cyclictest979-21snmpd20:50:342
179819953,1cyclictest979-21snmpd20:46:122
179819953,1cyclictest979-21snmpd20:40:022
179819953,1cyclictest979-21snmpd20:30:152
179819953,1cyclictest979-21snmpd20:25:112
179819953,1cyclictest979-21snmpd20:20:412
179819953,1cyclictest979-21snmpd20:05:432
179819953,1cyclictest979-21snmpd19:41:112
179819953,1cyclictest979-21snmpd19:36:502
179819953,1cyclictest979-21snmpd19:23:402
179819953,1cyclictest979-21snmpd19:16:332
179819953,1cyclictest30338-21ntpd19:30:092
179819953,1cyclictest2933-21grep19:50:052
179819953,1cyclictest18554-21cat20:35:082
179819952,2cyclictest0-21swapper/220:00:122
179819950,1cyclictest25386-21/usr/sbin/munin21:00:152
179819950,1cyclictest0-21swapper/219:27:372
179789953,1cyclictest9969-21timerwakeupswit20:05:140
179789953,1cyclictest979-21snmpd20:28:150
179789953,1cyclictest979-21snmpd20:13:060
179789953,1cyclictest979-21snmpd20:02:130
179789953,1cyclictest979-21snmpd19:48:100
179789953,1cyclictest979-21snmpd19:40:040
179789953,1cyclictest979-21snmpd19:31:250
179789953,1cyclictest979-21snmpd19:22:090
179789953,1cyclictest664-21avahi-daemon19:27:090
179789953,1cyclictest645-21dbus-daemon20:30:000
179789953,1cyclictest5083-21mii-tool19:55:050
179789953,1cyclictest22088-21users19:15:130
179789953,1cyclictest13263-21cat20:15:070
179789952,2cyclictest979-21snmpd20:55:090
179919942,1cyclictest980-21runrttasks21:06:273
179819942,1cyclictest647-21expr19:45:032
179819942,1cyclictest11026-21forks20:10:052
179819942,1cyclictest0-21swapper/219:10:022
179819940,3cyclictest4612-21/usr/sbin/munin19:55:032
179789942,1cyclictest23112-21grep20:50:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional