You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-01-20 - 10:24

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack1slot0.osadl.org (updated Wed Jan 20, 2021 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
241952970,0sleep30-21swapper/300:17:053
298832890,0sleep10-21swapper/122:18:251
112652650,0sleep30-21swapper/322:19:393
66952590,2sleep32042899cyclictest23:46:563
320652590,0sleep00-21swapper/021:34:490
163422590,1sleep116321-21sshd23:13:051
138332590,0sleep00-21swapper/000:30:440
81062580,0sleep28091-21sshd22:46:102
267712580,0sleep20-21swapper/200:05:502
267712580,0sleep20-21swapper/200:05:502
50272570,1sleep25024-21sshd21:49:502
241672570,0sleep20-21swapper/221:42:452
2005125634,7sleep10-21swapper/119:06:411
190512560,0sleep3401ktimersoftd/321:42:183
122632560,0sleep012266-21bash22:22:370
115392560,0sleep111520-21sshd00:13:071
115392560,0sleep111520-21sshd00:13:071
256812550,0sleep00-21swapper/022:56:350
255652550,0sleep20-21swapper/223:28:252
242102550,0sleep00-21swapper/023:34:080
21752550,0sleep00-21swapper/022:12:590
74812540,0sleep10-21swapper/122:34:071
64152540,0sleep20-21swapper/223:06:212
61842540,0sleep20-21swapper/222:34:002
244862540,0sleep10-21swapper/123:39:571
242992540,0sleep3793-21dbus-daemon22:41:563
20132520,0sleep20-21swapper/221:29:032
1956425240,7sleep20-21swapper/219:05:132
235762510,1sleep123566-21sshd22:03:111
217082510,0sleep1241ktimersoftd/121:42:321
282052490,0sleep00-21swapper/022:15:160
239492490,2sleep22041999cyclictest21:34:032
2019324836,7sleep00-21swapper/019:08:170
2015624837,7sleep30-21swapper/319:07:483
206262210,0sleep20-21swapper/200:28:302
279662190,1sleep3816-21polkitd22:09:303
174202190,0sleep10-21swapper/121:56:421
59272180,0sleep30-21swapper/323:15:043
311242180,0sleep3401ktimersoftd/300:29:273
308432170,2sleep32042899cyclictest23:51:533
18122170,1sleep0786-21rngd22:33:340
318022160,0sleep1231rcuc/100:00:331
204162150,1sleep0111rcuc/000:05:160
204162150,1sleep0111rcuc/000:05:160
116732150,1sleep31-21systemd23:44:323
289502140,0sleep10-21swapper/121:25:331
221642140,0sleep0101ktimersoftd/023:51:060
180502130,1sleep118046-21sshd00:34:101
2042899129,2cyclictest15814-21sshd21:12:063
20428991211,0cyclictest19592-21lspci23:39:303
2042899120,1cyclictest12866-21bash22:37:503
2042899120,1cyclictest12866-21bash22:37:503
2042899120,11cyclictest793-21dbus-daemon21:58:363
2042899120,11cyclictest14024-21lspci23:07:043
20419991210,1cyclictest0-21swapper/223:43:432
2041999120,1cyclictest8688-21grepconf.sh21:35:312
2041999120,11cyclictest921-21gdbus22:04:542
2041999120,11cyclictest2388-21sshd22:42:512
20415991210,1cyclictest793-21dbus-daemon22:40:091
2041599120,1cyclictest32587-21lspci23:34:531
2041599120,11cyclictest27161-21lspci00:37:431
2041599120,11cyclictest1-21systemd21:18:471
2041399120,1cyclictest9466-21bash23:06:400
2041399120,11cyclictest0-21swapper/021:44:390
124172120,0sleep00-21swapper/021:21:030
206592110,2sleep32042899cyclictest22:56:053
20428991110,0cyclictest919-21lspci21:20:033
20428991110,0cyclictest20222-21lspci22:47:203
20428991110,0cyclictest0-21swapper/323:13:503
2042899110,10cyclictest30719-21kworker/3:300:08:213
2042899110,10cyclictest30719-21kworker/3:300:08:203
2042899110,10cyclictest0-21swapper/323:00:173
2042899110,10cyclictest0-21swapper/300:33:173
2042899110,0cyclictest0-21swapper/321:19:033
2041999119,1cyclictest10019-21lspci21:56:022
2041999119,1cyclictest0-21swapper/200:00:032
2041999110,10cyclictest0-21swapper/222:57:232
2041999110,0cyclictest0-21swapper/222:25:062
2041599119,1cyclictest8455-21lspci21:47:121
2041599119,1cyclictest31552-21sshd23:26:001
2041599119,1cyclictest0-21swapper/122:28:411
2041599110,10cyclictest0-21swapper/122:47:541
2041599110,10cyclictest0-21swapper/122:23:411
2041599110,10cyclictest0-21swapper/121:14:351
2041399119,1cyclictest32273-21sshd23:20:140
20413991110,0cyclictest0-21swapper/023:13:270
20413991110,0cyclictest0-21swapper/021:53:120
2041399110,10cyclictest25209-21sshd22:47:460
2041399110,10cyclictest0-21swapper/023:48:000
2041399110,10cyclictest0-21swapper/022:41:190
176122110,0sleep00-21swapper/021:39:180
212382100,0sleep10-21swapper/123:59:401
2042899109,0cyclictest0-21swapper/321:49:533
2042899108,1cyclictest41-21ksoftirqd/300:03:043
2042899108,1cyclictest15745-21sshd00:10:313
2042899108,1cyclictest15745-21sshd00:10:313
2042899103,3cyclictest41-21ksoftirqd/323:59:493
2042899100,9cyclictest31935-21kworker/3:122:31:143
2042899100,0cyclictest131rcu_sched21:29:163
2042899100,0cyclictest0-21swapper/323:24:493
2041999109,0cyclictest0-21swapper/220:35:182
2041999109,0cyclictest0-21swapper/200:14:432
2041999109,0cyclictest0-21swapper/200:14:432
2041999108,1cyclictest23811-21sshd22:38:582
2041999108,1cyclictest23811-21sshd22:38:582
2041999108,1cyclictest16166-21sshd22:20:052
2041999108,1cyclictest1-21systemd22:16:222
2041999100,9cyclictest0-21swapper/221:12:072
2041999100,0cyclictest0-21swapper/222:53:212
2041999100,0cyclictest0-21swapper/221:52:192
2041599100,0cyclictest0-21swapper/121:54:371
2041399109,0cyclictest0-21swapper/022:28:340
2041399109,0cyclictest0-21swapper/021:58:120
2041399109,0cyclictest0-21swapper/021:13:000
2041399109,0cyclictest0-21swapper/019:15:180
2041399109,0cyclictest0-21swapper/000:17:340
2041399108,1cyclictest0-21swapper/022:53:060
2041399100,0cyclictest0-21swapper/021:26:240
204289998,0cyclictest0-21swapper/322:53:303
204289998,0cyclictest0-21swapper/321:53:363
204289998,0cyclictest0-21swapper/321:09:213
204289998,0cyclictest0-21swapper/320:30:173
204289998,0cyclictest0-21swapper/300:39:253
204289996,2cyclictest41-21ksoftirqd/321:36:253
204289991,1cyclictest41-21ksoftirqd/323:26:413
204289991,1cyclictest131rcu_sched22:11:413
204289990,8cyclictest3873-21kworker/3:022:25:283
204289990,0cyclictest0-21swapper/320:44:203
204289990,0cyclictest0-21swapper/319:20:033
204199998,0cyclictest0-21swapper/223:14:592
204199998,0cyclictest0-21swapper/223:01:252
204199998,0cyclictest0-21swapper/221:17:362
204199998,0cyclictest0-21swapper/219:22:292
204199998,0cyclictest0-21swapper/200:16:102
204199997,1cyclictest22818-21sshd00:34:352
204199990,0cyclictest0-21swapper/223:58:342
204159998,0cyclictest0-21swapper/123:46:401
204159998,0cyclictest0-21swapper/123:04:031
204159998,0cyclictest0-21swapper/122:38:521
204159998,0cyclictest0-21swapper/122:38:521
204159998,0cyclictest0-21swapper/120:20:201
204159998,0cyclictest0-21swapper/100:05:531
204159998,0cyclictest0-21swapper/100:05:531
204159997,1cyclictest23631-21sshd22:14:581
204159997,1cyclictest0-21swapper/123:23:061
204159990,0cyclictest0-21swapper/123:52:401
204159990,0cyclictest0-21swapper/123:00:011
204139998,0cyclictest1062-21sshd21:49:290
204139998,0cyclictest0-21swapper/023:56:530
204139998,0cyclictest0-21swapper/022:38:440
204139998,0cyclictest0-21swapper/022:38:440
204139990,0cyclictest0-21swapper/023:40:340
30224280,0sleep00-21swapper/022:00:450
204289986,1cyclictest41-21ksoftirqd/323:31:523
204289986,1cyclictest41-21ksoftirqd/322:00:393
204289986,1cyclictest41-21ksoftirqd/300:21:423
204289980,7cyclictest4142-21sshd21:32:123
204199980,7cyclictest974-21sshd22:12:532
204199980,7cyclictest20110-21sshd23:36:362
204199980,7cyclictest12150-21sshd00:36:242
204159986,1cyclictest24760-21lspci00:28:521
204159986,1cyclictest18618-21kworker/1:119:16:301
204159980,7cyclictest8202-21sshd23:18:111
204159980,7cyclictest6756-21sshd21:38:201
204159980,7cyclictest5309-21sshd22:07:221
204159980,7cyclictest4529-21sshd00:21:091
204159980,7cyclictest2238-21sshd00:15:071
204159980,7cyclictest21686-21sshd23:07:461
204159980,7cyclictest14052-21sshd23:44:441
204139987,0cyclictest0-21swapper/023:17:290
204139987,0cyclictest0-21swapper/023:03:550
204139986,1cyclictest32158-21lspci23:37:410
204139980,7cyclictest7570-21sshd00:38:490
204139980,7cyclictest6893-21sshd00:12:390
204139980,7cyclictest6893-21sshd00:12:390
204139980,7cyclictest1990-21sshd00:20:550
204139980,7cyclictest19668-21sshd00:25:250
204139980,7cyclictest10470-21sshd21:17:480
10490280,0sleep0111rcuc/023:27:020
204289975,1cyclictest1420-21snmpd19:50:073
204289971,0cyclictest131rcu_sched22:20:033
204289970,6cyclictest0-21swapper/319:41:223
204289970,0cyclictest0-21swapper/320:55:483
204199975,1cyclictest3335-21lspci21:20:122
204199975,1cyclictest13189-21lspci23:50:182
204199974,2cyclictest812-21systemd-logind23:30:122
204199970,6cyclictest5253-21sshd19:57:292
204199970,6cyclictest31598-21sshd00:20:412
204199970,6cyclictest30970-21sshd23:17:192
204199970,6cyclictest22105-21sshd19:12:242
204199970,6cyclictest18299-21sshd23:45:052
204199970,6cyclictest17038-21sshd22:05:262
204199970,6cyclictest15677-21sshd20:27:362
204199970,6cyclictest14231-21sshd23:21:352
204159976,1cyclictest15887-21kworker/1:320:35:141
204159976,0cyclictest0-21swapper/120:04:301
204159975,1cyclictest29548-21turbostat21:05:001
204159975,1cyclictest18618-21kworker/1:119:12:301
204159975,1cyclictest15887-21kworker/1:320:40:301
204159970,6cyclictest20126-21sshd22:50:111
204159970,6cyclictest16385-21sshd21:30:221
204159970,6cyclictest15698-21sshd21:21:241
204139976,0cyclictest15795-21kworker/0:120:32:140
204139975,1cyclictest9-21ksoftirqd/022:05:090
204139975,1cyclictest17789-21kworker/0:220:40:060
204139975,1cyclictest11021-21lspci00:01:360
204139974,2cyclictest1420-21snmpd20:02:460
204139970,6cyclictest9167-21sshd20:07:350
204289965,0cyclictest0-21swapper/320:53:133
204289965,0cyclictest0-21swapper/319:27:583
204289963,2cyclictest1421-21snmpd20:20:313
204289963,2cyclictest1421-21snmpd20:18:103
204289963,2cyclictest1420-21snmpd20:25:293
204289963,2cyclictest1420-21snmpd19:57:583
204289963,2cyclictest1420-21snmpd19:17:393
204289960,5cyclictest9601-21kworker/3:020:12:343
204289960,5cyclictest15888-21kworker/3:120:39:293
204289960,5cyclictest0-21swapper/320:46:553
204199964,1cyclictest23468-21irqstats19:15:062
204199963,2cyclictest1420-21snmpd20:18:112
204199963,2cyclictest1420-21snmpd19:53:552
204199963,1cyclictest29542-21latency_hist21:05:002
204159966,0cyclictest31135-21kworker/1:019:40:321
204159966,0cyclictest24742-21kworker/1:219:35:121
204159966,0cyclictest24742-21kworker/1:219:20:121
204159966,0cyclictest21518-21kworker/1:020:50:041
204159966,0cyclictest15887-21kworker/1:320:30:201
204159966,0cyclictest14996-21kworker/1:120:27:021
204159965,1cyclictest7411-21kworker/1:220:11:181
204159964,1cyclictest1420-21snmpd20:08:111
204159963,2cyclictest1421-21snmpd20:46:301
204159963,2cyclictest1421-21snmpd20:16:071
204159963,2cyclictest1420-21snmpd19:54:541
204139966,0cyclictest31202-21kworker/0:219:40:150
204139966,0cyclictest29211-21kworker/0:120:20:160
204139966,0cyclictest29211-21kworker/0:120:11:260
204139966,0cyclictest24509-21kworker/0:019:35:460
204139966,0cyclictest24509-21kworker/0:019:30:560
204139966,0cyclictest24509-21kworker/0:019:26:120
204139966,0cyclictest24509-21kworker/0:019:21:000
204139966,0cyclictest20026-21kworker/0:019:10:540
204139966,0cyclictest15088-21kworker/0:220:25:340
204139965,0cyclictest0-21swapper/019:46:180
204139963,2cyclictest1421-21snmpd20:46:400
204139963,2cyclictest1421-21snmpd20:35:300
204139963,2cyclictest1421-21snmpd19:57:240
204139963,2cyclictest1420-21snmpd20:15:380
204289953,1cyclictest8471-21ntp_states20:05:123
204289953,1cyclictest29943-21cat19:35:063
204289953,1cyclictest28020-21cat19:30:053
204289953,1cyclictest21043-21cat19:10:073
204289953,1cyclictest1421-21snmpd20:00:303
204289953,1cyclictest1420-21snmpd19:47:013
204289953,1cyclictest1226-21runrttasks21:00:463
204199953,1cyclictest892-21timerwakeupswit19:45:122
204199953,1cyclictest26696-21latency_hist19:25:002
204199953,1cyclictest24785-21proc_pri20:50:102
204199953,1cyclictest1421-21snmpd19:59:592
204199953,1cyclictest1420-21snmpd21:05:032
204199953,1cyclictest1420-21snmpd20:55:212
204199953,1cyclictest1420-21snmpd20:40:422
204199953,1cyclictest1420-21snmpd20:30:562
204199953,1cyclictest1420-21snmpd20:11:052
204199953,1cyclictest1420-21snmpd20:05:102
204199953,1cyclictest1420-21snmpd19:40:132
204199953,1cyclictest1420-21snmpd19:31:242
204199953,1cyclictest13916-21cat20:20:062
204199952,2cyclictest1421-21snmpd19:35:022
204199952,2cyclictest1420-21snmpd20:45:122
204159953,1cyclictest30172-21cat21:05:061
204159953,1cyclictest27789-21cstates19:30:031
204159953,1cyclictest27164-21sshd20:55:281
204159953,1cyclictest1421-21snmpd19:45:031
204159953,1cyclictest1421-21snmpd19:25:141
204159953,1cyclictest1420-21snmpd19:55:491
204139953,1cyclictest2206-21cat19:50:060
204139953,1cyclictest1421-21snmpd21:00:140
204139953,1cyclictest1421-21snmpd20:56:080
204139953,1cyclictest1420-21snmpd20:50:300
204139952,2cyclictest31005-21sed21:05:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional