You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-07-28 - 05:11

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.4.39-rt50+ (Profile)

Latency plot of system in rack #7, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack7slot0.osadl.org (updated Wed Jul 28, 2021 00:45:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
110050680,0irq/25-eth00-21swapper/119:05:211
117950650,0irq/26-eth1-rx-0-21swapper/319:05:003
2607624822,0sleep00-21swapper/019:07:090
2419924421,0sleep20-21swapper/219:05:032
117950260,0irq/26-eth1-rx-0-21swapper/019:10:010
351180,0ktimersoftd/313420-21taskset23:14:583
351180,0ktimersoftd/311405-21sshd21:53:433
26563991514,0cyclictest381-21sshd22:05:283
2656399151,0cyclictest0-21swapper/322:57:173
26562991413,0cyclictest0-21swapper/223:16:442
2656299140,0cyclictest0-21swapper/221:58:132
2656299140,0cyclictest0-21swapper/221:25:132
2656199140,0cyclictest0-21swapper/122:10:211
2656099140,0cyclictest0-21swapper/000:00:000
26563991313,0cyclictest0-21swapper/323:21:203
26563991312,0cyclictest5224-21sshd22:42:113
26563991312,0cyclictest16620-21sshd22:03:223
2656399130,0cyclictest7106-21bash21:12:513
2656399130,0cyclictest0-21swapper/300:26:373
26562991312,0cyclictest5865-21sshd21:21:302
26562991312,0cyclictest10912-21bash22:34:062
26562991312,0cyclictest0-21swapper/221:52:172
26562991310,0cyclictest15484-21bash23:37:392
2656299130,0cyclictest0-21swapper/223:11:312
2656299130,0cyclictest0-21swapper/219:35:172
26561991312,0cyclictest788-40kipmi023:50:191
26561991312,0cyclictest0-21swapper/122:05:041
26561991311,0cyclictest18187-21diskmemload00:21:331
26561991311,0cyclictest15222-21sshd00:04:431
2656199130,0cyclictest0-21swapper/121:16:171
26560991312,0cyclictest0-21swapper/021:39:470
26560991311,0cyclictest19820-21sshd23:38:170
26560991311,0cyclictest15220-21sshd00:31:290
26560991311,0cyclictest0-21swapper/023:20:510
2656099130,0cyclictest18187-21diskmemload21:16:060
2656099130,0cyclictest0-21swapper/023:56:520
2656099130,0cyclictest0-21swapper/021:25:180
191130,0ktimersoftd/10-21swapper/123:56:141
351120,0ktimersoftd/30-21swapper/322:10:353
26563991212,0cyclictest0-21swapper/323:06:283
26563991211,0cyclictest19434-21sshd23:51:393
26563991210,0cyclictest0-21swapper/323:46:183
26563991210,0cyclictest0-21swapper/323:39:413
26563991210,0cyclictest0-21swapper/321:41:473
26563991210,0cyclictest0-21swapper/321:26:403
2656399120,0cyclictest0-21swapper/300:17:543
26562991211,0cyclictest548-21sshd23:58:052
26562991211,0cyclictest0-21swapper/223:34:102
26562991211,0cyclictest0-21swapper/222:50:222
26562991210,0cyclictest5248-21sshd22:55:362
26562991210,0cyclictest0-21swapper/223:26:162
26562991210,0cyclictest0-21swapper/221:46:222
2656299120,0cyclictest12472-21sshd00:22:082
2656299120,0cyclictest0-21swapper/223:54:012
2656299120,0cyclictest0-21swapper/223:46:412
2656299120,0cyclictest0-21swapper/222:38:032
2656299120,0cyclictest0-21swapper/222:38:032
2656299120,0cyclictest0-21swapper/222:13:472
2656299120,0cyclictest0-21swapper/222:09:142
2656299120,0cyclictest0-21swapper/222:04:442
2656299120,0cyclictest0-21swapper/221:10:562
26561991212,0cyclictest0-21swapper/121:10:071
26561991211,0cyclictest9685-21sshd21:35:271
26561991211,0cyclictest15150-21sshd22:52:351
26561991211,0cyclictest0-21swapper/123:06:591
26561991211,0cyclictest0-21swapper/122:25:371
26561991211,0cyclictest0-21swapper/122:18:391
26561991211,0cyclictest0-21swapper/122:18:391
26561991210,0cyclictest18328-21sshd23:33:351
26561991210,0cyclictest18187-21diskmemload00:32:431
26561991210,0cyclictest12132-21sshd21:49:221
2656199120,0cyclictest5895-21bash22:01:491
2656199120,0cyclictest23395-21sshd21:55:191
2656199120,0cyclictest1662-21sshd21:30:061
2656199120,0cyclictest0-21swapper/123:37:041
2656199120,0cyclictest0-21swapper/122:57:481
26560991211,0cyclictest18187-21diskmemload22:53:050
26560991211,0cyclictest0-21swapper/023:34:480
26560991211,0cyclictest0-21swapper/022:09:460
26560991211,0cyclictest0-21swapper/021:48:030
26560991210,0cyclictest0-21swapper/021:59:540
2656099120,0cyclictest28968-21sshd21:24:560
2656099120,0cyclictest18187-21diskmemload23:25:370
2656099120,0cyclictest16407-21sshd21:32:010
2656099120,0cyclictest110050irq/25-eth023:53:440
2656099120,0cyclictest0-21swapper/022:40:260
271110,0ktimersoftd/20-21swapper/220:20:102
26563991111,0cyclictest0-21swapper/323:41:533
26563991111,0cyclictest0-21swapper/321:39:053
26563991110,0cyclictest10258-21sshd00:12:543
26563991110,0cyclictest0-21swapper/321:22:163
2656399110,0cyclictest0-21swapper/323:16:343
2656399110,0cyclictest0-21swapper/321:17:163
26562991111,0cyclictest0-21swapper/221:30:202
26562991111,0cyclictest0-21swapper/221:17:052
26562991111,0cyclictest0-21swapper/220:27:072
26562991111,0cyclictest0-21swapper/220:00:312
26562991111,0cyclictest0-21swapper/200:30:302
26562991111,0cyclictest0-21swapper/200:02:092
26562991110,0cyclictest0-21swapper/220:45:232
2656299110,0cyclictest5025-21df_abs22:15:172
2656299110,0cyclictest5025-21df_abs22:15:172
2656299110,0cyclictest0-21swapper/219:46:352
2656299110,0cyclictest0-21swapper/219:33:142
26561991111,0cyclictest0-21swapper/122:30:301
26561991111,0cyclictest0-21swapper/121:40:471
26561991111,0cyclictest0-21swapper/100:15:381
26561991110,0cyclictest23412-21sshd00:05:391
26561991110,0cyclictest0-21swapper/123:48:061
26561991110,0cyclictest0-21swapper/123:20:551
26561991110,0cyclictest0-21swapper/122:40:511
26561991110,0cyclictest0-21swapper/121:25:411
2656199110,0cyclictest18187-21diskmemload22:20:101
2656199110,0cyclictest18187-21diskmemload22:20:101
26560991111,0cyclictest0-21swapper/022:34:030
26560991110,0cyclictest27541-21sshd21:11:060
26560991110,0cyclictest26724-21sshd00:28:420
26560991110,0cyclictest2582-21smartctl19:25:170
26560991110,0cyclictest0-21swapper/022:21:510
26560991110,0cyclictest0-21swapper/022:21:510
26560991110,0cyclictest0-21swapper/021:53:580
26560991110,0cyclictest0-21swapper/000:24:030
26560991110,0cyclictest0-21swapper/000:18:080
2656099110,0cyclictest0-21swapper/023:41:460
2656099110,0cyclictest0-21swapper/021:08:120
2656099110,0cyclictest0-21swapper/020:36:500
2656099110,0cyclictest0-21swapper/020:19:110
110050110,0irq/25-eth00-21swapper/300:38:213
110050110,0irq/25-eth00-21swapper/300:09:253
110050110,0irq/25-eth00-21swapper/222:25:242
110050110,0irq/25-eth00-21swapper/200:06:342
41100,0ktimersoftd/00-21swapper/019:17:290
351100,0ktimersoftd/30-21swapper/321:56:593
271100,0ktimersoftd/20-21swapper/223:01:452
2656399109,0cyclictest15743-21sshd22:21:093
2656399109,0cyclictest15743-21sshd22:21:093
2656399109,0cyclictest0-21swapper/322:32:253
2656399109,0cyclictest0-21swapper/320:31:443
2656399109,0cyclictest0-21swapper/300:34:383
26563991010,0cyclictest0-21swapper/323:28:183
26563991010,0cyclictest0-21swapper/300:02:553
2656299109,0cyclictest0-21swapper/222:48:362
2656299109,0cyclictest0-21swapper/222:24:212
2656299109,0cyclictest0-21swapper/222:24:212
2656299109,0cyclictest0-21swapper/200:17:082
26562991010,0cyclictest0-21swapper/221:43:402
26562991010,0cyclictest0-21swapper/220:15:522
2656299100,0cyclictest0-21swapper/221:38:152
2656299100,0cyclictest0-21swapper/221:09:112
2656299100,0cyclictest0-21swapper/219:41:152
2656299100,0cyclictest0-21swapper/219:16:102
2656299100,0cyclictest0-21swapper/200:11:132
2656199109,0cyclictest18187-21diskmemload00:10:381
2656199109,0cyclictest0-21swapper/123:27:531
2656199109,0cyclictest0-21swapper/121:21:151
26561991010,0cyclictest0-21swapper/122:36:541
26561991010,0cyclictest0-21swapper/122:36:541
2656099109,0cyclictest0-21swapper/023:13:180
2656099109,0cyclictest0-21swapper/022:29:100
2656099109,0cyclictest0-21swapper/020:05:010
2656099109,0cyclictest0-21swapper/019:42:140
26560991010,0cyclictest18187-21diskmemload22:58:190
26560991010,0cyclictest0-21swapper/023:08:260
26560991010,0cyclictest0-21swapper/022:03:080
26560991010,0cyclictest0-21swapper/019:47:340
2656099100,0cyclictest10345-21bash00:39:590
2656099100,0cyclictest0-21swapper/022:38:100
2656099100,0cyclictest0-21swapper/022:38:100
2656099100,0cyclictest0-21swapper/019:39:340
110050100,0irq/25-eth025686-21sshd22:18:083
110050100,0irq/25-eth025686-21sshd22:18:083
4190,0ktimersoftd/08974-21sshd22:15:410
4190,0ktimersoftd/08974-21sshd22:15:400
4190,0ktimersoftd/05072-21sshd23:18:140
4190,0ktimersoftd/030364-21bash23:48:470
4190,0ktimersoftd/021947-21sshd23:02:310
4190,0ktimersoftd/00-21swapper/020:34:170
265639999,0cyclictest0-21swapper/323:03:043
265639998,0cyclictest0-21swapper/322:51:083
265639998,0cyclictest0-21swapper/322:29:593
265639998,0cyclictest0-21swapper/321:33:463
265639998,0cyclictest0-21swapper/319:37:153
265629999,0cyclictest0-21swapper/220:05:112
265629999,0cyclictest0-21swapper/219:57:522
265629998,0cyclictest16717-21sshd22:43:512
265629998,0cyclictest13025-21sshd23:05:412
265629990,0cyclictest0-21swapper/219:13:082
265619999,0cyclictest0-21swapper/120:45:591
265619998,0cyclictest0-21swapper/122:46:181
265619990,0cyclictest0-21swapper/119:30:551
265609999,0cyclictest0-21swapper/022:11:170
265609999,0cyclictest0-21swapper/019:50:150
265609998,0cyclictest18341-21sshd00:09:370
265609995,0cyclictest29058-21bash00:10:550
265609994,0cyclictest22814-21munin-node20:10:210
19190,0ktimersoftd/10-21swapper/120:34:161
11005090,0irq/25-eth07263-21sshd21:48:403
11005090,0irq/25-eth034-21rcuc/323:33:173
11005090,0irq/25-eth016428-21sshd23:55:403
11005090,0irq/25-eth012578-21sshd23:41:421
11005090,0irq/25-eth010-21rcuc/021:42:370
995080,0irq/24-0000:00:18190-21date21:10:001
265639987,0cyclictest0-21swapper/319:56:103
265639987,0cyclictest0-21swapper/319:45:583
265629988,0cyclictest0-21swapper/223:21:302
265629988,0cyclictest0-21swapper/219:24:172
265629988,0cyclictest0-21swapper/200:39:272
265619988,0cyclictest0-21swapper/120:20:241
265619988,0cyclictest0-21swapper/119:50:111
265619987,0cyclictest4710-21bash23:00:151
265619987,0cyclictest0-21swapper/121:52:381
265619987,0cyclictest0-21swapper/120:41:321
265619987,0cyclictest0-21swapper/119:49:351
265619986,0cyclictest0-21swapper/120:27:211
265619986,0cyclictest0-21swapper/100:26:051
265619985,0cyclictest0-21swapper/119:23:271
265619985,0cyclictest0-21swapper/119:15:561
265619983,0cyclictest0-21swapper/100:35:471
265619982,0cyclictest0-21swapper/120:12:391
265619980,0cyclictest0-21swapper/119:11:231
265609988,0cyclictest0-21swapper/019:32:070
265609980,0cyclictest7108-21df20:45:130
265609980,0cyclictest0-21swapper/022:45:330
265609980,0cyclictest0-21swapper/020:00:220
149980,0migration/06524-21perf20:45:000
11005080,0irq/25-eth0993-21bash00:20:273
11005080,0irq/25-eth029856-21sshd22:45:313
11005080,0irq/25-eth015938-21sshd23:10:361
11005080,0irq/25-eth01512-21sshd22:35:033
11005080,0irq/25-eth01512-21sshd22:35:033
4170,0ktimersoftd/00-21swapper/020:27:010
4170,0ktimersoftd/00-21swapper/019:22:100
265639976,0cyclictest36-21ksoftirqd/320:10:003
265639976,0cyclictest0-21swapper/320:48:273
265639976,0cyclictest0-21swapper/319:51:043
265639973,0cyclictest0-21swapper/319:42:283
265629977,0cyclictest0-21swapper/220:44:132
265629970,0cyclictest0-21swapper/220:35:202
265619977,0cyclictest0-21swapper/119:58:491
265619976,0cyclictest0-21swapper/120:50:251
265619976,0cyclictest0-21swapper/120:09:261
265619975,0cyclictest0-21swapper/121:00:591
265619975,0cyclictest0-21swapper/120:01:011
265619974,0cyclictest0-21swapper/120:16:481
265619970,0cyclictest0-21swapper/120:37:051
265619970,0cyclictest0-21swapper/119:40:171
265609970,0cyclictest0-21swapper/019:55:180
259970,0migration/24008-21nfsd400:25:272
11005070,0irq/25-eth025132-21cp23:43:322
11005070,0irq/25-eth015124-21bash23:15:111
339960,0migration/332441-21fschecks_count19:20:153
339960,0migration/327325-21munin-node19:10:153
339960,0migration/314199-21fschecks_count21:00:143
265639965,0cyclictest18764-21fschecks_time20:00:163
265639962,0cyclictest0-21swapper/321:05:183
265639962,0cyclictest0-21swapper/320:55:193
265639962,0cyclictest0-21swapper/320:40:183
265639962,0cyclictest0-21swapper/320:25:193
265639962,0cyclictest0-21swapper/320:20:163
265639962,0cyclictest0-21swapper/320:15:193
265639962,0cyclictest0-21swapper/320:10:173
265639962,0cyclictest0-21swapper/319:30:153
265639962,0cyclictest0-21swapper/319:15:013
265639960,0cyclictest0-21swapper/320:35:153
265629966,0cyclictest0-21swapper/219:52:312
265629965,0cyclictest22944-21cut20:10:122
265629962,0cyclictest0-21swapper/221:00:172
265629962,0cyclictest0-21swapper/220:55:192
265629962,0cyclictest0-21swapper/220:50:162
265629962,0cyclictest0-21swapper/219:25:182
265619966,0cyclictest0-21swapper/119:29:281
265619962,0cyclictest0-21swapper/120:55:181
265619960,0cyclictest0-21swapper/119:35:111
265609965,0cyclictest13727-21munin-node21:00:120
265609962,0cyclictest0-21swapper/020:55:190
265609962,0cyclictest0-21swapper/020:20:160
265639950,0cyclictest0-21swapper/320:50:023
265639950,0cyclictest0-21swapper/319:25:063
265609951,0cyclictest0-21swapper/020:50:130
11005050,0irq/25-eth028-21ksoftirqd/220:30:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional