You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-23 - 18:31

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.4.39-rt50+ (Profile)

Latency plot of system in rack #7, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack7slot0.osadl.org (updated Sat Oct 23, 2021 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
110050650,0irq/25-eth00-21swapper/107:06:431
110050590,0irq/25-eth00-21swapper/307:06:473
110050560,0irq/25-eth00-21swapper/207:06:292
325524321,0sleep00-21swapper/007:08:340
110050270,0irq/25-eth00-21swapper/307:10:003
363699202,0cyclictest11175-21sshd12:21:042
3637991818,0cyclictest34-21rcuc/312:27:013
271170,0ktimersoftd/25815-21sshd12:27:012
3636991412,0cyclictest28459-21sshd11:43:012
41130,0ktimersoftd/00-21swapper/009:45:520
3637991312,0cyclictest0-21swapper/311:12:063
3637991312,0cyclictest0-21swapper/310:21:143
363799130,0cyclictest0-21swapper/312:15:183
363699130,0cyclictest0-21swapper/210:27:382
363699130,0cyclictest0-21swapper/209:00:152
363599133,0cyclictest0-21swapper/108:35:211
363599132,0cyclictest0-21swapper/109:25:471
363599132,0cyclictest0-21swapper/109:25:461
3635991312,0cyclictest11321-21consoletype11:18:291
3635991311,0cyclictest0-21swapper/111:03:371
3634991312,0cyclictest12035-21timerandwakeup11:25:240
3634991312,0cyclictest0-21swapper/012:23:510
3634991312,0cyclictest0-21swapper/009:29:570
3634991312,0cyclictest0-21swapper/009:29:560
3634991311,0cyclictest0-21swapper/012:35:280
3634991311,0cyclictest0-21swapper/010:31:160
363499130,0cyclictest0-21swapper/011:04:090
110050130,0irq/25-eth00-21swapper/010:56:550
41120,0ktimersoftd/015378-21sshd11:47:080
3637991212,0cyclictest36-21ksoftirqd/312:05:263
3637991211,0cyclictest0-21swapper/312:32:153
3637991211,0cyclictest0-21swapper/310:44:133
3637991211,0cyclictest0-21swapper/309:21:593
3637991211,0cyclictest0-21swapper/309:14:483
3637991210,0cyclictest0-21swapper/312:13:273
3637991210,0cyclictest0-21swapper/310:55:363
3637991210,0cyclictest0-21swapper/310:25:323
363799120,0cyclictest34-21rcuc/311:53:343
3636991212,0cyclictest0-21swapper/211:10:302
3636991212,0cyclictest0-21swapper/209:39:402
3636991211,0cyclictest0-21swapper/211:01:102
3636991211,0cyclictest0-21swapper/210:35:172
363699121,0cyclictest0-21swapper/211:56:362
3636991210,0cyclictest13401-21munin-run07:30:002
363699120,0cyclictest0-21swapper/212:18:272
363699120,0cyclictest0-21swapper/211:28:372
363699120,0cyclictest0-21swapper/210:12:522
363699120,0cyclictest0-21swapper/208:20:272
363599122,0cyclictest788-40kipmi007:30:261
3635991212,0cyclictest0-21swapper/112:15:241
3635991210,0cyclictest0-21swapper/110:51:311
3635991210,0cyclictest0-21swapper/110:36:181
3635991210,0cyclictest0-21swapper/109:11:301
363599120,0cyclictest0-21swapper/112:25:441
363599120,0cyclictest0-21swapper/112:07:441
363599120,0cyclictest0-21swapper/111:29:591
363599120,0cyclictest0-21swapper/109:37:391
3634991211,0cyclictest0-21swapper/009:50:090
363499120,0cyclictest0-21swapper/010:26:590
363499120,0cyclictest0-21swapper/009:31:280
363499120,0cyclictest0-21swapper/009:31:280
271120,0ktimersoftd/20-21swapper/209:53:352
191120,0ktimersoftd/10-21swapper/110:42:491
110050120,0irq/25-eth024371-21bash10:04:350
110050120,0irq/25-eth013247-21sshd11:05:000
110050120,0irq/25-eth00-21swapper/309:39:313
41110,0ktimersoftd/027597-21sshd10:54:100
41110,0ktimersoftd/00-21swapper/007:45:360
3637991111,0cyclictest34-21rcuc/312:38:333
3637991111,0cyclictest34-21rcuc/311:05:093
3637991111,0cyclictest34-21rcuc/310:16:553
3637991111,0cyclictest0-21swapper/311:20:013
3637991110,0cyclictest351ktimersoftd/311:34:253
3637991110,0cyclictest0-21swapper/311:00:233
3637991110,0cyclictest0-21swapper/309:48:283
3637991110,0cyclictest0-21swapper/309:30:433
3637991110,0cyclictest0-21swapper/309:30:433
3637991110,0cyclictest0-21swapper/309:25:363
3637991110,0cyclictest0-21swapper/309:25:353
3637991110,0cyclictest0-21swapper/309:18:233
363799110,0cyclictest0-21swapper/308:20:323
3636991111,0cyclictest0-21swapper/210:46:192
3636991111,0cyclictest0-21swapper/210:42:582
3636991111,0cyclictest0-21swapper/210:31:002
3636991111,0cyclictest0-21swapper/210:01:232
3636991110,0cyclictest27689-21diskmemload10:57:302
3636991110,0cyclictest0-21swapper/212:02:122
3636991110,0cyclictest0-21swapper/210:05:542
3636991110,0cyclictest0-21swapper/209:56:502
3636991110,0cyclictest0-21swapper/209:17:432
3636991110,0cyclictest0-21swapper/208:18:342
3636991110,0cyclictest0-21swapper/207:18:172
363699110,0cyclictest0-21swapper/212:38:112
363699110,0cyclictest0-21swapper/210:18:072
363699110,0cyclictest0-21swapper/209:12:002
363699110,0cyclictest0-21swapper/207:10:242
3635991111,0cyclictest0-21swapper/112:22:361
3635991111,0cyclictest0-21swapper/111:10:351
3635991111,0cyclictest0-21swapper/110:58:511
3635991111,0cyclictest0-21swapper/109:50:191
3635991111,0cyclictest0-21swapper/109:48:471
3635991111,0cyclictest0-21swapper/109:31:021
3635991111,0cyclictest0-21swapper/109:31:011
3635991111,0cyclictest0-21swapper/109:22:191
3635991111,0cyclictest0-21swapper/109:18:421
3635991111,0cyclictest0-21swapper/108:55:281
3635991111,0cyclictest0-21swapper/108:29:141
3635991111,0cyclictest0-21swapper/107:50:041
3635991111,0cyclictest0-21swapper/107:14:521
3635991110,0cyclictest27689-21diskmemload10:32:011
363599110,0cyclictest0-21swapper/109:07:551
363599110,0cyclictest0-21swapper/108:20:371
363499112,0cyclictest0-21swapper/008:45:250
3634991111,0cyclictest0-21swapper/011:11:070
3634991111,0cyclictest0-21swapper/010:39:530
3634991111,0cyclictest0-21swapper/007:33:400
3634991110,0cyclictest27689-21diskmemload12:00:580
3634991110,0cyclictest0-21swapper/011:30:520
3634991110,0cyclictest0-21swapper/010:16:310
363499110,0cyclictest0-21swapper/012:10:360
363499110,0cyclictest0-21swapper/010:44:100
363499110,0cyclictest0-21swapper/009:24:490
363499110,0cyclictest0-21swapper/008:28:230
110050110,0irq/25-eth030378-21consoletype11:57:230
110050110,0irq/25-eth00-21swapper/012:25:140
110050110,0irq/25-eth00-21swapper/010:23:370
363799109,0cyclictest21383-21hddtemp_smartct07:45:223
363799109,0cyclictest0-21swapper/312:23:593
363799109,0cyclictest0-21swapper/311:48:173
3637991010,0cyclictest36-21ksoftirqd/311:59:593
3637991010,0cyclictest0-21swapper/311:35:423
3637991010,0cyclictest0-21swapper/310:54:213
3637991010,0cyclictest0-21swapper/310:05:393
3637991010,0cyclictest0-21swapper/308:06:563
3637991010,0cyclictest0-21swapper/307:44:593
3637991010,0cyclictest0-21swapper/307:20:223
363799100,0cyclictest0-21swapper/309:52:053
363799100,0cyclictest0-21swapper/308:30:043
363699109,0cyclictest0-21swapper/211:24:272
3636991010,0cyclictest0-21swapper/212:08:552
3636991010,0cyclictest0-21swapper/211:16:112
3636991010,0cyclictest0-21swapper/211:07:012
3636991010,0cyclictest0-21swapper/209:31:322
3636991010,0cyclictest0-21swapper/209:31:312
3636991010,0cyclictest0-21swapper/209:26:252
3636991010,0cyclictest0-21swapper/209:26:252
363699100,0cyclictest22740-21perf07:50:002
363699100,0cyclictest19647-21sshd11:48:082
363599109,0cyclictest1620-21runrttasks08:49:401
363599109,0cyclictest0-21swapper/109:42:241
363599109,0cyclictest0-21swapper/108:19:421
3635991010,0cyclictest23791-21sshd11:07:051
3635991010,0cyclictest0-21swapper/112:36:051
3635991010,0cyclictest0-21swapper/111:31:161
3635991010,0cyclictest0-21swapper/111:20:401
3635991010,0cyclictest0-21swapper/110:47:421
3635991010,0cyclictest0-21swapper/110:14:481
3635991010,0cyclictest0-21swapper/109:02:471
3635991010,0cyclictest0-21swapper/108:10:461
3635991010,0cyclictest0-21swapper/108:00:521
3635991010,0cyclictest0-21swapper/107:22:051
363599100,0cyclictest0-21swapper/107:47:171
363599100,0cyclictest0-21swapper/107:40:311
363499109,0cyclictest0-21swapper/009:11:010
363499105,0cyclictest0-21swapper/008:30:260
3634991010,0cyclictest0-21swapper/009:56:120
3634991010,0cyclictest0-21swapper/009:07:240
3634991010,0cyclictest0-21swapper/008:59:520
3634991010,0cyclictest0-21swapper/008:13:160
3634991010,0cyclictest0-21swapper/008:01:330
3634991010,0cyclictest0-21swapper/007:21:270
363499100,0cyclictest0-21swapper/011:35:250
363499100,0cyclictest0-21swapper/008:35:200
191100,0ktimersoftd/10-21swapper/107:26:511
110050100,0irq/25-eth027689-21diskmemload11:17:520
110050100,0irq/25-eth024452-21ipmi_temp10:25:231
110050100,0irq/25-eth00-21swapper/312:04:033
110050100,0irq/25-eth00-21swapper/011:20:370
110050100,0irq/25-eth00-21swapper/009:35:520
995090,0irq/24-0000:00:0-21swapper/010:10:170
4190,0ktimersoftd/00-21swapper/008:06:060
36379999,0cyclictest0-21swapper/311:27:403
36379999,0cyclictest0-21swapper/310:33:113
36379999,0cyclictest0-21swapper/308:38:183
36379999,0cyclictest0-21swapper/307:17:133
36379998,0cyclictest0-21swapper/308:53:173
36379990,0cyclictest0-21swapper/311:40:233
36369999,0cyclictest0-21swapper/210:22:242
36369999,0cyclictest0-21swapper/207:58:212
36369999,0cyclictest0-21swapper/207:38:442
36369998,0cyclictest0-21swapper/207:43:432
36369990,0cyclictest0-21swapper/212:10:182
36369990,0cyclictest0-21swapper/208:39:152
36359999,0cyclictest0-21swapper/112:02:161
36359999,0cyclictest0-21swapper/111:56:491
36359999,0cyclictest0-21swapper/108:31:391
36349999,0cyclictest0-21swapper/009:44:070
36349999,0cyclictest0-21swapper/008:41:310
36349998,0cyclictest0-21swapper/012:06:250
36349990,0cyclictest0-21swapper/011:42:130
36349990,0cyclictest0-21swapper/009:02:170
36349990,0cyclictest0-21swapper/007:28:400
11795090,0irq/26-eth1-rx-0-21swapper/110:08:571
11005090,0irq/25-eth028794-21sshd10:05:220
11005090,0irq/25-eth00-21swapper/112:10:391
4180,0ktimersoftd/027689-21diskmemload09:15:220
36379988,0cyclictest0-21swapper/310:10:123
36379988,0cyclictest0-21swapper/309:07:353
36379988,0cyclictest0-21swapper/308:49:553
36379988,0cyclictest0-21swapper/308:44:083
36379988,0cyclictest0-21swapper/308:19:373
36379987,0cyclictest0-21swapper/310:02:383
36379980,0cyclictest0-21swapper/309:00:023
36369988,0cyclictest0-21swapper/212:32:522
36369988,0cyclictest0-21swapper/209:07:282
36369988,0cyclictest0-21swapper/208:57:432
36369988,0cyclictest0-21swapper/208:08:572
36369988,0cyclictest0-21swapper/207:20:302
36369980,0cyclictest0-21swapper/211:35:182
36369980,0cyclictest0-21swapper/209:40:322
36369980,0cyclictest0-21swapper/208:10:222
36359988,0cyclictest0-21swapper/112:32:561
36359988,0cyclictest0-21swapper/111:35:051
36359988,0cyclictest0-21swapper/110:23:251
36359980,0cyclictest0-21swapper/110:15:241
36349988,0cyclictest0-21swapper/012:30:060
36349982,0cyclictest0-21swapper/011:50:170
11005080,0irq/25-eth00-21swapper/111:42:181
36379977,0cyclictest0-21swapper/307:55:123
36379975,0cyclictest0-21swapper/310:36:343
36379975,0cyclictest0-21swapper/308:55:453
36369977,0cyclictest0-21swapper/209:49:182
36369977,0cyclictest0-21swapper/209:21:182
36369977,0cyclictest0-21swapper/208:49:432
36369977,0cyclictest0-21swapper/208:44:282
36369977,0cyclictest0-21swapper/208:33:062
36369976,0cyclictest0-21swapper/210:50:022
36369970,0cyclictest0-21swapper/207:50:222
36359977,0cyclictest0-21swapper/108:09:491
36359977,0cyclictest0-21swapper/107:18:571
36349977,0cyclictest0-21swapper/008:53:070
36349975,0cyclictest0-21swapper/007:40:240
36349973,0cyclictest25732-21diskstats07:55:230
36349970,0cyclictest0-21swapper/008:15:210
36349970,0cyclictest0-21swapper/007:50:240
36349970,0cyclictest0-21swapper/007:35:260
149970,0migration/07777-21nfsd407:15:270
11795070,0irq/26-eth1-rx-0-21swapper/010:45:170
11005070,0irq/25-eth04262-21bash11:30:402
11005070,0irq/25-eth016203-21bash12:15:170
11005070,0irq/25-eth014927-21latency09:55:221
11005070,0irq/25-eth00-21swapper/111:50:441
11005070,0irq/25-eth00-21swapper/008:22:240
36379965,0cyclictest36-21ksoftirqd/310:45:453
36379965,0cyclictest14706-21bash09:55:203
36379962,0cyclictest0-21swapper/311:15:203
36379962,0cyclictest0-21swapper/308:10:223
36379962,0cyclictest0-21swapper/307:50:223
36379962,0cyclictest0-21swapper/307:30:273
36379961,0cyclictest0-21swapper/307:35:113
36369965,0cyclictest28740-21munin-node11:50:142
36369962,0cyclictest0-21swapper/208:50:192
36369962,0cyclictest0-21swapper/208:25:212
36369962,0cyclictest0-21swapper/208:00:182
36369962,0cyclictest0-21swapper/207:30:262
36359966,0cyclictest0-21swapper/108:41:251
36359965,0cyclictest25882-21fschecks_time07:55:251
36359963,0cyclictest788-40kipmi007:35:261
36359962,0cyclictest0-21swapper/108:50:201
36349962,0cyclictest0-21swapper/007:10:190
179960,0migration/112642-21bash11:46:291
11005060,0irq/25-eth036-21ksoftirqd/309:40:353
11005060,0irq/25-eth017696-21sshd10:03:011
36379950,0cyclictest0-21swapper/308:00:553
36379950,0cyclictest0-21swapper/307:25:113
11795050,0irq/26-eth1-rx-14344-21kworker/3:108:25:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional