You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-04-15 - 00:07

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.4.39-rt50+ (Profile)

Latency plot of system in rack #7, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack7slot0.osadl.org (updated Wed Apr 14, 2021 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
117950660,0irq/26-eth1-rx-0-21swapper/307:07:463
110050660,0irq/25-eth00-21swapper/107:07:391
839324824,0sleep00-21swapper/007:07:520
829824826,0sleep20-21swapper/207:06:442
881199203,0cyclictest0-21swapper/111:49:531
881199201,0cyclictest30634-21expr07:55:021
881199182,0cyclictest0-21swapper/112:06:011
351180,0ktimersoftd/330038-21sshd12:33:543
271180,0ktimersoftd/20-21swapper/210:15:232
191180,0ktimersoftd/1954-21sshd10:15:241
881199173,0cyclictest0-21swapper/112:01:451
8813991614,0cyclictest0-21swapper/309:10:003
8813991613,0cyclictest32573-21diskmemload12:36:403
881199162,0cyclictest4711-21bash11:28:391
881199160,0cyclictest0-21swapper/112:25:051
881199150,0cyclictest0-21swapper/111:59:091
110050150,0irq/25-eth020-21ksoftirqd/111:16:151
8813991411,0cyclictest0-21swapper/312:28:523
8813991411,0cyclictest0-21swapper/311:56:283
881199143,0cyclictest0-21swapper/111:34:361
8811991411,0cyclictest0-21swapper/109:29:121
881199141,0cyclictest0-21swapper/111:00:141
881199140,0cyclictest7001-21sshd12:10:311
8810991412,0cyclictest0-21swapper/011:28:540
8810991412,0cyclictest0-21swapper/010:44:320
8813991312,0cyclictest22723-21sshd11:48:003
8813991312,0cyclictest12815-21sshd11:36:223
8813991312,0cyclictest0-21swapper/311:33:153
8813991312,0cyclictest0-21swapper/310:53:583
8813991312,0cyclictest0-21swapper/310:31:203
8813991312,0cyclictest0-21swapper/310:31:203
8813991312,0cyclictest0-21swapper/309:38:013
8813991311,0cyclictest9876-21expr08:20:043
8813991311,0cyclictest0-21swapper/312:01:283
881399130,0cyclictest8981-21sshd09:45:133
881399130,0cyclictest0-21swapper/311:53:193
8812991312,0cyclictest0-21swapper/211:22:192
8812991312,0cyclictest0-21swapper/209:26:022
8812991311,0cyclictest10786-21ntpq08:20:092
881299130,0cyclictest6032-21sshd10:26:202
881299130,0cyclictest0-21swapper/212:06:132
881199133,0cyclictest0-21swapper/111:50:291
881199133,0cyclictest0-21swapper/108:33:011
881199133,0cyclictest0-21swapper/107:43:161
8811991312,0cyclictest0-21swapper/110:34:301
8811991312,0cyclictest0-21swapper/110:34:291
8811991312,0cyclictest0-21swapper/110:01:301
8811991311,0cyclictest2303-21sshd09:58:211
8811991311,0cyclictest0-21swapper/111:13:111
881199130,0cyclictest10036-21ipmitool12:35:071
881199130,0cyclictest0-21swapper/111:35:191
8810991312,0cyclictest16848-21bash09:25:170
351130,0ktimersoftd/30-21swapper/311:13:303
1599130,0watchdog/02922-21ls10:50:100
8813991211,0cyclictest20007-21consoletype09:15:223
8813991210,0cyclictest0-21swapper/310:22:513
8813991210,0cyclictest0-21swapper/309:53:213
8813991210,0cyclictest0-21swapper/309:25:063
8813991210,0cyclictest0-21swapper/309:22:553
881399120,0cyclictest0-21swapper/311:20:413
881399120,0cyclictest0-21swapper/311:06:523
881399120,0cyclictest0-21swapper/307:39:173
8812991212,0cyclictest0-21swapper/211:08:302
8812991211,0cyclictest27596-21sshd12:16:122
8812991211,0cyclictest17020-21bash10:24:072
8812991211,0cyclictest13997-21sshd10:13:262
8812991211,0cyclictest0-21swapper/211:45:322
8812991211,0cyclictest0-21swapper/210:51:112
8812991211,0cyclictest0-21swapper/209:50:122
8812991211,0cyclictest0-21swapper/209:43:002
881299120,0cyclictest0-21swapper/212:35:302
881299120,0cyclictest0-21swapper/211:19:112
881299120,0cyclictest0-21swapper/210:04:562
881199123,0cyclictest0-21swapper/108:12:121
881199123,0cyclictest0-21swapper/107:16:401
881199122,0cyclictest0-21swapper/109:07:101
881199122,0cyclictest0-21swapper/108:25:161
881199122,0cyclictest0-21swapper/108:08:451
8811991211,0cyclictest5784-21sshd11:42:411
8811991211,0cyclictest0-21swapper/110:49:351
8811991210,0cyclictest1406-21sshd10:36:171
8811991210,0cyclictest0-21swapper/110:42:031
8810991212,0cyclictest0-21swapper/011:12:520
8810991211,0cyclictest4831-21bash10:22:480
8810991211,0cyclictest32131-21bash09:13:190
8810991211,0cyclictest0-21swapper/010:15:150
8810991211,0cyclictest0-21swapper/009:53:530
8810991210,0cyclictest0-21swapper/008:31:300
881099120,0cyclictest32573-21diskmemload11:20:240
881099120,0cyclictest0-21swapper/012:38:220
881099120,0cyclictest0-21swapper/011:42:430
881099120,0cyclictest0-21swapper/011:05:180
881099120,0cyclictest0-21swapper/010:34:460
881099120,0cyclictest0-21swapper/010:34:460
881099120,0cyclictest0-21swapper/010:04:340
881099120,0cyclictest0-21swapper/009:31:550
41120,0ktimersoftd/07471-21sshd12:03:440
41120,0ktimersoftd/05069-21consoletype11:53:000
110050120,0irq/25-eth05010-21sshd12:17:160
110050120,0irq/25-eth00-21swapper/311:28:223
881399119,0cyclictest0-21swapper/311:15:213
881399119,0cyclictest0-21swapper/307:59:583
8813991111,0cyclictest29163-21strings09:00:083
8813991111,0cyclictest0-21swapper/312:06:283
8813991111,0cyclictest0-21swapper/310:38:523
8813991111,0cyclictest0-21swapper/309:56:283
8813991110,0cyclictest32573-21diskmemload12:16:273
8813991110,0cyclictest0-21swapper/310:28:113
881399110,0cyclictest11538-21bash10:16:353
8812991111,0cyclictest17477-21sshd09:56:282
8812991111,0cyclictest0-21swapper/211:42:242
8812991111,0cyclictest0-21swapper/207:48:142
8812991110,0cyclictest1512-21sshd11:11:562
8812991110,0cyclictest0-21swapper/211:33:562
8812991110,0cyclictest0-21swapper/209:35:482
881299110,0cyclictest20647-21sshd12:25:512
881299110,0cyclictest0-21swapper/208:40:162
881299110,0cyclictest0-21swapper/208:05:452
881199119,0cyclictest0-21swapper/108:49:291
881199116,0cyclictest110050irq/25-eth010:20:401
881199116,0cyclictest0-21swapper/111:09:411
881199112,0cyclictest0-21swapper/107:53:531
881199112,0cyclictest0-21swapper/107:39:151
8811991111,0cyclictest0-21swapper/111:21:391
8811991111,0cyclictest0-21swapper/110:13:071
8811991111,0cyclictest0-21swapper/109:36:451
8811991111,0cyclictest0-21swapper/109:11:531
881199111,0cyclictest0-21swapper/109:00:191
881199111,0cyclictest0-21swapper/108:19:221
881199111,0cyclictest0-21swapper/107:47:021
881199111,0cyclictest0-21swapper/107:12:381
8811991110,0cyclictest2868-21sshd10:57:091
8811991110,0cyclictest21838-21sshd12:15:331
8811991110,0cyclictest0-21swapper/109:40:501
8811991110,0cyclictest0-21swapper/109:34:331
8810991111,0cyclictest0-21swapper/012:08:450
8810991111,0cyclictest0-21swapper/011:45:520
8810991110,0cyclictest0-21swapper/009:23:060
881099110,0cyclictest0-21swapper/011:35:100
881099110,0cyclictest0-21swapper/009:59:120
41110,0ktimersoftd/024653-21sshd12:12:270
41110,0ktimersoftd/02035-21bash10:05:110
110050110,0irq/25-eth08-21rcu_preempt09:49:381
110050110,0irq/25-eth029663-21sshd10:56:312
110050110,0irq/25-eth010-21rcuc/011:17:420
110050110,0irq/25-eth00-21swapper/310:03:553
110050110,0irq/25-eth00-21swapper/210:06:002
110050110,0irq/25-eth00-21swapper/011:04:000
881399109,0cyclictest11159-21sshd09:42:043
8813991010,0cyclictest0-21swapper/310:46:263
8813991010,0cyclictest0-21swapper/308:05:123
881399100,0cyclictest0-21swapper/310:11:113
881299109,0cyclictest13262-21bash12:11:132
881299109,0cyclictest0-21swapper/209:23:502
881299109,0cyclictest0-21swapper/207:17:412
8812991010,0cyclictest7995-21sshd11:25:262
8812991010,0cyclictest0-21swapper/211:00:562
8812991010,0cyclictest0-21swapper/210:41:252
8812991010,0cyclictest0-21swapper/210:31:402
8812991010,0cyclictest0-21swapper/210:31:392
8812991010,0cyclictest0-21swapper/209:10:562
8812991010,0cyclictest0-21swapper/209:02:052
8812991010,0cyclictest0-21swapper/208:35:092
8812991010,0cyclictest0-21swapper/207:51:282
881299100,0cyclictest32573-21diskmemload10:36:212
881299100,0cyclictest23198-21sshd12:01:522
881299100,0cyclictest0-21swapper/211:57:422
881299100,0cyclictest0-21swapper/211:35:122
881199109,0cyclictest0-21swapper/109:52:051
881199101,0cyclictest0-21swapper/107:25:441
8811991010,0cyclictest0-21swapper/112:34:501
8811991010,0cyclictest0-21swapper/110:26:571
8811991010,0cyclictest0-21swapper/109:17:151
881199100,0cyclictest0-21swapper/112:21:361
881199100,0cyclictest0-21swapper/109:23:521
881099109,0cyclictest29288-21sshd10:56:290
881099109,0cyclictest26586-21sshd12:26:300
881099109,0cyclictest0-21swapper/009:40:230
8810991010,0cyclictest0-21swapper/012:32:260
8810991010,0cyclictest0-21swapper/011:59:410
8810991010,0cyclictest0-21swapper/009:02:170
8810991010,0cyclictest0-21swapper/008:54:240
8810991010,0cyclictest0-21swapper/008:03:580
8810991010,0cyclictest0-21swapper/007:35:590
881099100,0cyclictest11384-21sshd09:35:080
881099100,0cyclictest0-21swapper/011:31:380
881099100,0cyclictest0-21swapper/008:35:210
881099100,0cyclictest0-21swapper/007:52:570
41100,0ktimersoftd/03502-21bash10:12:160
41100,0ktimersoftd/02094-21sshd10:29:270
41100,0ktimersoftd/00-21swapper/012:20:120
41100,0ktimersoftd/00-21swapper/008:10:360
995090,0irq/24-0000:00:0-21swapper/009:45:270
88139999,0cyclictest7226-21sshd11:04:403
88139999,0cyclictest0-21swapper/308:57:343
88139999,0cyclictest0-21swapper/308:35:313
88139999,0cyclictest0-21swapper/307:41:163
88139999,0cyclictest0-21swapper/307:25:213
88139998,0cyclictest0-21swapper/310:59:203
88139990,0cyclictest0-21swapper/312:21:413
88139990,0cyclictest0-21swapper/312:10:263
88129999,0cyclictest0-21swapper/209:16:162
88129999,0cyclictest0-21swapper/209:06:302
88129999,0cyclictest0-21swapper/208:46:182
88129998,0cyclictest1512-21sshd10:47:442
88119999,0cyclictest0-21swapper/110:54:561
88119999,0cyclictest0-21swapper/107:30:081
88119993,0cyclictest0-21swapper/108:50:561
88109999,0cyclictest0-21swapper/008:28:580
88109999,0cyclictest0-21swapper/007:34:210
88109999,0cyclictest0-21swapper/007:23:470
88109998,0cyclictest110050irq/25-eth008:45:130
88109998,0cyclictest0-21swapper/009:18:560
88109990,0cyclictest0-21swapper/007:55:040
88109990,0cyclictest0-21swapper/007:15:010
4190,0ktimersoftd/05372-21cp10:47:050
4190,0ktimersoftd/00-21swapper/008:43:280
27190,0ktimersoftd/20-21swapper/208:54:252
11005090,0irq/25-eth01512-21sshd11:40:223
11005090,0irq/25-eth01512-21sshd09:33:153
88139988,0cyclictest0-21swapper/308:40:583
88139987,0cyclictest0-21swapper/310:05:533
88139987,0cyclictest0-21swapper/308:28:253
88129988,0cyclictest0-21swapper/209:33:342
88129988,0cyclictest0-21swapper/208:17:372
88129988,0cyclictest0-21swapper/207:38:062
88129988,0cyclictest0-21swapper/207:28:352
88129987,0cyclictest0-21swapper/212:24:342
88129980,0cyclictest0-21swapper/209:47:002
88119988,0cyclictest0-21swapper/107:21:111
88119983,0cyclictest0-21swapper/110:06:121
88119983,0cyclictest0-21swapper/108:55:081
88119983,0cyclictest0-21swapper/108:40:261
88119983,0cyclictest0-21swapper/108:37:181
88119983,0cyclictest0-21swapper/108:20:091
88119983,0cyclictest0-21swapper/108:01:111
88109988,0cyclictest0-21swapper/008:56:360
88109988,0cyclictest0-21swapper/007:41:130
88109988,0cyclictest0-21swapper/007:13:130
88109987,0cyclictest0-21swapper/007:28:060
4180,0ktimersoftd/00-21swapper/009:05:090
11005080,0irq/25-eth031915-21sshd10:36:030
11005080,0irq/25-eth012200-21sshd11:50:132
88139977,0cyclictest0-21swapper/307:20:003
88129977,0cyclictest0-21swapper/208:33:522
88129977,0cyclictest0-21swapper/208:04:092
88109973,0cyclictest0-21swapper/008:05:100
149970,0migration/027404-21munin-node07:45:130
11005070,0irq/25-eth036-21ksoftirqd/310:41:463
11005070,0irq/25-eth02340-21sshd12:30:502
88139965,0cyclictest21904-21hwlatdetect08:45:063
88139965,0cyclictest19840-21nfsd407:30:123
88139962,0cyclictest0-21swapper/309:05:093
88139962,0cyclictest0-21swapper/308:15:073
88139962,0cyclictest0-21swapper/308:10:083
88139962,0cyclictest0-21swapper/308:00:063
88139962,0cyclictest0-21swapper/307:50:073
88139962,0cyclictest0-21swapper/307:45:073
88139960,0cyclictest0-21swapper/308:31:193
88129962,0cyclictest0-21swapper/208:55:102
88129962,0cyclictest0-21swapper/208:10:082
88129962,0cyclictest0-21swapper/207:30:092
88129962,0cyclictest0-21swapper/207:20:072
88129962,0cyclictest0-21swapper/207:10:072
88109962,0cyclictest0-21swapper/008:15:060
329960,0watchdog/39047-21tr07:10:033
159960,0watchdog/09667-21atenall_r7power08:20:040
88139950,0cyclictest0-21swapper/307:15:013
88129954,0cyclictest11811-21cat08:25:012
88129951,0cyclictest0-21swapper/207:40:072
88129950,0cyclictest0-21swapper/207:55:042
11005050,0irq/25-eth00-21swapper/308:50:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional