You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-05 - 17:19

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackaslot0.osadl.org (updated Tue Jul 05, 2022 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
112562890,0sleep00-21swapper/009:30:080
152752620,0sleep20-21swapper/210:32:082
85102610,1sleep28464-21sshd12:34:572
28762590,1sleep1201ktimersoftd/111:06:071
2214725924,18sleep00-21swapper/007:08:470
2198225824,18sleep20-21swapper/207:06:482
68962560,1sleep36886-21sshd12:08:143
241122560,0sleep10-21swapper/109:54:521
225852550,0sleep20-21swapper/212:06:412
135842550,0sleep30-21swapper/312:20:363
146432540,1sleep214620-21sshd09:18:452
319392520,1sleep10-21swapper/111:20:291
286612520,0sleep00-21swapper/010:04:050
266312520,1sleep30-21swapper/309:37:293
2193525226,8sleep10-21swapper/107:06:141
87442500,0sleep00-21swapper/009:26:550
50592500,0sleep00-21swapper/011:47:290
50592500,0sleep00-21swapper/011:47:290
257492500,0sleep10-21swapper/109:31:211
2194225028,18sleep30-21swapper/307:06:213
308672490,0sleep030865-21bash11:08:430
264372490,0sleep00-21swapper/012:27:430
222882490,1sleep30-21swapper/309:51:433
190992490,1sleep30-21swapper/312:12:153
185982490,1sleep118595-21bash10:17:481
62932480,1sleep115200-21diskmemload12:31:451
49752480,1sleep228-21rcuc/209:55:582
141492480,1sleep314123-21sshd11:27:443
245542470,0sleep10-21swapper/112:06:531
184012470,0sleep30-21swapper/311:31:013
184012470,0sleep30-21swapper/311:31:013
169462470,0sleep339-21ksoftirqd/312:03:183
79722460,1sleep17968-21sshd11:59:361
83072360,0sleep20-21swapper/207:45:172
49072220,0sleep00-21swapper/011:29:550
22498992220,1cyclictest0-21swapper/311:15:073
22494992220,1cyclictest0-21swapper/211:43:182
22487992220,1cyclictest31850-21sshd10:39:330
22487992220,1cyclictest0-21swapper/009:47:220
22487992210,4cyclictest4820-21sshd09:23:420
22494992119,1cyclictest0-21swapper/211:35:452
22494992119,1cyclictest0-21swapper/209:49:552
249102200,0sleep3381ktimersoftd/310:33:013
22489991917,1cyclictest20875-21sshd12:24:191
82342180,1sleep28233-21bash09:21:012
45752180,1sleep04571-21sshd10:28:160
22494991816,1cyclictest0-21swapper/211:22:382
22494991816,1cyclictest0-21swapper/210:58:092
2248799180,13cyclictest0-21swapper/011:37:280
2248799179,7cyclictest9635-21sshd12:02:370
2248799178,8cyclictest22936-21sshd11:13:530
2248799178,8cyclictest0-21swapper/012:23:410
2248799175,11cyclictest0-21swapper/011:40:120
2248799171,15cyclictest41ktimersoftd/011:17:510
2248799171,13cyclictest7341-21sshd12:34:510
2248799170,16cyclictest41ktimersoftd/011:21:480
61052160,1sleep215200-21diskmemload11:47:342
61052160,1sleep215200-21diskmemload11:47:342
242872160,0sleep124280-21sshd10:53:291
2249899168,5cyclictest0-21swapper/310:09:433
22498991614,1cyclictest0-21swapper/309:59:353
22489991614,1cyclictest21553-21sshd11:19:391
2248799168,2cyclictest22619-21bash11:55:020
2248799167,8cyclictest0-21swapper/009:44:540
2248799162,1cyclictest3-21ksoftirqd/010:10:320
2248799161,3cyclictest7481-21bash09:59:130
2248799160,15cyclictest806-21sshd11:00:120
213172160,2sleep20-21swapper/209:37:002
99752150,0sleep09976-21bash09:18:180
2249899155,9cyclictest15200-21diskmemload09:49:393
22498991513,1cyclictest0-21swapper/311:02:493
22489991512,2cyclictest12742-21sshd10:34:541
2248999150,2cyclictest0-21swapper/110:47:011
2248799159,5cyclictest10-21rcuc/010:30:530
2248799151,9cyclictest10-21rcuc/012:11:570
2248799151,3cyclictest0-21swapper/011:53:440
22487991512,2cyclictest21658-21bash10:20:570
2248799151,13cyclictest41ktimersoftd/009:12:430
178492150,0sleep10-21swapper/109:13:071
243142140,1sleep10-21swapper/110:56:221
2249899149,2cyclictest0-21swapper/309:16:473
22498991412,1cyclictest8727-21cp11:36:003
2249899140,1cyclictest23117-21sshd09:43:023
2249899140,1cyclictest0-21swapper/311:59:083
2249899140,13cyclictest11989-21sshd10:49:313
2249899140,13cyclictest0-21swapper/311:09:493
2249899140,13cyclictest0-21swapper/310:21:453
2249499149,2cyclictest14528-21bash11:15:582
2249499144,9cyclictest10942-21sshd12:26:162
22494991412,1cyclictest415-21sshd09:26:112
2249499140,13cyclictest10434-21sshd11:00:572
2249499140,13cyclictest0-21swapper/210:43:532
2249499140,13cyclictest0-21swapper/209:10:452
2248999149,4cyclictest0-21swapper/109:36:321
2248999149,2cyclictest0-21swapper/111:10:281
2248999144,9cyclictest21-21ksoftirqd/112:36:081
2248999144,9cyclictest21-21ksoftirqd/109:20:461
22489991413,1cyclictest21-21ksoftirqd/111:31:161
22489991413,1cyclictest21-21ksoftirqd/111:31:161
22489991412,1cyclictest0-21swapper/110:40:211
2248999140,13cyclictest15513-21sshd09:42:181
2248999140,13cyclictest1260-21sshd11:50:101
2248999140,10cyclictest0-21swapper/109:18:591
2248799149,1cyclictest0-21swapper/010:15:060
2248799148,5cyclictest0-21swapper/012:15:030
2248799148,3cyclictest0-21swapper/012:05:270
2248799143,10cyclictest25739-21rm11:34:430
2248799143,10cyclictest25739-21rm11:34:430
22487991410,1cyclictest2108-21sshd10:45:360
2248799140,13cyclictest9691-21sshd12:35:030
2248799140,13cyclictest10858-21sshd09:36:000
2248799140,13cyclictest0-21swapper/010:58:070
2248799140,13cyclictest0-21swapper/010:43:140
2248799140,13cyclictest0-21swapper/010:06:560
208982140,0sleep120897-21bash12:27:121
15072140,0sleep31503-21sshd10:39:463
2249899139,1cyclictest0-21swapper/310:52:363
2249899139,1cyclictest0-21swapper/310:01:463
2249899131,1cyclictest0-21swapper/309:30:193
22498991311,1cyclictest3573-21sshd10:25:163
2249899131,11cyclictest0-21swapper/309:05:123
2249899131,11cyclictest0-21swapper/307:25:113
22498991310,2cyclictest32213-21sshd12:34:083
2249899130,12cyclictest7256-21bash10:16:453
2249899130,12cyclictest20832-21sshd09:10:243
2249899130,12cyclictest19339-21sshd11:10:303
2249899130,12cyclictest15200-21diskmemload11:22:093
2249899130,12cyclictest0-21swapper/312:38:153
2249899130,12cyclictest0-21swapper/312:16:133
2249899130,12cyclictest0-21swapper/311:51:123
2249899130,12cyclictest0-21swapper/310:41:423
2249899130,12cyclictest0-21swapper/310:12:003
2249899130,10cyclictest24608-21sshd09:22:323
2249899130,10cyclictest19676-21id11:42:563
2249499139,3cyclictest0-21swapper/207:35:152
2249499138,4cyclictest11223-21munin-node12:35:222
2249499138,4cyclictest0-21swapper/212:12:002
22494991311,1cyclictest1810-21sshd11:29:372
22494991311,1cyclictest15200-21diskmemload10:09:402
2249499131,11cyclictest0-21swapper/211:14:082
22494991311,1cyclictest0-21swapper/209:54:102
2249499130,4cyclictest19887-21sshd12:18:202
2249499130,2cyclictest4233-21sshd09:41:172
2249499130,2cyclictest30248-21sshd10:21:472
2249499130,1cyclictest28234-21sshd10:01:022
2249499130,12cyclictest8203-21sshd10:16:512
2249499130,12cyclictest26118-21sshd11:52:212
2249499130,12cyclictest21193-21sshd10:35:332
2249499130,12cyclictest1534-21nfsd09:32:032
2249499130,12cyclictest0-21swapper/212:01:032
2249499130,12cyclictest0-21swapper/210:27:142
2249499130,12cyclictest0-21swapper/210:13:122
2249499130,10cyclictest0-21swapper/211:56:372
2248999139,1cyclictest16421-21sshd09:27:381
2248999139,1cyclictest118850irq/25-eth010:35:011
2248999139,1cyclictest0-21swapper/111:01:111
2248999138,2cyclictest0-21swapper/112:13:451
22489991312,1cyclictest21-21ksoftirqd/111:25:171
22489991311,1cyclictest20077-21sshd11:37:051
2248999130,4cyclictest18535-21sshd10:06:001
2248999130,12cyclictest18304-21sshd11:42:491
2248999130,12cyclictest17780-21bash10:20:351
2248999130,12cyclictest0-21swapper/112:01:181
2248999130,12cyclictest0-21swapper/111:45:591
2248999130,12cyclictest0-21swapper/111:45:591
2248999130,12cyclictest0-21swapper/110:27:101
2248999130,12cyclictest0-21swapper/109:55:381
22487991312,1cyclictest3-21ksoftirqd/009:50:140
2248799131,1cyclictest41ktimersoftd/010:52:090
2249899129,2cyclictest0-21swapper/312:25:143
2249899129,2cyclictest0-21swapper/311:46:443
2249899129,2cyclictest0-21swapper/311:46:433
2249899129,2cyclictest0-21swapper/309:26:353
2249899129,2cyclictest0-21swapper/308:15:153
2249899129,2cyclictest0-21swapper/308:00:143
2249899121,10cyclictest0-21swapper/310:55:133
2249899121,10cyclictest0-21swapper/307:55:153
2249899121,10cyclictest0-21swapper/307:30:153
2249499129,2cyclictest0-21swapper/211:30:172
2249499129,2cyclictest0-21swapper/211:30:172
2249499129,2cyclictest0-21swapper/208:45:132
2249499129,2cyclictest0-21swapper/208:40:142
2249499128,3cyclictest0-21swapper/212:20:372
2249499120,12cyclictest0-21swapper/208:22:332
2249499120,11cyclictest24366-21sshd11:05:142
2249499120,11cyclictest0-21swapper/210:50:012
2249499120,11cyclictest0-21swapper/208:55:122
2248999129,2cyclictest0-21swapper/107:50:111
2248999121,10cyclictest0-21swapper/108:55:121
2248999121,10cyclictest0-21swapper/108:00:151
22489991210,1cyclictest1875-21sshd10:01:351
2248999120,2cyclictest0-21swapper/110:10:341
2248999120,1cyclictest118850irq/25-eth012:16:071
2248999120,11cyclictest0-21swapper/109:45:431
2248799129,2cyclictest0-21swapper/007:30:140
2248799122,9cyclictest3-21ksoftirqd/009:00:140
2248799121,10cyclictest0-21swapper/007:55:130
2248799121,10cyclictest0-21swapper/007:25:130
2249899119,1cyclictest0-21swapper/308:45:143
2249899119,1cyclictest0-21swapper/308:40:143
2249899119,1cyclictest0-21swapper/307:35:153
2249899110,2cyclictest29605-21fw_forwarded_lo08:30:203
2249899110,11cyclictest0-21swapper/308:39:573
2249899110,10cyclictest0-21swapper/308:55:113
2249499119,1cyclictest0-21swapper/208:30:192
2249499111,1cyclictest0-21swapper/208:00:142
2249499110,2cyclictest0-21swapper/210:45:062
2249499110,10cyclictest0-21swapper/208:05:122
2249499110,10cyclictest0-21swapper/207:40:232
2248999119,1cyclictest0-21swapper/108:35:141
2248999118,2cyclictest0-21swapper/107:35:111
2248999113,5cyclictest0-21swapper/108:09:411
2248999113,5cyclictest0-21swapper/107:45:451
2248999110,5cyclictest0-21swapper/108:48:181
2248999110,1cyclictest0-21swapper/107:56:491
2248999110,11cyclictest0-21swapper/109:00:571
2248999110,11cyclictest0-21swapper/107:22:401
2248999110,10cyclictest30978-21grep07:25:131
2248999110,10cyclictest1111-21apache_volume08:40:121
2248999110,10cyclictest0-21swapper/108:22:371
2248999110,10cyclictest0-21swapper/107:30:191
2248999110,10cyclictest0-21swapper/107:15:011
2248799119,1cyclictest0-21swapper/008:55:110
22487991110,1cyclictest3-21ksoftirqd/009:10:000
2248799110,2cyclictest0-21swapper/008:00:130
2248799110,2cyclictest0-21swapper/007:35:110
2248799110,2cyclictest0-21swapper/007:15:000
2249899108,1cyclictest24498-21diskstats08:20:133
2249899107,2cyclictest0-21swapper/308:25:143
2249899100,9cyclictest0-21swapper/307:45:153
2249899100,1cyclictest17369-21df_inode08:05:133
2249899100,1cyclictest0-21swapper/309:00:143
2249899100,1cyclictest0-21swapper/308:50:593
2249899100,10cyclictest0-21swapper/308:10:483
2249899100,10cyclictest0-21swapper/307:40:083
2249499109,1cyclictest0-21swapper/208:16:512
2249499109,1cyclictest0-21swapper/207:29:012
2249499108,1cyclictest26359-21fw_conntrack07:15:152
2249499108,1cyclictest12171-21apache_volume07:55:122
2249499108,1cyclictest0-21swapper/208:25:142
2249499108,1cyclictest0-21swapper/208:10:142
2249499100,9cyclictest11406-21tune2fs09:00:152
2249499100,9cyclictest0-21swapper/209:05:132
2249499100,2cyclictest0-21swapper/208:50:152
2249499100,10cyclictest0-21swapper/207:51:542
2249499100,10cyclictest0-21swapper/207:21:342
2248999109,1cyclictest0-21swapper/109:08:161
2248999109,1cyclictest0-21swapper/107:10:281
2248999108,1cyclictest7643-21users08:50:241
2248999100,1cyclictest19-21rcuc/108:30:201
2248999100,10cyclictest0-21swapper/107:42:241
2248799109,1cyclictest41ktimersoftd/007:50:130
2248799109,1cyclictest0-21swapper/008:45:580
2248799108,1cyclictest5905-21apache_volume08:50:120
2248799108,1cyclictest25805-21users08:20:240
2248799105,5cyclictest0-21swapper/007:20:120
2248799101,9cyclictest10-21rcuc/008:30:200
2248799100,9cyclictest0-21swapper/008:25:110
2248799100,9cyclictest0-21swapper/008:10:140
2248799100,7cyclictest0-21swapper/008:15:220
2248799100,1cyclictest17595-21fschecks_time08:05:140
224989998,1cyclictest0-21swapper/307:20:133
224989997,1cyclictest9802-21apache_volume07:50:123
224989990,1cyclictest0-21swapper/307:15:183
224949990,9cyclictest0-21swapper/207:10:162
224949990,1cyclictest0-21swapper/208:35:142
224949990,1cyclictest0-21swapper/207:30:112
224899998,1cyclictest0-21swapper/108:15:461
224899990,6cyclictest0-21swapper/108:12:301
224879997,1cyclictest8702-21ntp_states07:45:200
224879994,1cyclictest0-21swapper/008:35:220
224879993,3cyclictest0-21swapper/008:40:190
224879992,6cyclictest41ktimersoftd/007:40:170
224989987,1cyclictest0-21swapper/307:12:433
224879980,4cyclictest24672-21munin-run07:15:000
224899976,1cyclictest0-21swapper/108:28:461
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional