You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:02

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rackaslot0.osadl.org (updated Sun Jan 23, 2022 12:44:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2165521060,0sleep221654-21bash09:52:212
184282900,0sleep318427-21bash09:49:023
99082890,0sleep09895-21sshd10:55:370
182562870,1sleep118234-21sshd12:04:021
182562870,1sleep118234-21sshd12:04:011
4392850,0sleep00-21swapper/011:34:430
42842700,1sleep31785-21runrttasks11:53:253
101782630,1sleep00-21swapper/009:20:350
957226025,30sleep30-21swapper/307:07:173
291632590,0sleep20-21swapper/212:23:242
101572570,1sleep010156-21sshd09:57:250
243112540,1sleep20-21swapper/211:30:452
150052540,0sleep30-21swapper/310:34:413
50022530,0sleep10-21swapper/110:39:521
82022520,0sleep18185-21sshd09:35:421
289412520,1sleep21785-21runrttasks11:43:322
215982520,1sleep221596-21sshd09:18:402
968325128,9sleep10-21swapper/107:08:341
108542510,0sleep20-21swapper/212:18:362
283652500,0sleep20-21swapper/211:06:372
265192500,0sleep010-21rcuc/011:09:350
262042500,1sleep026201-21sshd09:46:390
767024926,18sleep20-21swapper/207:05:192
303432490,0sleep31785-21runrttasks11:59:023
303432490,0sleep31785-21runrttasks11:59:023
160702490,0sleep10-21swapper/110:44:011
944824825,19sleep00-21swapper/007:05:410
200202450,0sleep30-21swapper/312:13:203
297092440,1sleep329701-21rm10:45:193
60292300,0sleep20-21swapper/209:32:302
304082300,0sleep00-21swapper/010:36:010
183552250,1sleep32874-21diskmemload10:10:263
10055992422,1cyclictest29636-21sshd12:23:281
10055992321,1cyclictest8681-21sshd10:49:311
1005799211,19cyclictest798-21id09:53:303
10054992119,1cyclictest27183-21sshd10:45:040
10054992018,1cyclictest1563-21sshd10:33:230
206242180,2sleep11005599cyclictest12:35:051
10057991816,1cyclictest0-21swapper/310:25:143
10056991816,1cyclictest0-21swapper/211:50:482
10055991814,2cyclictest21-21ksoftirqd/109:19:011
287222170,1sleep328720-21bash11:40:233
166332170,1sleep22874-21diskmemload10:28:432
10057991715,1cyclictest0-21swapper/312:38:103
1005599170,13cyclictest0-21swapper/110:15:361
1005499170,14cyclictest0-21swapper/012:08:320
1005499170,14cyclictest0-21swapper/012:08:320
1005499170,14cyclictest0-21swapper/010:52:350
62262160,0sleep10-21swapper/111:10:371
308992160,1sleep20-21swapper/210:20:462
112822160,0sleep30-21swapper/310:06:413
10055991614,1cyclictest21-21ksoftirqd/111:41:471
10055991614,1cyclictest0-21swapper/112:06:361
10055991614,1cyclictest0-21swapper/112:06:361
1005499160,1cyclictest0-21swapper/011:36:050
1005499160,13cyclictest2919-21bash11:16:280
180962150,0sleep20-21swapper/210:04:192
160672150,1sleep00-21swapper/009:14:450
115382150,0sleep10-21swapper/111:08:081
10057991513,1cyclictest11774-21bash09:57:343
10057991513,1cyclictest0-21swapper/309:30:453
10057991511,3cyclictest0-21swapper/312:26:133
1005799150,13cyclictest0-21swapper/312:23:533
1005699150,14cyclictest25002-21sshd10:17:112
1005599155,1cyclictest21-21ksoftirqd/109:25:231
1005499159,3cyclictest0-21swapper/010:41:000
1005499159,3cyclictest0-21swapper/010:09:570
1005499158,4cyclictest29963-21rm09:50:090
1005499155,9cyclictest2874-21diskmemload09:17:280
10054991511,3cyclictest25288-21munin-plugin-st12:20:010
1005499150,1cyclictest28769-21sshd10:14:330
1005499150,14cyclictest0-21swapper/012:35:410
1005499150,14cyclictest0-21swapper/010:04:450
1005499150,12cyclictest0-21swapper/012:10:250
1005499150,12cyclictest0-21swapper/009:29:010
274972140,0sleep10-21swapper/110:02:051
1005799140,11cyclictest0-21swapper/310:01:533
1005799140,10cyclictest0-21swapper/310:24:553
1005699144,9cyclictest30-21ksoftirqd/212:10:232
1005699140,13cyclictest0-21swapper/209:43:222
1005699140,11cyclictest20863-21sshd12:01:082
1005699140,11cyclictest20863-21sshd12:01:082
1005599144,9cyclictest21-21ksoftirqd/109:44:331
1005599144,1cyclictest21-21ksoftirqd/110:22:491
10055991412,2cyclictest27498-21kworker/1:211:23:381
10055991412,1cyclictest21-21ksoftirqd/112:16:211
10055991412,1cyclictest21-21ksoftirqd/112:14:531
10055991410,1cyclictest21-21ksoftirqd/111:02:021
1005599140,1cyclictest19-21rcuc/110:31:371
1005499149,4cyclictest0-21swapper/011:25:340
1005499148,3cyclictest19796-21bash12:01:010
1005499148,3cyclictest19796-21bash12:01:010
1005499146,7cyclictest0-21swapper/010:18:280
1005499142,1cyclictest0-21swapper/010:21:210
1005499142,10cyclictest0-21swapper/012:29:460
10054991412,1cyclictest0-21swapper/011:55:090
10054991412,1cyclictest0-21swapper/011:55:090
10054991410,1cyclictest0-21swapper/011:52:170
1005499140,3cyclictest3545-21sshd11:47:120
1005499140,2cyclictest29066-21sshd09:31:350
1005499140,1cyclictest12420-21cp11:20:270
1005499140,13cyclictest30700-21sshd11:40:310
1005499140,13cyclictest0-21swapper/012:31:190
1005799139,1cyclictest0-21swapper/312:33:103
1005799138,4cyclictest11053-21sshd09:29:583
1005799138,4cyclictest0-21swapper/310:56:273
1005799138,2cyclictest0-21swapper/311:12:083
1005799138,2cyclictest0-21swapper/309:40:103
1005799133,9cyclictest21260-21sshd09:21:383
10057991312,1cyclictest20623-21kworker/3:110:43:543
1005799131,1cyclictest119950irq/25-eth011:36:433
10057991311,1cyclictest14338-21sshd10:37:383
10057991311,1cyclictest0-21swapper/311:25:423
1005799131,11cyclictest0-21swapper/311:06:593
1005799131,11cyclictest0-21swapper/311:03:393
1005799131,11cyclictest0-21swapper/309:05:203
1005799131,11cyclictest0-21swapper/307:35:163
10057991310,2cyclictest13586-21sshd11:45:093
10057991310,1cyclictest119950irq/25-eth012:18:173
1005799130,12cyclictest24795-21bash12:01:303
1005799130,12cyclictest24795-21bash12:01:303
1005799130,12cyclictest0-21swapper/311:21:483
1005799130,12cyclictest0-21swapper/310:51:563
1005699138,4cyclictest6411-21sshd11:16:482
1005699138,4cyclictest32678-21sshd10:33:162
1005699138,4cyclictest0-21swapper/211:39:252
1005699138,2cyclictest1075-21sshd11:56:112
1005699138,2cyclictest1075-21sshd11:56:112
1005699137,1cyclictest0-21swapper/210:54:522
1005699131,1cyclictest10250irq/24-ahci[00010:43:012
10056991311,1cyclictest7333-21sshd10:46:142
1005699130,4cyclictest1631-21sshd09:10:372
1005699130,12cyclictest24354-21sshd11:03:132
1005699130,12cyclictest0-21swapper/212:30:472
1005699130,12cyclictest0-21swapper/212:07:352
1005699130,12cyclictest0-21swapper/212:07:352
1005699130,12cyclictest0-21swapper/210:55:492
1005699130,12cyclictest0-21swapper/210:35:492
1005699130,12cyclictest0-21swapper/210:05:562
1005699130,12cyclictest0-21swapper/209:25:242
1005699130,10cyclictest2248-21bash12:36:192
1005599139,1cyclictest0-21swapper/109:30:371
1005599138,4cyclictest13680-21fschecks_count07:15:221
1005599138,2cyclictest15540-21bash09:45:351
1005599138,2cyclictest0-21swapper/111:45:541
1005599137,4cyclictest10250irq/24-ahci[00011:25:381
1005599133,1cyclictest21-21ksoftirqd/112:29:131
1005599133,10cyclictest18952-21kworker/1:211:52:361
1005599131,9cyclictest10250irq/24-ahci[00010:05:011
10055991312,1cyclictest21-21ksoftirqd/110:51:141
1005599131,1cyclictest119950irq/25-eth009:58:391
10055991311,1cyclictest21-21ksoftirqd/110:10:301
1005599131,11cyclictest0-21swapper/110:25:201
10055991310,2cyclictest252812chrt11:55:231
10055991310,2cyclictest252812chrt11:55:231
10055991310,1cyclictest0-21swapper/111:36:451
1005599130,12cyclictest8181-21sshd12:30:431
1005599130,12cyclictest18657-21sshd09:21:241
1005599130,12cyclictest0-21swapper/111:30:441
1005599130,12cyclictest0-21swapper/111:16:211
1005599130,12cyclictest0-21swapper/109:54:251
1005599130,10cyclictest0-21swapper/110:55:141
1005599130,10cyclictest0-21swapper/109:10:061
1005499139,3cyclictest0-21swapper/011:00:100
1005499138,2cyclictest6365-21sshd11:10:370
1005499131,1cyclictest8126-21bash09:35:420
1005499130,4cyclictest24926-21sshd09:40:260
1005499130,1cyclictest0-21swapper/010:28:010
1005499130,11cyclictest0-21swapper/012:24:410
168172120,0sleep10-21swapper/107:20:281
168172120,0sleep10-21swapper/107:20:281
1005799129,2cyclictest0-21swapper/312:06:023
1005799129,2cyclictest0-21swapper/312:06:023
1005799129,2cyclictest0-21swapper/311:30:053
1005799129,2cyclictest0-21swapper/310:15:363
1005799129,2cyclictest0-21swapper/309:15:023
1005799129,2cyclictest0-21swapper/309:10:063
1005799129,2cyclictest0-21swapper/308:50:183
1005799129,2cyclictest0-21swapper/308:25:343
1005799129,2cyclictest0-21swapper/308:15:223
1005799129,2cyclictest0-21swapper/307:40:153
1005799122,9cyclictest10782-21sshd09:39:073
1005799120,1cyclictest0-21swapper/311:16:163
1005699129,2cyclictest0-21swapper/212:25:252
1005699129,2cyclictest0-21swapper/211:20:172
1005699129,2cyclictest0-21swapper/211:10:082
1005699129,2cyclictest0-21swapper/209:55:322
1005699129,2cyclictest0-21swapper/209:20:392
1005699129,2cyclictest0-21swapper/208:55:182
1005699129,2cyclictest0-21swapper/208:30:232
1005699127,4cyclictest0-21swapper/210:10:182
1005699127,4cyclictest0-21swapper/209:35:342
1005699121,2cyclictest0-21swapper/208:20:222
1005699121,2cyclictest0-21swapper/207:55:212
1005699120,1cyclictest0-21swapper/209:45:032
1005699120,11cyclictest0-21swapper/211:48:152
1005699120,11cyclictest0-21swapper/211:25:582
1005599129,2cyclictest0-21swapper/108:55:211
1005599129,2cyclictest0-21swapper/108:15:181
1005599129,2cyclictest0-21swapper/108:10:191
1005599129,2cyclictest0-21swapper/108:00:171
1005599129,2cyclictest0-21swapper/107:40:121
1005599121,10cyclictest0-21swapper/107:45:171
1005499129,2cyclictest0-21swapper/008:55:270
1005499128,3cyclictest1917-21latency_hist08:00:010
1005499121,2cyclictest0-21swapper/008:50:220
1005499121,2cyclictest0-21swapper/008:05:220
1005499121,2cyclictest0-21swapper/007:30:180
1005499121,10cyclictest0-21swapper/007:40:140
1005499120,12cyclictest0-21swapper/007:20:250
1005499120,12cyclictest0-21swapper/007:20:240
1005499120,11cyclictest0-21swapper/008:10:190
1005499120,11cyclictest0-21swapper/008:00:170
1005799119,1cyclictest0-21swapper/307:50:163
1005799119,1cyclictest0-21swapper/307:25:163
1005799118,2cyclictest0-21swapper/308:55:203
1005799118,2cyclictest0-21swapper/308:40:293
1005799118,2cyclictest0-21swapper/308:00:213
1005799110,10cyclictest0-21swapper/308:30:183
1005799110,10cyclictest0-21swapper/308:10:233
1005799110,10cyclictest0-21swapper/308:05:283
1005799110,10cyclictest0-21swapper/307:15:023
1005699119,1cyclictest0-21swapper/207:50:162
1005699118,2cyclictest0-21swapper/209:05:212
1005699110,2cyclictest0-21swapper/207:15:012
1005699110,10cyclictest0-21swapper/208:50:272
1005699110,10cyclictest0-21swapper/208:00:212
1005699110,10cyclictest0-21swapper/207:35:252
1005599119,1cyclictest0-21swapper/108:50:201
1005599118,2cyclictest0-21swapper/108:25:191
1005599118,2cyclictest0-21swapper/107:35:141
1005599110,10cyclictest29806-21apache_processe07:50:131
1005599110,10cyclictest0-21swapper/109:05:181
1005599110,10cyclictest0-21swapper/107:55:241
1005599110,10cyclictest0-21swapper/107:25:161
1005499119,1cyclictest0-21swapper/007:50:010
1005499110,9cyclictest0-21swapper/008:45:310
1005499110,8cyclictest14256-21grep07:15:290
1005799108,1cyclictest24042-21df_inode08:45:243
1005799108,1cyclictest10086-21apache_volume07:10:173
1005799100,9cyclictest0-21swapper/308:35:193
1005799100,9cyclictest0-21swapper/307:55:253
1005799100,1cyclictest16118-21fschecks_count07:20:213
1005799100,1cyclictest16118-21fschecks_count07:20:213
1005699108,1cyclictest14072-21apache_processe08:25:182
1005699108,1cyclictest0-21swapper/207:15:012
1005699100,9cyclictest6937-21apache_processe08:10:192
1005699100,9cyclictest5689-21nfsd408:05:282
1005699100,9cyclictest31520-21fw_conntrack09:00:232
1005699100,9cyclictest27858-21df_inode07:45:152
1005699100,9cyclictest18839-21apache_volume08:35:182
1005699100,9cyclictest0-21swapper/208:40:232
1005699100,9cyclictest0-21swapper/207:20:172
1005699100,9cyclictest0-21swapper/207:20:172
1005599109,1cyclictest0-21swapper/108:40:251
1005599108,1cyclictest10867-21conntrack07:10:201
1005599100,1cyclictest5639-21ls08:05:271
1005599100,1cyclictest21072-21fw_conntrack07:30:211
1005599100,1cyclictest12004-21cstates08:20:211
1005599100,1cyclictest0-21swapper/108:30:221
1005499109,1cyclictest0-21swapper/008:41:520
1005499108,1cyclictest11701-21apache_processe08:20:190
1005499107,2cyclictest0-21swapper/008:30:180
1005499107,2cyclictest0-21swapper/007:25:150
1005499100,9cyclictest14716-21fschecks_time08:25:220
1005499100,9cyclictest0-21swapper/009:00:210
1005499100,9cyclictest0-21swapper/007:50:200
1005499100,1cyclictest19572-21fschecks_time08:35:230
1005499100,1cyclictest10096-21fw_conntrack08:15:220
1005499100,1cyclictest0-21swapper/007:35:150
1005499100,10cyclictest0-21swapper/007:11:290
100579998,1cyclictest0-21swapper/308:20:223
100579998,1cyclictest0-21swapper/307:45:243
100579998,1cyclictest0-21swapper/307:30:263
100579997,1cyclictest630-21latency_hist09:05:003
100569998,1cyclictest0-21swapper/207:30:262
100569998,1cyclictest0-21swapper/207:25:162
100569990,9cyclictest0-21swapper/208:15:182
100569990,1cyclictest0-21swapper/207:40:122
100559998,1cyclictest19-21rcuc/109:00:221
100559998,1cyclictest19218-21df08:35:191
100559996,3cyclictest25053-21kworker/1:108:45:201
100549997,1cyclictest0-21swapper/009:05:000
100569976,1cyclictest0-21swapper/208:46:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional