You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:37

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #4, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack4slot0.osadl.org (updated Sun Jan 23, 2022 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1215621190,1sleep225839-21nfsd11:08:362
2281921040,0sleep10-21swapper/110:04:541
95042760,1sleep09501-21lspci12:36:340
296452760,0sleep0101ktimersoftd/011:57:310
308222750,0sleep3391rcuc/311:15:513
146352750,0sleep30-21swapper/312:00:063
196582740,0sleep10-21swapper/109:12:241
153392740,1sleep015337-21sshd09:40:290
32212720,1sleep13218-21sshd09:48:031
280712710,0sleep10-21swapper/110:28:591
141652650,1sleep225840-21nfsd10:36:172
38202620,2sleep22003199cyclictest12:35:442
1808125823,10sleep20-21swapper/207:05:042
1952325525,9sleep10-21swapper/107:06:401
189792540,0sleep20-21swapper/210:18:182
286312530,0sleep20-21swapper/211:10:532
1944824723,17sleep30-21swapper/307:05:503
1961424530,9sleep00-21swapper/007:07:440
25242390,1sleep341-21ksoftirqd/312:30:533
2003099258,2cyclictest0-21swapper/110:55:481
2003099258,2cyclictest0-21swapper/110:46:121
2003099258,2cyclictest0-21swapper/110:46:121
290512240,0sleep30-21swapper/309:47:053
2003799233,2cyclictest27732-21lspci09:19:013
2003799233,2cyclictest0-21swapper/308:30:413
2003799233,2cyclictest0-21swapper/307:59:463
2003099233,2cyclictest25877-21lspci10:05:201
2003799222,2cyclictest0-21swapper/309:35:283
2003799222,2cyclictest0-21swapper/309:35:283
2003799222,2cyclictest0-21swapper/307:49:343
2003099229,2cyclictest21336-21sshd09:41:191
37062210,0sleep20-21swapper/210:06:442
2003799212,3cyclictest25732-21nscd10:26:093
2003799212,2cyclictest0-21swapper/310:45:003
2003799211,2cyclictest0-21swapper/311:55:413
2003799211,2cyclictest0-21swapper/308:21:123
2003799211,2cyclictest0-21swapper/308:00:463
2003799211,2cyclictest0-21swapper/307:12:553
2003799210,3cyclictest0-21swapper/312:05:293
2003799210,3cyclictest0-21swapper/311:37:363
2003799210,3cyclictest0-21swapper/310:48:543
2003799210,3cyclictest0-21swapper/310:48:543
2003799210,3cyclictest0-21swapper/309:50:173
2003799210,3cyclictest0-21swapper/308:51:033
2003799210,3cyclictest0-21swapper/308:43:453
2003799210,2cyclictest0-21swapper/311:27:013
2003799210,1cyclictest0-21swapper/310:50:213
2003799210,1cyclictest0-21swapper/310:50:213
2003099213,2cyclictest0-21swapper/108:38:451
2003099212,4cyclictest0-21swapper/107:48:251
2003099210,3cyclictest18172-21sshd10:50:481
2003099210,3cyclictest18172-21sshd10:50:481
2003099210,3cyclictest0-21swapper/112:30:181
2003099210,3cyclictest0-21swapper/111:51:111
2003099210,3cyclictest0-21swapper/108:07:321
2003099210,2cyclictest23752-21munin-run12:20:001
2003799202,1cyclictest0-21swapper/312:15:013
2003799202,0cyclictest0-21swapper/307:31:303
2003799200,2cyclictest5880-21sshd11:30:493
2003799200,2cyclictest22576-21lspci10:23:323
2003799200,2cyclictest0-21swapper/308:18:203
2003799200,2cyclictest0-21swapper/307:54:253
2003799200,2cyclictest0-21swapper/307:23:053
2003099201,1cyclictest10463-21lspci12:36:431
2003099200,2cyclictest0-21swapper/112:24:211
2003099200,2cyclictest0-21swapper/109:58:141
2003099200,2cyclictest0-21swapper/109:34:441
2003099200,2cyclictest0-21swapper/108:58:271
2003099200,2cyclictest0-21swapper/107:31:451
2003799192,2cyclictest24765-21lspci12:29:303
2003799192,2cyclictest0-21swapper/312:24:283
2003799192,2cyclictest0-21swapper/311:01:453
2003799191,2cyclictest25817-21lspci11:52:173
2003799191,2cyclictest0-21swapper/308:45:483
2003799191,2cyclictest0-21swapper/307:36:473
2003799191,0cyclictest0-21swapper/307:25:413
2003799190,4cyclictest0-21swapper/308:14:393
2003799190,4cyclictest0-21swapper/308:14:393
2003799190,3cyclictest0-21swapper/310:01:203
2003799190,1cyclictest0-21swapper/309:40:463
2003799190,1cyclictest0-21swapper/309:31:463
2003799190,1cyclictest0-21swapper/309:21:433
2003799190,1cyclictest0-21swapper/309:09:523
2003099192,4cyclictest16455-21sshd10:13:171
2003099190,4cyclictest0-21swapper/111:13:401
2003099190,3cyclictest17422-21sshd11:32:321
2003099190,1cyclictest0-21swapper/112:08:521
2003099190,1cyclictest0-21swapper/111:26:161
2003099190,1cyclictest0-21swapper/111:16:041
2003099190,1cyclictest0-21swapper/111:06:581
2003099190,1cyclictest0-21swapper/110:33:001
2003099190,1cyclictest0-21swapper/110:24:261
2003099190,1cyclictest0-21swapper/109:24:371
2003099190,1cyclictest0-21swapper/109:19:041
2003099190,1cyclictest0-21swapper/108:47:001
2003099190,1cyclictest0-21swapper/107:15:541
52212180,0sleep00-21swapper/010:16:160
2003799181,2cyclictest0-21swapper/311:21:013
2003799180,3cyclictest0-21swapper/310:05:023
2003799180,1cyclictest0-21swapper/310:17:143
2003799180,1cyclictest0-21swapper/308:38:543
2003799180,0cyclictest0-21swapper/311:13:553
2003799180,0cyclictest0-21swapper/310:10:333
2003799180,0cyclictest0-21swapper/309:29:223
2003799180,0cyclictest0-21swapper/308:28:093
2003799180,0cyclictest0-21swapper/308:08:413
2003799180,0cyclictest0-21swapper/307:43:263
2003099183,2cyclictest0-21swapper/108:03:381
2003099180,3cyclictest25343-21lspci10:19:141
2003099180,3cyclictest0-21swapper/111:57:061
2003099180,2cyclictest22803-21sshd12:01:121
2003099180,1cyclictest0-21swapper/111:41:441
2003099180,1cyclictest0-21swapper/107:56:531
2003099180,1cyclictest0-21swapper/107:21:341
2003099180,0cyclictest0-21swapper/109:00:211
100192180,0sleep20-21swapper/212:27:202
58922170,2sleep0101ktimersoftd/010:11:430
204002170,0sleep20-21swapper/209:17:542
2003799173,2cyclictest0-21swapper/309:00:483
2003199178,8cyclictest24283-21kworker/2:111:57:342
2003199170,16cyclictest10342-21kworker/2:012:18:332
2003099170,4cyclictest0-21swapper/109:51:141
2003099170,2cyclictest6954-21bash09:39:191
2003099170,2cyclictest6954-21bash09:39:191
2003099170,2cyclictest0-21swapper/109:08:101
2003099170,2cyclictest0-21swapper/108:30:591
2003099170,2cyclictest0-21swapper/107:43:551
2002999170,15cyclictest20134-21kworker/0:311:14:170
80922160,0sleep00-21swapper/011:40:280
66792160,0sleep00-21swapper/010:07:100
2003799160,1cyclictest9301-21sshd09:58:173
2003799160,1cyclictest12845-21lspci10:36:063
2003799160,1cyclictest0-21swapper/311:47:383
2003799160,1cyclictest0-21swapper/310:31:363
2003199160,15cyclictest6826-21kworker/2:310:58:472
2003199160,15cyclictest23769-21kworker/2:312:22:032
2003199160,15cyclictest19718-21kworker/2:011:28:522
2003199160,15cyclictest15456-21kworker/2:209:46:582
2003099161,1cyclictest9560-21lspci12:27:151
2003099160,2cyclictest0-21swapper/107:36:411
2003099160,1cyclictest21105-21lspci12:10:201
2003099160,1cyclictest1850-21sshd11:20:581
2003099160,1cyclictest15597-21sshd09:26:311
2003099160,1cyclictest0-21swapper/111:02:571
2003099160,1cyclictest0-21swapper/110:42:371
2003099160,1cyclictest0-21swapper/108:50:391
2003099160,1cyclictest0-21swapper/108:16:281
2003099160,1cyclictest0-21swapper/107:54:101
20029991614,1cyclictest16244-21sshd10:57:260
2002999160,15cyclictest4697-21kworker/0:209:14:360
2003799151,2cyclictest0-21swapper/312:17:273
2003799150,2cyclictest0-21swapper/311:42:263
2003799150,2cyclictest0-21swapper/311:06:523
2003799150,1cyclictest20274-21sshd12:38:103
2003799150,1cyclictest11245-21sshd09:10:583
2003799150,0cyclictest0-21swapper/308:58:003
2003199157,7cyclictest26533-21kworker/2:311:53:172
2003199156,8cyclictest6826-21kworker/2:311:23:512
20031991514,0cyclictest722-40kipmi010:10:222
20031991513,1cyclictest18602-21sshd11:46:362
20031991513,1cyclictest18245-21bash10:22:542
2003199150,14cyclictest3309-21kworker/2:210:30:312
2003199150,14cyclictest25839-21nfsd11:35:462
2003199150,14cyclictest21444-21kworker/2:109:56:242
2003099150,3cyclictest7059-21lspci11:49:381
2002999155,9cyclictest30522-21sshd09:33:220
20029991513,1cyclictest14157-21lspci12:09:200
20029991513,1cyclictest11300-21sshd12:04:170
2002999150,6cyclictest25074-21kworker/0:009:57:130
2002999150,1cyclictest121rcu_preempt10:40:480
2002999150,1cyclictest0-21swapper/012:12:310
2002999150,14cyclictest32446-21kworker/0:011:34:080
2002999150,14cyclictest26149-21kworker/0:012:29:470
2002999150,14cyclictest15951-21kworker/0:112:19:240
2003199144,1cyclictest25840-21nfsd11:41:392
2003199144,1cyclictest21181-21bash10:04:402
20031991413,0cyclictest321ktimersoftd/212:10:382
20031991412,1cyclictest6326-21sshd11:30:532
20031991412,1cyclictest33-21ksoftirqd/212:30:512
20031991412,1cyclictest16244-21sshd09:52:252
2003199140,13cyclictest24283-21kworker/2:112:01:502
20030991412,1cyclictest31811-21sshd10:38:531
2002999148,1cyclictest121rcu_preempt09:46:540
2002999146,7cyclictest5490-21kworker/0:010:38:120
2002999144,1cyclictest11693-21bash11:50:190
20029991413,0cyclictest0-21swapper/011:35:460
20029991412,1cyclictest16244-21sshd10:52:110
20029991412,1cyclictest16244-21sshd10:52:100
2002999140,5cyclictest15951-21kworker/0:112:22:030
2003199137,1cyclictest18919-21lspci10:41:382
2003199133,1cyclictest33-21ksoftirqd/209:28:102
2003199132,10cyclictest25991-21sshd09:23:232
20031991311,1cyclictest33-21ksoftirqd/211:05:002
20031991311,1cyclictest321ktimersoftd/212:07:572
2003199131,10cyclictest23910-21grep10:51:392
2003199131,10cyclictest23910-21grep10:51:392
2002999136,6cyclictest28399-21kworker/0:111:29:020
20029991311,1cyclictest37650irq/25-ahci[00010:25:260
20029991311,1cyclictest22955-21rt-features11:05:280
20029991311,1cyclictest101ktimersoftd/010:46:290
20029991311,1cyclictest101ktimersoftd/010:46:290
20029991311,1cyclictest0-21swapper/011:49:170
2002999130,1cyclictest13835-21lspci09:16:430
2002999130,1cyclictest0-21swapper/010:21:530
179582130,0sleep00-21swapper/011:00:120
2003799120,2cyclictest14705-21lspci10:55:033
2003799120,0cyclictest0-21swapper/307:15:303
2003199122,1cyclictest25415-21if_enp2s008:40:232
2003199121,9cyclictest29084-21sed08:50:012
2003199120,2cyclictest5513-21sshd09:10:102
2003199120,2cyclictest4484-21sshd10:25:312
2003199120,2cyclictest31934-21switchtime07:35:232
2003199120,2cyclictest24656-21ksmtuned07:20:082
2003199120,2cyclictest18015-21grepconf.sh10:46:102
2003199120,2cyclictest18015-21grepconf.sh10:46:102
2003199120,2cyclictest14822-21grep09:35:422
2003199120,2cyclictest14822-21grep09:35:422
2003199120,2cyclictest11579-21sshd09:30:332
2003199120,2cyclictest10492-21latency_hist08:05:002
2003199120,10cyclictest7058-21idleruntime07:55:242
2003199120,10cyclictest29669-21grep11:15:402
2003199120,10cyclictest21420-21sed07:10:292
2003199120,10cyclictest20873-21grep08:30:012
2003199120,10cyclictest12759-21/usr/sbin/munin08:10:202
2003199120,10cyclictest12759-21/usr/sbin/munin08:10:202
2003199120,10cyclictest12195-21grepconf.sh09:40:072
2003199120,10cyclictest1101-21runrttasks07:40:162
2003099120,1cyclictest0-21swapper/108:28:361
2003099120,1cyclictest0-21swapper/108:20:261
2003099120,1cyclictest0-21swapper/107:29:241
2003099120,10cyclictest8143-21sshd11:35:471
2003099120,10cyclictest20536-21hddtemp_smartct07:10:191
2003099120,10cyclictest0-21swapper/108:10:191
2003099120,10cyclictest0-21swapper/108:10:191
2002999128,3cyclictest0-21swapper/012:30:510
2002999128,1cyclictest24117-21bash10:33:050
2002999121,10cyclictest29107-21sshd11:24:570
20029991210,1cyclictest9-21ksoftirqd/010:00:370
20029991210,1cyclictest37650irq/25-ahci[00009:51:010
20029991210,1cyclictest19442-21tty09:36:230
20029991210,1cyclictest19442-21tty09:36:230
20029991210,1cyclictest18812-21kworker/0:208:30:230
2002999120,1cyclictest17036-21sshd09:26:440
2002999120,11cyclictest23898-21bash09:23:050
2003199119,1cyclictest25306-21gdbus08:30:012
2003199118,2cyclictest4637-21cpuspeed07:50:132
2003199118,2cyclictest3783-21timerwakeupswit07:45:222
2003199118,2cyclictest29427-21df_inode08:50:192
2003199118,2cyclictest23172-21cstates08:35:182
2003199110,2cyclictest11119-21basename08:05:202
2003199110,10cyclictest29927-21timerandwakeup07:30:292
2003199110,10cyclictest28661-21cat07:30:012
2003199110,10cyclictest23278-21missed_timers07:15:252
2003199110,10cyclictest1949-21timerandwakeup09:00:312
2003099119,1cyclictest25248-21cstates08:40:201
2002999119,1cyclictest10492-21latency_hist08:05:000
2002999119,1cyclictest0-21swapper/008:40:350
2002999119,1cyclictest0-21swapper/007:30:390
2002999111,1cyclictest25838-21nfsd07:40:270
2002999110,2cyclictest28550-21munin-run07:30:010
2002999110,10cyclictest7781-21cat07:55:330
2002999110,10cyclictest30412-21bash11:15:470
2002999110,10cyclictest23310-21missed_timers07:15:250
2003199108,1cyclictest0-21swapper/208:15:192
2003199100,9cyclictest31441-21hddtemp_smartct08:55:202
2003199100,9cyclictest0-21swapper/209:05:012
2003199100,9cyclictest0-21swapper/208:20:232
2002999108,1cyclictest31663-21ls08:55:220
2002999108,1cyclictest0-21swapper/009:05:230
2002999108,1cyclictest0-21swapper/007:50:130
2002999107,1cyclictest0-21swapper/008:50:190
2002999100,9cyclictest2900-21df_inode07:45:140
2002999100,9cyclictest27102-21cpuspeed_turbos08:45:210
2002999100,9cyclictest25833-21nfsd08:25:040
2002999100,9cyclictest0-21swapper/009:00:190
2002999100,9cyclictest0-21swapper/008:35:190
2002999100,9cyclictest0-21swapper/008:15:190
2002999100,9cyclictest0-21swapper/008:10:240
2002999100,9cyclictest0-21swapper/008:10:240
2002999100,9cyclictest0-21swapper/008:05:200
2002999100,9cyclictest0-21swapper/007:10:130
2002999100,1cyclictest31115-21cat07:35:140
2002999100,1cyclictest0-21swapper/008:20:360
2002999100,1cyclictest0-21swapper/007:20:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional