You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-05-12 - 14:29

Intel(R) Xeon(R) CPU E31220L @ 2.20GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack5slot0.osadl.org (updated Wed May 12, 2021 12:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305182980,2sleep10-21swapper/110:35:101
89052940,2sleep10-21swapper/109:17:121
300622930,1sleep20-21swapper/211:36:312
74482920,3sleep10-21swapper/111:19:351
242292900,6sleep30-21swapper/310:05:273
112382900,1sleep20-21swapper/209:56:532
23802860,0sleep00-21swapper/012:24:260
130382860,1sleep013027-21sshd12:07:180
1823228532,49sleep10-21swapper/107:07:251
49422830,1sleep30-21swapper/309:20:183
326272830,2sleep132614-21sshd11:04:151
64512820,2sleep10-21swapper/110:14:231
1843528133,44sleep30-21swapper/307:09:473
78792800,1sleep27867-21sshd11:12:192
63332790,1sleep00-21swapper/011:23:030
45972790,1sleep30-21swapper/309:38:143
25152790,1sleep02121-21runrttasks10:28:250
185452780,2sleep30-21swapper/310:59:053
107282780,1sleep10-21swapper/110:22:041
202632770,2sleep30-21swapper/311:49:583
323892750,0sleep00-21swapper/010:56:580
314572750,1sleep10-21swapper/112:24:031
133522740,1sleep10-21swapper/109:57:071
314722730,2sleep30-21swapper/309:55:293
225522670,1sleep322554-21bash11:53:513
216752660,2sleep221674-21sshd11:31:572
315242650,2sleep331526-21bash11:22:123
1840525932,4sleep00-21swapper/007:09:240
1836025733,12sleep20-21swapper/207:08:502
111482330,31sleep3138550irq/27-eth010:40:143
37382320,2sleep23731-21sshd10:39:252
80382290,2sleep20-21swapper/209:38:372
82032280,26sleep1138550irq/27-eth011:12:221
222362280,3sleep10-21swapper/112:15:331
116262280,2sleep30-21swapper/309:28:163
312032260,3sleep20-21swapper/212:13:012
254752260,4sleep21870499cyclictest09:11:522
23422260,2sleep20-21swapper/211:40:362
189382260,24sleep1138550irq/27-eth010:48:181
29972250,2sleep22996-21nfsd410:10:182
258252250,4sleep31987-21sshd09:51:173
1870499250,10cyclictest138550irq/27-eth012:04:132
299522240,3sleep20-21swapper/212:34:542
299522240,3sleep20-21swapper/212:34:542
239872240,2sleep30-21swapper/311:35:473
1870499240,11cyclictest11088-21bash09:46:012
38312230,2sleep23829-21sshd09:16:372
1870499234,9cyclictest20369-21sshd10:41:102
1870499233,9cyclictest22439-21rm09:25:482
1870499231,10cyclictest0-21swapper/210:00:302
1870499230,13cyclictest138550irq/27-eth011:02:332
1870499230,11cyclictest1035-21sshd12:16:522
1870499228,12cyclictest16011-21sshd09:50:132
1870499223,17cyclictest26794-21sshd11:25:162
1870499221,9cyclictest138550irq/27-eth012:09:412
1870499221,9cyclictest138550irq/27-eth009:23:422
1870499221,13cyclictest32659-21sshd11:22:202
1870499221,11cyclictest8244-21sshd10:07:202
1870499221,11cyclictest30422-21sshd10:45:522
1870499220,20cyclictest7560-21sshd10:54:142
1870499220,20cyclictest31589-21bash09:30:242
1870499220,20cyclictest27747-21sshd11:07:182
1870499220,13cyclictest445-21jbd2/dm-0-810:23:302
1870499220,13cyclictest24893-21missed_timers09:40:262
1870499220,12cyclictest22384-21sshd10:30:312
95062210,2sleep10-21swapper/112:28:551
67732210,0sleep30-21swapper/311:15:433
1870499211,9cyclictest138550irq/27-eth011:45:112
1870499211,15cyclictest138550irq/27-eth008:28:152
1870499211,0cyclictest0-21swapper/210:27:192
1870499210,19cyclictest6964-21sshd12:28:372
1870499210,19cyclictest15808-21sshd10:15:222
1870499210,18cyclictest445-21jbd2/dm-0-810:55:212
1870499210,12cyclictest5998-21sshd11:19:242
98612200,18sleep3138550irq/27-eth010:47:153
91632200,1sleep00-21swapper/011:16:000
296702200,1sleep00-21swapper/011:07:310
1870499200,18cyclictest9279-21sshd11:52:192
1870499200,18cyclictest15129-21chrt08:12:232
1869599200,1cyclictest27169-21cp11:25:190
91962190,2sleep10-21swapper/112:14:131
1870899190,13cyclictest23013-21idleruntime-cro07:20:003
1870499198,9cyclictest1877-21nfsd12:20:132
1870499193,9cyclictest19147-21sshd11:57:032
1870499190,18cyclictest1877-21nfsd08:15:472
1870499190,17cyclictest7472-21chrt09:06:552
1870499190,17cyclictest3462-21munin-node09:00:122
1870499190,17cyclictest29962-21munin-node07:35:202
1870499190,17cyclictest16885-21sshd12:37:002
1870499190,10cyclictest1877-21nfsd07:30:202
1869599197,10cyclictest3292-21sshd09:30:490
18695991915,3cyclictest0-21swapper/011:12:280
1869599190,1cyclictest4884-21bash09:45:230
1869599190,1cyclictest13612-21sshd10:07:570
1869599190,1cyclictest0-21swapper/012:00:430
1870899180,6cyclictest0-21swapper/312:10:133
1870899180,14cyclictest11133-21sshd10:54:393
1870499180,9cyclictest0-21swapper/208:45:122
1870499180,1cyclictest0-21swapper/208:35:492
1870499180,1cyclictest0-21swapper/207:56:142
1870499180,1cyclictest0-21swapper/207:14:472
1870499180,17cyclictest27429-21taskset07:29:242
1870499180,17cyclictest0-21swapper/208:30:492
1869999186,6cyclictest22860-21sshd11:24:561
1869999180,0cyclictest0-21swapper/109:40:251
1869599181,15cyclictest24869-21memory09:40:260
1869599180,1cyclictest5627-21sshd09:16:490
1869599180,1cyclictest0-21swapper/010:01:170
1869599180,17cyclictest0-21swapper/010:16:570
1869599180,15cyclictest0-21swapper/007:40:080
1869599180,12cyclictest0-21swapper/010:24:500
1869599180,10cyclictest839-21sshd10:46:120
133192180,2sleep1138550irq/27-eth009:46:171
65872170,15sleep1138550irq/27-eth010:03:371
259082170,15sleep3138550irq/27-eth012:23:263
246862170,15sleep328750irq/26-ahci[00011:32:203
201752170,15sleep1138550irq/27-eth010:05:111
1870899171,11cyclictest22678-21bash11:57:293
1870899170,13cyclictest1876-21nfsd10:36:243
1870499170,9cyclictest0-21swapper/208:20:052
1870499170,8cyclictest0-21swapper/208:55:162
1870499170,8cyclictest0-21swapper/208:05:382
1870499170,2cyclictest0-21swapper/207:40:172
1870499170,1cyclictest0-21swapper/208:41:002
1870499170,1cyclictest0-21swapper/208:00:112
1870499170,1cyclictest0-21swapper/207:20:262
1870499170,1cyclictest0-21swapper/207:15:132
1870499170,16cyclictest8142sleep208:54:112
1870499170,16cyclictest0-21swapper/207:45:392
1870499170,15cyclictest5195-21memory07:50:172
1869599177,9cyclictest1877-21nfsd09:36:520
1869599173,6cyclictest10156-21bash12:25:160
1869599173,5cyclictest0-21swapper/011:35:250
1869599171,2cyclictest29844-21sshd09:55:210
1869599171,1cyclictest0-21swapper/012:12:040
18695991711,5cyclictest0-21swapper/010:50:280
1869599170,9cyclictest0-21swapper/008:45:340
1869599170,1cyclictest24623-21sshd09:51:070
1869599170,1cyclictest0-21swapper/011:02:350
1869599170,1cyclictest0-21swapper/010:30:470
1869599170,1cyclictest0-21swapper/010:11:190
1869599170,1cyclictest0-21swapper/009:22:050
1869599170,1cyclictest0-21swapper/008:40:530
5732160,14sleep1138550irq/27-eth009:23:301
308992160,14sleep1138550irq/27-eth010:27:581
1870899168,3cyclictest0-21swapper/310:11:233
18708991613,2cyclictest7355-21sshd11:26:443
18708991611,3cyclictest16808-21bash12:15:043
1870899160,15cyclictest445-21jbd2/dm-0-810:30:203
1870899160,14cyclictest24977-21bash11:10:303
1870899160,12cyclictest138550irq/27-eth011:00:153
1870899160,11cyclictest18173-21bash12:00:303
1869999164,4cyclictest3401-21bash09:34:321
1869999164,3cyclictest13060-21sshd11:38:161
1869999163,5cyclictest0-21swapper/110:41:561
1869999162,6cyclictest5666-21bash11:26:311
1869999160,9cyclictest26073-21sshd12:05:111
1869999160,9cyclictest15501-21sshd11:53:011
1869599162,12cyclictest0-21swapper/010:40:180
1869599161,14cyclictest0-21swapper/011:33:060
18695991610,5cyclictest0-21swapper/007:10:350
1869599160,1cyclictest0-21swapper/011:57:170
1869599160,1cyclictest0-21swapper/009:00:030
1869599160,1cyclictest0-21swapper/008:36:130
1869599160,1cyclictest0-21swapper/008:11:460
1869599160,15cyclictest0-21swapper/010:36:430
1869599160,15cyclictest0-21swapper/008:05:040
1869599160,12cyclictest8811-21sshd11:48:390
1869599160,12cyclictest0-21swapper/007:55:150
1869599160,11cyclictest0-21swapper/012:15:260
1869599160,10cyclictest0-21swapper/007:50:000
211882150,2sleep10-21swapper/109:11:241
1870899157,3cyclictest0-21swapper/309:47:323
1870899156,1cyclictest0-21swapper/309:41:483
18708991511,3cyclictest28194-21sshd11:43:353
18708991511,3cyclictest1769-21bash12:31:393
18708991511,3cyclictest1769-21bash12:31:383
18708991511,3cyclictest138550irq/27-eth009:15:203
1870899150,4cyclictest25262-21sshd10:23:443
1870899150,14cyclictest1876-21nfsd09:31:453
1870899150,14cyclictest138550irq/27-eth009:12:383
1870899150,13cyclictest0-21swapper/307:53:323
1870899150,10cyclictest0-21swapper/310:15:233
1870899150,10cyclictest0-21swapper/310:03:023
1869999154,3cyclictest17651-21id10:30:071
1869999153,4cyclictest0-21swapper/112:30:321
1869999153,4cyclictest0-21swapper/112:30:311
1869999153,3cyclictest18972-21sshd09:25:241
1869999153,3cyclictest0-21swapper/111:34:521
1869999152,6cyclictest0-21swapper/111:06:141
1869999152,4cyclictest23698-21sshd09:51:011
1869999151,5cyclictest26938-21sshd10:56:191
1869599158,6cyclictest0-21swapper/007:30:140
1869599157,7cyclictest0-21swapper/008:02:060
1869599152,12cyclictest31775-21sshd11:51:100
1869599151,12cyclictest0-21swapper/011:40:170
1869599151,12cyclictest0-21swapper/009:25:260
1869599151,12cyclictest0-21swapper/009:10:430
1869599150,9cyclictest0-21swapper/012:30:060
1869599150,9cyclictest0-21swapper/012:30:060
1869599150,9cyclictest0-21swapper/008:15:150
1869599150,14cyclictest0-21swapper/012:35:260
1869599150,14cyclictest0-21swapper/009:05:340
1869599150,14cyclictest0-21swapper/008:32:440
1869599150,14cyclictest0-21swapper/008:20:030
1869599150,14cyclictest0-21swapper/007:52:050
326782140,2sleep30-21swapper/312:05:493
18708991411,2cyclictest0-21swapper/307:38:143
18708991410,3cyclictest28750irq/26-ahci[00007:15:003
18708991410,3cyclictest138550irq/27-eth011:05:113
1870899140,13cyclictest0-21swapper/308:24:453
1870899140,12cyclictest8155-21bash10:25:213
1870899140,12cyclictest2640-21sshd12:35:203
1870899140,12cyclictest0-21swapper/307:46:133
1870899140,11cyclictest0-21swapper/308:05:293
1869999143,4cyclictest16208-21sshd10:19:061
1869999142,3cyclictest4312-21sshd12:35:301
1869999142,2cyclictest0-21swapper/112:01:221
18699991412,1cyclictest0-21swapper/111:41:081
1869999141,1cyclictest5072-21df10:50:211
1869999141,12cyclictest138550irq/27-eth008:45:461
1869999140,5cyclictest28770-21sshd11:58:111
1869599141,10cyclictest0-21swapper/008:25:160
1869599140,3cyclictest0-21swapper/007:20:280
1869599140,1cyclictest0-21swapper/008:55:040
1869599140,1cyclictest0-21swapper/008:50:010
1869599140,13cyclictest0-21swapper/007:36:000
1869599140,13cyclictest0-21swapper/007:25:120
1869599140,13cyclictest0-21swapper/007:15:150
1870899139,3cyclictest138550irq/27-eth008:26:233
1870899137,4cyclictest5719-21munin-node09:05:153
1870899132,10cyclictest138550irq/27-eth009:03:173
1870899132,10cyclictest138550irq/27-eth008:31:083
1870899130,5cyclictest0-21swapper/308:40:223
1870899130,4cyclictest138550irq/27-eth012:25:133
1870899130,12cyclictest138550irq/27-eth008:15:153
1870899130,12cyclictest0-21swapper/308:55:153
1870899130,12cyclictest0-21swapper/308:35:033
1870899130,12cyclictest0-21swapper/308:10:223
1870899130,12cyclictest0-21swapper/307:40:253
1870899130,12cyclictest0-21swapper/307:21:133
1870899130,11cyclictest7302-21seq07:55:143
1870899130,11cyclictest30834-21sshd08:46:573
1869999139,3cyclictest138550irq/27-eth007:57:491
1869999139,3cyclictest138550irq/27-eth007:45:541
1869999133,2cyclictest25681-21grep07:25:121
1869999133,2cyclictest12937-21sshd11:49:081
1869999132,3cyclictest17926-21sshd09:36:041
1869999131,4cyclictest0-21swapper/107:20:311
1869999131,10cyclictest0-21swapper/108:15:591
1869999130,10cyclictest7785-21munin-run09:10:001
1870899128,3cyclictest0-21swapper/307:30:283
1870899128,2cyclictest0-21swapper/308:50:013
1870899122,9cyclictest0-21swapper/308:00:013
1870899120,11cyclictest0-21swapper/307:25:183
1869999128,3cyclictest138550irq/27-eth008:20:031
1869999127,4cyclictest138550irq/27-eth008:31:071
1869999122,4cyclictest0-21swapper/108:10:221
1869999122,3cyclictest138550irq/27-eth007:15:161
1869999122,1cyclictest0-21swapper/107:10:231
1869999122,0cyclictest0-21swapper/108:00:141
1869999121,3cyclictest138550irq/27-eth008:56:091
1869999121,3cyclictest138550irq/27-eth008:36:031
1869999121,3cyclictest138550irq/27-eth007:41:111
1869999121,10cyclictest138550irq/27-eth008:08:441
1869999121,10cyclictest138550irq/27-eth007:50:181
1869999121,10cyclictest0-21swapper/108:25:491
1869999120,10cyclictest0-21swapper/109:00:551
1869999117,3cyclictest138550irq/27-eth008:42:381
1869999117,3cyclictest138550irq/27-eth007:30:041
1869999116,4cyclictest0-21swapper/108:51:361
1869999115,5cyclictest0-21swapper/107:35:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional