You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 22:02

Intel(R) Xeon(R) CPU E31220L @ 2.20GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack5slot0.osadl.org (updated Sun Oct 17, 2021 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
431121370,0sleep24305-21sshd11:06:252
2172721160,0sleep00-21swapper/011:15:430
221882970,1sleep041ktimersoftd/011:37:590
306712910,2sleep10-21swapper/111:02:031
306712910,2sleep10-21swapper/111:02:021
123332880,2sleep112336-21unixbench-2d10:30:251
307792850,1sleep30-21swapper/310:47:183
307792850,1sleep30-21swapper/310:47:173
74192840,2sleep20-21swapper/209:45:572
202052840,1sleep10-21swapper/112:25:451
315542830,1sleep00-21swapper/009:48:490
5092820,4sleep32307899cyclictest10:40:133
2260328132,45sleep30-21swapper/307:07:293
2256628032,44sleep10-21swapper/107:06:591
15962800,2sleep30-21swapper/312:27:243
23462780,3sleep22307299cyclictest12:20:102
116832780,2sleep32307899cyclictest12:24:563
234082770,1sleep10-21swapper/109:25:451
27502760,2sleep30-21swapper/310:55:153
194312760,2sleep32121-21runrttasks10:38:413
133942760,0sleep00-21swapper/009:35:370
117782740,2sleep10-21swapper/109:46:281
327222730,3sleep10-21swapper/109:23:161
31012730,1sleep10-21swapper/111:39:341
187572730,2sleep10-21swapper/111:08:071
58322720,1sleep00-21swapper/009:20:160
47282720,2sleep24725-21sshd09:56:402
2255326033,10sleep20-21swapper/207:06:522
2270025632,12sleep00-21swapper/007:08:420
79902550,1sleep30-21swapper/312:31:483
116912500,0sleep011693-21bash11:14:430
123292360,0sleep312325-21rm11:18:273
50082282,24sleep2138550irq/27-eth010:18:442
269412280,2sleep10-21swapper/111:49:401
291812270,1sleep10-21swapper/111:27:431
263312270,1sleep00-21swapper/009:18:550
58092261,23sleep2138550irq/27-eth010:29:512
33122260,2sleep10-21swapper/109:52:531
320522250,0sleep032046-21sshd11:50:150
154312250,23sleep228750irq/26-ahci[00010:23:342
2306399244,11cyclictest0-21swapper/010:46:150
2306399244,11cyclictest0-21swapper/010:46:150
27012220,2sleep20-21swapper/211:10:012
23069992220,1cyclictest0-21swapper/110:13:251
201672220,3sleep22307299cyclictest12:14:492
94612210,3sleep30-21swapper/309:13:203
58682210,2sleep10-21swapper/110:40:421
2306999217,13cyclictest27356-21perf07:20:001
210182210,0sleep00-21swapper/010:35:130
2307299202,12cyclictest0-21swapper/207:27:382
30582190,3sleep30-21swapper/311:02:373
30582190,3sleep30-21swapper/311:02:373
2307299193,13cyclictest0-21swapper/211:36:132
2307299192,14cyclictest0-21swapper/212:08:252
2307299192,11cyclictest28608-21sshd11:46:022
2307299191,9cyclictest0-21swapper/211:59:532
2306399191,7cyclictest5156-21sshd10:44:270
185972190,2sleep20-21swapper/212:29:242
55222180,1sleep20-21swapper/212:16:442
291102180,16sleep2138550irq/27-eth011:31:222
2307299182,13cyclictest33-21ksoftirqd/210:55:002
2307299182,10cyclictest0-21swapper/210:52:332
2307299181,7cyclictest0-21swapper/211:40:152
2307299181,14cyclictest0-21swapper/209:20:412
2306399180,3cyclictest0-21swapper/010:25:320
2306399180,17cyclictest0-21swapper/011:24:150
2306399180,14cyclictest0-21swapper/010:31:110
28942170,1sleep10-21swapper/109:16:121
287562170,15sleep228750irq/26-ahci[00010:06:422
23078991711,4cyclictest1877-21nfsd10:22:263
2307299173,3cyclictest138550irq/27-eth010:38:232
2307299172,13cyclictest0-21swapper/209:18:492
2307299170,9cyclictest33-21ksoftirqd/210:00:422
23069991713,3cyclictest1877-21nfsd11:34:131
2306399176,10cyclictest0-21swapper/009:54:390
23078991613,2cyclictest0-21swapper/311:51:593
2307899160,15cyclictest0-21swapper/308:15:193
2307299163,9cyclictest138550irq/27-eth012:00:582
2307299163,8cyclictest138550irq/27-eth009:36:132
2307299163,8cyclictest10434-21sshd09:50:052
2307299163,11cyclictest5431-21sshd09:31:062
2307299162,13cyclictest0-21swapper/211:20:202
2307299160,14cyclictest0-21swapper/209:01:022
2306999168,7cyclictest13695-21sshd10:23:201
2306999166,2cyclictest0-21swapper/108:00:151
2306999163,3cyclictest0-21swapper/110:06:051
2306999163,12cyclictest138550irq/27-eth008:09:421
2306999161,7cyclictest0-21swapper/111:14:071
2306999161,14cyclictest20105-21sshd11:56:091
23069991611,4cyclictest15369-21sshd11:51:591
2306999160,14cyclictest0-21swapper/110:56:491
2306399168,5cyclictest0-21swapper/008:55:270
2306399160,5cyclictest0-21swapper/007:55:190
2306399160,2cyclictest21897-21bash12:07:320
215182160,14sleep2138550irq/27-eth011:26:482
194572160,14sleep2138550irq/27-eth011:04:332
194572160,14sleep2138550irq/27-eth011:04:322
16492160,14sleep1138550irq/27-eth012:20:051
70932150,2sleep10-21swapper/110:37:141
2307899156,8cyclictest0-21swapper/308:12:033
23078991512,2cyclictest26571-21bash11:12:413
2307899151,13cyclictest27050-21bash12:11:493
23078991511,3cyclictest21274-21sshd11:48:593
2307899150,14cyclictest9892-21sshd11:32:553
2307899150,13cyclictest0-21swapper/309:56:383
2307299157,6cyclictest1987-21sshd11:52:422
2307299157,4cyclictest3687-21sshd10:14:562
2307299153,10cyclictest6450-21sshd10:48:152
2307299153,10cyclictest6450-21sshd10:48:152
2307299153,0cyclictest0-21swapper/208:55:212
2307299152,8cyclictest5984-21kworker/u8:112:31:012
2307299152,8cyclictest0-21swapper/208:06:092
2307299152,5cyclictest0-21swapper/209:42:272
2307299152,1cyclictest27146-21sshd12:37:482
2307299152,10cyclictest0-21swapper/209:06:182
2307299152,10cyclictest0-21swapper/208:18:462
2307299150,8cyclictest33-21ksoftirqd/209:10:052
2307299150,3cyclictest18369-21munin-node08:10:212
2307299150,12cyclictest20865-21cp09:25:342
2306999159,5cyclictest5467-21sshd12:05:351
23069991512,2cyclictest5412-21sshd09:34:521
23069991512,2cyclictest20388-21sshd12:33:171
23069991512,2cyclictest0-21swapper/110:48:351
23069991512,2cyclictest0-21swapper/110:48:341
2306999151,13cyclictest20020-21sshd09:10:471
2306999151,13cyclictest19963-21munin-node08:15:211
2306999151,13cyclictest15686-21sshd10:27:131
2306999151,13cyclictest0-21swapper/110:17:581
2306999151,12cyclictest26882-21ls12:15:311
23069991510,4cyclictest147450irq/28-eth1-rx-07:53:211
2306399158,3cyclictest0-21swapper/010:54:420
2306399157,4cyclictest0-21swapper/011:55:540
2306399157,4cyclictest0-21swapper/008:36:300
2306399154,6cyclictest20604-21sshd10:09:300
23063991510,2cyclictest0-21swapper/008:10:190
2306399150,3cyclictest0-21swapper/012:25:010
2306399150,14cyclictest0-21swapper/007:30:150
2306399150,11cyclictest0-21swapper/011:28:350
214142150,13sleep2138550irq/27-eth011:15:432
2307899148,5cyclictest27256-21sshd10:10:173
2307899148,4cyclictest28111-21sshd11:05:253
2307899147,6cyclictest0-21swapper/308:09:443
2307899141,4cyclictest1921-21nscd09:40:233
2307899141,11cyclictest5370-21sshd12:35:163
2307899140,5cyclictest1987-21sshd12:15:063
2307899140,5cyclictest15980-21sshd10:08:583
2307899140,4cyclictest1987-21sshd10:01:043
2307899140,1cyclictest1877-21nfsd08:21:123
2307899140,13cyclictest3196-21bash10:51:303
2307899140,13cyclictest14496-21sshd09:32:103
2307899140,13cyclictest10527-21sshd10:30:173
2307899140,13cyclictest0-21swapper/310:18:543
2307899140,13cyclictest0-21swapper/309:52:463
2307899140,13cyclictest0-21swapper/307:50:493
2307899140,12cyclictest712-21sshd10:25:253
2307899140,12cyclictest31061-21sshd11:35:183
2307899140,12cyclictest3028-21sshd12:05:213
2307899140,12cyclictest22425-21sshd12:00:153
2307899140,12cyclictest19816-21sshd11:56:083
2307899140,12cyclictest1436-21sshd09:23:243
2307299147,6cyclictest0-21swapper/208:35:282
2307299143,8cyclictest11431-21ntp_states10:30:202
2307299142,8cyclictest0-21swapper/208:26:372
2307299141,8cyclictest0-21swapper/208:33:252
2307299141,8cyclictest0-21swapper/207:58:572
2307299141,8cyclictest0-21swapper/207:36:012
2307299140,10cyclictest138550irq/27-eth008:50:372
2306999148,5cyclictest0-21swapper/109:06:211
2306999148,5cyclictest0-21swapper/108:57:441
2306999148,5cyclictest0-21swapper/107:55:211
2306999143,4cyclictest11057-21sshd12:02:361
2306999142,3cyclictest16019-21processes12:10:311
2306999142,3cyclictest0-21swapper/109:42:061
2306999141,5cyclictest0-21swapper/107:35:041
2306999141,4cyclictest20719-21sshd09:36:311
2306999141,4cyclictest20007-21sshd10:02:031
2306999141,4cyclictest0-21swapper/111:15:091
2306999141,12cyclictest29573-21sshd09:55:461
2306999141,12cyclictest1877-21nfsd08:45:521
2306999141,12cyclictest1877-21nfsd08:30:341
2306999141,12cyclictest1877-21nfsd08:21:141
2306999141,12cyclictest1877-21nfsd07:21:281
2306999141,12cyclictest0-21swapper/110:53:131
2306999141,12cyclictest0-21swapper/109:00:041
2306999141,12cyclictest0-21swapper/108:53:021
2306999141,12cyclictest0-21swapper/107:47:141
2306999140,7cyclictest0-21swapper/112:38:211
2306999140,5cyclictest0-21swapper/111:21:471
2306999140,13cyclictest30590-21ipmi_temp07:25:211
2306999140,13cyclictest0-21swapper/107:43:021
2306399148,3cyclictest0-21swapper/009:05:500
2306399147,3cyclictest0-21swapper/012:16:490
2306399143,6cyclictest27651-21sshd10:21:110
2306399142,11cyclictest0-21swapper/008:21:040
23063991412,1cyclictest0-21swapper/011:40:030
2306399140,9cyclictest1845-21bash12:20:070
2306399140,9cyclictest0-21swapper/009:25:130
2306399140,9cyclictest0-21swapper/007:20:370
2306399140,8cyclictest0-21swapper/012:13:000
2306399140,6cyclictest0-21swapper/011:45:060
2306399140,3cyclictest0-21swapper/012:32:120
2306399140,13cyclictest0-21swapper/011:01:270
2306399140,13cyclictest0-21swapper/011:01:270
2306399140,11cyclictest0-21swapper/011:06:420
2306399140,10cyclictest7391-21ls07:45:260
2306399140,10cyclictest0-21swapper/010:58:400
2306399140,10cyclictest0-21swapper/010:17:350
30232130,3sleep30-21swapper/309:45:313
2307899138,4cyclictest26045-21sshd09:37:083
2307899138,4cyclictest1877-21nfsd07:15:293
2307899138,4cyclictest15022-21sshd09:17:363
2307899132,2cyclictest0-21swapper/308:51:033
23078991311,1cyclictest0-21swapper/307:10:533
2307899130,5cyclictest1163-21jbd2/dm-2-807:55:173
2307899130,5cyclictest0-21swapper/307:30:553
2307899130,4cyclictest7233-21sshd11:43:413
2307899130,4cyclictest24565-21sshd09:25:533
2307899130,12cyclictest986-21munin-node08:45:203
2307899130,12cyclictest0-21swapper/311:25:043
2307899130,12cyclictest0-21swapper/311:20:283
2307899130,12cyclictest0-21swapper/309:05:173
2307899130,12cyclictest0-21swapper/309:02:483
2307899130,12cyclictest0-21swapper/308:55:463
2307899130,12cyclictest0-21swapper/308:40:293
2307899130,12cyclictest0-21swapper/308:35:573
2307899130,12cyclictest0-21swapper/308:31:463
2307899130,12cyclictest0-21swapper/308:27:403
2307899130,12cyclictest0-21swapper/308:00:213
2307899130,12cyclictest0-21swapper/307:45:453
2307899130,12cyclictest0-21swapper/307:44:303
2307899130,12cyclictest0-21swapper/307:35:133
2307899130,11cyclictest29832-21munin-node07:25:213
2307899130,11cyclictest27928-21df_inode07:20:303
2307299138,4cyclictest1921-21nscd10:41:192
2307299137,3cyclictest138550irq/27-eth007:24:322
2307299135,6cyclictest0-21swapper/207:40:192
2307299133,1cyclictest0-21swapper/207:30:212
2307299132,11cyclictest138550irq/27-eth007:51:382
2307299131,8cyclictest0-21swapper/208:41:382
2307299130,12cyclictest28750irq/26-ahci[00007:47:032
2307299130,12cyclictest138550irq/27-eth008:21:242
2307299130,12cyclictest0-21swapper/208:45:312
2306999138,4cyclictest0-21swapper/108:25:101
2306999134,0cyclictest0-21swapper/111:40:191
2306999131,11cyclictest0-21swapper/108:10:221
2306999131,11cyclictest0-21swapper/107:31:301
2306999130,12cyclictest0-21swapper/108:41:151
2306999130,12cyclictest0-21swapper/108:35:041
2306399137,3cyclictest31091-21sshd10:14:230
2306399137,3cyclictest0-21swapper/008:47:400
2306399133,4cyclictest0-21swapper/012:35:160
2306399132,8cyclictest0-21swapper/009:00:590
2306399131,7cyclictest30062-21bash09:30:180
23063991311,1cyclictest0-21swapper/009:11:080
2306399130,3cyclictest0-21swapper/011:31:180
2306399130,3cyclictest0-21swapper/007:25:410
2306399130,2cyclictest8505-21id12:02:170
2306399130,1cyclictest8981-21df07:50:190
2306399130,1cyclictest0-21swapper/009:40:510
2306399130,12cyclictest6779-21sshd10:04:150
2306399130,12cyclictest0-21swapper/008:05:020
2306399130,12cyclictest0-21swapper/007:38:050
2306399130,12cyclictest0-21swapper/007:12:480
2306399130,10cyclictest0-21swapper/009:55:050
2307299122,5cyclictest0-21swapper/207:11:372
2307299120,11cyclictest0-21swapper/208:00:012
2307299120,10cyclictest26533-21proc_pri07:15:272
2306999129,2cyclictest23761-21smartctl07:10:221
2306399128,1cyclictest0-21swapper/008:01:560
23063991210,1cyclictest0-21swapper/008:15:460
2306399120,11cyclictest0-21swapper/008:30:200
2306399120,11cyclictest0-21swapper/008:26:000
2306399120,10cyclictest0-21swapper/008:50:010
2306399120,10cyclictest0-21swapper/008:40:540
2306399120,10cyclictest0-21swapper/007:15:040
2306399110,10cyclictest0-21swapper/007:40:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional