You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-17 - 06:12

Intel(R) Xeon(R) CPU E31220L @ 2.20GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack5slot0.osadl.org (updated Sun Oct 17, 2021 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3114121410,0sleep331129-21sshd21:52:163
3110721250,1sleep11811399cyclictest21:26:591
347621200,1sleep20-21swapper/200:23:362
3073821010,0sleep01875-21nfsd23:38:380
500421000,0sleep00-21swapper/023:09:520
92362980,0sleep00-21swapper/021:46:120
165252950,1sleep00-21swapper/021:18:110
1592429232,47sleep30-21swapper/319:05:103
109772910,0sleep00-21swapper/021:50:050
177642890,2sleep317761-21bash22:05:163
122482890,2sleep20-21swapper/222:22:492
304532860,2sleep20-21swapper/221:41:222
1791728633,49sleep10-21swapper/119:09:481
53992830,1sleep00-21swapper/022:58:490
319122790,2sleep20-21swapper/223:20:162
318372780,1sleep231836-21bash00:34:262
26922780,1sleep31812199cyclictest23:13:163
265432770,2sleep226544-21sshd23:04:582
233692770,2sleep323359-21sshd23:48:513
225232770,0sleep00-21swapper/023:26:310
198822760,2sleep30-21swapper/322:27:203
191652760,1sleep10-21swapper/123:15:091
32882750,2sleep30-21swapper/300:08:453
25352750,2sleep30-21swapper/321:45:273
178622750,2sleep20-21swapper/221:25:272
292512730,2sleep10-21swapper/123:42:061
47572710,4sleep31812199cyclictest23:28:143
239422710,2sleep01811199cyclictest23:34:080
140232700,1sleep20-21swapper/221:10:362
129362700,2sleep20-21swapper/222:11:562
25722690,1sleep12121-21runrttasks21:52:441
195352670,4sleep11811399cyclictest23:59:301
194482670,4sleep11811399cyclictest21:22:061
36052620,1sleep30-21swapper/321:16:423
194722620,1sleep019459-21sshd00:10:350
268352610,4sleep11811399cyclictest22:17:081
1771826132,16sleep20-21swapper/219:07:252
1766626133,4sleep00-21swapper/019:06:470
57582290,3sleep20-21swapper/222:44:092
32292282,23sleep3138550irq/27-eth021:13:063
304432280,3sleep30-21swapper/300:30:253
209952270,25sleep3138550irq/27-eth000:03:213
255012260,2sleep225500-21bash21:15:302
4242250,2sleep20-21swapper/223:35:102
1811899252,21cyclictest16500-21sshd00:10:162
141872251,21sleep3138550irq/27-eth023:03:303
7932240,1sleep00-21swapper/000:19:370
301852230,2sleep10-21swapper/123:12:421
289172230,0sleep00-21swapper/022:54:100
253562230,3sleep10-21swapper/123:49:041
1812199230,15cyclictest15846-21bash21:36:073
1811899230,21cyclictest0-21swapper/223:56:192
107272230,21sleep3138550irq/27-eth021:28:213
62952220,2sleep26294-21bash23:06:132
247882220,20sleep228750irq/26-ahci[00000:07:292
218332220,2sleep10-21swapper/123:00:361
1812199220,2cyclictest0-21swapper/319:49:103
1811899222,15cyclictest33-21ksoftirqd/222:38:142
1811899222,15cyclictest33-21ksoftirqd/222:38:132
143482220,20sleep1138550irq/27-eth023:33:031
1811899211,15cyclictest0-21swapper/220:10:412
1811899210,2cyclictest0-21swapper/220:03:302
18113992115,4cyclictest20602-21sshd00:06:591
114582210,2sleep10-21swapper/122:22:451
324192200,1sleep00-21swapper/022:32:260
324192200,1sleep00-21swapper/022:32:250
194672200,18sleep1138550irq/27-eth021:14:571
18113992018,1cyclictest0-21swapper/121:43:361
86332190,2sleep38632-21sshd21:42:353
27832190,2sleep20-21swapper/221:56:162
1812199190,15cyclictest0-21swapper/300:26:433
1811899199,7cyclictest0-21swapper/223:16:292
1811899192,11cyclictest0-21swapper/222:48:532
1811199190,18cyclictest0-21swapper/020:37:080
150082190,17sleep3138550irq/27-eth023:51:313
1811899183,3cyclictest27413-21sshd22:06:152
1811899182,8cyclictest0-21swapper/219:36:052
1811899182,7cyclictest22807-21cp00:29:382
1811899182,11cyclictest1562-21sshd00:04:522
1811899182,10cyclictest4539-21bash22:32:582
1811899182,10cyclictest4539-21bash22:32:572
1811899182,10cyclictest24337-21sshd23:48:572
1811899180,1cyclictest0-21swapper/222:50:512
73022170,15sleep1138550irq/27-eth000:24:031
287022170,4sleep20-21swapper/221:30:192
1812199178,5cyclictest0-21swapper/300:36:583
1812199170,15cyclictest6319-21sshd23:35:423
1812199170,14cyclictest0-21swapper/320:16:313
1811899171,13cyclictest0-21swapper/222:55:302
1811899171,11cyclictest0-21swapper/221:02:172
1811899171,11cyclictest0-21swapper/219:16:052
18113991713,3cyclictest30732-21sshd21:59:281
1811399170,15cyclictest0-21swapper/122:25:181
244212161,14sleep3138550irq/27-eth021:22:403
185332160,14sleep3138550irq/27-eth023:44:393
1812199165,2cyclictest0-21swapper/322:22:503
1812199161,3cyclictest0-21swapper/323:57:003
18121991612,3cyclictest14200-21bash00:24:523
1812199160,15cyclictest0-21swapper/322:03:423
1811899169,5cyclictest0-21swapper/220:26:572
1811899163,8cyclictest0-21swapper/223:40:562
1811899163,8cyclictest0-21swapper/221:54:492
1811899163,7cyclictest24614-21sshd00:37:162
1811899162,11cyclictest11664-21sshd22:26:212
1811899162,11cyclictest0-21swapper/222:04:332
18118991611,4cyclictest25-21rcuop/121:37:152
1811899160,11cyclictest0-21swapper/219:40:292
18113991610,5cyclictest138550irq/27-eth023:27:591
1811399160,14cyclictest0-21swapper/119:40:261
1811399160,11cyclictest3285-21sshd22:07:131
1811199160,8cyclictest0-21swapper/022:11:040
179872160,3sleep10-21swapper/100:14:081
136612160,0sleep1211ktimersoftd/122:04:491
106302160,0sleep00-21swapper/022:40:540
1812199158,3cyclictest27398-21bash22:31:503
1812199158,3cyclictest27398-21bash22:31:493
1812199155,8cyclictest1987-21sshd22:12:113
1812199152,3cyclictest0-21swapper/322:40:023
1812199151,13cyclictest0-21swapper/300:10:083
1812199151,11cyclictest19468-21idleruntime-cro22:19:593
18121991510,4cyclictest15044-21sshd23:18:213
1812199150,6cyclictest0-21swapper/323:22:543
1812199150,3cyclictest25575-21sshd22:50:063
1812199150,13cyclictest1987-21sshd22:35:043
1812199150,13cyclictest1987-21sshd22:35:043
1812199150,13cyclictest0-21swapper/321:31:103
1812199150,11cyclictest0-21swapper/320:47:233
1811899153,9cyclictest32659-21bash23:31:222
1811899152,8cyclictest2074-21sshd23:50:082
1811899152,8cyclictest0-21swapper/220:31:292
1811899152,5cyclictest0-21swapper/200:15:572
1811899152,10cyclictest25381-21kworker/u8:223:25:572
1811899152,10cyclictest21976-21bash21:47:422
1811899151,12cyclictest1943-21snmpd20:47:182
1811899151,11cyclictest0-21swapper/219:51:062
1811899151,10cyclictest0-21swapper/221:24:012
1811899150,14cyclictest33-21ksoftirqd/219:14:082
1811899150,13cyclictest10901-21sshd22:15:192
1811399157,7cyclictest0-21swapper/100:33:001
1811399157,3cyclictest0-21swapper/100:25:241
1811399156,4cyclictest138550irq/27-eth021:45:181
1811399150,9cyclictest21609-21munin-run20:30:001
1811399150,9cyclictest0-21swapper/122:43:101
1811399150,6cyclictest0-21swapper/100:01:071
1811399150,14cyclictest0-21swapper/100:19:051
1811399150,13cyclictest25163-21sshd00:37:211
1811399150,13cyclictest0-21swapper/122:12:361
1811399150,11cyclictest3670-21sshd21:34:511
1811399150,11cyclictest27904-21sshd22:50:171
1811399150,11cyclictest138550irq/27-eth022:48:101
1811399150,10cyclictest9462-21ls21:35:221
1811199153,6cyclictest0-21swapper/021:14:390
18111991512,2cyclictest14313-21latency23:40:250
1811199150,1cyclictest0-21swapper/023:03:090
1811199150,13cyclictest0-21swapper/021:44:010
1811199150,12cyclictest0-21swapper/020:16:470
1811199150,11cyclictest0-21swapper/021:39:050
1812199148,5cyclictest20856-21sshd21:58:223
1812199146,3cyclictest0-21swapper/300:16:103
1812199142,10cyclictest22948-21munin-node23:30:183
18121991411,2cyclictest0-21swapper/320:07:423
1812199140,8cyclictest0-21swapper/320:39:303
1812199140,6cyclictest0-21swapper/319:35:593
1812199140,12cyclictest4744-21sshd22:55:063
1812199140,12cyclictest19522-21latency_hist20:25:003
1812199140,12cyclictest0-21swapper/323:05:493
1812199140,10cyclictest0-21swapper/321:01:193
1811899149,4cyclictest0-21swapper/220:15:382
1811899147,1cyclictest0-21swapper/220:41:302
1811899142,10cyclictest0-21swapper/219:30:292
1811899141,8cyclictest0-21swapper/220:06:212
1811899141,8cyclictest0-21swapper/219:56:452
1811899141,12cyclictest138550irq/27-eth023:11:092
1811899140,1cyclictest0-21swapper/219:49:292
1811399148,1cyclictest1877-21nfsd20:55:561
1811399145,2cyclictest18664-21bash22:56:351
1811399144,4cyclictest0-21swapper/122:30:221
1811399144,4cyclictest0-21swapper/122:30:221
1811399142,11cyclictest0-21swapper/120:54:101
18113991411,2cyclictest21029-21bash22:38:311
18113991411,2cyclictest21029-21bash22:38:311
1811399140,5cyclictest26013-21sshd23:38:041
1811399140,13cyclictest8882-21sshd23:50:461
1811399140,13cyclictest20281-21sshd23:07:531
1811399140,12cyclictest36-21rcuop/219:20:191
1811399140,12cyclictest30695-21sshd23:20:081
1811399140,12cyclictest1943-21snmpd20:12:201
1811199148,3cyclictest0-21swapper/023:23:050
1811199148,3cyclictest0-21swapper/020:47:540
1811199144,5cyclictest28862-21sshd22:17:210
1811199140,9cyclictest0-21swapper/021:58:530
1811199140,2cyclictest10909-21diskstats20:05:160
1811199140,13cyclictest32049-21sshd22:25:060
1811199140,13cyclictest0-21swapper/022:21:120
1811199140,13cyclictest0-21swapper/000:32:570
1811199140,12cyclictest27452-21ls23:45:300
1811199140,12cyclictest26367-21sshd00:00:170
173632140,1sleep00-21swapper/023:51:470
1812199139,3cyclictest0-21swapper/321:05:433
1812199137,5cyclictest0-21swapper/319:40:593
1812199137,5cyclictest0-21swapper/319:10:233
1812199132,3cyclictest138550irq/27-eth020:50:283
18121991311,1cyclictest0-21swapper/320:55:013
18121991311,1cyclictest0-21swapper/319:20:593
1812199130,5cyclictest0-21swapper/320:30:513
1812199130,4cyclictest7980-21chrt19:58:453
1812199130,4cyclictest17589-21sshd22:45:283
1812199130,4cyclictest13570-21iostat20:10:213
1812199130,12cyclictest0-21swapper/320:44:133
1812199130,12cyclictest0-21swapper/320:29:293
1812199130,12cyclictest0-21swapper/319:31:293
1812199130,12cyclictest0-21swapper/319:27:413
1812199130,12cyclictest0-21swapper/319:17:193
1811899133,5cyclictest17923-21perl20:20:172
1811899133,2cyclictest0-21swapper/221:06:212
1811899131,7cyclictest0-21swapper/220:37:232
1811899131,11cyclictest138550irq/27-eth019:26:302
1811899130,12cyclictest0-21swapper/220:50:112
1811899130,12cyclictest0-21swapper/219:21:562
1811399137,5cyclictest0-21swapper/120:02:101
1811399137,5cyclictest0-21swapper/119:16:081
1811399132,10cyclictest138550irq/27-eth020:05:391
18113991311,1cyclictest0-21swapper/119:59:041
18113991311,1cyclictest0-21swapper/119:50:421
1811399130,9cyclictest6108-21sed21:05:211
1811399130,5cyclictest0-21swapper/119:11:451
1811399130,12cyclictest0-21swapper/120:43:421
1811399130,12cyclictest0-21swapper/119:45:241
1811399130,12cyclictest0-21swapper/119:37:261
1811399130,12cyclictest0-21swapper/119:30:231
1811399130,12cyclictest0-21swapper/119:25:351
1811399130,11cyclictest3212-21df21:00:141
1811399130,11cyclictest29232-21hwlatdetect20:45:171
1811399130,11cyclictest22643-21diskstats21:15:151
1811199133,6cyclictest26642-21bash00:26:130
1811199132,8cyclictest0-21swapper/021:05:200
1811199132,2cyclictest0-21swapper/019:17:280
18111991311,1cyclictest0-21swapper/021:28:440
18111991311,1cyclictest0-21swapper/019:40:380
1811199130,9cyclictest0-21swapper/023:59:170
1811199130,9cyclictest0-21swapper/021:22:070
1811199130,4cyclictest1877-21nfsd22:35:250
1811199130,4cyclictest1877-21nfsd22:35:250
1811199130,1cyclictest0-21swapper/023:16:430
1811199130,1cyclictest0-21swapper/022:04:060
1811199130,1cyclictest0-21swapper/020:01:430
1811199130,12cyclictest0-21swapper/022:08:090
1811199130,12cyclictest0-21swapper/021:32:310
1811199130,12cyclictest0-21swapper/019:50:500
1811199130,12cyclictest0-21swapper/019:38:450
1811199130,12cyclictest0-21swapper/019:11:580
1811199130,12cyclictest0-21swapper/000:38:400
1811199130,10cyclictest0-21swapper/021:04:570
1811199130,10cyclictest0-21swapper/019:25:290
1812199120,11cyclictest1875-21nfsd20:00:353
1812199120,11cyclictest138550irq/27-eth019:50:033
1811899120,3cyclictest321ktimersoftd/220:55:302
1811399128,3cyclictest1877-21nfsd20:35:001
1811399120,1cyclictest1876-21nfsd20:20:011
1811399120,10cyclictest0-21swapper/120:30:171
1811399120,10cyclictest0-21swapper/120:15:121
1811199129,2cyclictest0-21swapper/020:51:130
1811199127,2cyclictest0-21swapper/019:21:240
1811199123,4cyclictest27001-21sshd22:46:290
18111991210,1cyclictest0-21swapper/000:20:310
1811199120,8cyclictest0-21swapper/023:10:270
1811199120,3cyclictest0-21swapper/019:55:320
1811199120,1cyclictest0-21swapper/020:57:030
1811199120,12cyclictest0-21swapper/000:06:110
1811199120,11cyclictest14536-21seq20:11:240
1811199120,11cyclictest0-21swapper/020:40:180
1811199120,11cyclictest0-21swapper/020:30:280
1811199120,11cyclictest0-21swapper/020:25:160
1811199120,11cyclictest0-21swapper/020:20:180
1811199120,11cyclictest0-21swapper/019:46:110
1811199120,11cyclictest0-21swapper/019:30:280
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional