You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-09-29 - 04:10

Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackaslot0.osadl.org (updated Wed Sep 29, 2021 00:44:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
72342890,1sleep125008-21diskmemload21:30:201
257242690,0sleep00-21swapper/023:13:340
83152630,1sleep21785-21runrttasks23:24:382
256532600,1sleep11785-21runrttasks23:04:021
195072600,0sleep00-21swapper/021:18:530
143252560,0sleep225008-21diskmemload21:18:232
314112550,0sleep30-21swapper/323:10:533
12482550,0sleep31246-21sshd23:45:593
184782530,0sleep20-21swapper/223:03:202
287662520,0sleep00-21swapper/021:13:330
211232520,0sleep30-21swapper/323:57:453
76842510,1sleep07663-21sshd23:46:400
74302510,0sleep10-21swapper/123:43:281
306042510,1sleep330602-21sshd21:13:443
223342510,0sleep00-21swapper/022:41:340
115502510,0sleep00-21swapper/021:37:000
98222500,1sleep00-21swapper/023:40:270
3181125028,8sleep30-21swapper/319:07:123
59412490,1sleep15935-21sshd21:27:041
3181024927,8sleep20-21swapper/219:07:112
3180924924,21sleep10-21swapper/119:07:101
3170224928,18sleep00-21swapper/019:05:490
272202490,0sleep20-21swapper/223:16:502
272202490,0sleep20-21swapper/223:16:502
157832490,0sleep00-21swapper/023:22:040
157102490,0sleep00-21swapper/000:10:060
311602480,1sleep31785-21runrttasks00:11:393
252212480,0sleep00-21swapper/022:38:380
306552470,0sleep00-21swapper/022:55:030
115912470,0sleep20-21swapper/200:30:042
257972460,0sleep00-21swapper/023:26:170
308142450,1sleep230809-21rm21:48:322
255152450,0sleep10-21swapper/121:19:301
121332450,0sleep30-21swapper/322:30:553
53692420,1sleep15344-21sshd00:29:271
310832220,0sleep03-21ksoftirqd/022:26:310
83332200,1sleep28331-21sshd22:17:522
65582200,1sleep16550-21sshd23:27:381
32292992019,1cyclictest0-21swapper/222:08:502
3229299200,1cyclictest1544-21nfsd22:38:222
32290991917,1cyclictest11219-21sshd21:43:290
3229399180,17cyclictest0-21swapper/320:08:193
3229199181,5cyclictest0-21swapper/122:28:411
3229099188,9cyclictest0-21swapper/022:30:060
32290991816,1cyclictest0-21swapper/021:54:160
233402180,0sleep20-21swapper/200:21:032
32293991716,1cyclictest0-21swapper/323:25:233
32293991713,1cyclictest39-21ksoftirqd/321:26:533
3229399170,1cyclictest22813-21bash22:47:573
3229399170,11cyclictest0-21swapper/322:02:523
32292991716,1cyclictest0-21swapper/223:41:012
3229199176,4cyclictest13928-21sshd23:53:491
315692170,1sleep30-21swapper/300:32:063
233522170,1sleep123347-21sshd21:54:031
3229399166,1cyclictest0-21swapper/321:16:443
3229399163,1cyclictest119950irq/25-eth000:22:273
3229399160,9cyclictest0-21swapper/323:42:083
32292991614,1cyclictest0-21swapper/223:34:172
3229199168,4cyclictest18541-21kworker/1:121:46:581
3229199168,4cyclictest12444-21kworker/1:000:30:321
3229099169,4cyclictest0-21swapper/023:56:500
70162150,0sleep121-21ksoftirqd/123:30:471
3229399157,3cyclictest11464-21bash22:21:203
32293991513,1cyclictest39-21ksoftirqd/322:58:483
3229399151,1cyclictest37-21rcuc/300:18:193
3229399150,1cyclictest119950irq/25-eth022:05:563
3229299152,9cyclictest119950irq/25-eth022:10:362
3229299150,14cyclictest28-21rcuc/222:23:462
3229199159,3cyclictest5342-21kworker/1:122:03:241
3229199159,3cyclictest19-21rcuc/123:06:101
3229199155,1cyclictest23931-21bash21:44:451
3229199152,3cyclictest0-21swapper/121:56:361
3229199150,14cyclictest0-21swapper/122:48:041
3229099159,5cyclictest0-21swapper/023:35:040
3229099158,4cyclictest0-21swapper/021:45:210
3229099150,5cyclictest0-21swapper/022:24:570
3229099150,14cyclictest30221-21sshd22:07:250
3229099150,14cyclictest0-21swapper/000:38:350
3229099150,13cyclictest0-21swapper/000:25:370
3229099150,12cyclictest0-21swapper/023:30:300
263032150,1sleep10-21swapper/122:32:221
255162150,0sleep025497-21sshd23:16:390
255162150,0sleep025497-21sshd23:16:390
89802140,0sleep30-21swapper/300:26:243
3229399148,2cyclictest0-21swapper/322:18:153
3229399147,2cyclictest0-21swapper/323:50:093
3229399145,9cyclictest39-21ksoftirqd/300:03:223
3229399145,0cyclictest37-21rcuc/321:56:223
3229399144,9cyclictest39-21ksoftirqd/322:35:553
3229399144,9cyclictest39-21ksoftirqd/321:42:073
32293991413,1cyclictest39-21ksoftirqd/323:38:563
32293991413,1cyclictest39-21ksoftirqd/322:53:083
32293991413,1cyclictest39-21ksoftirqd/321:53:193
32293991413,1cyclictest39-21ksoftirqd/321:20:493
32293991413,1cyclictest39-21ksoftirqd/300:37:543
32293991413,1cyclictest381ktimersoftd/323:08:303
3229399141,1cyclictest37-21rcuc/323:20:573
3229399140,1cyclictest119950irq/25-eth022:12:023
3229399140,13cyclictest23443-21sshd23:03:493
3229399140,13cyclictest0-21swapper/321:35:163
3229299148,5cyclictest1543-21nfsd23:09:502
32292991412,1cyclictest30-21ksoftirqd/222:27:332
32292991412,1cyclictest30-21ksoftirqd/200:35:222
3229299140,13cyclictest0-21swapper/222:32:362
3229299140,13cyclictest0-21swapper/200:26:582
3229299140,13cyclictest0-21swapper/200:12:172
3229199149,4cyclictest0-21swapper/122:15:371
3229199148,3cyclictest14700-21kworker/1:223:36:141
3229199147,3cyclictest806-40kipmi023:10:211
3229199146,4cyclictest29666-21kworker/1:123:55:251
3229199141,10cyclictest0-21swapper/122:44:161
3229199140,6cyclictest11852-21sshd21:24:341
3229199140,5cyclictest1526-21sshd00:02:071
3229199140,2cyclictest0-21swapper/122:55:481
3229199140,13cyclictest32340-21atenall_rapower00:15:161
3229199140,13cyclictest0-21swapper/122:08:361
3229199140,13cyclictest0-21swapper/121:39:001
3229199140,13cyclictest0-21swapper/100:35:331
3229199140,12cyclictest0-21swapper/122:10:531
3229099149,4cyclictest0-21swapper/022:50:320
3229099148,3cyclictest0-21swapper/021:21:100
3229099144,1cyclictest10457-21sshd21:55:530
3229099143,10cyclictest8190-21rm00:22:550
3229099143,10cyclictest0-21swapper/021:33:330
3229099142,1cyclictest0-21swapper/021:25:250
32290991410,1cyclictest21519-21cp23:06:480
3229099140,13cyclictest0-21swapper/023:50:270
3229099140,13cyclictest0-21swapper/000:16:220
214112140,0sleep20-21swapper/222:00:172
195072140,1sleep20-21swapper/219:50:002
3229399134,9cyclictest39-21ksoftirqd/323:15:073
3229399134,9cyclictest39-21ksoftirqd/323:15:073
3229399134,9cyclictest39-21ksoftirqd/321:47:113
3229399134,1cyclictest39-21ksoftirqd/322:25:013
3229399133,9cyclictest12576-21sshd21:30:483
32293991312,1cyclictest39-21ksoftirqd/323:32:293
32293991312,1cyclictest39-21ksoftirqd/322:42:093
3229399130,2cyclictest5385-21sshd00:05:423
3229299139,1cyclictest119950irq/25-eth023:27:572
3229299139,1cyclictest0-21swapper/223:50:462
3229299138,4cyclictest18748-21sshd23:35:062
3229299137,5cyclictest0-21swapper/221:52:212
3229299133,9cyclictest18937-21sshd21:25:152
3229299133,1cyclictest30-21ksoftirqd/221:12:442
32292991312,1cyclictest30-21ksoftirqd/223:46:552
32292991312,1cyclictest30-21ksoftirqd/200:05:262
32292991311,1cyclictest0-21swapper/222:40:382
3229299130,12cyclictest28-21rcuc/200:00:102
3229299130,12cyclictest23711-21sshd21:41:202
3229299130,12cyclictest17415-21sshd23:57:212
3229299130,12cyclictest10078-21cp21:36:512
3229299130,12cyclictest0-21swapper/223:12:592
3229299130,12cyclictest0-21swapper/222:50:132
3229299130,10cyclictest15095-21sshd22:47:112
3229199139,1cyclictest0-21swapper/123:45:031
3229199138,4cyclictest9461-21sshd22:37:041
3229199138,2cyclictest26943-21sshd22:51:261
3229199138,2cyclictest16797-21sshd00:20:231
3229199135,8cyclictest0-21swapper/120:39:431
3229199135,4cyclictest0-21swapper/119:20:141
3229199132,9cyclictest119950irq/25-eth000:12:111
3229199132,4cyclictest0-21swapper/120:24:531
3229199130,6cyclictest0-21swapper/119:14:541
3229199130,12cyclictest0-21swapper/123:21:291
3229199130,12cyclictest0-21swapper/123:15:151
3229199130,12cyclictest0-21swapper/123:15:151
3229199130,12cyclictest0-21swapper/100:06:351
3229199130,10cyclictest0-21swapper/121:12:061
3229099139,3cyclictest0-21swapper/000:30:470
3229099138,2cyclictest26612-21sshd22:00:440
32290991312,1cyclictest0-21swapper/022:15:090
3229099130,9cyclictest0-21swapper/000:05:360
3229099130,2cyclictest0-21swapper/000:03:180
3229099130,12cyclictest25855-21sshd22:45:100
3229099130,12cyclictest0-21swapper/023:00:200
3229099130,12cyclictest0-21swapper/022:11:060
3229299128,1cyclictest1752-21bash00:15:232
3229299123,9cyclictest30-21ksoftirqd/221:23:502
3229299120,1cyclictest32726-21bash22:55:172
3229299120,1cyclictest0-21swapper/221:30:212
3229299120,11cyclictest0-21swapper/221:55:132
3229199129,2cyclictest0-21swapper/122:21:291
3229199125,4cyclictest25583-21kworker/1:220:10:001
3229199125,3cyclictest127850irq/26-eth1-rx-19:17:241
3229199124,4cyclictest0-21swapper/119:51:061
3229199123,7cyclictest0-21swapper/120:46:381
3229199123,7cyclictest0-21swapper/119:40:291
3229199121,10cyclictest0-21swapper/119:35:161
3229199120,11cyclictest0-21swapper/119:30:181
3229099124,4cyclictest0-21swapper/020:30:250
32292991110,1cyclictest0-21swapper/220:57:132
32292991110,1cyclictest0-21swapper/220:41:122
3229299110,11cyclictest0-21swapper/221:00:292
3229299110,11cyclictest0-21swapper/220:21:202
3229299110,11cyclictest0-21swapper/219:29:312
3229199118,2cyclictest0-21swapper/120:40:241
3229199113,8cyclictest0-21swapper/120:31:431
3229199113,1cyclictest0-21swapper/119:28:141
3229199112,7cyclictest0-21swapper/119:56:481
3229199111,6cyclictest127850irq/26-eth1-rx-20:04:141
3229199111,1cyclictest19-21rcuc/120:15:131
3229199110,10cyclictest0-21swapper/121:07:001
3229099110,11cyclictest0-21swapper/020:45:010
3229099110,11cyclictest0-21swapper/019:55:190
3229399108,1cyclictest0-21swapper/319:20:193
32293991010,0cyclictest0-21swapper/319:30:293
3229399100,9cyclictest0-21swapper/320:00:173
3229299109,1cyclictest0-21swapper/221:06:302
3229299109,1cyclictest0-21swapper/220:37:012
3229299109,1cyclictest0-21swapper/220:07:082
3229299109,1cyclictest0-21swapper/220:00:102
3229299109,1cyclictest0-21swapper/219:13:002
3229299100,9cyclictest0-21swapper/219:20:192
3229299100,1cyclictest0-21swapper/220:19:142
3229299100,10cyclictest0-21swapper/220:46:392
3229199108,1cyclictest0-21swapper/120:55:181
3229199102,4cyclictest21-21ksoftirqd/120:10:261
3229199102,4cyclictest0-21swapper/119:45:391
3229199100,9cyclictest0-21swapper/121:00:161
3229199100,10cyclictest0-21swapper/120:53:241
3229099102,5cyclictest0-21swapper/020:15:230
3229099100,9cyclictest0-21swapper/021:10:010
3229099100,8cyclictest0-21swapper/021:01:220
3229099100,7cyclictest0-21swapper/019:50:260
3229099100,7cyclictest0-21swapper/019:40:280
3229099100,1cyclictest2391-21diskstats19:15:010
5348290,2sleep00-21swapper/019:20:150
322939999,0cyclictest0-21swapper/320:25:263
322939998,1cyclictest0-21swapper/319:15:093
322929998,1cyclictest0-21swapper/220:34:552
322929998,1cyclictest0-21swapper/219:59:072
322929998,1cyclictest0-21swapper/219:50:122
322929998,1cyclictest0-21swapper/219:17:462
322929996,2cyclictest17577-21sshd20:51:532
322929990,9cyclictest0-21swapper/220:25:172
322919991,4cyclictest0-21swapper/120:26:011
322909997,1cyclictest16401-21perl20:50:190
322909994,3cyclictest0-21swapper/020:20:280
322909991,5cyclictest0-21swapper/020:10:260
322909990,9cyclictest0-21swapper/019:25:330
322909990,8cyclictest0-21swapper/020:00:170
322909990,8cyclictest0-21swapper/019:35:150
322909990,7cyclictest0-21swapper/019:45:280
322909990,6cyclictest0-21swapper/020:05:030
322939987,1cyclictest0-21swapper/321:08:113
322939987,1cyclictest0-21swapper/319:50:063
322939987,1cyclictest0-21swapper/319:45:203
322939987,1cyclictest0-21swapper/319:41:373
322939981,6cyclictest0-21swapper/320:50:193
322939981,6cyclictest0-21swapper/320:30:233
322939981,6cyclictest0-21swapper/320:20:283
322939981,6cyclictest0-21swapper/319:25:433
322939980,8cyclictest0-21swapper/320:45:103
322909987,1cyclictest0-21swapper/020:35:450
322909981,6cyclictest0-21swapper/020:25:130
322909980,8cyclictest0-21swapper/020:41:050
322909980,5cyclictest10108-21munin-node19:30:210
322909980,1cyclictest32597-21cstates19:10:170
322939976,1cyclictest0-21swapper/320:40:183
322939975,1cyclictest31859-21df20:15:173
322939975,1cyclictest18352-21cat20:55:163
322939970,6cyclictest0-21swapper/321:00:143
322939970,6cyclictest0-21swapper/320:35:283
322939970,6cyclictest0-21swapper/320:10:153
322939970,6cyclictest0-21swapper/319:55:173
322939970,6cyclictest0-21swapper/319:35:163
322939970,6cyclictest0-21swapper/319:10:163
322929971,5cyclictest0-21swapper/219:35:132
322929970,5cyclictest0-21swapper/220:14:022
322929970,5cyclictest0-21swapper/219:40:222
322929970,2cyclictest0-21swapper/219:30:052
322909970,6cyclictest0-21swapper/020:55:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional