You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-07 - 13:34

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack5slot0.osadl.org (updated Tue Feb 07, 2023 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36921650,3sleep02178799cyclictest21:23:120
2914821470,1sleep129131-21sshd21:20:051
2803321400,3sleep02178799cyclictest23:09:300
479221360,0sleep24793-21sshd23:01:512
1625621350,4sleep12179199cyclictest23:59:051
1520921340,1sleep015205-21sshd22:48:460
1520921340,1sleep015205-21sshd22:48:450
1425921310,3sleep02178799cyclictest00:31:020
2040221290,2sleep10-21swapper/121:47:341
3022121250,0sleep00-21swapper/021:59:440
561821080,1sleep05619-21sshd22:28:170
1632821030,2sleep20-21swapper/222:26:242
384721000,1sleep23846-21sshd22:42:112
25652980,2sleep32121-21runrttasks00:12:263
2136729833,48sleep10-21swapper/119:06:471
179582980,2sleep20-21swapper/222:18:142
266072970,1sleep226606-21sshd22:35:412
249642970,2sleep10-21swapper/120:25:271
75782950,0sleep00-21swapper/021:18:120
2130829432,57sleep30-21swapper/319:06:143
274852930,0sleep00-21swapper/022:19:040
280622920,2sleep30-21swapper/321:11:263
280622920,2sleep30-21swapper/321:11:263
24132920,2sleep12410-21sshd23:29:471
91822910,0sleep00-21swapper/023:58:280
302462910,1sleep030243-21sshd00:23:450
252122900,3sleep325208-21bash22:55:143
192572900,1sleep30-21swapper/322:04:233
190392900,2sleep119038-21bash23:48:031
153562900,2sleep12179199cyclictest21:58:251
133542890,3sleep10-21swapper/122:28:591
321642880,2sleep00-21swapper/022:50:130
134952870,2sleep313494-21timerwakeupswit20:00:263
77992860,2sleep10-21swapper/122:50:521
53552860,1sleep05354-21seq23:44:040
182572840,4sleep02178799cyclictest23:50:420
54982830,5sleep22179699cyclictest23:24:222
307082820,2sleep20-21swapper/222:24:552
185312820,3sleep018520-21rm22:07:060
106942820,2sleep20-21swapper/223:55:412
246532810,1sleep10-21swapper/123:20:211
18852810,1sleep30-21swapper/323:18:273
123662810,2sleep212364-21sshd23:05:192
46812770,1sleep20-21swapper/223:10:162
199912770,4sleep22179699cyclictest23:48:082
80732760,3sleep38072-21bash21:49:213
43852760,0sleep00-21swapper/022:11:270
308912760,2sleep330865-21sshd23:04:073
300692740,0sleep00-21swapper/022:33:150
283522740,3sleep12179199cyclictest22:58:171
130982740,0sleep00-21swapper/000:19:150
230112730,0sleep00-21swapper/023:23:050
198122680,1sleep10-21swapper/100:25:381
65302650,1sleep30-21swapper/323:13:143
5162640,0sleep21875-21nfsd22:30:372
278972620,2sleep127887-21sshd22:07:551
2156025833,13sleep20-21swapper/219:09:142
2155825128,11sleep00-21swapper/019:09:110
115902420,1sleep021782-21cyclictest23:10:460
260642410,1sleep226060-21rm23:43:032
51242392,35sleep1138550irq/27-eth021:32:051
282582360,1sleep08-21rcu_preempt22:41:290
155622360,34sleep1138550irq/27-eth021:38:421
259862350,32sleep3138550irq/27-eth022:07:463
243922320,2sleep324389-21sshd23:40:073
135542320,30sleep228750irq/26-ahci[00023:30:412
240772310,29sleep2138550irq/27-eth022:55:102
2179699303,22cyclictest7370-21sshd22:50:492
112102300,3sleep20-21swapper/221:10:012
112102300,3sleep20-21swapper/221:10:002
135082290,2sleep113507-21sshd22:01:021
228782280,1sleep122855-21sshd23:09:031
199772280,1sleep30-21swapper/300:16:543
20822270,2sleep10-21swapper/100:35:551
49462260,1sleep10-21swapper/123:32:471
254522260,2sleep225451-21bash22:49:402
254522260,2sleep225451-21bash22:49:402
22822260,3sleep02178799cyclictest21:48:500
2179699264,12cyclictest0-21swapper/221:26:472
103172260,23sleep3138550irq/27-eth022:40:003
77072250,3sleep37704-21sshd22:28:283
21962250,2sleep22190-21sshd21:20:312
206462250,2sleep10-21swapper/100:31:391
156232250,1sleep015620-21sshd22:20:440
2179699246,8cyclictest29893-21sshd21:17:172
2179699243,8cyclictest21757-21bash23:37:032
2179699241,14cyclictest2449-21sshd00:30:012
2179699240,2cyclictest0-21swapper/220:45:262
209852240,0sleep041ktimersoftd/023:00:240
273722231,20sleep1138550irq/27-eth021:25:321
2179699236,8cyclictest12211-21bash21:41:102
2179699233,8cyclictest0-21swapper/221:52:512
2179699233,8cyclictest0-21swapper/200:38:072
203082230,2sleep11875-21nfsd22:46:231
203082230,2sleep11875-21nfsd22:46:221
177672230,21sleep2138550irq/27-eth023:53:342
141392230,5sleep31921-21nscd22:23:283
259422220,2sleep00-21swapper/023:34:390
2179699222,16cyclictest0-21swapper/220:35:472
2179699213,0cyclictest0-21swapper/222:11:522
2179699211,16cyclictest0-21swapper/220:12:242
2179699210,17cyclictest0-21swapper/219:58:502
5302200,18sleep3138550irq/27-eth000:24:013
2179699206,2cyclictest0-21swapper/221:56:152
2179699206,2cyclictest0-21swapper/200:21:232
2179699206,1cyclictest0-21swapper/220:06:322
2179699205,2cyclictest0-21swapper/222:08:582
2179699205,12cyclictest147450irq/28-eth1-rx-20:43:382
2179699204,11cyclictest1346-21id21:34:362
2179699204,10cyclictest2121-21runrttasks19:50:192
2179699203,16cyclictest12516-21sshd21:10:072
2179699203,16cyclictest12516-21sshd21:10:062
2179699202,6cyclictest18199-21sshd21:36:002
2179699201,0cyclictest0-21swapper/221:45:592
180472200,18sleep3138550irq/27-eth021:19:093
107392202,16sleep1138550irq/27-eth000:07:201
2180099198,3cyclictest0-21swapper/321:26:463
2180099191,8cyclictest0-21swapper/322:53:323
21800991910,4cyclictest138550irq/27-eth022:17:383
2179699196,3cyclictest5459-21sshd23:18:442
2179699194,14cyclictest0-21swapper/220:19:422
2179699194,14cyclictest0-21swapper/219:41:282
2179699192,2cyclictest0-21swapper/220:24:422
2179699192,2cyclictest0-21swapper/219:12:082
2179699192,13cyclictest0-21swapper/223:25:512
9212180,16sleep3138550irq/27-eth023:24:013
244162180,16sleep3138550irq/27-eth023:28:543
2180099188,4cyclictest2134-21bash22:44:533
2180099187,6cyclictest30280-21bash21:37:073
2180099180,9cyclictest26400-21sshd21:50:523
2180099180,6cyclictest0-21swapper/323:49:133
2179699185,2cyclictest0-21swapper/219:37:202
2179699184,8cyclictest0-21swapper/220:01:042
2179699183,10cyclictest0-21swapper/222:02:592
2179699182,8cyclictest28183-21sshd00:17:402
2179699182,1cyclictest0-21swapper/219:34:472
2179699181,2cyclictest0-21swapper/220:31:512
2179699180,2cyclictest0-21swapper/220:25:072
2179699180,12cyclictest0-21swapper/200:14:592
2178799185,5cyclictest20144-21sshd23:17:130
2178799180,6cyclictest16549-21sshd21:35:520
143052180,3sleep10-21swapper/122:20:381
2180099174,4cyclictest13836-21sshd21:24:243
21800991715,1cyclictest0-21swapper/319:42:043
2180099170,5cyclictest25933-21sshd23:06:263
2180099170,12cyclictest3042-21sshd22:33:423
2180099170,12cyclictest24770-21sshd00:35:083
2179699176,2cyclictest0-21swapper/200:28:552
2179699175,9cyclictest147450irq/28-eth1-rx-21:02:392
2179699174,7cyclictest13029-21sshd00:01:372
2179699174,7cyclictest0-21swapper/219:25:512
2179699170,2cyclictest0-21swapper/220:51:032
2179699170,2cyclictest0-21swapper/219:15:592
2179199170,7cyclictest29088-21cp22:13:371
2179199170,3cyclictest9568-21sshd23:41:351
2178799178,5cyclictest47-21rcuop/321:54:160
2178799176,6cyclictest0-21swapper/019:50:190
139962170,15sleep3138550irq/27-eth000:07:383
2180099164,4cyclictest0-21swapper/323:50:193
2180099164,11cyclictest26726-21sshd00:32:153
2180099163,12cyclictest3053-21sshd22:47:433
2180099163,12cyclictest3053-21sshd22:47:433
21800991610,1cyclictest7512-21sshd23:58:193
2180099160,8cyclictest0-21swapper/321:30:203
2180099160,7cyclictest1921-21nscd21:43:153
2180099160,14cyclictest138550irq/27-eth023:37:033
2180099160,10cyclictest0-21swapper/323:32:583
2179699164,3cyclictest147450irq/28-eth1-rx-00:05:332
2179699163,8cyclictest27829-21ls19:20:232
2179199164,1cyclictest17722-21sshd00:10:491
2179199164,10cyclictest5702-21bash00:15:341
2179199163,1cyclictest14130-21sshd21:15:511
2179199162,1cyclictest18612-21sshd21:41:451
2179199160,3cyclictest7411-21bash00:21:381
2179199160,3cyclictest0-21swapper/122:30:121
2179199160,14cyclictest0-21swapper/122:18:381
2179199160,11cyclictest138550irq/27-eth023:18:541
2178799163,12cyclictest30527-21sshd21:11:400
2178799163,12cyclictest30527-21sshd21:11:400
2178799160,2cyclictest25818-21bash23:45:460
2178799160,15cyclictest0-21swapper/022:57:560
2178799160,12cyclictest0-21swapper/021:32:190
2180099158,5cyclictest20528-21sshd22:10:063
2180099157,7cyclictest0-21swapper/300:27:063
2180099157,3cyclictest0-21swapper/300:01:053
2180099152,12cyclictest0-21swapper/319:39:113
2180099150,13cyclictest10866-21sshd21:55:123
2180099150,13cyclictest0-21swapper/319:47:413
2180099150,10cyclictest29871-21munin-run20:39:593
2179699154,10cyclictest5322-21aten_r5power_fr19:45:152
2179199154,1cyclictest0-21swapper/123:03:051
2179199153,11cyclictest138550irq/27-eth022:38:171
21791991512,2cyclictest13589-21sshd00:01:411
2179199150,8cyclictest18671-21sshd21:50:141
2179199150,5cyclictest9958-21sshd23:10:391
2179199150,3cyclictest24601-21sshd23:51:161
2179199150,2cyclictest1987-21sshd22:42:061
2179199150,10cyclictest0-21swapper/123:37:231
2178799157,3cyclictest0-21swapper/022:02:050
2178799150,4cyclictest0-21swapper/019:55:400
2178799150,1cyclictest5601-21sshd00:00:550
2178799150,12cyclictest0-21swapper/020:52:160
2178799150,11cyclictest0-21swapper/023:38:420
2178799150,11cyclictest0-21swapper/019:20:120
2178799150,10cyclictest18829-21sshd00:37:330
2180099140,9cyclictest138550irq/27-eth021:06:233
2180099140,9cyclictest138550irq/27-eth021:06:233
2180099140,13cyclictest9555-21taskset19:54:533
2179199140,9cyclictest11836-21perf19:59:591
2179199140,3cyclictest138550irq/27-eth020:31:111
2179199140,2cyclictest138550irq/27-eth020:50:171
2179199140,12cyclictest31658-21ipmi_fans19:30:201
2179199140,12cyclictest20607-21sshd21:10:441
2179199140,12cyclictest20607-21sshd21:10:431
2178799145,8cyclictest6733-21sshd21:26:320
2178799140,5cyclictest3688-21bash22:36:330
2178799140,4cyclictest3467-21bash21:40:240
2178799140,3cyclictest0-21swapper/023:25:170
2178799140,2cyclictest11485-21df_inode00:10:200
2178799140,13cyclictest24826-21sshd00:29:070
2178799140,10cyclictest0-21swapper/000:07:320
2180099139,2cyclictest0-21swapper/320:50:303
2180099138,2cyclictest147450irq/28-eth1-rx-19:17:073
2180099137,5cyclictest0-21swapper/320:25:043
2180099137,5cyclictest0-21swapper/320:16:333
2180099132,2cyclictest28022-21users19:20:243
2180099130,12cyclictest0-21swapper/320:21:483
2180099130,12cyclictest0-21swapper/320:05:493
2180099130,12cyclictest0-21swapper/319:34:283
2180099130,12cyclictest0-21swapper/319:25:063
2180099130,11cyclictest9805-21munin-node19:55:213
2180099130,11cyclictest7478-21iostat21:00:203
2180099130,11cyclictest22753-21memory19:10:173
2179699131,3cyclictest4813-21diskstats20:55:192
2179199138,3cyclictest9863-21memory21:05:251
2179199138,3cyclictest9863-21memory21:05:251
2179199137,5cyclictest0-21swapper/120:10:251
2179199137,4cyclictest1943-21snmpd19:45:061
2179199132,10cyclictest138550irq/27-eth020:00:581
21791991311,1cyclictest0-21swapper/119:15:311
2179199130,5cyclictest26401-21munin-node19:20:141
2179199130,4cyclictest1943-21snmpd20:36:291
2179199130,12cyclictest1987-21sshd19:36:361
2179199130,12cyclictest0-21swapper/121:00:391
2179199130,12cyclictest0-21swapper/120:40:331
2179199130,12cyclictest0-21swapper/120:20:101
2179199130,12cyclictest0-21swapper/120:15:031
2179199130,12cyclictest0-21swapper/120:05:011
2179199130,12cyclictest0-21swapper/119:26:261
2179199130,11cyclictest7556-21munin-node19:50:211
2179199130,11cyclictest1124-21processes20:45:271
2178799139,2cyclictest0-21swapper/020:27:320
2178799137,5cyclictest0-21swapper/019:25:340
21787991311,1cyclictest0-21swapper/020:20:020
2178799130,8cyclictest0-21swapper/021:00:170
2178799130,5cyclictest0-21swapper/020:47:360
2178799130,12cyclictest0-21swapper/019:48:580
2178799130,12cyclictest0-21swapper/019:34:250
2178799130,11cyclictest0-21swapper/021:05:260
2178799130,11cyclictest0-21swapper/021:05:250
2178799130,10cyclictest0-21swapper/020:10:250
2180099128,3cyclictest5459-21munin-node20:55:253
2180099120,3cyclictest705-21munin-node20:45:233
2180099120,11cyclictest0-21swapper/320:10:393
2180099120,10cyclictest30363-21munin-node20:40:153
2180099120,10cyclictest25920-21munin-node20:30:213
2179199120,3cyclictest21923-21aten_r5power_fr19:10:111
2179199120,11cyclictest0-21swapper/119:40:071
2179199120,10cyclictest0-21swapper/120:55:171
2178799127,4cyclictest0-21swapper/020:07:270
2178799120,1cyclictest0-21swapper/019:16:020
2178799120,11cyclictest0-21swapper/020:55:230
2178799120,11cyclictest0-21swapper/020:37:090
2178799120,11cyclictest0-21swapper/020:30:040
2178799120,11cyclictest0-21swapper/020:17:510
2178799120,11cyclictest0-21swapper/020:01:150
2178799120,11cyclictest0-21swapper/019:35:340
2178799120,10cyclictest0-21swapper/019:42:490
2178799113,4cyclictest0-21swapper/019:10:190
2178799110,1cyclictest0-21swapper/020:40:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional