You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-02 - 10:57

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack5slot0.osadl.org (updated Fri Jun 02, 2023 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1952621250,1sleep30-21swapper/300:39:093
1174021170,1sleep011738-21bash00:10:570
1074521170,4sleep1134699cyclictest23:44:131
1101210334,48sleep30-21swapper/319:08:513
2481421000,1sleep00-21swapper/023:06:050
51302970,0sleep00-21swapper/023:33:460
111629732,44sleep10-21swapper/119:09:021
18152940,0sleep01813-21sshd22:57:030
267632910,2sleep10-21swapper/121:45:541
41922890,5sleep10-21swapper/123:36:561
202182880,0sleep00-21swapper/023:25:250
105982870,3sleep30-21swapper/322:47:323
58912860,2sleep10-21swapper/123:33:511
50492860,2sleep30-21swapper/322:01:153
266832860,1sleep026680-21sshd22:42:160
318792850,3sleep10-21swapper/123:46:171
116632850,1sleep00-21swapper/021:44:220
253632800,1sleep10-21swapper/100:39:481
247352800,1sleep00-21swapper/023:39:030
284112790,1sleep128408-21sshd21:28:241
287412780,2sleep328673-21sshd21:28:253
159682780,2sleep30-21swapper/323:08:333
120492770,1sleep00-21swapper/000:28:050
321162760,1sleep00-21swapper/021:32:200
147092760,2sleep30-21swapper/322:12:583
147092760,2sleep30-21swapper/322:12:583
8912750,2sleep2888-21sshd21:35:572
271782750,1sleep127179-21bash22:14:201
271782750,1sleep127179-21bash22:14:201
269262750,2sleep30-21swapper/300:26:113
89712730,2sleep30-21swapper/322:57:503
5542730,2sleep20-21swapper/221:32:252
80122680,0sleep00-21swapper/000:37:540
44262670,2sleep3134899cyclictest22:53:543
112726223,9sleep20-21swapper/219:09:102
301912590,5sleep0134599cyclictest00:09:370
95125732,14sleep00-21swapper/019:07:000
122242440,1sleep20-21swapper/222:44:152
275512340,7sleep20-21swapper/223:19:382
94932330,1sleep00-21swapper/023:47:210
110572320,0sleep00-21swapper/000:34:490
111582280,26sleep1138550irq/27-eth021:33:381
284382270,3sleep10-21swapper/123:59:161
280402270,2sleep20-21swapper/200:09:232
68162260,2sleep30-21swapper/321:50:503
89992250,2sleep20-21swapper/223:44:012
15542250,1sleep02121-21runrttasks20:20:240
15542250,1sleep02121-21runrttasks20:20:240
147432250,0sleep00-21swapper/021:12:350
134599251,12cyclictest15955-21bash22:30:320
84452240,21sleep1138550irq/27-eth021:11:531
134799242,14cyclictest0-21swapper/223:38:142
134799241,9cyclictest1877-21nfsd22:22:592
76982230,2sleep17697-21sshd22:01:341
310092230,2sleep231010-21id00:02:542
244292230,21sleep228750irq/26-ahci[00000:32:482
134799231,10cyclictest14026-21sshd22:58:222
181012220,3sleep20-21swapper/219:45:232
134799225,3cyclictest5122-21sshd22:29:312
134799225,13cyclictest147450irq/28-eth1-rx-20:53:392
134799223,14cyclictest8633-21cp21:54:412
134799222,11cyclictest20028-21fschecks_count21:20:192
254062210,19sleep3138550irq/27-eth023:45:373
238932211,18sleep1138550irq/27-eth023:12:361
226322210,11sleep3138550irq/27-eth022:17:203
183072210,3sleep30-21swapper/323:21:543
134799215,8cyclictest5347-21sshd22:18:582
134799212,9cyclictest0-21swapper/221:59:392
312742200,1sleep131273-21id23:06:451
306622200,0sleep00-21swapper/022:25:160
134799205,2cyclictest0-21swapper/222:00:592
134799204,2cyclictest0-21swapper/223:30:022
134799203,14cyclictest30621-21sshd00:36:502
134799203,10cyclictest0-21swapper/200:15:342
134699201,16cyclictest4950-21grep20:30:001
134599201,6cyclictest27816-21sshd21:46:030
134599200,3cyclictest7037-21cp23:50:250
34212190,17sleep1138550irq/27-eth022:32:491
31712190,0sleep00-21swapper/000:16:540
134899191,3cyclictest24654-21rm21:13:433
134799194,7cyclictest1875-21nfsd21:11:312
134799194,2cyclictest0-21swapper/220:59:552
134799193,11cyclictest3326-21sshd00:23:472
134799193,11cyclictest0-21swapper/220:38:542
134799193,11cyclictest0-21swapper/220:38:542
134799192,12cyclictest0-21swapper/221:06:072
134599198,10cyclictest0-21swapper/021:22:200
134599194,9cyclictest8228-21sshd23:14:190
134599194,14cyclictest3-21ksoftirqd/021:15:330
134599190,7cyclictest14900-21sshd23:57:490
215442182,14sleep1138550irq/27-eth000:12:041
214742181,15sleep1138550irq/27-eth000:29:071
134799183,14cyclictest138550irq/27-eth000:13:462
134799182,7cyclictest31735-21sshd23:56:112
134799182,7cyclictest0-21swapper/223:06:392
134799181,4cyclictest0-21swapper/221:17:232
134799181,11cyclictest0-21swapper/223:21:512
134799180,9cyclictest0-21swapper/221:41:542
134699180,16cyclictest0-21swapper/122:39:531
134599187,5cyclictest27631-21sshd22:38:550
134599187,10cyclictest0-21swapper/021:26:080
134599180,4cyclictest0-21swapper/019:20:220
134599180,14cyclictest11143-21munin-node20:40:340
134599180,13cyclictest0-21swapper/022:21:010
134599180,12cyclictest0-21swapper/019:42:070
293112170,15sleep2138550irq/27-eth020:10:252
278702170,1sleep10-21swapper/122:45:501
134899170,14cyclictest1987-21sshd00:14:533
134799175,3cyclictest0-21swapper/222:09:232
134799173,11cyclictest28113-21sshd23:26:112
1347991713,3cyclictest24296-21sshd23:02:502
1347991712,3cyclictest29398-21sshd23:13:102
134799171,13cyclictest0-21swapper/222:48:042
134799171,11cyclictest0-21swapper/222:14:242
134799171,11cyclictest0-21swapper/222:14:242
134799171,11cyclictest0-21swapper/219:13:452
134799171,11cyclictest0-21swapper/200:27:292
134799170,8cyclictest1877-21nfsd20:21:152
134799170,8cyclictest1877-21nfsd20:21:152
134799170,6cyclictest15343-21sshd22:30:312
134799170,12cyclictest0-21swapper/221:46:032
134799170,12cyclictest0-21swapper/219:20:052
134699170,15cyclictest0-21swapper/123:00:001
134699170,12cyclictest15450-21sshd21:19:511
134699170,11cyclictest18835-21bash23:15:231
134599179,3cyclictest0-21swapper/020:45:280
134599177,9cyclictest1987-21sshd23:03:340
134599176,8cyclictest0-21swapper/019:50:420
1345991713,3cyclictest0-21swapper/000:00:190
134599170,5cyclictest0-21swapper/021:54:470
134599170,4cyclictest6504-21sshd22:08:350
134599170,3cyclictest0-21swapper/020:56:000
134599170,1cyclictest2505-21sshd23:23:370
134599170,1cyclictest2121-21runrttasks19:57:080
134599170,16cyclictest5644-21sshd00:20:310
134599170,16cyclictest0-21swapper/023:18:010
134599170,16cyclictest0-21swapper/022:52:560
134599170,13cyclictest19115-21sshd23:41:440
134599170,13cyclictest0-21swapper/021:37:080
134599170,12cyclictest30561-21bash22:46:090
193762160,14sleep1138550irq/27-eth000:05:061
134899169,5cyclictest16588-21sshd23:58:013
134899168,3cyclictest2834-21sshd21:43:233
134899164,4cyclictest0-21swapper/323:29:333
134899161,3cyclictest17413-21sshd21:23:353
134899161,10cyclictest0-21swapper/322:30:103
1348991610,4cyclictest1877-21nfsd23:30:263
134799165,6cyclictest147450irq/28-eth1-rx-21:04:432
134799165,5cyclictest147450irq/28-eth1-rx-20:04:032
134799165,10cyclictest147450irq/28-eth1-rx-19:36:042
134799163,8cyclictest5838-21sshd23:50:192
134799163,8cyclictest445-21jbd2/dm-0-819:50:312
134799163,8cyclictest0-21swapper/222:52:462
134799163,10cyclictest0-21swapper/220:06:022
134799162,7cyclictest4114-21bash22:36:192
134799161,14cyclictest147450irq/28-eth1-rx-19:18:552
134799161,13cyclictest12837-21sshd23:47:422
134799161,10cyclictest30809-21aten_r5power_fr20:15:152
134699168,3cyclictest10992-21sshd22:16:001
134699167,7cyclictest0-21swapper/120:40:231
134699165,6cyclictest0-21swapper/122:22:581
134699163,4cyclictest22310-21sshd21:20:301
1346991612,3cyclictest1877-21nfsd19:21:341
134699160,4cyclictest29236-21bash23:03:191
134699160,14cyclictest29478-21sshd22:25:091
134699160,11cyclictest28014-21sshd23:29:331
134599165,10cyclictest1167-21bash22:15:020
134599163,1cyclictest0-21swapper/021:55:260
134599160,3cyclictest0-21swapper/019:26:520
134599160,2cyclictest0-21swapper/021:01:040
134599160,15cyclictest0-21swapper/022:01:360
134599160,15cyclictest0-21swapper/019:34:550
134599160,15cyclictest0-21swapper/019:11:290
134599160,10cyclictest18572-21sshd22:13:240
134599160,10cyclictest18572-21sshd22:13:240
270642150,13sleep1138550irq/27-eth021:56:381
134899159,4cyclictest13855-21hddtemp_smartct21:30:183
134899158,6cyclictest138550irq/27-eth000:00:193
134899158,3cyclictest138550irq/27-eth022:41:413
134899151,5cyclictest22931-21bash21:45:313
134899151,5cyclictest19162-21sshd23:38:293
134899151,1cyclictest28391-21sshd21:35:263
134899151,1cyclictest22114-21sshd23:42:033
134899151,13cyclictest1365-21sshd00:33:483
134899151,13cyclictest0-21swapper/323:03:003
134899150,13cyclictest0-21swapper/319:17:303
134799155,9cyclictest147450irq/28-eth1-rx-20:25:552
134799155,9cyclictest147450irq/28-eth1-rx-19:41:282
134799154,10cyclictest147450irq/28-eth1-rx-20:47:112
134799154,10cyclictest147450irq/28-eth1-rx-19:33:562
134799152,8cyclictest0-21swapper/220:30:572
134799150,14cyclictest1877-21nfsd21:26:022
134699157,3cyclictest12098-21switchtime22:05:341
134699151,2cyclictest9704-21sshd00:34:411
1346991511,3cyclictest18468-21sshd00:18:331
1346991510,3cyclictest11404-21bash23:50:451
134699150,14cyclictest1877-21nfsd00:21:221
134699150,14cyclictest1877-21nfsd00:00:281
134699150,14cyclictest14479-21sshd22:44:301
134699150,13cyclictest4426-21sshd21:36:221
134699150,13cyclictest0-21swapper/121:54:521
134599158,4cyclictest0-21swapper/020:25:230
134599157,4cyclictest0-21swapper/020:00:280
134599151,1cyclictest0-21swapper/020:15:400
134599151,1cyclictest0-21swapper/020:10:590
134599150,3cyclictest0-21swapper/019:35:080
134599150,1cyclictest0-21swapper/020:50:490
134599150,14cyclictest0-21swapper/019:15:020
134899148,2cyclictest18906-21sshd00:15:133
134899142,10cyclictest0-21swapper/300:07:393
134899141,8cyclictest0-21swapper/323:50:323
134899141,4cyclictest12161-21sshd21:15:473
134899141,4cyclictest10383-21sshd22:05:253
134899141,11cyclictest25145-21sshd00:22:383
134899141,11cyclictest20552-21sshd23:15:313
134899141,11cyclictest16200-21bash22:20:113
134899140,5cyclictest7226-21sshd22:36:393
134899140,13cyclictest28750irq/26-ahci[00020:00:093
134899140,10cyclictest1462-21cpu23:10:223
134799145,5cyclictest0-21swapper/219:25:432
134799144,8cyclictest138550irq/27-eth020:40:302
134699147,6cyclictest1877-21nfsd20:30:201
134699145,4cyclictest19848-21bash22:52:011
134699144,2cyclictest1034-21bash21:43:101
134699140,8cyclictest14834-21awk19:39:591
134699140,13cyclictest31637-21sshd23:20:021
134699140,13cyclictest138550irq/27-eth019:18:301
134699140,13cyclictest0-21swapper/120:56:011
134699140,12cyclictest3362-21runrttasks19:12:331
134599140,1cyclictest0-21swapper/020:35:090
134599140,1cyclictest0-21swapper/020:35:090
134599140,13cyclictest0-21swapper/021:05:340
134599140,13cyclictest0-21swapper/020:32:370
134599140,13cyclictest0-21swapper/020:06:460
134599140,13cyclictest0-21swapper/019:45:440
134899138,4cyclictest12493-21chrt19:34:113
134899134,3cyclictest0-21swapper/320:45:493
134899131,3cyclictest17582-21users21:55:303
134899131,3cyclictest0-21swapper/320:25:283
134899131,11cyclictest31788-21sshd22:28:543
134899131,11cyclictest0-21swapper/321:06:183
134899131,11cyclictest0-21swapper/320:50:193
1348991311,1cyclictest0-21swapper/320:17:553
1348991311,1cyclictest0-21swapper/320:06:383
134899131,11cyclictest0-21swapper/319:51:393
134899131,11cyclictest0-21swapper/319:47:193
134899130,12cyclictest0-21swapper/321:00:103
134899130,12cyclictest0-21swapper/320:40:173
134899130,12cyclictest0-21swapper/320:36:463
134899130,12cyclictest0-21swapper/320:36:453
134899130,12cyclictest0-21swapper/320:30:103
134899130,12cyclictest0-21swapper/320:20:113
134899130,12cyclictest0-21swapper/320:20:113
134899130,12cyclictest0-21swapper/319:38:013
134899130,12cyclictest0-21swapper/319:29:263
134899130,11cyclictest6375-21df_inode19:20:183
134899130,11cyclictest1836-21df_inode19:10:243
134899130,11cyclictest0-21swapper/319:55:203
134799133,5cyclictest0-21swapper/219:56:452
134699138,4cyclictest22204-21fschecks_count19:55:171
134699137,5cyclictest0-21swapper/121:09:521
134699137,5cyclictest0-21swapper/120:10:451
134699132,3cyclictest138550irq/27-eth019:41:111
134699132,10cyclictest15146-21netstat20:50:271
1346991311,1cyclictest0-21swapper/120:00:011
134699130,4cyclictest1543-21needreboot20:20:241
134699130,4cyclictest1543-21needreboot20:20:231
134699130,2cyclictest0-21swapper/120:45:161
134699130,12cyclictest0-21swapper/120:05:521
134699130,12cyclictest0-21swapper/119:50:561
134699130,12cyclictest0-21swapper/119:46:371
134699130,12cyclictest0-21swapper/119:30:041
134699130,11cyclictest9036-21iostat19:25:181
134699130,11cyclictest7701-21df20:35:201
134699130,11cyclictest7701-21df20:35:201
134699130,11cyclictest30796-21munin-node20:15:271
134899120,3cyclictest29674-21ntp_states20:10:263
134899120,11cyclictest0-21swapper/320:55:213
134699120,11cyclictest0-21swapper/121:00:021
134899110,10cyclictest16330-21munin-node19:40:293
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional