You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 10:51

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack5slot0.osadl.org (updated Sun Dec 04, 2022 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2823021420,1sleep328227-21sshd22:00:263
1078121350,5sleep0642799cyclictest22:40:140
1565921300,4sleep0642799cyclictest23:55:140
492121250,2sleep34884-21sshd21:55:213
918621130,3sleep2643099cyclictest23:36:392
3212621030,2sleep30-21swapper/322:36:133
814821000,1sleep3643199cyclictest00:00:333
105742970,3sleep110559-21sshd00:20:071
599529532,50sleep10-21swapper/119:06:511
4872950,3sleep30-21swapper/300:06:143
161762950,2sleep116173-21sshd21:11:501
161882930,3sleep20-21swapper/222:25:542
193742910,2sleep119369-21sshd23:01:481
137952910,2sleep313779-21sshd23:10:153
147682900,2sleep20-21swapper/223:22:172
3812890,0sleep03-21ksoftirqd/022:36:180
322812890,2sleep30-21swapper/300:35:013
191332890,2sleep10-21swapper/121:27:011
87542880,2sleep30-21swapper/322:16:213
264522860,4sleep1642999cyclictest23:50:131
241722860,2sleep21875-21nfsd21:33:272
196882860,1sleep319685-21bash22:43:583
165582860,1sleep016557-21seq23:46:150
277882850,2sleep327787-21sshd21:54:343
605928433,42sleep30-21swapper/319:07:403
57112840,2sleep21875-21nfsd22:42:412
236072840,2sleep223604-21sshd22:59:162
248322800,2sleep124824-21sshd21:30:301
274172790,1sleep11877-21nfsd21:18:521
312232760,1sleep331224-21sshd21:45:553
27032760,5sleep1642999cyclictest22:06:571
240132760,2sleep30-21swapper/322:20:433
164762760,2sleep316470-21sshd23:58:223
81112750,2sleep10-21swapper/100:29:291
65962720,1sleep16584-21sshd22:36:521
52192710,2sleep10-21swapper/100:16:191
608027032,9sleep20-21swapper/219:07:562
279922700,2sleep20-21swapper/221:36:452
303112660,5sleep3643199cyclictest22:50:523
294802630,1sleep029474-21sshd23:20:350
314742620,3sleep1642999cyclictest22:42:041
74292590,3sleep325029-21kworker/3:121:25:553
599925633,11sleep00-21swapper/019:06:530
187682460,0sleep00-21swapper/021:41:510
24892370,2sleep21921-21nscd22:12:512
212722320,30sleep128750irq/26-ahci[00000:08:211
185942290,2sleep30-21swapper/322:11:203
166042290,26sleep1138550irq/27-eth023:16:271
642999270,25cyclictest24853-21sshd21:48:211
157352270,1sleep00-21swapper/023:43:150
109422270,25sleep3138550irq/27-eth022:31:213
224172260,2sleep30-21swapper/323:20:023
642999250,23cyclictest24084-21sshd23:29:091
642799252,11cyclictest18383-21sshd22:17:150
25902250,2sleep22573-21sshd22:04:022
119642250,1sleep011959-21sshd23:25:030
46002240,2sleep0642799cyclictest23:18:230
44222240,22sleep228750irq/26-ahci[00023:06:222
220622240,4sleep1642999cyclictest22:17:351
643099222,14cyclictest16276-21bash22:34:522
642799223,8cyclictest4107-21sshd22:13:000
642799221,2cyclictest0-21swapper/021:48:520
642799221,1cyclictest474-21sshd23:50:450
288622220,2sleep10-21swapper/123:20:311
288412220,19sleep3138550irq/27-eth021:36:503
236632220,2sleep30-21swapper/323:08:113
161962220,4sleep3643199cyclictest23:16:243
80692210,3sleep20-21swapper/223:03:462
73392210,19sleep1138550irq/27-eth000:38:551
643099212,9cyclictest0-21swapper/222:24:192
643099212,10cyclictest138550irq/27-eth021:21:152
643099210,16cyclictest2035-21lldpd-portmirro21:06:592
643099210,16cyclictest2035-21lldpd-portmirro21:06:592
642799210,7cyclictest12247-21bash23:13:040
642799210,16cyclictest0-21swapper/000:13:300
231902210,0sleep223191-21bash23:31:592
133852210,0sleep00-21swapper/000:04:230
643199205,2cyclictest0-21swapper/322:29:253
6431992018,1cyclictest0-21swapper/323:46:283
643099203,14cyclictest13580-21bash21:50:142
643099202,7cyclictest10871-21sshd21:26:142
642799200,2cyclictest27914-21sshd21:39:480
642799200,2cyclictest0-21swapper/021:32:480
642799200,19cyclictest0-21swapper/022:45:110
642799200,14cyclictest19864-21sshd22:23:210
137652200,4sleep2643099cyclictest23:55:032
104712200,18sleep3138550irq/27-eth021:20:193
643199191,11cyclictest138550irq/27-eth021:42:153
643099193,5cyclictest20087-21bash23:52:352
643099193,11cyclictest4003-21sshd22:39:372
643099193,11cyclictest0-21swapper/223:14:212
643099192,14cyclictest25581-21sshd23:26:152
6430991914,4cyclictest15345-21sshd00:10:532
6429991914,3cyclictest3917-21sshd23:57:081
642999190,11cyclictest12482-21sshd22:01:581
642799198,6cyclictest0-21swapper/021:58:090
642799198,5cyclictest0-21swapper/000:21:170
642799198,10cyclictest6652-21sshd00:35:320
642799190,9cyclictest1585-21sshd23:30:030
642799190,1cyclictest0-21swapper/022:25:380
642799190,13cyclictest0-21swapper/000:17:190
642799190,12cyclictest3185-21sshd23:09:170
292902190,2sleep10-21swapper/123:14:431
643099183,13cyclictest12617-21sshd00:17:032
643099182,11cyclictest4080-21sshd22:51:272
643099182,11cyclictest26036-21sshd00:05:322
643099182,11cyclictest25704-21sshd00:27:592
643099181,12cyclictest10685-21sshd22:46:032
642799182,13cyclictest0-21swapper/023:35:190
642799181,9cyclictest0-21swapper/020:30:290
642799181,14cyclictest0-21swapper/023:03:570
642799180,2cyclictest0-21swapper/021:05:130
642799180,2cyclictest0-21swapper/021:05:130
642799180,17cyclictest0-21swapper/022:07:440
642799180,15cyclictest3281-21sshd00:25:450
642799180,13cyclictest610-21sshd22:54:110
642799180,13cyclictest0-21swapper/022:30:180
642799180,12cyclictest0-21swapper/022:56:370
642799180,11cyclictest12042-21sshd21:20:240
275492180,16sleep3138550irq/27-eth000:28:123
643199170,8cyclictest0-21swapper/321:12:153
643099173,11cyclictest12159-21sshd23:16:012
643099173,11cyclictest11717-21cp21:41:112
643099173,10cyclictest0-21swapper/219:10:012
643099172,5cyclictest0-21swapper/221:47:372
643099171,1cyclictest4043-21smartctl20:15:152
643099171,13cyclictest1876-21nfsd00:00:082
643099171,11cyclictest8978-21sshd00:35:472
643099171,11cyclictest22496-21sshd21:18:252
643099171,11cyclictest14807-21sshd21:11:432
643099170,15cyclictest18657-21sshd22:17:172
642999173,1cyclictest15006-21sshd22:34:441
642999172,5cyclictest0-21swapper/122:52:591
642999170,12cyclictest1987-21sshd22:14:361
642999170,12cyclictest1260-21sshd00:31:551
642999170,11cyclictest14211-21sshd21:50:181
642799172,1cyclictest31142-21sshd21:10:150
642799172,1cyclictest0-21swapper/021:17:290
642799171,2cyclictest0-21swapper/000:32:140
642799170,16cyclictest0-21swapper/019:46:530
642799170,14cyclictest0-21swapper/020:04:020
642799170,13cyclictest0-21swapper/019:31:290
642799170,12cyclictest10709-21rm21:52:570
642799170,11cyclictest23089-21sshd22:02:560
642799170,11cyclictest0-21swapper/021:25:160
642799170,10cyclictest0-21swapper/000:05:460
299272170,15sleep1138550irq/27-eth023:08:461
643199168,6cyclictest7843-21rm23:03:443
643199164,4cyclictest24655-21sshd23:50:023
643199161,6cyclictest0-21swapper/322:58:233
643099167,3cyclictest4278-21sshd21:55:182
643099163,8cyclictest20917-21runrttasks20:52:172
643099163,8cyclictest19797-21idleruntime-cro19:40:002
643099163,5cyclictest0-21swapper/219:46:232
643099163,10cyclictest23945-21sshd23:40:592
643099163,10cyclictest0-21swapper/219:25:052
643099162,13cyclictest0-21swapper/220:25:152
643099162,11cyclictest4089-21sshd23:45:112
643099162,11cyclictest27676-21sshd22:06:152
643099162,10cyclictest0-21swapper/220:20:412
6430991610,5cyclictest21787-21sshd00:21:092
643099160,10cyclictest0-21swapper/220:32:192
642999168,3cyclictest19479-21sshd22:23:181
642999163,11cyclictest11868-21sshd22:46:101
642999160,4cyclictest14116-21sshd23:40:091
642999160,3cyclictest7505-21sshd21:22:591
642999160,3cyclictest0-21swapper/123:31:091
642999160,3cyclictest0-21swapper/121:57:221
642999160,12cyclictest0-21swapper/100:04:541
642799160,1cyclictest0-21swapper/019:26:430
642799160,1cyclictest0-21swapper/019:20:410
642799160,15cyclictest0-21swapper/019:40:540
643199156,4cyclictest25206-21sshd00:37:263
643199151,13cyclictest0-21swapper/300:17:473
643199151,12cyclictest31010-21bash21:19:123
643199151,12cyclictest22012-21sshd23:28:583
6431991510,3cyclictest401-21sshd22:09:453
643199150,13cyclictest0-21swapper/300:21:533
643099152,8cyclictest0-21swapper/220:00:052
643099152,7cyclictest18263-21cat00:30:252
643099152,10cyclictest0-21swapper/220:11:112
643099151,8cyclictest0-21swapper/219:55:292
643099151,11cyclictest23774-21cpu21:00:142
642999158,3cyclictest8801-21sshd21:40:541
6429991511,3cyclictest12809-21bash22:55:161
6429991510,1cyclictest11673-21sshd22:28:291
642999150,6cyclictest138550irq/27-eth023:48:051
642999150,6cyclictest0-21swapper/100:12:471
642999150,3cyclictest0-21swapper/121:38:581
642999150,13cyclictest3223-21sshd23:36:061
642799151,1cyclictest0-21swapper/020:40:230
642799150,2cyclictest0-21swapper/020:16:500
642799150,1cyclictest0-21swapper/020:07:000
642799150,14cyclictest0-21swapper/020:37:520
194402150,13sleep3138550irq/27-eth000:14:373
186432150,13sleep2138550irq/27-eth020:46:562
643199149,3cyclictest5217-21sshd23:36:173
643199148,5cyclictest0-21swapper/319:26:323
643199145,4cyclictest7839-21sshd23:42:303
643199143,2cyclictest7912-21ls23:30:323
643199142,4cyclictest0-21swapper/322:49:523
643199141,4cyclictest23361-21chrt20:59:423
643199141,12cyclictest0-21swapper/320:48:143
643199141,12cyclictest0-21swapper/320:15:453
643199141,12cyclictest0-21swapper/319:34:413
643199141,10cyclictest138550irq/27-eth021:34:093
643099148,1cyclictest0-21swapper/220:55:352
643099143,6cyclictest0-21swapper/219:42:572
643099143,5cyclictest0-21swapper/219:52:132
643099143,10cyclictest1877-21nfsd19:20:532
643099141,8cyclictest138550irq/27-eth019:16:042
643099140,12cyclictest16762-21munin-node19:30:212
642999145,3cyclictest147450irq/28-eth1-rx-20:17:501
642999142,11cyclictest138550irq/27-eth020:08:181
642999140,13cyclictest0-21swapper/119:35:011
642999140,12cyclictest15762-21sh20:40:201
642999140,11cyclictest28781-21perf19:59:591
642999140,10cyclictest16783-21munin-node19:30:211
642799148,5cyclictest0-21swapper/019:10:140
642799146,7cyclictest10691-21chrt19:18:510
642799142,8cyclictest19659-21fschecks_time20:50:140
642799141,8cyclictest3447-21kworker/0:019:57:310
642799141,8cyclictest138550irq/27-eth020:45:530
642799140,1cyclictest18519-21irqstats19:35:160
642799140,1cyclictest0-21swapper/021:00:150
642799140,13cyclictest0-21swapper/020:55:030
642799140,13cyclictest0-21swapper/020:25:430
642799140,13cyclictest0-21swapper/020:20:170
642799140,11cyclictest0-21swapper/019:52:370
642799140,10cyclictest0-21swapper/020:12:350
643199138,3cyclictest22580-21df_inode19:45:133
643199131,4cyclictest0-21swapper/320:20:203
643199131,3cyclictest9879-21munin-node19:15:193
643199131,11cyclictest1877-21nfsd19:40:393
643199131,11cyclictest0-21swapper/321:02:323
643199131,11cyclictest0-21swapper/319:56:143
643199131,11cyclictest0-21swapper/319:22:563
643199131,10cyclictest1790-21munin-node20:10:163
6431991310,2cyclictest1877-21nfsd20:35:023
643199130,12cyclictest0-21swapper/320:05:123
643199130,12cyclictest0-21swapper/319:50:013
643199130,12cyclictest0-21swapper/319:35:073
643199130,11cyclictest29330-21diskstats20:00:133
643199130,11cyclictest26604-21munin-node21:05:153
643199130,11cyclictest26604-21munin-node21:05:153
643099133,5cyclictest0-21swapper/220:44:292
643099132,8cyclictest12980-21perl20:35:142
643099130,12cyclictest0-21swapper/220:05:532
642999139,3cyclictest138550irq/27-eth019:45:451
642999130,5cyclictest0-21swapper/120:00:531
642999130,4cyclictest26516-21crond19:55:001
642999130,4cyclictest24897-21sensors_temp21:00:231
642999130,12cyclictest138550irq/27-eth020:25:551
642999130,12cyclictest0-21swapper/120:22:331
642999130,12cyclictest0-21swapper/120:10:021
642999130,12cyclictest0-21swapper/119:25:481
642999130,12cyclictest0-21swapper/119:21:591
642999130,12cyclictest0-21swapper/119:14:481
642999130,11cyclictest20278-21df19:40:141
643199121,2cyclictest0-21swapper/320:30:233
643199120,3cyclictest9152-21munin-node20:25:233
643199120,10cyclictest7668-21open_inodes19:10:203
643199120,10cyclictest15290-21hddtemp_smartct20:40:153
642999120,3cyclictest27662-21chrt21:07:091
642999120,3cyclictest27662-21chrt21:07:091
642999120,3cyclictest21704-21munin-node20:55:121
642999120,3cyclictest0-21swapper/120:35:141
642999120,11cyclictest1877-21nfsd20:46:081
642999120,11cyclictest1877-21nfsd20:30:011
642999120,11cyclictest0-21swapper/120:50:261
642999120,11cyclictest0-21swapper/119:15:031
643199110,2cyclictest19401-21taskset20:50:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional