You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-09 - 00:10

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot0.osadl.org (updated Mon Aug 08, 2022 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1749621680,2sleep33075499cyclictest10:27:533
973321500,3sleep03074299cyclictest12:25:320
3086321420,2sleep13074599cyclictest11:44:351
946221320,1sleep20-21swapper/211:54:462
3170521310,2sleep20-21swapper/211:44:412
1917321290,1sleep10-21swapper/110:40:071
1965221270,3sleep23075199cyclictest09:19:022
1099021270,2sleep03074299cyclictest11:27:350
1158321260,1sleep047-21rcuop/311:06:220
546421210,2sleep041ktimersoftd/010:02:450
3171621200,4sleep13074599cyclictest12:12:101
2241321200,3sleep03074299cyclictest11:34:410
1596121190,1sleep10-21swapper/111:06:481
1319021110,4sleep20-21swapper/210:15:252
1472721080,1sleep13074599cyclictest09:24:341
254722990,2sleep125457-21sshd09:13:281
191672980,2sleep119168-21sshd12:17:061
80202960,1sleep00-21swapper/009:56:560
47812940,2sleep20-21swapper/209:32:382
3025829432,50sleep30-21swapper/307:07:043
125612930,1sleep10-21swapper/109:27:181
155812920,1sleep00-21swapper/012:19:590
183252890,3sleep10-21swapper/110:45:551
28712880,2sleep12870-21id11:38:541
3031828733,49sleep10-21swapper/107:07:501
190202860,5sleep30-21swapper/311:49:343
42992850,3sleep04277-21sshd12:21:540
42992850,3sleep04277-21sshd12:21:540
307302850,2sleep30-21swapper/311:50:353
143792850,3sleep214368-21sshd11:39:592
287832840,1sleep328785-21bash11:59:413
27402840,2sleep10-21swapper/110:32:321
134362840,4sleep23075199cyclictest09:12:212
132062840,1sleep00-21swapper/011:21:450
76892830,4sleep13074599cyclictest12:00:391
50912830,2sleep10-21swapper/112:09:421
105612820,1sleep20-21swapper/212:37:562
233432810,2sleep323346-21sshd10:49:313
89372800,1sleep33075499cyclictest10:39:073
216062790,2sleep321600-21bash10:07:143
19352790,1sleep00-21swapper/012:30:570
133822790,2sleep10-21swapper/110:24:311
109352780,2sleep30-21swapper/312:10:153
103532770,1sleep010347-21sshd09:45:060
48952760,1sleep24874-21sshd09:59:422
3045627432,13sleep20-21swapper/207:09:312
298302740,2sleep329827-21sshd09:31:533
189512740,1sleep233-21ksoftirqd/211:46:292
165102740,2sleep316511-21sshd11:34:073
121852740,3sleep23075199cyclictest10:36:222
5672730,2sleep30-21swapper/310:20:203
25182710,0sleep00-21swapper/009:26:210
234652670,1sleep20-21swapper/211:22:452
326552660,3sleep23075199cyclictest12:09:142
271812660,0sleep00-21swapper/011:44:140
274542640,1sleep027442-21bash09:10:340
157752620,1sleep20-21swapper/210:54:502
3031725532,11sleep00-21swapper/007:07:490
108652530,0sleep110866-21bash11:00:231
3074299460,14cyclictest13213-21sshd10:30:290
3074299430,3cyclictest17161-21bash10:09:510
3074299421,2cyclictest0-21swapper/009:21:190
3074299410,9cyclictest12964-21id12:07:200
47892400,2sleep34787-21bash09:53:383
3074299401,14cyclictest0-21swapper/009:34:200
3074299385,3cyclictest1542-21bash09:53:190
30751993731,4cyclictest21430-21bash12:11:092
3074299377,7cyclictest29817-21sshd10:41:000
3074299371,12cyclictest0-21swapper/011:56:080
3074299370,34cyclictest8036-21sshd11:18:170
3074299370,15cyclictest30294-21sshd10:14:030
119982370,34sleep1138550irq/27-eth010:54:301
50512360,1sleep25052-21sshd10:02:432
3074299361,2cyclictest16979-21sshd10:54:570
3074299361,10cyclictest0-21swapper/010:17:550
89332340,0sleep28932-21sshd11:03:122
30742993411,2cyclictest0-21swapper/010:46:270
3074299321,29cyclictest0-21swapper/010:24:230
217102320,2sleep3431ktimersoftd/310:55:193
3402310,1sleep1341-21bash09:32:111
30751993127,3cyclictest19330-21sshd11:10:132
3074299311,13cyclictest0-21swapper/009:43:340
198932310,1sleep30-21swapper/312:35:403
30751993022,4cyclictest22385-21sshd10:13:182
30751992924,3cyclictest28720-21sshd11:08:002
30751992916,10cyclictest13398-21sshd09:24:262
3074599290,26cyclictest22-21ksoftirqd/109:47:351
3074299290,14cyclictest0-21swapper/011:35:220
205342290,27sleep1138550irq/27-eth010:13:091
104072290,4sleep33075499cyclictest11:36:333
30751992821,5cyclictest7648-21sshd10:41:572
30751992819,5cyclictest17441-21sshd12:26:182
30751992814,11cyclictest27943-21sshd09:52:442
3074299280,22cyclictest22923-21bash12:35:590
3074299280,10cyclictest2378-21bash12:00:160
30751992721,3cyclictest17899-21sshd12:32:292
30751992715,6cyclictest27420-21bash11:59:332
30751992713,9cyclictest30144-21sshd12:15:072
205972271,24sleep3138550irq/27-eth011:19:303
30751992622,2cyclictest30416-21sshd10:08:032
30751992621,3cyclictest757-21id09:26:112
30751992621,3cyclictest20549-21sshd10:55:162
30751992621,3cyclictest1987-21sshd09:37:102
30751992621,3cyclictest17787-21sed11:25:142
30751992620,3cyclictest2810-21sshd10:26:292
30751992618,4cyclictest4774-21sshd12:21:582
30751992618,4cyclictest4774-21sshd12:21:572
30751992613,10cyclictest0-21swapper/210:23:182
3074599260,17cyclictest22-21ksoftirqd/110:38:231
298442260,1sleep20-21swapper/212:02:492
75502250,23sleep1138550irq/27-eth011:45:251
48712250,23sleep128750irq/26-ahci[00010:02:431
3075499250,9cyclictest12056-21bash10:03:223
30751992520,3cyclictest7509-21sshd09:47:482
30751992520,3cyclictest30154-21sshd10:32:042
30751992520,3cyclictest13423-21sshd09:42:212
30751992519,4cyclictest26829-21sshd11:16:592
30751992518,4cyclictest27726-21sshd11:32:072
30751992518,4cyclictest24042-21sshd10:46:282
3074599257,13cyclictest10725-21sshd12:25:381
30745992513,5cyclictest138550irq/27-eth011:20:271
197192250,23sleep3138550irq/27-eth009:15:553
3075499248,1cyclictest0-21swapper/312:15:283
30754992418,4cyclictest7946-21sshd12:28:313
30751992417,4cyclictest4922-21turbostat.cron07:24:592
3074599241,20cyclictest13893-21bash09:42:231
3074299241,13cyclictest0-21swapper/009:17:360
3074299240,20cyclictest0-21swapper/011:45:520
226662241,21sleep128750irq/26-ahci[00012:32:581
47452230,21sleep3138550irq/27-eth009:41:323
30754992313,3cyclictest0-21swapper/310:50:303
30745992312,4cyclictest8442-21sshd10:27:011
30745992312,4cyclictest47-21rcuop/311:14:291
3074299236,8cyclictest0-21swapper/009:39:320
3074299231,9cyclictest0-21swapper/008:00:330
263252230,22sleep128750irq/26-ahci[00011:53:191
76362220,2sleep30-21swapper/311:21:123
3075499229,5cyclictest138550irq/27-eth011:40:173
3075499222,11cyclictest0-21swapper/311:29:393
30754992213,2cyclictest138550irq/27-eth012:07:533
30754992211,4cyclictest11650-21bash12:22:373
30754992211,4cyclictest11650-21bash12:22:373
30745992212,4cyclictest17629-21sshd09:51:451
249922220,2sleep10-21swapper/109:37:291
3075499215,12cyclictest0-21swapper/307:14:523
3074599218,11cyclictest2629-21sshd10:05:281
30745992116,3cyclictest30995-21id11:26:261
30745992114,5cyclictest25-21rcuop/110:18:591
30745992111,7cyclictest25725-21bash09:19:371
3074299211,15cyclictest0-21swapper/010:36:370
3075499209,2cyclictest14731-21munin-node10:30:353
3075499207,1cyclictest32402-21sshd10:44:173
30754992014,4cyclictest18033-21sshd10:12:543
3075499200,18cyclictest44-21ksoftirqd/309:47:023
3075499200,18cyclictest27390-21sshd12:02:343
3075499200,10cyclictest31596-21sshd11:02:163
3074599209,4cyclictest138550irq/27-eth009:57:151
30745992016,2cyclictest2790-21consoletype11:17:461
30745992015,3cyclictest12526-21sshd11:58:071
30745992014,4cyclictest26666-21sshd10:55:441
30745992012,4cyclictest0-21swapper/112:20:261
30745992012,4cyclictest0-21swapper/112:20:251
3075499199,2cyclictest15440-21sshd11:12:553
3075499196,5cyclictest2121-21runrttasks09:21:583
30754991913,4cyclictest0-21swapper/312:30:303
30754991911,5cyclictest24081-21bash09:10:193
3075499190,8cyclictest0-21swapper/309:39:433
3075499190,15cyclictest2811-21sshd09:29:273
3075199192,13cyclictest0-21swapper/208:34:372
30751991917,1cyclictest7168-21sh07:30:002
30745991911,5cyclictest13664-21sshd12:35:111
3074599190,13cyclictest11670-21sshd11:33:401
3074299195,7cyclictest0-21swapper/011:03:540
30742991914,3cyclictest0-21swapper/010:27:140
3074299191,2cyclictest7292-21sshd10:57:000
3074299191,15cyclictest9184-21sshd12:13:050
3075499180,15cyclictest0-21swapper/307:15:223
3075499180,10cyclictest31841-21sshd10:17:103
3075199183,10cyclictest1877-21nfsd08:05:472
3074299180,17cyclictest18302-21sshd11:10:070
3074299180,15cyclictest10348-21sshd11:51:460
30754991710,6cyclictest138550irq/27-eth011:05:533
3075199173,11cyclictest0-21swapper/207:54:072
3075199171,13cyclictest0-21swapper/207:16:172
3075499167,4cyclictest138550irq/27-eth009:55:353
3075499160,13cyclictest0-21swapper/307:26:223
3075199169,4cyclictest7698-21grep08:40:242
3075199161,14cyclictest33-21ksoftirqd/208:45:572
30745991612,3cyclictest147450irq/28-eth1-rx-08:33:291
3074299160,15cyclictest0-21swapper/008:26:090
3075499150,6cyclictest0-21swapper/307:49:463
3075499150,10cyclictest20662-21munin-run08:00:013
3075499150,10cyclictest19796-21munin-run09:10:003
3075199159,5cyclictest0-21swapper/208:23:132
3075199157,5cyclictest0-21swapper/209:03:072
3075199151,13cyclictest0-21swapper/208:04:252
3074599152,12cyclictest0-21swapper/108:07:371
3074599150,6cyclictest0-21swapper/107:17:291
3074599150,12cyclictest138550irq/27-eth007:31:241
3074299150,11cyclictest0-21swapper/007:41:230
324722140,12sleep2138550irq/27-eth007:11:122
3075499147,6cyclictest0-21swapper/309:01:073
3075499146,4cyclictest13042-21perf08:55:003
3075499143,1cyclictest0-21swapper/308:55:353
3075499140,5cyclictest0-21swapper/308:39:333
3075499140,12cyclictest0-21swapper/307:40:303
3075499140,10cyclictest138550irq/27-eth007:30:573
3075199148,5cyclictest0-21swapper/209:08:112
3075199147,5cyclictest0-21swapper/208:25:172
3075199141,12cyclictest0-21swapper/207:42:572
3075199140,2cyclictest0-21swapper/208:58:372
3075199140,13cyclictest33-21ksoftirqd/207:46:472
3074599146,7cyclictest15705-21unixbench_multi07:45:351
3074599143,10cyclictest31712-21chrt08:22:421
30745991410,3cyclictest6835-21runrttasks07:26:071
3074599140,5cyclictest0-21swapper/108:51:511
3074599140,13cyclictest0-21swapper/108:01:531
3074299147,4cyclictest0-21swapper/008:57:050
3074299140,13cyclictest0-21swapper/009:03:080
3074299140,13cyclictest0-21swapper/008:41:190
3074299140,13cyclictest0-21swapper/008:14:480
3075499138,4cyclictest1877-21nfsd08:20:503
3075499132,1cyclictest0-21swapper/307:21:353
3075499132,10cyclictest138550irq/27-eth008:31:243
3075499131,11cyclictest138550irq/27-eth008:17:033
3075499130,12cyclictest0-21swapper/308:41:483
3075499130,12cyclictest0-21swapper/308:26:473
3075499130,12cyclictest0-21swapper/307:51:353
3075499130,12cyclictest0-21swapper/307:35:173
3075499130,11cyclictest9510-21sh08:45:193
3075199131,3cyclictest0-21swapper/208:10:192
3075199131,11cyclictest10864-21proc_pri07:35:352
3075199131,11cyclictest0-21swapper/207:30:252
3075199130,12cyclictest0-21swapper/208:51:092
3075199130,12cyclictest0-21swapper/208:18:552
3075199130,12cyclictest0-21swapper/207:55:512
3074599139,3cyclictest138550irq/27-eth008:36:021
3074599132,2cyclictest0-21swapper/108:43:021
3074599132,1cyclictest0-21swapper/109:02:031
3074599130,5cyclictest0-21swapper/107:35:181
3074599130,2cyclictest138550irq/27-eth008:25:511
3074599130,12cyclictest0-21swapper/109:08:251
3074599130,12cyclictest0-21swapper/107:55:321
3074599130,12cyclictest0-21swapper/107:51:201
3074599130,12cyclictest0-21swapper/107:40:101
3074599130,12cyclictest0-21swapper/107:24:511
3074599130,12cyclictest0-21swapper/107:10:241
3074299139,3cyclictest0-21swapper/008:53:390
3074299138,4cyclictest0-21swapper/007:10:270
30742991311,1cyclictest0-21swapper/008:45:520
30742991311,1cyclictest0-21swapper/008:20:580
3074299130,9cyclictest6015-21memory07:25:230
3074299130,5cyclictest0-21swapper/008:15:020
3074299130,3cyclictest0-21swapper/007:21:410
3074299130,2cyclictest0-21swapper/007:50:260
3074299130,2cyclictest0-21swapper/007:40:010
3074299130,12cyclictest0-21swapper/007:55:320
3074299130,12cyclictest0-21swapper/007:15:130
30754991210,1cyclictest0-21swapper/308:00:043
3075499120,11cyclictest27120-21taskset08:11:383
3075499120,11cyclictest24822-21runrttasks08:06:013
3075199120,10cyclictest5426-21munin-node08:35:262
3074599121,8cyclictest9050-21df_abs08:45:161
3074599120,3cyclictest14470-21munin-node08:55:211
3074599120,11cyclictest25382-21runrttasks08:10:031
3074599120,10cyclictest0-21swapper/108:15:181
3074299128,3cyclictest0-21swapper/008:30:250
3074299128,2cyclictest0-21swapper/008:06:020
3074299128,1cyclictest138550irq/27-eth008:37:200
3074299120,8cyclictest7644-21cpuspeed_turbos07:30:150
3074299120,11cyclictest0-21swapper/009:05:040
3074299120,11cyclictest0-21swapper/007:46:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional