You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-05-27 - 18:30

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack5slot0.osadl.org (updated Fri May 27, 2022 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1711121380,1sleep017109-21bash11:40:440
588021140,0sleep35879-21sshd10:24:073
1178921090,1sleep211786-21sshd09:20:062
29810210633,46sleep10-21swapper/107:06:111
30090210232,41sleep30-21swapper/307:09:343
388321000,1sleep00-21swapper/011:09:540
9142990,4sleep30-21swapper/310:12:583
297832990,3sleep30-21swapper/311:50:013
73892980,7sleep20-21swapper/212:01:422
67492970,2sleep30-21swapper/312:01:403
240922940,3sleep30-21swapper/312:35:473
85412920,1sleep20-21swapper/212:20:512
51642920,2sleep15160-21rm09:11:271
277442900,0sleep00-21swapper/012:38:520
274572890,1sleep20-21swapper/210:31:162
223272880,2sleep222332-21sshd11:22:152
119422880,2sleep30-21swapper/309:20:063
92442870,4sleep33030499cyclictest11:02:113
282622870,2sleep23030399cyclictest11:41:432
160612870,1sleep016050-21sshd12:10:340
43522860,2sleep10-21swapper/111:37:001
207552850,2sleep320758-21bash12:24:403
231892840,2sleep223188-21sshd11:08:472
192452840,2sleep119244-21bash12:29:591
145052840,2sleep114497-21sshd10:32:561
43712830,4sleep13030299cyclictest12:04:141
99512820,0sleep00-21swapper/012:15:320
271432820,0sleep00-21swapper/009:29:250
314952810,3sleep20-21swapper/209:29:472
217472810,1sleep30-21swapper/309:42:233
100722810,2sleep10-21swapper/110:54:081
172432800,2sleep317242-21sshd11:43:313
242412790,2sleep124240-21munin-plugin-st11:54:591
13742790,2sleep10-21swapper/111:23:131
70002780,2sleep20-21swapper/212:28:562
304652780,1sleep230462-21sshd09:32:202
84982770,4sleep03030199cyclictest11:12:590
233642770,4sleep33030499cyclictest12:30:183
22142770,2sleep11876-21nfsd10:40:011
221392770,0sleep222128-21sshd10:36:122
10162760,1sleep00-21swapper/012:33:550
312272750,4sleep23030399cyclictest10:15:252
301662750,4sleep13030299cyclictest08:20:151
289692740,1sleep30-21swapper/312:17:113
113772700,0sleep00-21swapper/011:15:500
10042690,1sleep10-21swapper/110:47:591
127472680,2sleep10-21swapper/111:48:351
228752620,2sleep10-21swapper/110:57:561
303592570,1sleep20-21swapper/212:17:172
2991525632,12sleep20-21swapper/207:07:322
2984425531,4sleep00-21swapper/007:06:390
156912470,0sleep215692-21id10:51:512
115822460,44sleep1138550irq/27-eth011:15:531
156502450,4sleep30-21swapper/310:49:123
124492440,0sleep112448-21bash10:11:111
30301993825,11cyclictest28450-21bash09:24:090
30301993529,3cyclictest0-21swapper/009:18:150
27332350,0sleep22731-21sshd11:31:232
144922350,0sleep114493-21sshd09:44:301
40072340,2sleep10-21swapper/110:42:491
36662340,0sleep23665-21sshd09:51:402
158212340,32sleep2138550irq/27-eth009:36:302
310582310,0sleep00-21swapper/011:36:320
300732300,28sleep1138550irq/27-eth010:26:061
30302992922,4cyclictest29662-21sshd11:58:101
30302992913,13cyclictest22-21ksoftirqd/110:08:441
40882280,3sleep24078-21sshd10:29:232
3030299289,5cyclictest7712-21sshd12:31:421
3030299286,19cyclictest22-21ksoftirqd/111:11:201
249442280,2sleep20-21swapper/212:38:382
215462280,2sleep321544-21bash10:28:093
30302992722,2cyclictest0-21swapper/111:32:101
240692270,2sleep00-21swapper/011:33:150
116652270,24sleep1138550irq/27-eth010:00:251
324732260,4sleep20-21swapper/211:52:572
30302992613,8cyclictest22-21ksoftirqd/109:51:191
97422250,2sleep39739-21bash10:51:213
54432250,23sleep3138550irq/27-eth011:10:023
32412250,3sleep03030199cyclictest09:48:580
312542250,23sleep1138550irq/27-eth009:56:411
30303992512,3cyclictest28138-21sshd12:30:392
30302992515,6cyclictest4909-21bash09:22:101
3030299250,21cyclictest22-21ksoftirqd/109:27:161
296452250,23sleep3138550irq/27-eth011:09:223
272192250,2sleep327218-21bash11:25:193
164612250,2sleep216460-21sshd09:47:192
90332240,2sleep20-21swapper/211:02:112
3030399242,3cyclictest21958-21sshd11:11:212
3030299245,11cyclictest22-21ksoftirqd/111:06:241
30302992417,5cyclictest22-21ksoftirqd/109:49:491
30302992416,1cyclictest22-21ksoftirqd/111:40:391
30302992415,6cyclictest31518-21sshd11:04:061
30302992415,5cyclictest1674-21needreboot12:20:181
299362240,2sleep229922-21sshd11:25:312
29142240,1sleep00-21swapper/010:59:000
222722240,22sleep3138550irq/27-eth009:37:023
177462240,2sleep30-21swapper/309:55:303
49832230,2sleep14985-21sshd09:19:331
30302992318,3cyclictest138550irq/27-eth012:07:031
30302992315,5cyclictest14581-21sshd09:33:431
30302992313,5cyclictest1987-21sshd11:25:461
3030299230,19cyclictest22-21ksoftirqd/112:39:301
199562230,2sleep20-21swapper/211:49:122
89822223,17sleep3138550irq/27-eth011:31:593
30304992213,5cyclictest23104-21bash10:39:033
3030399222,17cyclictest1876-21nfsd10:41:122
3030399222,15cyclictest33-21ksoftirqd/210:55:562
30302992216,4cyclictest22-21ksoftirqd/110:19:331
30302992213,6cyclictest28693-21sshd12:11:421
30302992212,6cyclictest31811-21sshd09:35:111
30302992211,7cyclictest8151-21sshd12:15:231
3030199220,13cyclictest20777-21sshd09:50:220
30304992116,3cyclictest36-21rcuop/210:31:563
30304992113,5cyclictest16535-21sshd11:48:543
30304992111,6cyclictest20377-21sshd10:57:443
30304992111,5cyclictest138550irq/27-eth009:31:223
3030399213,16cyclictest0-21swapper/212:09:412
30303992117,2cyclictest22850-21sshd12:11:112
30303992116,3cyclictest4128-21sshd11:55:572
30303992116,3cyclictest32211-21bash10:02:082
30303992116,3cyclictest2227-21sshd11:36:502
30303992116,3cyclictest1877-21nfsd09:57:302
3030199219,3cyclictest23482-21sshd10:12:080
3030199210,12cyclictest8683-21bash11:04:530
228182212,17sleep3138550irq/27-eth012:13:583
18262210,1sleep00-21swapper/010:53:270
3030499209,1cyclictest0-21swapper/311:55:393
3030499203,7cyclictest31922-21rm11:23:043
3030499200,2cyclictest0-21swapper/310:03:373
3030499200,17cyclictest32711-21sshd10:07:313
3030399209,4cyclictest138550irq/27-eth010:24:302
3030399202,12cyclictest0-21swapper/207:57:322
30303992013,5cyclictest18429-21bash09:12:342
30303992013,4cyclictest4243-21bash10:45:282
30303992013,4cyclictest18365-21cp10:11:402
30303992012,4cyclictest23841-21sshd09:15:382
30302992012,5cyclictest30358-21bash10:20:431
3030199208,3cyclictest4549-21sshd09:30:130
3030199204,13cyclictest16110-21sshd11:48:520
30301992016,2cyclictest0-21swapper/010:01:310
30301992015,3cyclictest3-21ksoftirqd/009:11:090
3030199201,11cyclictest14702-21sshd12:21:230
3030199200,19cyclictest1936-21bash11:23:160
3030199200,11cyclictest28498-21sshd09:56:260
186832200,1sleep00-21swapper/011:51:470
3030499199,8cyclictest10717-21sshd09:11:543
30303991915,2cyclictest1987-21sshd11:16:502
3030199190,17cyclictest0-21swapper/011:27:050
3030199190,15cyclictest22327-21bash09:39:450
3030199190,11cyclictest8572-21sshd10:37:490
252532190,17sleep3138550irq/27-eth011:19:483
3030499189,1cyclictest0-21swapper/312:29:153
30303991810,5cyclictest20121-21sshd10:09:122
30303991810,4cyclictest0-21swapper/208:49:242
3030199180,13cyclictest0-21swapper/010:30:140
298182180,0sleep00-21swapper/010:42:160
3030499179,5cyclictest21482-21sshd09:45:053
3030499178,5cyclictest32418-21sshd12:06:333
3030499176,4cyclictest20158-21sshd11:35:343
3030499175,3cyclictest12613-21sshd10:40:483
3030499173,0cyclictest0-21swapper/309:50:493
30304991713,3cyclictest20719-21bash09:26:073
30304991712,3cyclictest28156-21sshd10:15:133
3030499171,11cyclictest23283-21sshd09:15:373
30303991711,4cyclictest29406-21sshd09:40:182
3030199177,5cyclictest13730-21rm10:46:180
3030199175,10cyclictest0-21swapper/012:03:010
3030199171,9cyclictest0-21swapper/012:27:470
3030199171,14cyclictest0-21swapper/011:56:200
3030199170,14cyclictest32332-21sshd10:23:370
3030199170,14cyclictest0-21swapper/012:06:440
3030199170,14cyclictest0-21swapper/010:25:030
3030199170,14cyclictest0-21swapper/009:08:460
3030399162,11cyclictest7265-21ntp_states08:40:212
30303991612,2cyclictest5933-21cut07:25:222
3030399160,5cyclictest18162-21kernelversion09:05:192
3030199169,6cyclictest0-21swapper/010:19:080
3030199166,6cyclictest1271-21sshd09:40:380
3030199160,9cyclictest28301-21sshd10:09:520
3030499158,5cyclictest15481-21fschecks_count09:00:163
3030499150,13cyclictest0-21swapper/307:14:193
3030499150,12cyclictest0-21swapper/307:50:533
30303991510,4cyclictest0-21swapper/207:42:522
3030399150,13cyclictest0-21swapper/207:35:132
3030399150,13cyclictest0-21swapper/207:35:122
3030299151,9cyclictest15788-21munin-run07:50:001
3030299150,6cyclictest0-21swapper/108:30:211
3030499143,1cyclictest0-21swapper/308:41:543
3030399143,1cyclictest0-21swapper/208:22:082
3030399141,8cyclictest15190-21atenall_r5power09:00:122
30303991411,2cyclictest484-21df_abs07:15:112
30303991410,3cyclictest32488-21latency_hist07:15:012
3030399140,5cyclictest0-21swapper/208:58:082
3030399140,12cyclictest1955-21snmpd08:54:402
3030299147,6cyclictest22-21ksoftirqd/108:05:241
3030299142,10cyclictest7756-21munin-node07:30:141
3030299141,11cyclictest0-21swapper/107:55:221
3030299141,11cyclictest0-21swapper/107:25:161
3030299141,11cyclictest0-21swapper/107:10:231
30302991410,3cyclictest0-21swapper/108:26:461
3030199148,3cyclictest0-21swapper/008:55:540
3030199140,2cyclictest4700-21latency_hist07:25:010
3030499139,3cyclictest138550irq/27-eth008:10:583
3030499138,3cyclictest19271-21grep07:55:203
3030499137,5cyclictest0-21swapper/307:42:183
3030499137,5cyclictest0-21swapper/307:33:403
3030499136,6cyclictest138550irq/27-eth008:45:233
30304991310,2cyclictest0-21swapper/308:30:133
3030499130,5cyclictest0-21swapper/309:06:133
3030499130,5cyclictest0-21swapper/308:01:313
3030499130,4cyclictest0-21swapper/308:15:093
3030499130,12cyclictest1877-21nfsd08:55:173
3030499130,12cyclictest1877-21nfsd08:51:453
3030499130,12cyclictest0-21swapper/308:27:223
3030499130,12cyclictest0-21swapper/308:20:503
3030499130,12cyclictest0-21swapper/308:05:233
3030499130,12cyclictest0-21swapper/307:17:113
3030499130,11cyclictest5507-21iostat07:25:153
3030499130,11cyclictest4624-21iostat08:35:163
3030399137,5cyclictest0-21swapper/207:50:542
3030399131,11cyclictest138550irq/27-eth008:36:292
3030399131,11cyclictest138550irq/27-eth008:25:502
30303991311,1cyclictest0-21swapper/208:06:582
3030399130,5cyclictest138550irq/27-eth008:00:532
3030399130,12cyclictest0-21swapper/208:15:532
3030399130,12cyclictest0-21swapper/207:30:092
3030399130,10cyclictest2757-21cpuspeed_turbos07:20:132
3030299138,4cyclictest0-21swapper/108:55:051
3030299131,3cyclictest0-21swapper/109:06:271
30302991311,1cyclictest0-21swapper/108:15:201
3030299131,11cyclictest0-21swapper/107:39:011
3030299131,11cyclictest0-21swapper/107:39:001
30302991310,2cyclictest22-21ksoftirqd/108:45:201
30302991310,2cyclictest22-21ksoftirqd/107:40:181
3030299130,4cyclictest21628-21munin-node08:00:241
3030299130,12cyclictest0-21swapper/109:02:251
3030299130,12cyclictest0-21swapper/108:50:101
3030299130,12cyclictest0-21swapper/108:41:441
3030299130,12cyclictest0-21swapper/108:35:061
3030299130,12cyclictest0-21swapper/107:54:231
3030299130,12cyclictest0-21swapper/107:15:111
3030299130,11cyclictest3179-21hddtemp_smartct07:20:141
3030199139,1cyclictest0-21swapper/007:30:120
3030199137,3cyclictest0-21swapper/007:40:210
3030199130,9cyclictest8164-21latency_hist08:45:000
3030199130,1cyclictest138550irq/27-eth008:23:590
3030199130,12cyclictest0-21swapper/009:02:010
3030199130,12cyclictest0-21swapper/008:37:330
3030199130,12cyclictest0-21swapper/007:15:160
3030199130,10cyclictest0-21swapper/008:05:330
3030199130,10cyclictest0-21swapper/007:25:520
3030499128,3cyclictest1805-40kipmi007:45:173
3030499120,11cyclictest2527-21munin-node07:20:133
3030499120,11cyclictest0-21swapper/307:35:033
3030499120,11cyclictest0-21swapper/307:35:033
3030399120,11cyclictest14095-21diskstats07:45:152
3030399120,11cyclictest138550irq/27-eth008:10:012
3030399120,10cyclictest2501-21munin-node08:30:172
3030299120,11cyclictest0-21swapper/108:10:101
3030199128,3cyclictest0-21swapper/008:10:560
3030199127,4cyclictest0-21swapper/008:15:170
3030199124,4cyclictest0-21swapper/008:30:190
3030199120,2cyclictest0-21swapper/007:55:440
3030199120,2cyclictest0-21swapper/007:35:020
3030199120,2cyclictest0-21swapper/007:35:020
3030199120,11cyclictest0-21swapper/008:50:130
3030199120,11cyclictest0-21swapper/008:45:110
3030199120,11cyclictest0-21swapper/008:25:380
3030199120,11cyclictest0-21swapper/008:00:220
3030199120,11cyclictest0-21swapper/007:51:020
3030199120,11cyclictest0-21swapper/007:12:440
3030199110,10cyclictest0-21swapper/007:45:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional