You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:54

Intel(R) Xeon(R) CPU E31220L @ 2.20GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack5slot0.osadl.org (updated Sun Jan 23, 2022 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2265321330,3sleep02944799cyclictest11:15:100
1345721270,0sleep213456-21bash11:29:282
2145921250,4sleep22944999cyclictest10:02:432
1595021240,4sleep32945099cyclictest11:41:033
1190021230,4sleep32945099cyclictest11:10:053
2229821180,3sleep10-21swapper/110:21:341
37642920,1sleep30-21swapper/310:11:543
301252910,2sleep30-21swapper/310:30:103
2903228932,9sleep20-21swapper/207:06:452
48132880,2sleep10-21swapper/111:39:531
160822870,3sleep20-21swapper/209:28:082
85612860,2sleep10-21swapper/110:31:201
246612840,2sleep124662-21sshd11:49:501
2894028332,42sleep10-21swapper/107:05:361
81962810,1sleep20-21swapper/210:50:242
2922828133,44sleep30-21swapper/307:09:053
171032800,1sleep20-21swapper/211:56:332
26672790,1sleep10-21swapper/109:34:081
287712780,2sleep20-21swapper/210:11:012
316042770,1sleep32121-21runrttasks12:09:493
195452770,0sleep00-21swapper/011:56:510
141242770,1sleep114121-21cp10:58:531
190212760,4sleep32945099cyclictest09:43:333
314392740,2sleep131442-21fschecks_count08:25:211
235772740,2sleep31877-21nfsd09:25:173
9412730,1sleep1940-21bash11:20:171
299152700,2sleep30-21swapper/309:48:343
205592700,2sleep20-21swapper/212:16:092
10912690,4sleep22944999cyclictest09:41:222
64002660,2sleep30-21swapper/311:47:363
55992660,4sleep12944899cyclictest11:01:381
233902660,1sleep20-21swapper/211:19:062
88782520,1sleep20-21swapper/212:33:562
111742510,2sleep12944899cyclictest09:46:191
2900324133,4sleep00-21swapper/007:06:220
123142400,1sleep30-21swapper/312:22:513
2944799360,34cyclictest0-21swapper/010:33:290
213232350,32sleep2138550irq/27-eth011:53:172
29450992921,5cyclictest12788-21sshd11:25:263
23212290,0sleep00-21swapper/010:38:190
29450992721,4cyclictest16039-21sshd12:27:063
29450992717,6cyclictest0-21swapper/312:12:373
29449992723,3cyclictest23950-21sshd09:14:132
29449992712,3cyclictest138550irq/27-eth010:21:352
29448992713,5cyclictest138550irq/27-eth011:55:381
29450992617,5cyclictest0-21swapper/309:38:423
148152260,3sleep02944799cyclictest10:43:370
29450992521,3cyclictest3066-21sshd10:45:573
29450992515,7cyclictest2121-21runrttasks10:21:413
29450992515,6cyclictest25950-21sshd11:04:063
29450992515,6cyclictest1987-21sshd10:17:563
29450992515,6cyclictest19620-21sshd11:18:383
29449992517,6cyclictest25-21rcuop/111:00:422
2945099249,11cyclictest19285-21unixbench-2d10:55:313
2945099242,3cyclictest138550irq/27-eth009:13:563
29450992419,3cyclictest0-21swapper/309:30:323
29450992416,4cyclictest8671-21bash11:59:273
2944999248,14cyclictest33-21ksoftirqd/209:20:512
229742240,1sleep010-21rcuc/012:08:450
98192230,2sleep20-21swapper/211:44:142
32652230,3sleep02944799cyclictest10:26:580
314932230,2sleep20-21swapper/209:37:272
29450992319,3cyclictest554-21sshd09:56:293
29450992319,2cyclictest27109-21sshd09:52:013
29450992317,4cyclictest12589-21sshd10:35:343
29450992311,7cyclictest6330-21sshd12:18:223
29450992310,9cyclictest8002-21sshd11:21:003
2944999238,13cyclictest33-21ksoftirqd/211:33:442
29449992311,5cyclictest16043-21sshd09:31:522
2944799231,14cyclictest0-21swapper/009:29:440
2944799231,11cyclictest10107-21sshd11:09:530
29447992310,8cyclictest3599-21nfsd409:45:280
189722230,2sleep30-21swapper/312:38:583
189722230,2sleep30-21swapper/312:38:573
132052230,0sleep00-21swapper/011:14:030
29450992217,4cyclictest10175-21bash11:05:573
29450992217,3cyclictest5124-21sshd11:32:163
29450992215,5cyclictest5719-21bash10:27:163
29450992214,4cyclictest0-21swapper/312:02:493
29450992211,7cyclictest1501-21bash09:19:023
2944999223,16cyclictest138550irq/27-eth011:10:252
29449992215,5cyclictest33-21ksoftirqd/209:17:112
221992220,1sleep10-21swapper/111:53:231
29450992117,3cyclictest0-21swapper/311:50:123
29450992117,2cyclictest0-21swapper/309:22:193
29450992116,3cyclictest0-21swapper/312:31:113
29450992116,3cyclictest0-21swapper/311:36:353
29450992114,4cyclictest0-21swapper/310:06:573
29450992114,3cyclictest0-21swapper/310:40:403
29450992110,8cyclictest26922-21sshd10:03:233
29449992116,3cyclictest8922-21sshd12:03:172
2944899218,4cyclictest12444-21sshd09:27:431
2944799210,12cyclictest16697-21sshd11:45:030
126882210,4sleep20-21swapper/212:22:542
29450992013,5cyclictest7674-21sshd10:50:213
2944999200,3cyclictest138550irq/27-eth012:09:092
29448992014,3cyclictest23745-21sshd12:08:511
29448992013,5cyclictest31463-21sshd10:00:121
2945099199,5cyclictest32124-21ntp_kernel_err08:25:273
2944999192,13cyclictest0-21swapper/211:21:532
2944999192,11cyclictest138550irq/27-eth012:25:172
29449991914,3cyclictest28820-21sshd09:55:592
29449991914,3cyclictest17702-21sshd11:45:102
29449991914,2cyclictest138550irq/27-eth009:48:282
29449991913,4cyclictest15243-21cp10:35:562
29449991912,4cyclictest0-21swapper/211:07:522
2944999190,17cyclictest33-21ksoftirqd/210:28:192
29448991914,3cyclictest22621-21sshd10:29:171
29448991910,5cyclictest1876-21nfsd10:48:511
2944799193,9cyclictest0-21swapper/009:11:110
2944799191,15cyclictest0-21swapper/011:35:190
2944799190,11cyclictest32632-21sshd09:22:340
221602190,2sleep122149-21sshd09:51:231
182752190,1sleep10-21swapper/111:41:211
134742190,2sleep32121-21runrttasks07:45:153
34612180,1sleep00-21swapper/009:15:280
2944999187,4cyclictest7322-21sshd12:14:402
2944999183,10cyclictest1877-21nfsd07:10:512
2944999181,6cyclictest9163-21sshd10:46:412
29449991814,2cyclictest4362-21bash10:08:162
29449991812,4cyclictest4045-21sshd11:39:462
29449991812,4cyclictest20011-21sshd10:32:432
2944999180,13cyclictest1876-21nfsd08:21:282
2944899188,6cyclictest23640-21bash11:19:081
2944899186,7cyclictest23082-21sshd10:10:231
29448991812,3cyclictest0-21swapper/109:14:471
2944799188,2cyclictest0-21swapper/010:24:110
29447991812,2cyclictest0-21swapper/008:16:590
2944799181,11cyclictest0-21swapper/010:56:150
2944799180,8cyclictest20765-21fschecks_count12:35:190
2944799180,8cyclictest20765-21fschecks_count12:35:190
2944799180,10cyclictest2063-21sshd12:02:270
15462180,4sleep12944899cyclictest10:15:251
2945099178,7cyclictest19534-21latency_hist08:00:003
2945099173,10cyclictest7847-21cpuspeed_turbos08:45:223
2945099171,11cyclictest18776-21munin-plugin-st09:10:013
29450991710,4cyclictest5984-21crond07:29:593
2945099170,15cyclictest0-21swapper/307:50:343
2945099170,14cyclictest0-21swapper/308:10:583
2944999174,7cyclictest0-21swapper/208:42:302
2944999172,12cyclictest0-21swapper/208:52:522
2944999172,12cyclictest0-21swapper/207:24:522
2944999171,8cyclictest0-21swapper/210:42:342
29449991712,3cyclictest10170-21taskset07:36:112
2944999171,11cyclictest0-21swapper/207:43:192
29449991710,5cyclictest15086-21sshd10:16:592
2944999170,15cyclictest33-21ksoftirqd/212:38:082
2944999170,15cyclictest33-21ksoftirqd/212:38:082
2944899179,1cyclictest138550irq/27-eth009:40:541
2944899177,2cyclictest30709-21sshd11:08:271
2944899174,0cyclictest0-21swapper/109:22:291
29448991712,3cyclictest0-21swapper/112:39:131
29448991712,3cyclictest0-21swapper/112:39:131
2944899170,2cyclictest0-21swapper/109:56:001
2944899170,15cyclictest0-21swapper/112:32:551
2944899170,13cyclictest138550irq/27-eth010:06:541
2944799178,2cyclictest0-21swapper/009:58:270
2944799172,9cyclictest0-21swapper/011:32:190
2944799171,8cyclictest0-21swapper/011:26:150
2944799171,8cyclictest0-21swapper/010:52:520
2944799171,2cyclictest28614-21sshd09:52:120
2944799171,12cyclictest12886-21sshd09:35:190
2944799171,11cyclictest0-21swapper/010:05:040
29447991710,3cyclictest0-21swapper/012:18:540
2944799170,1cyclictest0-21swapper/011:21:270
101802170,15sleep2138550irq/27-eth009:50:062
2945099163,9cyclictest22992-21nfsd408:05:283
2944999169,4cyclictest0-21swapper/208:31:232
2944999163,8cyclictest0-21swapper/208:55:392
2944999163,8cyclictest0-21swapper/208:17:272
2944999163,8cyclictest0-21swapper/207:55:192
29449991611,3cyclictest23025-21sshd10:59:582
29449991610,4cyclictest31516-21fschecks_time08:25:202
2944899169,5cyclictest14208-21sshd11:29:341
29448991614,1cyclictest0-21swapper/112:25:491
29448991613,1cyclictest0-21swapper/109:17:231
2944899160,11cyclictest1422-21perf07:19:591
2944799168,2cyclictest0-21swapper/008:14:240
2944799162,7cyclictest0-21swapper/012:24:410
2944799161,13cyclictest0-21swapper/010:47:230
2944799161,10cyclictest0-21swapper/009:32:410
2944799160,3cyclictest1877-21nfsd11:01:490
2944799160,11cyclictest2835-21sshd11:54:580
2945099158,4cyclictest891-21grep07:15:333
2945099150,13cyclictest28181-21taskset08:15:553
2945099150,12cyclictest10499-21munin-run07:40:003
2944999153,8cyclictest0-21swapper/207:32:182
2944999153,7cyclictest0-21swapper/207:47:382
2944999152,8cyclictest0-21swapper/208:47:382
2944999150,12cyclictest138550irq/27-eth008:36:582
2944999150,12cyclictest0-21swapper/207:54:102
2944899158,4cyclictest0-21swapper/112:14:301
29448991512,2cyclictest0-21swapper/111:30:071
29448991510,3cyclictest0-21swapper/109:35:381
2944899150,3cyclictest1687-21bash10:42:021
2944899150,14cyclictest0-21swapper/110:51:231
2944899150,13cyclictest0-21swapper/107:35:201
2944799159,5cyclictest0-21swapper/012:26:540
2944799154,2cyclictest0-21swapper/012:33:330
29447991510,1cyclictest0-21swapper/010:02:180
2944799150,9cyclictest13362-21bash11:40:440
2944799150,9cyclictest0-21swapper/009:41:010
2944799150,7cyclictest0-21swapper/010:17:560
2945099148,5cyclictest0-21swapper/307:30:443
2945099140,12cyclictest21451-21taskset08:01:153
2945099140,12cyclictest12826-21gltestperf08:55:203
2944999148,3cyclictest22200-21tr08:05:182
2944999143,10cyclictest0-21swapper/209:01:062
2944999141,8cyclictest0-21swapper/208:10:142
2944999140,5cyclictest0-21swapper/207:25:152
2944899147,6cyclictest0-21swapper/112:00:021
2944899143,7cyclictest25603-21sshd11:11:371
2944899142,11cyclictest0-21swapper/108:00:191
2944899141,10cyclictest0-21swapper/110:35:251
2944899140,5cyclictest29886-21munin-node08:20:271
2944899140,3cyclictest138550irq/27-eth009:03:441
2944899140,12cyclictest26896-21rm12:20:401
2944899140,12cyclictest12720-21bash12:15:191
2944799148,3cyclictest0-21swapper/008:50:340
2944799141,4cyclictest0-21swapper/007:11:090
2944799140,3cyclictest19220-21sshd12:12:080
2944799140,1cyclictest25102-21sshd10:10:340
2944799140,13cyclictest0-21swapper/008:39:290
2945099135,6cyclictest1877-21nfsd07:40:123
2945099133,8cyclictest1094-21df08:30:203
29450991311,1cyclictest0-21swapper/308:40:043
2945099130,4cyclictest3163-21munin-node08:35:233
2945099130,12cyclictest0-21swapper/307:22:113
2945099130,11cyclictest30685-21ntp_kernel_pll_07:10:263
2945099130,11cyclictest15147-21if_eth009:00:223
2945099130,11cyclictest10662-21iostat08:50:233
2944999130,12cyclictest0-21swapper/207:16:012
2944899131,11cyclictest138550irq/27-eth007:46:251
2944899130,12cyclictest0-21swapper/109:05:091
2944899130,12cyclictest0-21swapper/108:47:101
2944899130,12cyclictest0-21swapper/108:15:561
2944899130,12cyclictest0-21swapper/108:05:251
2944899130,12cyclictest0-21swapper/107:55:481
2944899130,12cyclictest0-21swapper/107:50:061
2944899130,12cyclictest0-21swapper/107:41:031
2944899130,12cyclictest0-21swapper/107:25:051
2944899130,12cyclictest0-21swapper/107:23:521
2944899130,12cyclictest0-21swapper/107:11:431
2944799130,1cyclictest0-21swapper/007:55:210
2944799130,12cyclictest0-21swapper/009:00:200
2944799130,12cyclictest0-21swapper/007:15:170
2944799130,11cyclictest0-21swapper/008:02:290
2944799130,10cyclictest0-21swapper/008:25:410
2944799130,10cyclictest0-21swapper/008:20:340
2944799130,10cyclictest0-21swapper/007:40:010
2944799130,10cyclictest0-21swapper/007:25:490
2945099120,11cyclictest28770-21munin-node08:20:173
2944999123,4cyclictest0-21swapper/209:06:182
2944999120,3cyclictest20605-21memory08:00:252
2944899127,3cyclictest6972-21iostat_ios07:30:211
2944899120,3cyclictest1586-21iostat_ios08:30:221
2944899120,11cyclictest138550irq/27-eth008:40:031
2944899120,10cyclictest3642-21munin-node08:35:191
2944899120,10cyclictest0-21swapper/108:55:211
2944899120,10cyclictest0-21swapper/108:50:181
2944899120,10cyclictest0-21swapper/108:10:221
2944799126,5cyclictest0-21swapper/007:36:010
2944799122,9cyclictest0-21swapper/008:41:050
2944799120,4cyclictest825-21munin-node08:30:190
2944799120,2cyclictest0-21swapper/007:31:010
2944799120,1cyclictest0-21swapper/007:45:140
2944799120,11cyclictest0-21swapper/007:50:550
2944799119,1cyclictest0-21swapper/009:05:070
2944799116,3cyclictest0-21swapper/008:06:040
2944799112,5cyclictest8413-21iostat08:45:260
2944799110,1cyclictest0-21swapper/007:20:420
121482110,10sleep00-21swapper/008:55:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional