You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-24 - 17:13

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack5slot0.osadl.org (updated Fri Mar 24, 2023 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
628721880,8sleep10-21swapper/107:08:431
3229221330,3sleep0656799cyclictest10:17:470
6231210732,46sleep30-21swapper/307:08:103
1092621020,2sleep30-21swapper/309:53:293
262592970,3sleep20-21swapper/212:09:482
36582960,1sleep03657-21seq10:04:040
135782960,2sleep213573-21bash11:51:442
135782960,2sleep213573-21bash11:51:432
105572960,2sleep30-21swapper/311:12:083
128192940,1sleep012807-21sshd11:03:560
40912910,2sleep20-21swapper/210:54:442
254752910,1sleep225465-21bash11:49:592
53182900,1sleep05295-21sshd09:10:460
75342890,2sleep37525-21sshd09:22:153
286472890,1sleep20-21swapper/209:52:122
267932880,0sleep00-21swapper/012:15:270
245042880,2sleep124498-21sshd12:35:041
255742870,2sleep225566-21sshd10:05:552
242262870,1sleep024222-21sshd11:16:060
151832870,1sleep015182-21bash11:37:470
55292860,2sleep25525-21cp10:04:152
256232860,2sleep10-21swapper/109:23:501
116722860,2sleep20-21swapper/211:17:532
87342850,2sleep21877-21nfsd10:12:562
231482850,2sleep10-21swapper/110:39:321
60662840,0sleep06065-21bash10:26:460
60662840,0sleep06065-21bash10:26:450
227442840,2sleep20-21swapper/209:40:222
308722830,3sleep3657799cyclictest10:59:543
147502830,3sleep014741-21sshd12:31:180
104262830,2sleep310425-21sshd09:36:303
620028232,46sleep20-21swapper/207:07:452
326812820,2sleep232651-21sshd10:37:332
139152820,3sleep013906-21sshd09:39:390
95582810,0sleep2321ktimersoftd/211:59:532
81712810,2sleep30-21swapper/311:28:493
25562810,0sleep00-21swapper/010:09:370
216892810,2sleep10-21swapper/111:13:071
121582810,2sleep112156-21sshd12:08:341
310642780,2sleep10-21swapper/110:54:151
240402780,2sleep124035-21sshd11:52:381
240402780,2sleep124035-21sshd11:52:381
64922760,1sleep16494-21sshd09:55:521
300702760,1sleep20-21swapper/212:10:072
147602760,2sleep114759-21sshd09:34:071
124512760,2sleep312443-21cp12:31:053
282912750,1sleep328290-21sshd12:18:223
67892740,2sleep36777-21sshd11:45:273
253162720,2sleep20-21swapper/209:32:132
127672710,2sleep30-21swapper/310:16:023
104912710,1sleep30-21swapper/311:59:583
105812700,2sleep30-21swapper/312:22:263
193572640,1sleep30-21swapper/309:59:513
152962610,2sleep0656799cyclictest09:34:100
608025732,13sleep00-21swapper/007:06:150
212162560,0sleep021217-21sshd10:50:300
308892520,0sleep030890-21bash11:30:410
170302450,0sleep00-21swapper/012:00:280
170302450,0sleep00-21swapper/012:00:270
258972410,39sleep1138550irq/27-eth009:15:251
109982400,0sleep11877-21nfsd10:30:031
26202350,0sleep12619-21sshd09:38:411
22892350,1sleep20-21swapper/211:42:182
257852332,29sleep128750irq/26-ahci[00009:12:361
60582310,4sleep1657099cyclictest10:21:061
166382310,29sleep1138550irq/27-eth010:41:461
236832300,27sleep3138550irq/27-eth007:45:283
270992290,2sleep127081-21sshd11:30:231
260792280,2sleep20-21swapper/211:33:092
41362270,1sleep30-21swapper/309:44:283
302842270,0sleep330286-21id10:45:393
235692270,25sleep1138550irq/27-eth010:19:531
57912260,3sleep30-21swapper/312:10:453
657599253,14cyclictest30574-21sshd11:27:572
26782250,3sleep20-21swapper/209:21:502
135872250,3sleep0656799cyclictest12:39:470
57582240,22sleep1138550irq/27-eth012:24:551
313242240,0sleep00-21swapper/012:10:140
26252240,2sleep3657799cyclictest09:16:103
261152240,2sleep2445-21jbd2/dm-0-812:23:502
143132240,3sleep30-21swapper/312:25:323
657799230,12cyclictest0-21swapper/311:35:503
282812230,2sleep20-21swapper/210:28:442
282812230,2sleep20-21swapper/210:28:432
104352230,2sleep10-21swapper/111:43:001
657599229,9cyclictest18727-21sshd10:30:382
6575992217,4cyclictest8406-21sshd12:39:202
6575992215,5cyclictest7573-21sadc11:39:592
327232220,2sleep132722-21bash10:06:351
161032220,2sleep30-21swapper/309:45:283
87832210,19sleep128750irq/26-ahci[00012:16:401
657599211,9cyclictest32173-21cp10:20:312
6575992117,2cyclictest17919-21sshd09:11:542
105422210,1sleep00-21swapper/011:23:250
6575992014,3cyclictest14994-21sshd11:01:172
6575992012,6cyclictest1943-21snmpd12:17:122
296332200,3sleep1657099cyclictest11:36:121
657599192,5cyclictest30115-21sshd09:29:512
657599191,4cyclictest29545-21sshd09:57:542
6575991912,5cyclictest1943-21snmpd10:18:412
6575991912,5cyclictest1943-21snmpd09:49:262
6575991912,3cyclictest24156-21sshd12:26:262
6567991915,3cyclictest3-21ksoftirqd/009:56:050
245472190,0sleep00-21swapper/011:13:220
162542192,15sleep3138550irq/27-eth011:21:043
153812190,1sleep32121-21runrttasks09:25:413
657799187,6cyclictest842-21sshd10:20:373
657799183,5cyclictest0-21swapper/310:35:483
657799180,16cyclictest26542-21sshd11:16:193
657799180,12cyclictest0-21swapper/307:56:533
657599186,10cyclictest27057-21sshd10:48:172
6575991811,5cyclictest1943-21snmpd09:38:252
657099184,12cyclictest138550irq/27-eth012:00:121
657099184,12cyclictest138550irq/27-eth012:00:121
657099180,13cyclictest5637-21sshd12:33:221
656799180,9cyclictest9916-21bash10:24:150
657799171,14cyclictest29927-21rm10:54:103
657799170,9cyclictest6741-21perf10:09:593
657799170,8cyclictest0-21swapper/311:03:033
657799170,15cyclictest0-21swapper/309:06:093
657799170,14cyclictest0-21swapper/308:38:173
657599179,6cyclictest1943-21snmpd11:10:572
657599179,6cyclictest19168-21sshd10:41:592
657599178,6cyclictest31939-21sshd09:15:532
657599172,6cyclictest21566-21sshd10:56:142
657599171,5cyclictest23261-21sshd11:07:382
657599171,14cyclictest20623-21sshd11:21:262
6575991710,2cyclictest0-21swapper/208:01:032
657599170,15cyclictest12620-21sshd12:31:062
657099178,4cyclictest16748-21sshd10:27:421
657099178,4cyclictest16748-21sshd10:27:421
657099173,2cyclictest32327-21sshd09:46:511
657099173,1cyclictest31826-21bash11:25:161
657099172,9cyclictest4202-21sshd11:05:561
6570991713,3cyclictest0-21swapper/107:43:161
6570991712,3cyclictest0-21swapper/112:10:481
657099170,12cyclictest11269-21sshd09:50:381
657099170,11cyclictest22335-21bash09:29:091
6567991710,7cyclictest0-21swapper/007:55:170
223452170,1sleep00-21swapper/012:29:090
657799163,4cyclictest31343-21sshd11:53:173
657799163,4cyclictest31343-21sshd11:53:163
657799162,4cyclictest23518-21sshd10:11:243
657799162,2cyclictest21752-21sshd09:31:533
6577991610,4cyclictest24615-21sshd10:25:303
6577991610,4cyclictest24615-21sshd10:25:293
657799160,4cyclictest2064-21id10:32:033
657799160,4cyclictest0-21swapper/308:53:173
657799160,15cyclictest0-21swapper/308:59:173
657799160,13cyclictest0-21swapper/308:42:483
657599163,8cyclictest3173-21taskset08:13:322
657599162,11cyclictest19069-21perf08:50:012
6575991612,2cyclictest19554-21sshd12:00:422
6575991612,2cyclictest19554-21sshd12:00:412
657599161,13cyclictest2121-21runrttasks08:34:332
657099168,6cyclictest5822-21sshd11:00:271
657099168,3cyclictest0-21swapper/111:20:441
657099161,13cyclictest14047-21sshd12:25:311
657099161,13cyclictest0-21swapper/108:40:131
657099160,4cyclictest25577-21sshd10:45:191
657099160,4cyclictest12047-21sshd11:17:561
657099160,3cyclictest29202-21cp10:14:431
657099160,12cyclictest9762-21taskset08:26:351
657099160,11cyclictest2786-21bash10:01:091
656799166,5cyclictest5537-21sshd12:05:110
656799166,1cyclictest3-21ksoftirqd/010:38:020
656799163,9cyclictest0-21swapper/009:44:440
6567991613,2cyclictest16488-21sshd11:07:030
6567991611,4cyclictest3-21ksoftirqd/011:53:300
6567991611,4cyclictest3-21ksoftirqd/011:53:290
656799160,14cyclictest4867-21bash10:46:170
656799160,14cyclictest3939-21bash10:32:130
657799159,4cyclictest2337-21sshd12:38:473
657799156,4cyclictest6545-21sshd12:05:143
657799156,4cyclictest11939-21id11:06:383
657799156,0cyclictest0-21swapper/312:03:573
657799156,0cyclictest0-21swapper/312:03:573
657799154,6cyclictest0-21swapper/311:41:383
657799150,6cyclictest23247-21sshd11:32:533
657799150,3cyclictest10922-21sshd09:11:163
657799150,3cyclictest0-21swapper/310:43:253
657799150,10cyclictest0-21swapper/310:01:123
657599158,5cyclictest5557-21sadc08:20:012
657599157,5cyclictest138550irq/27-eth007:39:232
657599155,5cyclictest0-21swapper/207:58:532
657599152,8cyclictest0-21swapper/207:21:312
657599150,13cyclictest1877-21nfsd09:09:422
657599150,13cyclictest15929-21munin-node07:30:132
657599150,13cyclictest14683-21munin-node07:25:232
657599150,13cyclictest0-21swapper/208:55:192
657099155,4cyclictest2942-21bash11:56:261
6570991510,3cyclictest14263-21sshd10:55:321
657099150,14cyclictest1874-21nfsd11:48:201
657099150,13cyclictest0-21swapper/108:50:131
657099150,13cyclictest0-21swapper/108:30:101
657099150,13cyclictest0-21swapper/107:26:141
657099150,11cyclictest24655-21munin-node07:50:251
657099150,10cyclictest0-21swapper/109:41:381
656799159,1cyclictest11231-21rm10:58:100
656799156,1cyclictest15449-21basename07:30:000
656799150,14cyclictest0-21swapper/010:43:010
656799150,14cyclictest0-21swapper/009:26:230
656799150,13cyclictest6485-21sshd09:50:180
231952150,13sleep1138550irq/27-eth008:55:531
657799149,4cyclictest138550irq/27-eth007:35:143
657799143,3cyclictest138550irq/27-eth008:02:233
657799142,11cyclictest0-21swapper/308:45:413
657799140,12cyclictest9213-21pmu-power08:25:233
657799140,12cyclictest0-21swapper/309:04:173
657599148,5cyclictest1877-21nfsd08:41:392
657599140,6cyclictest1876-21nfsd08:25:042
657599140,13cyclictest8479-21taskset07:12:412
657599140,13cyclictest0-21swapper/208:05:012
657099148,5cyclictest965-21turbostat.cron08:14:551
657099141,12cyclictest0-21swapper/108:17:061
657099140,13cyclictest0-21swapper/107:45:281
656799142,4cyclictest21663-21sshd10:11:140
6567991411,2cyclictest3-21ksoftirqd/011:42:270
6567991411,2cyclictest3-21ksoftirqd/009:21:220
6567991411,2cyclictest1987-21sshd12:21:420
656799140,13cyclictest0-21swapper/007:45:410
657799139,2cyclictest0-21swapper/307:40:013
657799130,7cyclictest0-21swapper/307:23:233
657799130,12cyclictest0-21swapper/308:30:323
657799130,12cyclictest0-21swapper/308:06:123
657799130,12cyclictest0-21swapper/307:50:373
657799130,12cyclictest0-21swapper/307:17:013
657799130,11cyclictest3536-21munin-node08:15:213
657799130,11cyclictest15926-21munin-node07:30:133
657799130,11cyclictest1211-21munin-node08:10:193
657599139,3cyclictest13271-21iostat08:35:162
657599138,4cyclictest0-21swapper/209:01:352
657599137,5cyclictest1877-21nfsd07:15:412
657599132,10cyclictest0-21swapper/208:50:532
657599130,5cyclictest0-21swapper/207:40:012
657599130,12cyclictest0-21swapper/208:23:372
657599130,12cyclictest0-21swapper/207:48:332
657599130,11cyclictest24655-21munin-node07:50:162
657099139,3cyclictest138550irq/27-eth008:20:131
657099138,3cyclictest16317-21hddtemp_smartct07:30:151
657099132,10cyclictest138550irq/27-eth009:00:421
6570991311,1cyclictest0-21swapper/108:46:521
6570991311,1cyclictest0-21swapper/108:37:141
657099130,4cyclictest27893-21taskset09:08:171
657099130,4cyclictest18247-21forks07:35:131
657099130,12cyclictest0-21swapper/108:07:561
657099130,12cyclictest0-21swapper/108:03:161
657099130,12cyclictest0-21swapper/107:55:361
657099130,11cyclictest11308-21wc07:20:151
656799133,9cyclictest0-21swapper/011:28:250
656799132,7cyclictest0-21swapper/008:05:160
6567991310,2cyclictest3-21ksoftirqd/009:47:530
656799130,12cyclictest9278-21sshd11:45:410
656799130,12cyclictest4243-21munin-node08:15:150
656799130,12cyclictest1987-21sshd09:16:030
656799130,12cyclictest0-21swapper/011:55:190
656799130,12cyclictest0-21swapper/008:42:390
657799120,3cyclictest6991-21df_inode07:10:143
657799120,11cyclictest138550irq/27-eth007:25:013
657799120,11cyclictest0-21swapper/308:20:273
657099120,11cyclictest0-21swapper/107:15:341
657099120,11cyclictest0-21swapper/107:13:291
656799128,3cyclictest0-21swapper/008:25:300
656799127,4cyclictest0-21swapper/008:10:180
656799127,4cyclictest0-21swapper/007:15:180
656799127,2cyclictest0-21swapper/008:36:400
656799126,5cyclictest0-21swapper/008:00:130
656799120,11cyclictest22654-21ntp_states08:55:210
656799120,11cyclictest0-21swapper/008:30:170
656799120,11cyclictest0-21swapper/007:35:020
656799120,11cyclictest0-21swapper/007:30:160
172172120,1sleep041ktimersoftd/008:45:180
6567991110,1cyclictest0-21swapper/009:01:330
656799110,3cyclictest0-21swapper/008:21:260
656799110,3cyclictest0-21swapper/007:40:230
656799110,2cyclictest0-21swapper/007:20:240
656799110,1cyclictest0-21swapper/009:05:420
656799110,10cyclictest25579-21ls07:50:200
656799110,10cyclictest0-21swapper/008:50:200
656799100,9cyclictest0-21swapper/007:10:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional