You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-09 - 17:20

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack5slot0.osadl.org (updated Fri Dec 09, 2022 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1118821260,2sleep20-21swapper/210:46:122
2068721170,3sleep31844799cyclictest11:01:363
18171210733,44sleep10-21swapper/107:08:391
17964210633,44sleep30-21swapper/307:06:043
912521030,1sleep29122-21bash11:58:432
72522940,1sleep07253-21sshd10:37:090
154782940,2sleep115477-21bash11:07:001
148732930,2sleep10-21swapper/110:43:401
249512920,4sleep21844399cyclictest09:26:012
54532900,3sleep20-21swapper/209:38:522
37832880,1sleep03784-21sshd12:30:190
271692880,2sleep327170-21id09:58:173
98672860,3sleep10-21swapper/112:16:181
43332860,1sleep02121-21runrttasks12:12:550
226652860,2sleep222667-21sshd10:09:292
190702860,1sleep00-21swapper/011:10:160
92412840,2sleep19235-21sshd12:07:281
316052840,1sleep10-21swapper/110:45:121
152282840,2sleep20-21swapper/209:57:122
112162840,3sleep111208-21sshd09:10:171
189132830,1sleep20-21swapper/209:10:592
232412820,2sleep10-21swapper/110:03:441
34632800,3sleep12333-21munin-node11:55:181
317302800,3sleep131707-21sshd11:31:371
300552800,3sleep130043-21sshd12:32:411
266632790,2sleep20-21swapper/211:36:592
173362780,2sleep10-21swapper/110:09:011
280152760,4sleep31844799cyclictest09:49:393
321022750,2sleep10-21swapper/109:52:531
261712750,2sleep10-21swapper/110:56:181
72172740,2sleep37214-21sshd11:58:323
31752740,1sleep23173-21sshd11:31:572
297162740,1sleep30-21swapper/311:40:133
186652740,2sleep218659-21sshd11:42:042
33662730,1sleep23354-21sshd09:50:192
18062730,4sleep11843599cyclictest12:03:541
242132700,2sleep124206-21bash11:42:351
197642690,1sleep10-21swapper/111:24:521
165512690,2sleep316537-21sshd12:02:173
59872680,2sleep01843199cyclictest09:30:130
155692680,3sleep31844799cyclictest11:50:273
261942660,1sleep30-21swapper/310:38:523
153942610,1sleep30-21swapper/309:16:283
1826825732,22sleep00-21swapper/007:09:420
1802225632,12sleep20-21swapper/207:06:482
270512340,2sleep31877-21nfsd09:37:523
55342330,1sleep20-21swapper/212:04:162
230232320,0sleep323022-21bash11:39:373
143392310,29sleep328750irq/26-ahci[00011:06:553
321692300,28sleep2138550irq/27-eth012:06:382
210412300,3sleep221040-21bash10:18:042
167562300,28sleep2138550irq/27-eth011:47:462
156652300,0sleep015663-21sshd09:48:320
310992290,1sleep10-21swapper/111:14:141
231942280,4sleep03-21ksoftirqd/012:37:520
1844399277,9cyclictest1877-21nfsd11:17:292
57192260,2sleep30-21swapper/311:11:553
47772260,2sleep20-21swapper/209:32:592
266452260,5sleep20-21swapper/212:17:512
25662260,1sleep00-21swapper/011:29:040
101092260,3sleep225-21rcuop/110:51:582
229822250,3sleep222971-21sshd12:14:382
1844399254,0cyclictest0-21swapper/211:23:252
284952240,1sleep20-21swapper/210:41:562
1844399230,3cyclictest0-21swapper/210:57:012
121652230,21sleep1138550irq/27-eth011:18:191
316652220,2sleep131661-21sshd12:21:111
196322220,3sleep30-21swapper/309:14:013
19382220,1sleep01843199cyclictest10:45:230
1844399228,12cyclictest21852-21bash09:46:072
1844399226,8cyclictest138550irq/27-eth010:20:372
1844399226,8cyclictest138550irq/27-eth010:20:372
1844399224,9cyclictest138550irq/27-eth011:25:082
1844399223,0cyclictest0-21swapper/210:30:432
99342210,2sleep31876-21nfsd09:50:523
1844399218,11cyclictest1921-21nscd11:00:322
1844399217,8cyclictest138550irq/27-eth009:20:122
1844399217,7cyclictest138550irq/27-eth008:06:002
1844399217,11cyclictest10707-21sshd11:06:342
1844399216,13cyclictest19856-21bash10:03:252
1844399210,3cyclictest0-21swapper/209:16:222
1844399210,2cyclictest0-21swapper/210:25:432
1844399210,2cyclictest0-21swapper/210:25:422
1844399210,2cyclictest0-21swapper/210:13:412
1844399210,10cyclictest0-21swapper/211:14:102
18435992113,2cyclictest29200-21sshd09:49:451
1843199211,19cyclictest0-21swapper/007:24:330
12242210,1sleep00-21swapper/011:58:010
47092200,18sleep1138550irq/27-eth010:16:351
275412200,2sleep127540-21bash10:53:331
1844399207,11cyclictest21022-21sshd12:23:102
1844399200,9cyclictest0-21swapper/209:40:112
1844399200,3cyclictest0-21swapper/212:33:452
1844399200,2cyclictest0-21swapper/212:25:562
1844399200,18cyclictest0-21swapper/211:50:252
1844399200,18cyclictest0-21swapper/210:37:312
218802190,17sleep1138550irq/27-eth010:12:181
1844799195,12cyclictest31795-21sshd10:45:133
1844799192,8cyclictest0-21swapper/309:43:383
1844799190,16cyclictest0-21swapper/310:32:323
1844399190,3cyclictest0-21swapper/212:35:512
1844399190,3cyclictest0-21swapper/207:21:432
1844399190,2cyclictest0-21swapper/208:20:222
1844399190,2cyclictest0-21swapper/207:15:232
1843599195,12cyclictest22713-21sshd11:04:461
18435991914,4cyclictest10979-21sshd11:29:501
1843199193,10cyclictest28583-21sshd10:15:470
1843199190,17cyclictest0-21swapper/012:05:140
18443991816,1cyclictest0-21swapper/208:56:432
18443991816,1cyclictest0-21swapper/208:56:432
1844399180,9cyclictest0-21swapper/207:50:082
1844399180,2cyclictest0-21swapper/209:05:192
1844399180,2cyclictest0-21swapper/208:25:142
1844399180,2cyclictest0-21swapper/207:40:182
1844399180,1cyclictest0-21swapper/208:40:142
1843599180,13cyclictest22877-21sshd09:31:391
1844799177,3cyclictest30276-21bash09:32:213
1844799176,4cyclictest15092-21sshd12:25:283
1844799170,11cyclictest0-21swapper/311:46:083
1844799170,10cyclictest12540-21sshd09:24:593
1844399170,9cyclictest0-21swapper/207:26:152
1844399170,2cyclictest0-21swapper/208:30:142
1844399170,16cyclictest0-21swapper/207:14:462
1844399170,15cyclictest0-21swapper/207:39:192
1844399170,10cyclictest0-21swapper/207:45:222
1843599171,7cyclictest0-21swapper/110:20:591
1843599171,7cyclictest0-21swapper/110:20:591
1843599171,3cyclictest0-21swapper/109:59:501
1843599171,14cyclictest0-21swapper/112:36:191
1843599170,4cyclictest12636-21sshd09:39:321
1843599170,3cyclictest23098-21sshd09:40:251
1843599170,2cyclictest0-21swapper/109:16:101
1843599170,12cyclictest29173-21users07:30:251
1843599170,12cyclictest10749-21sshd12:25:111
1843199171,5cyclictest0-21swapper/011:51:310
18431991713,3cyclictest21419-21kworker/0:008:53:010
18431991710,3cyclictest12814-21bash09:56:570
1843199170,14cyclictest1625-21kworker/0:212:23:320
1843199170,12cyclictest17340-21sshd12:19:570
168782170,0sleep00-21swapper/009:42:470
1844799168,6cyclictest30991-21tr12:15:203
1844799161,6cyclictest11403-21sshd12:39:473
1844799160,9cyclictest15486-21sshd12:05:093
1844799160,7cyclictest0-21swapper/310:00:133
1844799160,2cyclictest8700-21bash10:51:513
1844799160,2cyclictest26893-21sshd12:12:013
1844799160,2cyclictest16895-21sshd11:18:453
1844799160,13cyclictest644-21bash10:22:033
1844799160,13cyclictest644-21bash10:22:033
1844399160,8cyclictest0-21swapper/208:46:182
1844399160,8cyclictest0-21swapper/207:55:042
1844399160,1cyclictest0-21swapper/209:00:182
1844399160,1cyclictest0-21swapper/208:16:062
1844399160,1cyclictest0-21swapper/208:00:102
1844399160,1cyclictest0-21swapper/207:30:152
1844399160,15cyclictest0-21swapper/208:11:342
1844399160,0cyclictest0-21swapper/208:50:382
1843599167,5cyclictest21381-21sshd10:38:261
1843599163,11cyclictest25583-21sshd12:14:521
1843599162,9cyclictest900-21sshd11:46:181
18435991611,4cyclictest3506-21sshd10:28:081
18435991611,4cyclictest3506-21sshd10:28:071
1843599161,13cyclictest0-21swapper/108:36:191
1843599160,7cyclictest3613-21bash09:21:131
1843599160,14cyclictest0-21swapper/111:52:381
18431991613,2cyclictest3-21ksoftirqd/011:22:580
18431991613,2cyclictest0-21swapper/012:04:420
1843199161,11cyclictest0-21swapper/010:20:520
1843199161,11cyclictest0-21swapper/010:20:510
18431991610,2cyclictest0-21swapper/010:52:580
1843199160,4cyclictest7179-21sshd09:15:410
1843199160,12cyclictest0-21swapper/012:27:100
1844799159,5cyclictest22212-21sshd10:15:173
1844799159,5cyclictest138550irq/27-eth007:50:413
1844799158,4cyclictest0-21swapper/307:35:553
1844799156,2cyclictest20276-21sshd12:34:443
1844799153,1cyclictest24053-21cpu11:25:133
1844799152,12cyclictest16342-21id10:05:573
18447991511,3cyclictest1805-40kipmi008:35:183
1844799150,8cyclictest147450irq/28-eth1-rx-10:10:193
1844799150,3cyclictest138550irq/27-eth009:28:453
1844799150,14cyclictest3013-21bash10:57:093
1844799150,14cyclictest11721-21sshd11:24:083
1844799150,13cyclictest0-21swapper/310:25:123
1844799150,13cyclictest0-21swapper/310:25:123
1844799150,10cyclictest0-21swapper/311:31:063
1844399150,1cyclictest0-21swapper/208:35:462
1843599150,3cyclictest0-21swapper/111:37:421
1843599150,13cyclictest31104-21sshd10:30:311
1843599150,13cyclictest1877-21nfsd09:25:441
1843599150,13cyclictest12789-21chrt08:08:191
1843199158,4cyclictest445-21jbd2/dm-0-811:30:320
1843199158,3cyclictest28937-21bash09:52:370
1843199158,3cyclictest0-21swapper/011:17:100
1843199157,5cyclictest0-21swapper/007:38:120
1843199157,4cyclictest3170-21cp09:12:300
1843199153,1cyclictest6350-21bash09:35:580
1843199153,11cyclictest3-21ksoftirqd/010:55:200
1843199152,1cyclictest0-21swapper/011:09:060
18431991513,1cyclictest3-21ksoftirqd/011:48:330
18431991512,2cyclictest0-21swapper/011:02:510
1843199150,4cyclictest0-21swapper/011:36:140
1843199150,4cyclictest0-21swapper/010:42:090
1843199150,14cyclictest623-21bash10:01:410
1843199150,14cyclictest0-21swapper/010:07:170
1843199150,12cyclictest0-21swapper/009:26:310
1844799140,9cyclictest0-21swapper/308:53:303
1844799140,5cyclictest1877-21nfsd12:20:083
1844799140,11cyclictest0-21swapper/308:25:283
1844799140,10cyclictest28301-21bash10:41:563
1843599147,6cyclictest17686-21aten_r5power_fr08:20:141
1843599144,9cyclictest21966-21idleruntime-cro08:30:001
1843199147,3cyclictest3-21ksoftirqd/010:10:270
18431991410,3cyclictest21419-21kworker/0:008:06:010
1843199140,4cyclictest30311-21sshd09:20:400
1843199140,1cyclictest0-21swapper/010:25:290
1843199140,1cyclictest0-21swapper/010:25:290
1843199140,13cyclictest0-21swapper/007:40:350
1843199140,10cyclictest29398-21bash11:43:030
1844799137,5cyclictest20374-21runrttasks07:12:393
1844799137,3cyclictest0-21swapper/308:56:143
1844799137,3cyclictest0-21swapper/308:56:133
1844799132,3cyclictest138550irq/27-eth008:41:363
1844799132,2cyclictest0-21swapper/308:08:463
1844799132,10cyclictest138550irq/27-eth008:15:033
1844799132,10cyclictest138550irq/27-eth008:10:583
18447991311,1cyclictest0-21swapper/307:25:283
1844799130,12cyclictest0-21swapper/309:05:383
1844799130,12cyclictest0-21swapper/309:00:363
1844799130,12cyclictest0-21swapper/308:46:163
1844799130,12cyclictest0-21swapper/307:46:483
1844799130,12cyclictest0-21swapper/307:40:183
1844799130,12cyclictest0-21swapper/307:34:033
1844799130,12cyclictest0-21swapper/307:15:393
1843599139,3cyclictest138550irq/27-eth008:53:151
1843599139,3cyclictest138550irq/27-eth007:12:371
1843599138,4cyclictest0-21swapper/107:20:001
1843599131,3cyclictest30538-21iostat_ios07:35:171
1843599130,12cyclictest15428-21munin-node08:15:231
1843599130,12cyclictest138550irq/27-eth007:45:281
1843599130,12cyclictest0-21swapper/109:00:351
1843599130,12cyclictest0-21swapper/107:55:031
1843599130,12cyclictest0-21swapper/107:52:071
1843599130,12cyclictest0-21swapper/107:25:361
1843599130,11cyclictest9407-21iostat08:00:171
1843599130,11cyclictest5893-21munin-node09:05:151
1843599130,11cyclictest327-21iostat_ios07:40:181
1843599130,11cyclictest22457-21grep08:30:121
18431991311,1cyclictest0-21swapper/008:35:030
18431991311,1cyclictest0-21swapper/007:17:490
1843199130,5cyclictest27631-21bash10:30:170
1843199130,5cyclictest0-21swapper/008:30:210
1843199130,5cyclictest0-21swapper/008:25:100
1843199130,12cyclictest26929-21users07:25:300
1843199130,10cyclictest0-21swapper/009:00:110
1844799126,5cyclictest0-21swapper/307:55:233
1844799120,11cyclictest8946-21munin-node08:00:123
1844799120,11cyclictest19435-21chrt08:21:593
1844799120,10cyclictest23329-21df07:20:153
1843599120,3cyclictest13501-21munin-node08:10:131
1843599120,3cyclictest1230-21df08:55:131
1843599120,3cyclictest1230-21df08:55:121
1843599120,11cyclictest29665-21iostat_ios08:45:191
1843599120,11cyclictest138550irq/27-eth008:40:141
1843599120,10cyclictest0-21swapper/107:15:161
1843199128,3cyclictest0-21swapper/009:08:370
1843199128,3cyclictest0-21swapper/008:15:550
1843199128,3cyclictest0-21swapper/008:00:330
1843199126,5cyclictest0-21swapper/008:55:300
1843199126,5cyclictest0-21swapper/008:55:290
1843199120,5cyclictest0-21swapper/007:45:210
1843199120,1cyclictest0-21swapper/007:56:290
1843199120,1cyclictest0-21swapper/007:30:200
1843199120,11cyclictest0-21swapper/008:20:160
1843199120,11cyclictest0-21swapper/008:12:590
1843199120,11cyclictest0-21swapper/007:51:120
1844799117,3cyclictest22505-21diskstats08:30:133
1843199118,2cyclictest0-21swapper/007:10:270
1843199117,3cyclictest0-21swapper/008:45:010
1843199110,10cyclictest0-21swapper/008:40:550
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional