You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-03 - 07:58

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rack5slot0.osadl.org (updated Sun Jul 03, 2022 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2323121680,4sleep21404699cyclictest21:57:512
91021600,1sleep0891-21sshd21:22:020
2305121250,3sleep21404699cyclictest21:33:182
483021240,1sleep20-21swapper/223:25:252
567121160,2sleep15672-21id21:34:451
3179621120,2sleep30-21swapper/321:58:403
585821110,0sleep25853-21rm00:02:482
1777021110,1sleep017769-21seq21:48:070
1519521040,1sleep215187-21bash23:35:392
437521030,1sleep041ktimersoftd/023:59:260
194421030,2sleep11945-21sshd00:21:311
1568021030,2sleep115676-21sshd23:04:521
2563621010,5sleep025623-21sshd21:27:270
1617921000,4sleep31405399cyclictest21:29:383
1363429731,38sleep30-21swapper/319:06:473
201132960,2sleep10-21swapper/123:20:431
166812940,1sleep10-21swapper/100:35:431
113512930,4sleep20-21swapper/221:50:292
98692920,0sleep00-21swapper/022:48:550
259982920,1sleep325997-21id21:15:123
28162900,2sleep12815-21bash22:23:351
91252890,2sleep30-21swapper/323:04:133
324002860,3sleep01404199cyclictest21:40:180
235632860,3sleep11404299cyclictest00:14:051
238032850,4sleep31405399cyclictest23:33:273
130482850,1sleep113047-21bash21:29:201
224062830,2sleep222405-21sshd21:45:252
206002830,1sleep020591-21sshd00:29:480
1362228332,47sleep10-21swapper/119:06:361
12852830,2sleep31284-21bash23:28:143
104912830,2sleep31876-21nfsd00:06:263
62192820,2sleep26212-21bash22:26:572
62192820,2sleep26212-21bash22:26:572
312322820,1sleep20-21swapper/222:17:032
104372820,3sleep10-21swapper/123:07:281
210752800,1sleep00-21swapper/023:54:490
206292790,2sleep320628-21sshd22:00:373
314312780,1sleep20-21swapper/222:32:262
178262770,4sleep31405399cyclictest21:45:063
230322760,2sleep10-21swapper/120:40:161
306932710,1sleep030692-21sshd22:26:110
306932710,1sleep030692-21sshd22:26:110
136172700,1sleep213614-21sshd00:16:152
235102690,4sleep31405399cyclictest23:42:413
111112690,4sleep21404699cyclictest23:56:552
310202650,1sleep031017-21sshd22:32:230
319692640,3sleep31405399cyclictest21:52:313
30982600,1sleep31405399cyclictest21:31:233
1359925731,14sleep00-21swapper/019:06:200
205502550,3sleep30-21swapper/322:59:083
1357325028,10sleep20-21swapper/219:06:062
138222480,0sleep013820-21bash23:17:050
178782370,0sleep00-21swapper/021:38:570
145162350,32sleep3138550irq/27-eth022:18:333
10712350,33sleep1138550irq/27-eth021:22:041
60162320,0sleep00-21swapper/021:31:400
276922320,1sleep30-21swapper/323:58:323
14041993228,3cyclictest31362-21bash23:00:110
23552310,2sleep22352-21sshd23:31:222
96912291,26sleep1138550irq/27-eth000:15:511
63222280,3sleep30-21swapper/321:22:333
14041992814,3cyclictest3-21ksoftirqd/022:21:430
64982270,3sleep20-21swapper/223:50:182
257912270,3sleep21987-21sshd00:33:262
167082270,1sleep20-21swapper/221:14:182
1404699278,6cyclictest0-21swapper/220:20:502
80572260,1sleep041ktimersoftd/023:31:560
301922260,3sleep00-21swapper/023:37:090
14046992621,4cyclictest30840-21sshd21:24:552
14041992615,4cyclictest3-21ksoftirqd/000:38:480
14041992613,12cyclictest3-21ksoftirqd/000:19:040
96372250,2sleep29640-21sshd23:44:292
1404199250,23cyclictest0-21swapper/023:29:370
263322240,1sleep00-21swapper/022:53:320
232102240,2sleep323201-21sshd22:53:153
175812240,2sleep20-21swapper/221:35:472
157982240,22sleep3138550irq/27-eth000:38:533
1404699241,8cyclictest28129-21sshd00:14:332
1404699240,18cyclictest0-21swapper/220:18:472
1404199249,2cyclictest18923-21sshd23:08:170
1404199245,5cyclictest3002-21sshd22:39:030
14041992413,2cyclictest3-21ksoftirqd/022:14:430
14041992412,3cyclictest3-21ksoftirqd/000:24:340
1404199240,2cyclictest10-21rcuc/021:54:170
114872240,3sleep30-21swapper/323:23:013
108792240,3sleep10-21swapper/123:44:361
86712230,1sleep08653-21sshd22:08:470
169462230,2sleep216943-21bash00:38:592
153402230,21sleep1138550irq/27-eth022:58:391
14053992317,4cyclictest11757-21id21:35:173
1405399230,7cyclictest0-21swapper/321:12:203
14041992315,2cyclictest3-21ksoftirqd/023:12:470
14041992314,3cyclictest3-21ksoftirqd/023:20:460
14041992313,2cyclictest3-21ksoftirqd/021:57:430
1404199231,2cyclictest14001-21sshd00:00:200
1404199231,15cyclictest0-21swapper/023:48:440
14041992311,3cyclictest3-21ksoftirqd/000:07:130
1404199230,4cyclictest29537-21sshd21:18:360
134352230,2sleep10-21swapper/123:29:251
12642230,2sleep11242-21sshd21:40:231
153012220,20sleep3138550irq/27-eth022:27:513
153012220,20sleep3138550irq/27-eth022:27:513
1404699222,7cyclictest29501-21sshd21:18:352
1404699220,10cyclictest0-21swapper/222:10:392
1404199229,3cyclictest3-21ksoftirqd/022:56:170
14041992211,4cyclictest3-21ksoftirqd/022:16:310
164932211,17sleep1138550irq/27-eth023:45:091
14053992115,4cyclictest9845-21bash22:05:453
14053992112,6cyclictest31980-21sshd00:18:093
14053992111,7cyclictest5599-21bash22:42:193
1404699211,2cyclictest0-21swapper/222:22:402
1404699211,14cyclictest0-21swapper/221:03:512
1404699211,13cyclictest11428-21sshd23:13:482
1404199211,6cyclictest348-21sshd23:40:260
14041992114,2cyclictest3-21ksoftirqd/021:13:400
14041992112,3cyclictest3-21ksoftirqd/000:32:150
237802201,17sleep1138550irq/27-eth022:13:181
191942200,18sleep1138550irq/27-eth023:51:301
147522200,18sleep2138550irq/27-eth022:03:132
1405399208,4cyclictest17469-21sshd23:35:533
1405399207,4cyclictest7014-21bash00:12:273
1405399207,1cyclictest0-21swapper/323:08:173
14053992010,6cyclictest27371-21sshd22:38:163
1405399200,6cyclictest26911-21id22:10:253
1404699207,7cyclictest138550irq/27-eth021:27:052
1404699201,7cyclictest0-21swapper/200:27:422
14046992015,3cyclictest1480-21bash23:00:212
1404699201,4cyclictest4390-21sshd21:40:392
1404699201,4cyclictest29213-21bash22:44:372
1404699201,2cyclictest0-21swapper/222:09:552
1404699201,13cyclictest26471-21sshd23:21:212
1404299203,9cyclictest21895-21sshd22:00:441
14041992013,2cyclictest3-21ksoftirqd/022:02:140
14041992012,2cyclictest3-21ksoftirqd/000:14:380
1404199201,13cyclictest28652-21sshd22:41:250
292502190,17sleep3138550irq/27-eth023:52:283
1405399197,3cyclictest7468-21bash23:13:253
1405399194,10cyclictest0-21swapper/320:20:163
14053991910,7cyclictest4310-21sshd22:20:353
1405399190,8cyclictest0-21swapper/323:18:373
1405399190,7cyclictest11618-21bash00:03:243
1404699195,3cyclictest8934-21bash23:07:192
1404699191,2cyclictest0-21swapper/219:31:302
1404699191,2cyclictest0-21swapper/219:20:552
1404699191,14cyclictest0-21swapper/222:51:412
1404699191,12cyclictest0-21swapper/220:06:292
1404699190,6cyclictest0-21swapper/220:42:592
1404299197,5cyclictest138550irq/27-eth000:33:061
1404299196,3cyclictest138550irq/27-eth022:06:541
14042991912,4cyclictest21514-21sshd00:04:241
1404299190,13cyclictest0-21swapper/122:46:421
1405399181,5cyclictest0-21swapper/322:32:463
14053991812,4cyclictest21798-21memory00:20:183
1405399181,1cyclictest0-21swapper/323:48:203
1405399180,7cyclictest29238-21sshd21:43:053
1405399180,4cyclictest0-21swapper/320:02:043
1404699189,4cyclictest27374-21sshd22:59:482
1404699188,2cyclictest0-21swapper/223:18:512
1404699182,2cyclictest0-21swapper/200:06:512
1404699182,13cyclictest22365-1kworker/2:2H23:46:052
1404699180,2cyclictest0-21swapper/220:37:112
1404299186,3cyclictest8485-21cp00:25:201
1404299185,6cyclictest138550irq/27-eth023:35:211
14042991813,4cyclictest7786-21rm21:37:581
1404299180,17cyclictest22-21ksoftirqd/121:48:451
1404299180,13cyclictest22080-21sshd21:51:341
228542170,3sleep10-21swapper/122:37:501
1405399178,6cyclictest0-21swapper/320:55:503
1405399173,5cyclictest26036-21sshd22:47:213
1405399171,7cyclictest7352-21bash00:31:353
1405399171,14cyclictest4934-21sshd00:25:033
1404699171,5cyclictest30507-21sed22:35:242
1404699171,4cyclictest14694-21sshd22:46:152
1404699171,13cyclictest1877-21nfsd19:36:552
1404699170,2cyclictest0-21swapper/220:45:152
1404299178,1cyclictest0-21swapper/122:43:531
1404299174,2cyclictest19631-21bash23:17:391
1404299171,13cyclictest11085-21sshd22:27:261
1404299171,13cyclictest11085-21sshd22:27:261
1404299170,9cyclictest5980-21sshd22:51:351
1404299170,15cyclictest0-21swapper/119:25:161
1404699162,8cyclictest0-21swapper/220:32:432
1404699162,6cyclictest32253-21bash00:24:342
1404699162,10cyclictest0-21swapper/220:14:492
1404699161,2cyclictest0-21swapper/219:10:392
1404299167,2cyclictest7157-21bash23:10:151
1404299165,4cyclictest138550irq/27-eth021:17:581
1404299164,1cyclictest372-21unixbench_multi00:05:241
1404299163,3cyclictest0-21swapper/123:32:051
1404299160,4cyclictest138550irq/27-eth020:15:051
1404299160,3cyclictest23887-21sshd23:55:041
1404299160,14cyclictest9477-21sshd22:15:011
1404299160,14cyclictest15262-21sshd22:34:011
1404299160,12cyclictest1987-21sshd21:11:481
1405399158,2cyclictest32304-21nfsd421:00:183
1405399152,5cyclictest0-21swapper/320:35:583
1405399150,13cyclictest2121-21runrttasks19:48:593
1405399150,13cyclictest0-21swapper/319:59:103
1404699152,8cyclictest0-21swapper/219:49:112
1404699151,1cyclictest0-21swapper/220:02:182
1404699150,12cyclictest0-21swapper/219:27:442
1404299158,3cyclictest138550irq/27-eth020:25:121
1404299152,12cyclictest0-21swapper/119:36:401
1404299150,13cyclictest30724-21munin-node19:45:211
1404299150,13cyclictest0-21swapper/121:57:521
1404199159,2cyclictest0-21swapper/020:25:150
1405399146,1cyclictest0-21swapper/319:15:223
1405399142,1cyclictest0-21swapper/320:06:293
1405399140,5cyclictest0-21swapper/320:47:143
1405399140,4cyclictest28048-21smart_sda20:50:223
1404699141,8cyclictest17281-21missed_timers19:15:172
1404699141,4cyclictest29022-21munin-node20:55:162
1404699141,12cyclictest3833-21runrttasks19:57:182
1404699141,12cyclictest1877-21nfsd19:50:352
1404699141,11cyclictest2121-21runrttasks20:25:332
1404699140,2cyclictest0-21swapper/221:05:032
1404299149,4cyclictest2035-21lldpd-portmirro20:48:551
1404299141,12cyclictest138550irq/27-eth020:31:061
1404299140,5cyclictest25173-21taskset19:35:011
1404199144,4cyclictest3-21ksoftirqd/021:05:100
1405399139,3cyclictest138550irq/27-eth021:05:373
1405399139,3cyclictest0-21swapper/320:30:203
1405399131,9cyclictest0-21swapper/320:40:103
1405399131,1cyclictest0-21swapper/319:44:293
1405399130,3cyclictest0-21swapper/319:25:143
1405399130,12cyclictest0-21swapper/320:25:223
1405399130,12cyclictest0-21swapper/320:15:553
1405399130,12cyclictest0-21swapper/320:11:003
1405399130,12cyclictest0-21swapper/319:39:583
1405399130,12cyclictest0-21swapper/319:20:393
1405399130,11cyclictest0-21swapper/319:30:223
1404699139,3cyclictest1877-21nfsd19:40:552
1404699138,4cyclictest0-21swapper/220:50:142
1404299139,3cyclictest138550irq/27-eth019:10:481
1404299137,5cyclictest1843-21munin-node21:05:161
1404299137,5cyclictest0-21swapper/120:50:211
1404299136,6cyclictest0-21swapper/120:10:171
1404299131,4cyclictest1877-21nfsd19:20:541
1404299131,11cyclictest138550irq/27-eth020:05:161
14042991311,1cyclictest0-21swapper/119:58:141
1404299130,2cyclictest138550irq/27-eth019:15:111
1404299130,12cyclictest0-21swapper/121:00:111
1404299130,12cyclictest0-21swapper/120:56:181
1404299130,12cyclictest0-21swapper/120:35:431
1404299130,12cyclictest0-21swapper/120:20:531
1404299130,12cyclictest0-21swapper/120:00:011
1404299130,12cyclictest0-21swapper/119:52:051
1404299130,10cyclictest295792sleep119:43:541
1404199130,8cyclictest16318-21latency_hist19:15:000
1404199130,3cyclictest0-21swapper/020:00:200
1404199130,2cyclictest3046-21nfsd419:55:180
1404199130,1cyclictest20773-21iostat_ios20:35:140
1404199130,1cyclictest0-21swapper/020:13:130
1405399127,3cyclictest32428-21munin-node19:50:103
1405399120,10cyclictest14442-21df19:10:123
1404199128,3cyclictest0-21swapper/019:36:300
1404199128,3cyclictest0-21swapper/019:16:250
1404199128,2cyclictest0-21swapper/019:20:220
1404199124,4cyclictest0-21swapper/020:45:120
1404199123,1cyclictest1656-21kworker/0:120:43:120
1404199121,1cyclictest138550irq/27-eth020:16:010
14041991211,1cyclictest0-21swapper/020:54:020
1404199120,2cyclictest0-21swapper/021:00:130
1404199120,11cyclictest0-21swapper/020:31:020
1404199120,11cyclictest0-21swapper/019:47:550
1404199120,11cyclictest0-21swapper/019:40:200
1404199120,10cyclictest0-21swapper/020:20:540
1404199120,10cyclictest0-21swapper/019:51:040
1404199114,3cyclictest20916-21munin-node19:25:130
1404199110,1cyclictest0-21swapper/020:55:010
1404199110,1cyclictest0-21swapper/020:05:180
1404199100,9cyclictest1943-21snmpd19:30:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional