You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 05:38

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack5slot0.osadl.org (updated Mon Jun 27, 2022 00:45:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
435021330,1sleep00-21swapper/000:15:260
1739321330,1sleep10-21swapper/123:12:131
2379121300,0sleep023774-21sshd22:52:240
1080221290,0sleep010801-21id22:02:030
3128721270,1sleep331288-21sshd21:20:333
298921230,1sleep30-21swapper/322:15:493
2293521180,1sleep11869099cyclictest23:35:591
3255221070,3sleep30-21swapper/322:53:113
2619621050,1sleep026192-21bash23:04:200
1861121010,4sleep21869499cyclictest00:16:482
1383321010,3sleep01868599cyclictest00:01:270
18269210033,54sleep10-21swapper/119:06:411
1596521000,1sleep31870099cyclictest00:31:233
169522990,3sleep016954-21bash23:29:410
170842980,1sleep017079-21bash23:58:510
1831129632,49sleep30-21swapper/319:07:123
181802950,1sleep018183-21cpuspeed_turbos20:20:170
286362930,1sleep10-21swapper/122:15:181
112532910,0sleep00-21swapper/021:21:420
309222890,4sleep31870099cyclictest21:26:183
210032890,2sleep11869099cyclictest22:31:531
121792880,2sleep20-21swapper/222:07:582
92192870,0sleep00-21swapper/021:12:560
233262860,1sleep10-21swapper/123:09:541
209382860,2sleep30-21swapper/323:03:493
258152850,1sleep225814-21sshd22:43:522
193192850,1sleep30-21swapper/321:10:553
93812840,2sleep29371-21sshd21:33:032
93812840,2sleep29371-21sshd21:33:022
13742830,2sleep31373-21bash00:18:143
110152830,4sleep31870099cyclictest21:38:573
110152830,4sleep31870099cyclictest21:38:563
107872830,2sleep210788-21sshd21:27:252
45472820,3sleep01868599cyclictest00:39:180
43212820,3sleep01868599cyclictest22:21:420
276122820,2sleep10-21swapper/121:11:411
201822820,1sleep20-21swapper/222:23:092
1849028232,45sleep20-21swapper/219:09:272
170332820,1sleep20-21swapper/200:10:382
89422810,2sleep28938-21bash22:04:492
54592790,4sleep31870099cyclictest22:44:593
189742780,4sleep31870099cyclictest23:15:163
189742780,4sleep31870099cyclictest23:15:163
140202780,1sleep10-21swapper/122:51:301
105732780,2sleep110572-21bash21:35:591
105732780,2sleep110572-21bash21:35:581
51662770,1sleep00-21swapper/000:24:310
100552770,3sleep20-21swapper/223:23:132
100552770,3sleep20-21swapper/223:23:122
248152760,1sleep20-21swapper/221:40:122
56402750,2sleep35641-21bash22:39:123
229912750,2sleep322986-21sshd22:00:183
151142730,1sleep218676-21cyclictest23:29:302
211182710,1sleep00-21swapper/000:31:530
95302700,3sleep11869099cyclictest22:28:011
167582680,4sleep31870099cyclictest23:47:073
321392670,2sleep30-21swapper/321:43:453
100592670,6sleep10-21swapper/121:59:071
301452660,2sleep10-21swapper/121:17:381
309572640,1sleep31870099cyclictest23:51:153
311042630,2sleep01868599cyclictest22:32:490
139502610,2sleep10-21swapper/122:37:021
35992580,2sleep33594-21sshd23:40:103
323312460,3sleep31877-21nfsd00:38:533
306272450,2sleep230622-21sshd21:55:112
119212430,36sleep2138550irq/27-eth022:36:532
18700994233,5cyclictest28793-21bash23:56:573
1727824032,4sleep00-21swapper/019:05:160
312642380,1sleep031261-21bash21:49:270
18685993726,4cyclictest0-21swapper/000:07:190
18700993617,10cyclictest0-21swapper/300:20:313
117132360,0sleep211716-21bash23:00:052
18700993530,3cyclictest2788-21sshd21:32:283
18700993530,3cyclictest2788-21sshd21:32:273
18700993427,5cyclictest23568-21sshd23:12:483
18700993425,5cyclictest17201-21sshd21:51:023
18700993425,5cyclictest17201-21sshd21:51:013
41162330,30sleep3138550irq/27-eth022:21:423
1870099328,9cyclictest18580-21bash00:07:513
18700993226,4cyclictest30491-21sshd23:25:043
316912310,0sleep031687-21bash22:06:470
18700993124,5cyclictest1126-21sshd22:12:493
18700993124,3cyclictest7496-21id22:59:423
298372300,1sleep20-21swapper/220:45:182
18700993022,5cyclictest27586-21sshd22:32:303
18700993017,10cyclictest0-21swapper/322:29:063
1869499304,15cyclictest18276-21sshd23:35:332
52382290,26sleep2138550irq/27-eth023:34:292
220492291,26sleep128750irq/26-ahci[00000:17:071
18700992921,6cyclictest29906-21sshd21:55:073
18700992920,6cyclictest29756-21sshd00:26:443
18700992920,5cyclictest30428-21sshd23:30:483
18700992919,7cyclictest32203-21sshd22:47:203
18700992919,7cyclictest0-21swapper/300:02:023
296842280,3sleep136-21rcuop/223:42:281
18700992823,3cyclictest10420-21sshd00:13:073
18700992821,5cyclictest7320-21sshd23:05:263
18700992820,5cyclictest18301-21sshd22:08:303
18700992813,4cyclictest1182-21sshd23:22:253
18700992813,4cyclictest1182-21sshd23:22:243
181252280,2sleep10-21swapper/121:22:191
161362280,3sleep216132-21sshd23:09:142
74952270,1sleep27497-21id00:03:542
271872270,1sleep027188-21sshd22:38:130
260582270,21sleep2138550irq/27-eth021:51:522
260582270,21sleep2138550irq/27-eth021:51:512
234052270,0sleep00-21swapper/023:50:320
18700992722,3cyclictest1987-21sshd23:35:123
18685992712,14cyclictest3-21ksoftirqd/000:27:260
253852260,3sleep125377-21sshd00:02:311
199662260,1sleep00-21swapper/023:41:350
18700992619,4cyclictest0-21swapper/321:47:263
22262250,23sleep1138550irq/27-eth022:56:161
1869099251,10cyclictest31556-21sshd00:12:041
1868599251,19cyclictest0-21swapper/021:02:560
18685992511,8cyclictest3-21ksoftirqd/021:57:540
1869499242,0cyclictest29659-21sshd21:17:362
1868599240,15cyclictest28176-21sshd22:46:580
252782230,1sleep00-21swapper/023:12:570
18700992310,9cyclictest14412-21sshd21:19:093
1869499239,1cyclictest0-21swapper/223:54:442
1869499232,8cyclictest0-21swapper/219:15:262
18685992313,8cyclictest3-21ksoftirqd/023:39:030
65882220,1sleep26591-21kernelversion21:05:172
295732220,2sleep129570-21cp23:48:161
283352220,20sleep2138550irq/27-eth023:42:212
210702220,0sleep021069-21seq22:55:060
1869499225,5cyclictest138550irq/27-eth022:48:352
1869499221,3cyclictest0-21swapper/221:49:482
1869499220,12cyclictest21826-21sshd23:18:292
1869499220,12cyclictest21826-21sshd23:18:282
18690992213,6cyclictest2027-21bash22:12:541
18685992213,3cyclictest3-21ksoftirqd/023:30:510
18685992211,5cyclictest3-21ksoftirqd/023:24:350
18685992211,5cyclictest3-21ksoftirqd/023:24:340
1869499218,1cyclictest0-21swapper/222:29:562
1869499216,3cyclictest9051-21bash00:39:432
1869499215,8cyclictest2133-21munin-node19:10:022
18694992116,4cyclictest12672-21sshd22:51:232
1869099218,9cyclictest2878-21sshd22:41:491
18690992118,2cyclictest0-21swapper/100:22:561
1869099210,9cyclictest0-21swapper/100:25:101
18685992116,3cyclictest3-21ksoftirqd/023:06:020
18685992111,4cyclictest3-21ksoftirqd/023:45:440
18685992110,5cyclictest3-21ksoftirqd/021:42:490
113552210,1sleep10-21swapper/122:45:241
1869499205,2cyclictest0-21swapper/221:01:222
1869499205,0cyclictest0-21swapper/200:31:102
1869499204,1cyclictest10123-21bash21:24:332
1869499201,2cyclictest4386-21sshd23:11:002
18694992011,2cyclictest7993-21sshd00:21:452
1869499200,2cyclictest0-21swapper/219:20:102
1869099208,5cyclictest14995-21sshd21:30:361
1869099208,5cyclictest14995-21sshd21:30:361
1869099207,11cyclictest28750irq/26-ahci[00023:20:291
1869099207,11cyclictest28750irq/26-ahci[00023:20:291
1869099201,9cyclictest445-21jbd2/dm-0-822:24:311
1869099200,18cyclictest30390-21bash21:43:351
18685992014,4cyclictest3-21ksoftirqd/021:53:400
18685992014,4cyclictest3-21ksoftirqd/021:53:390
1868599200,9cyclictest24972-21sshd00:14:270
106232209,7sleep1138550irq/27-eth021:50:261
106232209,7sleep1138550irq/27-eth021:50:261
1869499194,2cyclictest0-21swapper/220:52:382
1869499194,14cyclictest8921-21munin-node20:00:102
1869499193,15cyclictest138550irq/27-eth019:45:382
1869499193,10cyclictest10237-21sshd22:13:382
1869499192,2cyclictest0-21swapper/222:18:232
1869499191,0cyclictest0-21swapper/223:55:112
18694991910,1cyclictest4058-21sshd23:45:562
1869499190,17cyclictest33-21ksoftirqd/222:32:032
1869499190,12cyclictest22730-21sshd00:26:032
18690991915,2cyclictest10919-21sshd00:30:541
18690991912,5cyclictest24538-21sshd23:56:331
18690991911,5cyclictest27737-21cp23:16:031
18690991911,5cyclictest27737-21cp23:16:021
18685991914,3cyclictest3-21ksoftirqd/022:40:030
18685991914,3cyclictest3-21ksoftirqd/022:27:060
18685991913,4cyclictest3-21ksoftirqd/022:18:150
18685991912,6cyclictest3-21ksoftirqd/022:14:250
18685991912,3cyclictest3-21ksoftirqd/023:19:590
18685991912,3cyclictest3-21ksoftirqd/023:19:590
18685991910,4cyclictest3-21ksoftirqd/021:32:270
18685991910,4cyclictest3-21ksoftirqd/021:32:260
1869499188,3cyclictest14488-21rm21:36:192
1869499188,3cyclictest14488-21rm21:36:192
18694991814,3cyclictest1876-21nfsd22:55:032
1869499181,2cyclictest0-21swapper/220:25:472
1869499180,7cyclictest1875-21nfsd00:05:102
1869499180,2cyclictest0-21swapper/220:41:512
1869099189,5cyclictest29681-21sshd21:46:231
1869099189,1cyclictest0-21swapper/122:04:551
1869099181,14cyclictest1876-21nfsd00:36:271
1868599187,5cyclictest3-21ksoftirqd/021:17:440
1869499173,11cyclictest24874-21sshd21:11:262
1869499171,2cyclictest0-21swapper/219:30:222
1869099179,6cyclictest4372-21bash00:09:341
1869099175,4cyclictest23182-21sshd22:08:571
18690991713,3cyclictest23413-21fschecks_count23:30:131
18690991711,4cyclictest11844-21sshd23:03:001
18685991715,1cyclictest3-21ksoftirqd/021:38:120
18685991715,1cyclictest3-21ksoftirqd/021:38:110
1868599170,16cyclictest0-21swapper/021:25:280
18700991611,3cyclictest3316-21crond21:00:003
1869499164,2cyclictest0-21swapper/220:37:062
1869499162,6cyclictest0-21swapper/220:16:392
1869099169,4cyclictest0-21swapper/123:51:441
1869099161,4cyclictest0-21swapper/123:26:391
18690991614,1cyclictest0-21swapper/120:33:281
18690991612,2cyclictest0-21swapper/119:40:121
1869099161,14cyclictest0-21swapper/120:51:381
1869099160,13cyclictest31154-21bash21:29:161
1869099160,11cyclictest31198-21perf20:50:011
1870099159,3cyclictest0-21swapper/320:10:183
1870099150,13cyclictest0-21swapper/319:35:543
1869499154,2cyclictest0-21swapper/220:55:112
1869499153,7cyclictest0-21swapper/220:10:102
1869499152,5cyclictest138550irq/27-eth019:36:462
1869499151,13cyclictest147450irq/28-eth1-rx-19:29:582
1869099157,2cyclictest0-21swapper/120:41:111
1869099150,13cyclictest0-21swapper/120:01:411
1868599150,12cyclictest0-21swapper/019:59:000
1870099149,4cyclictest147450irq/28-eth1-rx-19:52:503
1870099149,4cyclictest147450irq/28-eth1-rx-19:52:493
1870099148,5cyclictest44-21ksoftirqd/320:43:453
1870099140,9cyclictest8706-21grep20:00:003
1870099140,6cyclictest21869-21latency19:15:173
1869499144,5cyclictest958-21ntp_states19:40:192
1869499142,4cyclictest138550irq/27-eth020:21:532
1869499142,11cyclictest138550irq/27-eth020:05:352
1869499141,0cyclictest0-21swapper/219:50:332
1869499141,0cyclictest0-21swapper/219:50:332
1869099148,5cyclictest0-21swapper/120:06:141
1869099145,1cyclictest0-21swapper/120:27:541
1869099140,9cyclictest26703-21idleruntime-cro20:40:001
1869099140,8cyclictest4419-21munin-node19:50:201
1869099140,8cyclictest4419-21munin-node19:50:201
1869099140,5cyclictest0-21swapper/119:48:251
1869099140,13cyclictest0-21swapper/119:39:571
1869099140,13cyclictest0-21swapper/119:27:401
1869099140,13cyclictest0-21swapper/119:21:341
1869099140,12cyclictest1943-21snmpd20:25:001
1868599147,6cyclictest0-21swapper/019:20:530
1868599142,10cyclictest0-21swapper/020:50:170
1868599140,12cyclictest0-21swapper/020:00:180
1870099139,3cyclictest0-21swapper/321:00:393
1870099138,4cyclictest0-21swapper/319:11:293
1870099132,2cyclictest32295-21munin-node19:40:163
1870099131,11cyclictest0-21swapper/320:30:283
1870099131,10cyclictest0-21swapper/320:05:143
1870099130,5cyclictest0-21swapper/319:32:493
1870099130,4cyclictest18585-21gltestperf20:20:193
1870099130,12cyclictest1875-21nfsd20:35:343
1870099130,12cyclictest0-21swapper/320:48:293
1870099130,12cyclictest0-21swapper/320:16:113
1870099130,12cyclictest0-21swapper/319:45:083
1870099130,12cyclictest0-21swapper/319:25:473
1870099130,12cyclictest0-21swapper/319:21:293
1870099130,11cyclictest22059-21taskset20:28:213
1870099130,11cyclictest10158-21rt-features20:00:193
1869499137,3cyclictest138550irq/27-eth020:30:362
1869499130,12cyclictest0-21swapper/219:55:072
1869099131,2cyclictest138550irq/27-eth019:30:011
18690991311,1cyclictest0-21swapper/121:06:171
1869099131,10cyclictest0-21swapper/120:13:271
1869099130,4cyclictest21789-21grep19:15:151
1869099130,12cyclictest0-21swapper/121:00:391
1869099130,12cyclictest0-21swapper/120:59:141
1869099130,12cyclictest0-21swapper/119:55:181
1869099130,12cyclictest0-21swapper/119:10:521
18685991311,1cyclictest0-21swapper/019:27:410
1868599130,3cyclictest0-21swapper/019:36:500
1868599130,12cyclictest1943-21snmpd20:45:260
1868599130,10cyclictest8-21rcu_preempt20:36:580
1870099120,10cyclictest5899-21munin-node21:05:133
1870099120,10cyclictest0-21swapper/320:50:123
1869099120,11cyclictest17500-21taskset20:17:401
1868599126,3cyclictest0-21swapper/019:50:210
1868599126,3cyclictest0-21swapper/019:50:210
1868599124,4cyclictest21035-21memory20:25:170
1868599120,2cyclictest0-21swapper/020:16:100
1868599120,1cyclictest0-21swapper/020:56:110
1868599120,1cyclictest0-21swapper/019:42:200
1868599120,1cyclictest0-21swapper/019:34:050
1868599120,11cyclictest1943-21snmpd20:30:010
1868599120,11cyclictest0-21swapper/019:15:020
1868599120,11cyclictest0-21swapper/019:11:480
1868599120,10cyclictest0-21swapper/021:06:050
1868599120,10cyclictest0-21swapper/020:06:030
1868599111,8cyclictest0-21swapper/020:10:320
1868599110,7cyclictest0-21swapper/020:40:150
1868599110,1cyclictest0-21swapper/019:45:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional