You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-21 - 19:35

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack5slot0.osadl.org (updated Tue Mar 21, 2023 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2556621270,4sleep31799599cyclictest10:51:503
707621220,1sleep20-21swapper/212:18:182
707621220,1sleep20-21swapper/212:18:182
17516210933,47sleep30-21swapper/307:05:583
1779329535,48sleep10-21swapper/107:09:221
307292940,2sleep30-21swapper/311:29:133
325492930,2sleep30-21swapper/307:40:273
303192920,1sleep10-21swapper/111:49:061
291552890,2sleep10-21swapper/110:40:461
279232890,2sleep227917-21sshd10:40:392
222842890,0sleep00-21swapper/009:17:160
219302880,2sleep121917-21sshd11:54:041
306722860,3sleep10-21swapper/110:15:291
66152850,2sleep30-21swapper/311:52:413
164642850,2sleep20-21swapper/211:13:442
55002840,1sleep30-21swapper/310:55:433
191712840,1sleep010-21rcuc/012:19:240
191712840,1sleep010-21rcuc/012:19:240
306482830,2sleep10-21swapper/110:12:391
293342820,1sleep30-21swapper/310:06:533
135542820,1sleep20-21swapper/209:16:262
320442810,2sleep10-21swapper/109:47:231
308822810,1sleep00-21swapper/010:55:110
141142810,2sleep214097-21sshd10:56:302
306062800,2sleep230601-21sshd09:35:582
283662800,2sleep228362-21sshd11:03:272
202112800,0sleep00-21swapper/009:20:040
147972800,0sleep00-21swapper/009:28:390
253802790,2sleep01799299cyclictest11:34:270
216032790,2sleep30-21swapper/310:48:413
147462790,2sleep30-21swapper/309:51:303
80772780,2sleep28065-21sshd12:04:112
220202780,0sleep00-21swapper/010:34:340
190152780,1sleep01875-21nfsd09:54:440
67302760,1sleep31799599cyclictest10:41:403
113362760,1sleep311335-21id09:39:593
11292760,2sleep30-21swapper/312:34:583
121202750,1sleep20-21swapper/211:50:202
79592650,2sleep20-21swapper/210:38:582
11522630,1sleep20-21swapper/210:46:522
209122610,3sleep21799499cyclictest09:11:082
1764426133,4sleep00-21swapper/007:07:350
1753025632,20sleep20-21swapper/207:06:102
218862380,3sleep30-21swapper/312:39:393
83632350,2sleep30-21swapper/311:44:203
310942330,1sleep02121-21runrttasks09:38:520
278022330,3sleep30-21swapper/310:23:443
70212310,2sleep10-21swapper/111:24:181
127692300,1sleep012766-21sshd10:53:350
317832290,2sleep331776-21bash12:06:113
101912291,26sleep1138550irq/27-eth010:33:331
204412280,26sleep1138550irq/27-eth012:22:221
141992280,2sleep314196-21sshd11:02:123
91992270,2sleep39186-21sshd11:38:433
269842260,24sleep1138550irq/27-eth012:11:281
269842260,24sleep1138550irq/27-eth012:11:281
254382260,2sleep10-21swapper/111:37:171
183542260,1sleep218347-21sshd11:22:272
277862250,1sleep20-21swapper/210:09:372
5032240,2sleep1501-21fschecks_time09:15:161
1799399240,4cyclictest138550irq/27-eth012:02:571
1799399240,3cyclictest138550irq/27-eth010:52:491
95402230,3sleep10-21swapper/109:34:081
57232230,2sleep20-21swapper/209:53:332
56322230,2sleep15631-21sshd10:21:461
42272230,21sleep1138550irq/27-eth012:35:171
25252230,21sleep128750irq/26-ahci[00011:04:031
1799599232,5cyclictest0-21swapper/309:19:173
1799599231,6cyclictest31314-21sshd10:12:443
51272220,4sleep30-21swapper/310:18:543
1799599221,3cyclictest30439-21sshd12:03:183
17995992210,4cyclictest1875-21nfsd11:57:293
107152220,1sleep041ktimersoftd/012:21:280
310582210,2sleep10-21swapper/110:38:091
1799599213,3cyclictest0-21swapper/309:28:543
1799599212,13cyclictest28750irq/26-ahci[00009:30:363
1799599212,11cyclictest21575-21bash09:46:273
1799599211,7cyclictest23692-21sshd12:19:483
1799599211,7cyclictest23692-21sshd12:19:483
1799599210,3cyclictest138550irq/27-eth010:29:083
1799599210,3cyclictest0-21swapper/311:47:143
17994992114,6cyclictest1955-21snmpd10:24:012
1799599202,6cyclictest10782-21bash11:10:223
1799599201,6cyclictest18932-21sshd11:19:423
1799599201,5cyclictest23069-21sshd09:55:033
176702200,2sleep12121-21runrttasks10:03:021
192412190,0sleep10-21swapper/109:37:491
1799599192,4cyclictest138550irq/27-eth012:23:253
1799599192,4cyclictest10270-21sshd10:02:233
1799599192,12cyclictest1061-21sshd12:29:153
1799599192,12cyclictest1061-21sshd12:29:153
1799599191,7cyclictest15407-21sshd11:33:343
1799599191,3cyclictest21067-21sshd12:10:543
1799599191,3cyclictest21067-21sshd12:10:543
1799599190,9cyclictest0-21swapper/310:31:273
1799599190,6cyclictest30799-21sshd09:24:083
1799599190,5cyclictest28812-21sshd09:11:533
1799599190,13cyclictest11853-21sshd11:21:533
1799299197,4cyclictest30531-21bash11:23:310
1799299192,14cyclictest0-21swapper/007:28:570
17992991916,2cyclictest1877-21nfsd10:12:160
1799299190,11cyclictest4905-21sshd12:26:410
1799299190,11cyclictest4905-21sshd12:26:410
84792180,2sleep28304-21sshd09:56:342
1799599182,11cyclictest0-21swapper/310:38:403
1799599181,14cyclictest19880-21bash09:40:393
1799599180,5cyclictest0-21swapper/311:05:183
17994991813,4cyclictest11632-21sshd10:19:302
17994991812,5cyclictest5377-21sshd12:23:532
1799399183,12cyclictest20696-21sshd12:16:361
1799399183,12cyclictest20696-21sshd12:16:361
1799399180,13cyclictest2416-21sshd10:07:221
1799399180,13cyclictest23680-21sshd10:29:031
1799299189,6cyclictest6227-21awk09:05:360
1799299189,1cyclictest0-21swapper/010:25:020
1799299184,5cyclictest0-21swapper/011:38:240
1799299180,9cyclictest11205-21bash11:27:290
1799299180,16cyclictest28266-21sshd09:41:250
1799299180,11cyclictest0-21swapper/009:56:140
57252171,14sleep1138550irq/27-eth011:44:071
240292170,15sleep128750irq/26-ahci[00011:34:201
1799599170,8cyclictest0-21swapper/307:39:593
1799499175,10cyclictest24608-21bash11:42:542
1799499175,10cyclictest0-21swapper/209:25:382
17994991712,4cyclictest25-21rcuop/111:15:342
1799399177,3cyclictest16545-21bash10:48:141
17993991713,3cyclictest21357-21sshd11:28:241
1799299178,1cyclictest0-21swapper/011:10:160
1799299175,6cyclictest0-21swapper/007:55:210
1799299173,9cyclictest10289-21bash11:01:500
1799299173,10cyclictest0-21swapper/012:37:200
1799299171,1cyclictest19970-21sshd12:30:490
1799299171,10cyclictest0-21swapper/009:47:440
1799299170,1cyclictest25146-21cp09:14:350
1799299170,1cyclictest0-21swapper/010:49:390
1799599168,6cyclictest6941-21taskset09:07:513
1799499161,14cyclictest0-21swapper/212:10:012
1799499161,14cyclictest0-21swapper/212:10:012
1799499160,15cyclictest31738-21sshd11:26:242
1799499160,14cyclictest13002-21sshd11:30:272
17993991610,5cyclictest684-21sshd11:12:221
1799399160,13cyclictest26896-21munin-run07:30:011
1799399160,12cyclictest23067-21sshd09:20:191
1799399160,11cyclictest23181-21sshd11:05:461
1799299169,6cyclictest3-21ksoftirqd/011:17:470
1799299168,4cyclictest19809-21sshd11:53:530
1799299168,4cyclictest0-21swapper/011:06:190
1799299168,3cyclictest2338-21sshd10:07:210
1799299164,6cyclictest0-21swapper/010:03:480
1799299163,12cyclictest3-21ksoftirqd/012:10:360
1799299163,12cyclictest3-21ksoftirqd/012:10:360
1799299162,1cyclictest0-21swapper/007:36:470
1799299161,6cyclictest0-21swapper/012:01:310
17992991611,5cyclictest0-21swapper/007:53:110
17992991611,4cyclictest0-21swapper/011:56:050
17992991611,4cyclictest0-21swapper/011:48:220
1799299161,11cyclictest0-21swapper/010:17:030
1799299160,2cyclictest448-21sshd11:43:380
1799299160,15cyclictest28111-21sshd12:08:450
1799299160,15cyclictest0-21swapper/010:22:290
1799299160,13cyclictest22803-21bash10:43:060
10402160,6sleep20-21swapper/209:33:222
232722150,13sleep1138550irq/27-eth009:43:521
1799599152,12cyclictest138550irq/27-eth008:06:513
1799599151,6cyclictest4025-21pmu-power09:00:213
1799599151,3cyclictest20191-21latency_hist07:15:003
1799599150,2cyclictest138550irq/27-eth007:25:483
1799599150,13cyclictest5124-21munin-node07:50:253
1799599150,13cyclictest3104-21munin-node07:45:273
1799599150,13cyclictest21651-21munin-node07:15:233
1799599150,13cyclictest0-21swapper/308:50:233
1799599150,13cyclictest0-21swapper/308:25:143
1799599150,10cyclictest147450irq/28-eth1-rx-07:56:583
1799499158,6cyclictest16292-21sshd12:07:432
1799499157,7cyclictest1876-21nfsd11:57:542
1799499154,4cyclictest138550irq/27-eth010:11:432
17994991512,2cyclictest22523-21sshd09:20:172
17994991512,2cyclictest19792-21sshd12:28:022
17994991512,2cyclictest19792-21sshd12:28:022
17994991512,2cyclictest1877-21nfsd10:03:312
17994991512,2cyclictest12617-21sshd09:42:532
17994991511,3cyclictest18880-21sshd11:39:352
17994991511,3cyclictest12511-21sshd12:35:542
17994991510,4cyclictest16259-21sshd11:45:012
1799499150,14cyclictest0-21swapper/208:10:162
1799499150,13cyclictest16350-21sshd09:48:502
1799399154,9cyclictest30918-21bash12:06:081
1799399153,10cyclictest24336-21bash12:25:301
1799399153,10cyclictest24336-21bash12:25:291
1799399152,1cyclictest22-21ksoftirqd/109:52:191
1799399152,12cyclictest0-21swapper/109:01:361
17993991510,3cyclictest1115-21sshd09:27:211
17993991510,0cyclictest0-21swapper/111:55:261
1799399150,6cyclictest16537-21sshd12:30:321
1799399150,4cyclictest12417-21sshd09:10:281
1799399150,13cyclictest26645-21sshd09:55:221
1799399150,11cyclictest1987-21sshd11:16:321
1799399150,10cyclictest0-21swapper/110:55:011
1799299159,1cyclictest0-21swapper/008:01:100
1799299157,7cyclictest0-21swapper/008:33:580
1799299154,4cyclictest0-21swapper/009:31:380
1799299153,4cyclictest0-21swapper/007:40:280
1799299153,1cyclictest3-21ksoftirqd/010:35:320
1799299151,13cyclictest0-21swapper/008:52:190
1799299151,11cyclictest0-21swapper/008:26:060
1799299151,10cyclictest0-21swapper/008:36:370
1799299150,12cyclictest0-21swapper/008:16:290
1799299150,12cyclictest0-21swapper/007:32:020
1799299150,12cyclictest0-21swapper/007:20:230
1799299150,12cyclictest0-21swapper/007:16:420
1799299150,11cyclictest0-21swapper/008:48:260
1799599148,4cyclictest9406-21pmu-power08:00:213
1799599146,5cyclictest138550irq/27-eth007:30:253
1799599143,2cyclictest0-21swapper/308:31:213
1799599142,10cyclictest0-21swapper/308:10:303
1799499148,5cyclictest138550irq/27-eth008:55:142
1799499142,10cyclictest0-21swapper/212:30:022
17994991411,2cyclictest5893-21sshd10:33:092
17994991410,3cyclictest27085-21sshd11:06:072
1799499140,6cyclictest0-21swapper/210:51:332
1799499140,6cyclictest0-21swapper/208:40:052
1799499140,13cyclictest0-21swapper/207:30:012
1799499140,12cyclictest5174-21bash10:27:262
1799399140,13cyclictest28750irq/26-ahci[00007:10:141
1799299148,3cyclictest0-21swapper/008:20:260
17992991411,1cyclictest0-21swapper/008:40:510
17992991411,1cyclictest0-21swapper/008:07:060
1799299140,9cyclictest0-21swapper/009:00:230
1799299140,1cyclictest0-21swapper/007:10:440
1799599130,6cyclictest24992-21awk08:35:243
1799599130,4cyclictest1605-21avahi-daemon08:40:413
1799599130,3cyclictest138550irq/27-eth007:21:493
1799599130,12cyclictest0-21swapper/308:21:183
1799599130,11cyclictest790-21df08:55:163
1799599130,11cyclictest15411-21sort08:15:223
1799499138,4cyclictest32545-21munin-node07:40:272
1799499138,4cyclictest0-21swapper/208:07:092
1799499137,5cyclictest0-21swapper/208:26:432
1799499130,5cyclictest0-21swapper/207:50:442
1799499130,2cyclictest28750irq/26-ahci[00007:55:262
1799499130,12cyclictest36-21rcuop/207:39:212
1799499130,12cyclictest1877-21nfsd08:51:022
1799499130,12cyclictest0-21swapper/209:06:012
1799499130,12cyclictest0-21swapper/209:01:252
1799499130,12cyclictest0-21swapper/208:45:252
1799499130,12cyclictest0-21swapper/208:20:142
1799499130,12cyclictest0-21swapper/208:16:352
1799499130,12cyclictest0-21swapper/208:02:152
1799499130,12cyclictest0-21swapper/207:45:492
1799499130,11cyclictest23362-21iostat_ios07:20:182
1799399139,3cyclictest138550irq/27-eth008:32:211
1799399139,3cyclictest0-21swapper/108:54:451
1799399132,10cyclictest138550irq/27-eth008:26:011
1799399132,10cyclictest138550irq/27-eth008:15:561
1799399131,11cyclictest25348-21awk08:35:281
1799399131,11cyclictest138550irq/27-eth007:50:571
17993991311,1cyclictest0-21swapper/107:40:301
1799399130,5cyclictest0-21swapper/107:47:541
1799399130,12cyclictest0-21swapper/109:06:111
1799399130,12cyclictest0-21swapper/108:12:401
1799399130,12cyclictest0-21swapper/108:00:131
1799399130,12cyclictest0-21swapper/107:55:011
1799399130,12cyclictest0-21swapper/107:36:301
1799399130,12cyclictest0-21swapper/107:34:201
1799399130,12cyclictest0-21swapper/107:22:001
1799399130,11cyclictest849-21df_inode08:55:171
1799399130,11cyclictest27489-21processes08:40:341
1799399130,11cyclictest22387-21latency_hist07:20:001
1799399130,11cyclictest11140-21perl08:05:231
1799399130,10cyclictest138550irq/27-eth008:20:531
1799599120,10cyclictest0-21swapper/308:45:183
1799499120,3cyclictest22396-21hddtemp_smartct08:30:172
1799499120,3cyclictest21225-21memory07:15:172
1799499120,11cyclictest24865-21munin-node07:25:132
1799499120,10cyclictest18609-21fschecks_time07:10:142
1799499120,10cyclictest0-21swapper/208:35:222
1799399120,11cyclictest0-21swapper/108:46:071
1799299117,3cyclictest0-21swapper/008:55:540
1799299111,9cyclictest0-21swapper/008:10:230
1799299101,8cyclictest0-21swapper/007:45:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional