You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 01:51

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackaslot0.osadl.org (updated Sun Jun 26, 2022 12:44:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
324792970,0sleep332481-21bash09:59:513
36542940,1sleep33642-21sshd09:34:153
289562610,1sleep30-21swapper/309:53:463
3069626023,33sleep00-21swapper/007:05:450
201672600,5sleep03-21ksoftirqd/010:41:500
161622580,2sleep212050irq/23-ehci_hcd10:41:282
320552570,0sleep00-21swapper/012:24:050
239112570,2sleep13129799cyclictest10:30:371
313022560,1sleep231282-21sshd09:10:302
194882560,1sleep119467-21sshd09:26:561
3092025533,18sleep10-21swapper/107:08:341
143482540,1sleep214347-21sshd11:27:302
143482540,1sleep214347-21sshd11:27:302
39272530,1sleep33926-21sshd11:03:313
11702530,5sleep230-21ksoftirqd/209:39:512
3095225230,18sleep30-21swapper/307:09:013
91492510,0sleep20-21swapper/212:39:252
311532510,1sleep131096-21sshd11:00:131
252452510,0sleep20-21swapper/211:51:342
242162510,1sleep024215-21sshd10:24:590
232302510,1sleep323217-21sshd11:45:343
306742500,1sleep130676-21id09:19:151
73322490,0sleep2291ktimersoftd/211:24:012
54372490,0sleep00-21swapper/012:07:100
305972490,1sleep030596-21sshd10:11:050
2931224926,18sleep20-21swapper/207:05:152
226382490,0sleep010-21rcuc/011:10:540
176822490,0sleep317679-21bash11:36:253
153592490,0sleep10-21swapper/110:53:001
60322480,0sleep3381ktimersoftd/311:41:093
188232480,1sleep118793-21sshd11:16:201
86302470,0sleep123996-21diskmemload10:00:291
286252470,1sleep328624-21sshd12:09:183
202162470,0sleep120194-21sshd11:36:391
49372460,2sleep010-21rcuc/010:31:520
237802450,0sleep20-21swapper/212:31:562
197692450,1sleep228-21rcuc/210:47:372
132892440,0sleep113288-21bash12:39:481
266122430,0sleep323996-21diskmemload10:28:033
143562430,0sleep20-21swapper/212:28:152
260292410,0sleep121-21ksoftirqd/112:09:041
130112350,0sleep00-21swapper/011:01:250
126092320,0sleep312605-21bash09:11:473
170692240,0sleep317068-21bash09:38:193
3129699220,21cyclictest11431-21bash12:10:330
31297992018,1cyclictest14380-21sshd12:13:481
31296992013,1cyclictest0-21swapper/012:16:010
31299991917,1cyclictest0-21swapper/311:11:163
31297991917,1cyclictest22502-21sshd09:47:231
74722180,1sleep00-21swapper/011:38:260
3129699180,15cyclictest22165-21sshd10:50:380
31299991714,2cyclictest11043-21sshd09:46:213
31298991716,1cyclictest0-21swapper/211:11:442
31298991715,1cyclictest0-21swapper/211:38:382
31297991715,2cyclictest118850irq/25-eth010:39:421
31297991714,3cyclictest118850irq/25-eth009:53:581
3129699170,16cyclictest0-21swapper/010:07:550
3129699170,15cyclictest8071-21sshd10:35:050
31299991613,2cyclictest0-21swapper/310:47:293
31298991615,1cyclictest118850irq/25-eth012:09:102
3129699169,4cyclictest0-21swapper/011:31:050
3129699169,4cyclictest0-21swapper/011:05:540
3129699161,12cyclictest0-21swapper/011:23:130
3129699160,15cyclictest0-21swapper/009:57:050
3129999155,9cyclictest28381-21sshd11:28:463
3129999155,9cyclictest28381-21sshd11:28:463
31299991511,1cyclictest39-21ksoftirqd/310:03:503
3129899153,1cyclictest10330-21sshd11:44:302
31297991512,2cyclictest0-21swapper/109:36:591
31296991511,1cyclictest0-21swapper/009:24:210
3129699150,4cyclictest0-21swapper/012:28:420
3129699150,14cyclictest0-21swapper/011:57:000
3129699150,14cyclictest0-21swapper/011:19:390
3129699150,12cyclictest0-21swapper/009:50:100
268232150,1sleep026812-21sshd10:16:300
137762150,1sleep113773-21bash09:32:121
122562150,0sleep10-21swapper/111:24:291
3129999144,9cyclictest39-21ksoftirqd/310:53:223
3129999144,9cyclictest39-21ksoftirqd/309:26:543
3129999144,9cyclictest2608-21sshd12:32:573
3129999144,1cyclictest4514-21sshd11:32:233
31299991413,1cyclictest39-21ksoftirqd/310:21:033
31299991413,1cyclictest0-21swapper/312:02:083
31299991412,1cyclictest39-21ksoftirqd/311:53:213
31299991412,1cyclictest39-21ksoftirqd/310:30:173
31299991412,1cyclictest39-21ksoftirqd/309:15:313
31299991412,1cyclictest0-21swapper/312:23:163
3129999141,1cyclictest0-21swapper/310:05:243
3129999140,5cyclictest0-21swapper/311:21:543
3129899141,1cyclictest118850irq/25-eth011:18:512
31298991410,3cyclictest1621-21sshd11:32:582
31298991410,1cyclictest3001-21sshd10:08:422
3129899140,13cyclictest0-21swapper/210:22:152
3129899140,13cyclictest0-21swapper/208:00:242
3129799149,2cyclictest1621-21sshd11:51:191
3129799142,2cyclictest118850irq/25-eth010:55:201
31297991412,2cyclictest118850irq/25-eth009:10:421
31297991412,1cyclictest17756-21sshd11:59:411
31297991412,1cyclictest0-21swapper/111:31:451
3129799141,1cyclictest0-21swapper/112:18:511
3129799141,1cyclictest0-21swapper/110:15:161
3129799140,3cyclictest0-21swapper/110:13:421
3129799140,2cyclictest3301-21sshd11:43:521
3129799140,13cyclictest6388-21sshd11:09:331
3129799140,13cyclictest0-21swapper/112:21:391
3129699149,4cyclictest0-21swapper/012:37:450
3129699149,1cyclictest0-21swapper/010:00:030
3129699149,1cyclictest0-21swapper/009:45:260
3129699144,1cyclictest0-21swapper/011:48:250
3129699140,3cyclictest8576-21sshd09:17:130
3129699140,3cyclictest31756-21sshd09:10:330
3129699140,2cyclictest0-21swapper/009:28:510
3129699140,13cyclictest0-21swapper/010:45:050
3129699140,11cyclictest0-21swapper/010:25:230
13802140,0sleep21374-21cp12:24:152
3129999139,2cyclictest0-21swapper/311:57:443
3129999138,2cyclictest7077-21sshd10:55:123
3129999133,9cyclictest39-21ksoftirqd/309:40:083
3129999133,1cyclictest18600-21sshd10:15:453
3129999131,1cyclictest118850irq/25-eth010:12:053
31299991311,1cyclictest39-21ksoftirqd/310:41:533
31299991311,1cyclictest39-21ksoftirqd/309:20:273
3129999131,11cyclictest0-21swapper/312:12:523
3129999130,12cyclictest3492-21sshd12:38:543
3129999130,12cyclictest24299-21sshd12:17:363
3129999130,12cyclictest0-21swapper/312:26:483
3129999130,10cyclictest12725-21sshd11:18:423
3129899139,1cyclictest28155-21ls09:50:442
3129899139,1cyclictest0-21swapper/209:55:022
3129899138,4cyclictest0-21swapper/209:26:022
3129899131,10cyclictest0-21swapper/210:50:152
3129899130,4cyclictest0-21swapper/211:57:162
3129899130,2cyclictest27155-21sshd11:08:282
3129899130,2cyclictest12954-21sshd12:10:422
3129899130,1cyclictest0-21swapper/210:16:062
3129899130,12cyclictest28912-21sshd09:42:132
3129899130,12cyclictest27800-21bash10:31:002
3129899130,12cyclictest24367-21sshd09:30:162
3129899130,12cyclictest0-21swapper/209:22:102
3129799138,4cyclictest0-21swapper/110:40:341
3129799138,1cyclictest0-21swapper/112:30:091
3129799131,1cyclictest118850irq/25-eth010:47:511
31297991311,1cyclictest15174-21id10:06:511
3129799131,11cyclictest0-21swapper/111:45:271
3129799131,11cyclictest0-21swapper/111:11:371
3129799130,2cyclictest1816-21sshd10:20:071
3129799130,12cyclictest753-21sshd12:03:561
3129799130,12cyclictest0-21swapper/111:28:201
3129799130,12cyclictest0-21swapper/111:28:201
3129699137,3cyclictest0-21swapper/011:54:190
3129699133,5cyclictest0-21swapper/009:40:060
3129699132,9cyclictest21057-21sshd10:56:200
31296991311,1cyclictest26221-21sshd12:00:210
31296991311,1cyclictest0-21swapper/009:36:580
3129699130,2cyclictest0-21swapper/012:32:230
3129699130,2cyclictest0-21swapper/011:42:150
3129699130,12cyclictest5627-21sshd09:31:270
3129699130,12cyclictest0-21swapper/011:25:080
3129699130,12cyclictest0-21swapper/011:25:080
3129999122,9cyclictest18684-21sshd10:35:553
3129999121,10cyclictest0-21swapper/307:40:153
3129899129,2cyclictest0-21swapper/212:15:172
3129899129,2cyclictest0-21swapper/211:45:162
3129899129,2cyclictest0-21swapper/211:01:032
3129899129,2cyclictest0-21swapper/210:55:082
3129899121,10cyclictest0-21swapper/210:35:272
3129899121,10cyclictest0-21swapper/210:00:142
31298991210,1cyclictest8211-21sshd12:04:362
31298991210,1cyclictest31894-21bash09:45:202
31298991210,1cyclictest0-21swapper/210:13:582
3129899120,12cyclictest0-21swapper/210:25:562
3129899120,11cyclictest2325-21bash09:16:392
3129799129,2cyclictest0-21swapper/112:26:081
3129799129,2cyclictest0-21swapper/109:55:121
3129799128,3cyclictest0-21swapper/110:26:131
3129799120,1cyclictest118850irq/25-eth009:41:451
3129799120,11cyclictest0-21swapper/109:20:251
3129699129,2cyclictest0-21swapper/009:05:220
3129699129,2cyclictest0-21swapper/008:30:140
3129999119,1cyclictest22739-21ipmitool09:05:183
3129999119,1cyclictest0-21swapper/308:10:163
3129999111,9cyclictest0-21swapper/311:05:083
3129999110,10cyclictest3460-21perl08:25:143
3129899119,1cyclictest0-21swapper/208:40:152
3129899119,1cyclictest0-21swapper/208:10:142
3129899116,1cyclictest0-21swapper/207:35:142
3129899111,1cyclictest1656-21forks07:15:022
3129899110,11cyclictest0-21swapper/209:06:022
3129899110,11cyclictest0-21swapper/208:28:582
3129899110,11cyclictest0-21swapper/207:55:382
3129799119,1cyclictest0-21swapper/108:15:181
3129799110,1cyclictest0-21swapper/108:50:221
3129799110,10cyclictest14527-21fschecks_time07:40:161
3129799110,10cyclictest0-21swapper/108:13:531
3129799110,10cyclictest0-21swapper/108:00:141
3129699118,2cyclictest0-21swapper/008:20:160
3129699114,4cyclictest0-21swapper/007:10:210
3129699110,8cyclictest0-21swapper/008:00:180
3129699110,8cyclictest0-21swapper/007:35:180
3129699110,1cyclictest0-21swapper/007:25:220
3129699110,11cyclictest0-21swapper/008:54:000
3129699110,11cyclictest0-21swapper/007:55:550
3129699110,11cyclictest0-21swapper/007:44:040
3129699110,10cyclictest3610-21conntrack08:25:150
3129699110,10cyclictest0-21swapper/009:00:140
3129699110,10cyclictest0-21swapper/008:05:100
3129699110,10cyclictest0-21swapper/007:50:230
3129699110,10cyclictest0-21swapper/007:20:100
3129999108,1cyclictest1657-21memory07:15:023
3129999103,6cyclictest0-21swapper/308:20:183
3129999100,9cyclictest5046-21grep07:20:133
3129999100,9cyclictest39-21ksoftirqd/307:45:133
3129999100,9cyclictest0-21swapper/308:40:153
3129999100,9cyclictest0-21swapper/307:55:123
3129999100,9cyclictest0-21swapper/307:30:153
3129899109,1cyclictest0-21swapper/208:55:552
3129899109,1cyclictest0-21swapper/208:22:562
3129899109,1cyclictest0-21swapper/207:10:412
3129899108,1cyclictest0-21swapper/207:45:152
3129899100,9cyclictest26507-21fschecks_time08:05:132
3129899100,9cyclictest20214-21perl09:00:142
3129899100,1cyclictest0-21swapper/208:30:142
3129899100,10cyclictest0-21swapper/207:52:232
3129899100,10cyclictest0-21swapper/207:41:562
3129799109,1cyclictest0-21swapper/109:08:431
3129799109,1cyclictest0-21swapper/108:08:051
3129799108,1cyclictest12204-21perl07:35:141
3129799108,1cyclictest0-21swapper/107:55:151
3129799108,1cyclictest0-21swapper/107:50:171
3129799108,1cyclictest0-21swapper/107:30:151
3129799100,9cyclictest19465-21apache_volume09:00:111
3129799100,9cyclictest0-21swapper/107:45:161
3129799100,4cyclictest0-21swapper/108:59:181
3129799100,10cyclictest0-21swapper/108:47:421
3129799100,10cyclictest0-21swapper/108:39:341
3129799100,10cyclictest0-21swapper/108:25:301
3129799100,10cyclictest0-21swapper/107:28:281
3129699109,1cyclictest0-21swapper/008:56:400
3129699108,1cyclictest10395-21diskstats08:40:140
3129699103,4cyclictest0-21swapper/008:10:150
3129699100,9cyclictest13099-21fw_conntrack08:45:150
3129699100,7cyclictest16230-21apache_volume07:45:120
3129699100,6cyclictest0-21swapper/008:15:180
3129699100,1cyclictest0-21swapper/007:15:020
312999998,1cyclictest0-21swapper/309:02:093
312999998,1cyclictest0-21swapper/308:57:303
312999998,1cyclictest0-21swapper/307:50:163
312999998,1cyclictest0-21swapper/307:27:293
312999990,9cyclictest0-21swapper/308:51:413
312999990,9cyclictest0-21swapper/308:47:443
312999990,9cyclictest0-21swapper/308:30:143
312999990,9cyclictest0-21swapper/308:01:583
312999990,1cyclictest0-21swapper/308:10:003
312989998,1cyclictest0-21swapper/208:16:462
312989998,1cyclictest0-21swapper/207:25:142
312989990,1cyclictest0-21swapper/208:50:142
312979999,0cyclictest0-21swapper/108:41:541
312979998,1cyclictest6006-21smartctl08:30:151
312979997,1cyclictest5124-21perl07:20:141
312979990,6cyclictest0-21swapper/107:10:041
312999987,1cyclictest0-21swapper/308:17:113
312999987,1cyclictest0-21swapper/307:35:113
312999987,1cyclictest0-21swapper/307:11:063
312989986,1cyclictest0-21swapper/208:35:152
312989982,5cyclictest0-21swapper/207:20:162
312989981,6cyclictest0-21swapper/208:45:142
312979981,6cyclictest0-21swapper/108:20:161
312979981,6cyclictest0-21swapper/107:15:011
312969983,3cyclictest0-21swapper/007:34:500
312999975,1cyclictest8395-21munin-node08:35:173
312989970,6cyclictest0-21swapper/207:30:202
312969970,7cyclictest0-21swapper/008:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional