You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-11-26 - 16:33

x86 Intel Core i7-2600K @3400 MHz, Linux 5.9.0-rc2-rt1 (Profile)

Latency plot of system in rack #4, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h200 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Sat Nov 26, 2022 12:44:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10419521200,103pltrace0-21swapper/407:10:014
7590521120,102pltrace0-21swapper/007:05:220
1041952870,84pltrace0-21swapper/207:10:012
266862810,0sleep30-21swapper/310:25:003
1544752810,6pltrace1092699cyclictest09:40:365
1068827963,12sleep70-21swapper/707:09:267
829627862,12sleep50-21swapper/507:05:075
1045327660,12sleep30-21swapper/307:06:003
136992740,1sleep755-21ksoftirqd/708:10:127
793652720,0pltrace0-21swapper/312:10:373
1061327245,12sleep60-21swapper/607:08:216
148502690,0sleep00-21swapper/009:40:270
356452670,0pltrace0-21swapper/110:00:361
228082670,0sleep70-21swapper/712:25:227
1052526650,12sleep10-21swapper/107:07:021
36072640,1sleep20-21swapper/210:00:392
25272580,0sleep60-21swapper/611:04:146
257342270,1sleep30-21swapper/307:35:013
2857152240,1pltrace28575-21pltrace-run12:30:383
2222552240,1pltrace22227-21tee09:15:360
36262220,0sleep10-21swapper/107:50:201
1091999220,1cyclictest0-21swapper/412:25:234
125202200,0chrt0-21swapper/211:13:382
70142190,0sleep60-21swapper/610:35:306
322752190,1sleep232277-21sendmail_mailqu11:00:242
5672180,0sleep60-21swapper/612:35:236
1831352180,2pltrace20359-21missed_timers08:20:244
1450052180,0pltrace0-21swapper/711:49:527
710252170,2pltrace10533-21df_inode10:40:165
710252170,2pltrace10533-21df_inode10:40:155
2999152170,2pltrace1441-21ssh12:05:184
2857152170,0pltrace0-21swapper/012:35:290
277752170,1pltrace0-21swapper/012:05:520
2674652170,0pltrace0-21swapper/008:32:240
2522952170,2pltrace29712-21irqstats09:55:244
249962170,1sleep10-21swapper/109:20:001
1309552170,9pltrace0-21swapper/212:20:142
1309552170,2pltrace16334-21cstates12:20:134
1226552170,2pltrace1089-21snmpd10:43:240
1226552170,2pltrace1089-21snmpd10:43:240
1093299170,16cyclictest0-21swapper/609:30:316
1041952170,1pltrace12284-21irqstats_LOC07:10:174
866852160,5pltrace0-21swapper/410:09:064
43252160,1pltrace0-21swapper/011:03:290
3144452160,1pltrace31660-21ssh11:31:000
3144452160,1pltrace0-21swapper/511:30:525
2999152160,1pltrace9-21ksoftirqd/012:04:030
2522952160,1pltrace0-21swapper/009:54:560
2409852160,1pltrace25354-21ssh10:22:534
2409852160,1pltrace0-21swapper/010:23:490
2342252160,1pltrace0-21swapper/012:30:240
1824552160,0pltrace0-21swapper/012:20:550
1709452160,6pltrace0-21swapper/009:10:560
1709452160,5pltrace0-21swapper/509:11:245
1709452160,1pltrace37-21ksoftirqd/409:11:444
1450052160,2pltrace15796-21ssh11:47:560
127652160,1pltrace43-21ksoftirqd/512:35:445
1091999161,14cyclictest0-21swapper/409:30:314
982152150,5pltrace0-21swapper/409:40:184
982152150,5pltrace0-21swapper/409:39:584
866852150,7pltrace1141-21runrttasks10:09:532
866852150,1pltrace0-21swapper/010:09:450
866852150,0pltrace0-21swapper/510:08:585
43252150,1pltrace1602-21ssh11:02:344
3244352150,0pltrace0-21swapper/209:26:202
3144452150,0pltrace0-21swapper/211:31:072
3100252150,6pltrace0-21swapper/007:44:070
3081452150,7pltrace30818-21tail09:55:360
277752150,0pltrace0-21swapper/512:05:455
2734852150,1pltrace31129-21gltestperf09:25:194
2734852150,0pltrace0-21swapper/009:25:190
2674652150,2pltrace37-21ksoftirqd/408:33:384
2522952150,6pltrace30389-21timerandwakeup09:55:302
2409852150,6pltrace24845-21ssh10:21:597
2409852150,1pltrace0-21swapper/510:22:325
2256952150,5pltrace0-21swapper/407:29:394
2256952150,2pltrace23811-21df_inode07:30:177
2256952150,0pltrace0-21swapper/007:30:240
1598552150,1pltrace0-21swapper/011:17:540
1598552150,0pltrace0-21swapper/711:17:077
1544752150,2pltrace15652-21taskset09:40:597
1329952150,0pltrace0-21swapper/007:15:010
1226552150,6pltrace0-21swapper/210:40:372
1226552150,6pltrace0-21swapper/210:40:372
1226552150,5pltrace0-21swapper/710:42:567
1226552150,5pltrace0-21swapper/710:42:567
1226552150,1pltrace12591-21sh10:41:134
1226552150,1pltrace12591-21sh10:41:134
1093299150,14cyclictest0-21swapper/612:07:406
1093299150,14cyclictest0-21swapper/610:26:286
1093299150,14cyclictest0-21swapper/609:24:246
1091999151,13cyclictest0-21swapper/411:45:224
356452140,11pltrace0-21swapper/010:00:590
3244352140,4pltrace0-21swapper/709:26:337
3237052140,0pltrace0-21swapper/008:45:170
3144452140,6pltrace0-21swapper/411:32:074
3081452140,5pltrace0-21swapper/509:55:385
2999152140,5pltrace0-21swapper/412:04:444
2999152140,5pltrace0-21swapper/212:04:592
2857152140,5pltrace0-21swapper/512:34:305
2857152140,0pltrace0-21swapper/212:35:162
278352140,6pltrace2861-21sshd08:46:375
2774152140,0pltrace0-21swapper/010:57:280
2674652140,0pltrace0-21swapper/508:33:585
2630152140,0pltrace0-21swapper/011:27:300
2538052140,1pltrace1141-21runrttasks07:30:382
2522952140,4pltrace30491-21ls09:55:307
2409852140,5pltrace0-21swapper/210:23:152
2112352140,10pltrace0-21swapper/008:25:250
1993052140,11pltrace0-21swapper/009:46:200
1709452140,1pltrace17708-21ssh09:11:437
1598552140,4pltrace0-21swapper/211:17:342
1598552140,0pltrace0-21swapper/411:17:154
1309552140,5pltrace0-21swapper/512:19:475
1309552140,4pltrace0-21swapper/412:19:404
1309552140,1pltrace17495-21pltrace12:20:217
1309552140,0pltrace0-21swapper/212:19:532
127652140,1pltrace1401-21ssh12:35:507
1268852140,4pltrace0-21swapper/008:06:590
1268852140,0pltrace0-21swapper/508:07:595
1226552140,0pltrace0-21swapper/310:42:433
1226552140,0pltrace0-21swapper/310:42:423
1093299142,7cyclictest2522952pltrace09:55:036
10932991410,3cyclictest0-21swapper/612:21:466
10932991410,3cyclictest0-21swapper/609:35:096
1093299140,4cyclictest0-21swapper/610:53:516
1093299140,13cyclictest0-21swapper/611:22:016
1093299140,13cyclictest0-21swapper/611:18:266
1093299140,13cyclictest0-21swapper/610:57:356
1091999140,12cyclictest13382-21ntp_states09:05:204
982152130,5pltrace0-21swapper/209:39:522
933352130,0pltrace0-21swapper/411:44:494
840652130,4pltrace0-21swapper/408:58:504
840652130,4pltrace0-21swapper/208:59:302
422952130,1pltrace1141-21runrttasks07:54:430
2999152130,0pltrace0-21swapper/712:04:307
2999152130,0pltrace0-21swapper/212:05:252
2857152130,0pltrace0-21swapper/412:35:034
2857152130,0pltrace0-21swapper/412:34:294
278352130,0pltrace0-21swapper/408:46:104
1895552130,6pltrace0-21swapper/010:18:100
1824552130,4pltrace0-21swapper/512:21:095
1709452130,5pltrace0-21swapper/209:11:362
1544752130,4pltrace0-21swapper/209:41:262
1450052130,4pltrace0-21swapper/511:47:585
1309552130,0pltrace0-21swapper/012:19:140
1268852130,5pltrace0-21swapper/408:07:124
1093299139,3cyclictest0-21swapper/611:52:276
1093299135,6cyclictest11076-21if_eth010:40:186
1093299135,6cyclictest11076-21if_eth010:40:186
1093299130,9cyclictest1450052pltrace11:48:106
1089699130,12cyclictest0-21swapper/110:54:261
1089099131,2cyclictest0-21swapper/010:30:010
1089099131,11cyclictest0-21swapper/011:13:050
1089099131,11cyclictest0-21swapper/009:35:380
1089099131,10cyclictest0-21swapper/007:20:010
1041952130,4pltrace0-21swapper/207:10:032
866852120,0pltrace0-21swapper/010:10:050
567852120,0pltrace0-21swapper/011:07:500
43252120,5pltrace0-21swapper/511:03:495
422952120,4pltrace0-21swapper/707:54:117
3237052120,5pltrace0-21swapper/008:45:000
3100252120,0pltrace0-21swapper/707:43:217
2999152120,3pltrace0-21swapper/712:05:187
2955552120,3pltrace31344-21iostat08:40:184
278352120,0pltrace0-21swapper/308:45:453
2630152120,4pltrace27941-21sh11:28:344
2258852120,6pltrace0-21swapper/010:55:010
2256952120,4pltrace0-21swapper/007:29:310
198252120,9pltrace1091499cyclictest10:30:423
198252120,0pltrace0-21swapper/010:35:040
1831352120,0pltrace0-21swapper/008:18:470
1598552120,0pltrace0-21swapper/311:17:013
1450052120,0pltrace0-21swapper/211:48:172
1309552120,0pltrace0-21swapper/712:19:547
10939991210,1cyclictest0-21swapper/711:59:167
1093999120,11cyclictest0-21swapper/710:18:047
1093299125,6cyclictest1141-21runrttasks08:20:146
1093299123,8cyclictest3244352pltrace09:26:256
10932991210,1cyclictest0-21swapper/610:21:186
1093299120,8cyclictest0-21swapper/608:59:106
1093299120,11cyclictest0-21swapper/612:15:376
1093299120,11cyclictest0-21swapper/610:05:166
1093299120,1cyclictest0-21swapper/609:12:506
10919991210,1cyclictest0-21swapper/411:58:364
10919991210,1cyclictest0-21swapper/411:21:014
1091999120,11cyclictest0-21swapper/411:35:064
1091999120,11cyclictest0-21swapper/410:30:304
1091999120,11cyclictest0-21swapper/408:39:474
1091999120,11cyclictest0-21swapper/407:40:014
10914991210,1cyclictest0-21swapper/312:29:563
10914991210,1cyclictest0-21swapper/312:15:503
10914991210,1cyclictest0-21swapper/311:47:213
10914991210,1cyclictest0-21swapper/310:57:483
10914991210,1cyclictest0-21swapper/310:52:083
10914991210,1cyclictest0-21swapper/310:25:163
10914991210,1cyclictest0-21swapper/310:09:443
1091499120,11cyclictest0-21swapper/312:37:533
1091499120,11cyclictest0-21swapper/312:00:483
1091499120,11cyclictest0-21swapper/311:42:493
1091499120,11cyclictest0-21swapper/311:32:463
1091499120,11cyclictest0-21swapper/309:56:073
1091499120,11cyclictest0-21swapper/309:19:523
1091499120,1cyclictest0-21swapper/310:17:203
1091499120,1cyclictest0-21swapper/309:29:153
10904991210,1cyclictest0-21swapper/212:31:082
10904991210,1cyclictest0-21swapper/212:10:552
10904991210,1cyclictest0-21swapper/211:08:432
10904991210,1cyclictest0-21swapper/210:59:592
10904991210,1cyclictest0-21swapper/209:47:072
1090499121,1cyclictest0-21swapper/211:29:382
1090499120,1cyclictest0-21swapper/212:26:462
1090499120,1cyclictest0-21swapper/211:43:132
1090499120,1cyclictest0-21swapper/210:27:362
10896991210,1cyclictest0-21swapper/112:37:421
10896991210,1cyclictest0-21swapper/112:16:311
1089699120,11cyclictest0-21swapper/112:27:591
1089699120,11cyclictest0-21swapper/111:23:331
1089699120,11cyclictest0-21swapper/111:08:391
1089699120,11cyclictest0-21swapper/109:37:021
1089699120,1cyclictest0-21swapper/109:41:241
1089099121,10cyclictest0-21swapper/007:25:000
1089099120,11cyclictest0-21swapper/011:42:290
43252110,4pltrace0-21swapper/711:02:217
422952110,0pltrace0-21swapper/307:53:573
278352110,3pltrace0-21swapper/708:45:387
1093999119,1cyclictest0-21swapper/711:42:077
1093999119,1cyclictest0-21swapper/711:24:087
1093999119,1cyclictest0-21swapper/709:49:557
1093999110,10cyclictest0-21swapper/711:11:017
1093999110,10cyclictest0-21swapper/710:39:077
1093999110,10cyclictest0-21swapper/710:07:047
1093999110,10cyclictest0-21swapper/709:23:017
1093999110,10cyclictest0-21swapper/707:15:177
1093999110,1cyclictest0-21swapper/711:28:307
1093999110,1cyclictest0-21swapper/710:25:137
1093299119,1cyclictest0-21swapper/610:17:056
1093299115,5cyclictest7127-21ntp_states12:10:196
1093299115,5cyclictest27176-21cut12:30:216
1093299114,6cyclictest27448-21cut07:35:186
1093299114,6cyclictest13300-21pltrace-run07:10:366
1093299114,6cyclictest1141-21runrttasks11:31:206
1093299114,5cyclictest13031-21sh11:14:336
1093299113,6cyclictest23466-21gltestperf11:55:186
1093299113,6cyclictest12399-21cat08:05:246
1093299110,7cyclictest982152pltrace09:40:056
1093299110,10cyclictest639-21expr10:30:216
1093299110,10cyclictest0-21swapper/611:06:206
1093299110,10cyclictest0-21swapper/609:47:196
1093299110,10cyclictest0-21swapper/609:18:486
1093299110,10cyclictest0-21swapper/608:33:296
1093299110,10cyclictest0-21swapper/608:17:366
1093299110,10cyclictest0-21swapper/607:24:406
1093299110,1cyclictest0-21swapper/610:45:556
1093299110,1cyclictest0-21swapper/610:45:556
1092699119,1cyclictest0-21swapper/512:28:495
1092699119,1cyclictest0-21swapper/511:22:255
1092699118,2cyclictest0-21swapper/511:18:525
1092699110,1cyclictest0-21swapper/510:59:165
1092699110,1cyclictest0-21swapper/510:39:225
1092699110,1cyclictest0-21swapper/509:47:335
1091999119,1cyclictest0-21swapper/410:58:214
1091999119,1cyclictest0-21swapper/410:47:504
1091999119,1cyclictest0-21swapper/410:47:504
1091999119,1cyclictest0-21swapper/409:22:124
1091999119,1cyclictest0-21swapper/408:54:394
1091999119,1cyclictest0-21swapper/407:58:394
1091999110,1cyclictest0-21swapper/409:47:164
1091499119,1cyclictest0-21swapper/312:06:083
1091499119,1cyclictest0-21swapper/311:29:133
1091499119,1cyclictest0-21swapper/311:08:203
1091499119,1cyclictest0-21swapper/309:11:163
1091499110,7cyclictest43252pltrace11:02:273
1091499110,10cyclictest0-21swapper/312:23:463
1091499110,1cyclictest0-21swapper/311:24:033
1091499110,1cyclictest0-21swapper/309:35:243
1090499119,1cyclictest0-21swapper/209:32:242
1090499110,10cyclictest0-21swapper/210:34:052
1090499110,10cyclictest0-21swapper/210:16:562
1090499110,1cyclictest0-21swapper/210:12:432
1089699119,1cyclictest0-21swapper/112:00:281
1089699119,1cyclictest0-21swapper/111:50:351
1089699119,1cyclictest0-21swapper/111:02:301
1089699119,1cyclictest0-21swapper/110:55:231
1089699119,1cyclictest0-21swapper/110:46:301
1089699119,1cyclictest0-21swapper/110:46:301
1089699119,1cyclictest0-21swapper/110:42:261
1089699119,1cyclictest0-21swapper/110:42:261
1089699119,1cyclictest0-21swapper/110:15:441
1089699119,1cyclictest0-21swapper/110:12:201
1089699119,1cyclictest0-21swapper/109:17:061
1089699119,1cyclictest0-21swapper/107:35:361
1089699113,7cyclictest19810-21chrt07:21:121
1089699112,7cyclictest28863-21tr09:55:181
1089699110,10cyclictest0-21swapper/111:58:411
1089699110,10cyclictest0-21swapper/111:48:491
1089699110,10cyclictest0-21swapper/109:30:531
1089699110,1cyclictest0-21swapper/109:25:421
1089099111,1cyclictest0-21swapper/008:35:180
1089099111,1cyclictest0-21swapper/008:00:000
1089099110,2cyclictest28593-21cat07:40:010
1089099110,10cyclictest0-21swapper/012:25:130
1089099110,10cyclictest0-21swapper/012:10:170
1089099110,10cyclictest0-21swapper/011:35:420
1089099110,10cyclictest0-21swapper/010:46:290
1089099110,10cyclictest0-21swapper/010:46:280
1089099110,10cyclictest0-21swapper/009:24:430
1089099110,10cyclictest0-21swapper/008:00:270
1089099110,1cyclictest0-21swapper/011:56:530
1089099110,1cyclictest0-21swapper/010:30:060
1041952110,1pltrace0-21swapper/507:10:175
840652100,0pltrace0-21swapper/008:58:310
559852100,7pltrace0-21swapper/008:54:470
3100252100,0pltrace0-21swapper/207:43:332
2674652100,0pltrace0-21swapper/208:32:372
2256952100,2pltrace23668-21cstates07:30:174
1831352100,0pltrace0-21swapper/008:20:120
1093999108,1cyclictest0-21swapper/712:32:567
1093999108,1cyclictest0-21swapper/712:11:457
1093999108,1cyclictest0-21swapper/711:33:517
1093999108,1cyclictest0-21swapper/709:37:467
1093999108,1cyclictest0-21swapper/708:40:567
1093299108,1cyclictest0-21swapper/608:46:556
1093299106,3cyclictest0-21swapper/608:51:566
1093299104,5cyclictest28150-21fschecks_time08:35:166
1093299104,5cyclictest21491-21cstates12:25:146
1093299104,5cyclictest14381-21diskmemload12:04:486
1093299103,6cyclictest9284-21processes08:00:266
1093299103,6cyclictest2417-21cpuspeed_turbos07:50:136
1093299103,6cyclictest24140-21fschecks_time07:30:196
1093299103,6cyclictest1976-21aten_r4power_po11:35:156
1093299103,6cyclictest1141-21runrttasks08:10:086
1093299103,5cyclictest9048-21timerwakeupswit11:40:256
1093299103,5cyclictest0-21swapper/607:25:176
1093299102,6cyclictest29139-21expr07:40:186
1093299102,6cyclictest24601-21grep11:25:136
1093299101,8cyclictest15121-21grep07:15:136
1093299100,5cyclictest704152pltrace07:55:356
1092699108,1cyclictest0-21swapper/511:51:455
1092699108,1cyclictest0-21swapper/510:19:365
1092699101,8cyclictest0-21swapper/507:56:105
1092699101,3cyclictest21210-21load09:15:185
1092699100,9cyclictest0-21swapper/512:10:505
1092699100,9cyclictest0-21swapper/512:01:465
1092699100,9cyclictest0-21swapper/511:42:315
1092699100,9cyclictest0-21swapper/511:38:475
1092699100,9cyclictest0-21swapper/510:26:055
1092699100,9cyclictest0-21swapper/509:22:475
1092699100,1cyclictest0-21swapper/511:25:595
1092699100,1cyclictest0-21swapper/510:33:425
1091999108,1cyclictest0-21swapper/411:07:074
1091999108,1cyclictest0-21swapper/410:26:474
1091999108,1cyclictest0-21swapper/410:02:404
1091999108,1cyclictest0-21swapper/409:53:164
1091999108,1cyclictest0-21swapper/409:18:484
1091999100,9cyclictest0-21swapper/408:13:224
1091999100,1cyclictest0-21swapper/407:20:464
1091499108,1cyclictest0-21swapper/310:45:113
1091499108,1cyclictest0-21swapper/310:45:103
1091499108,1cyclictest0-21swapper/309:33:373
1091499101,1cyclictest0-21swapper/308:58:003
1091499100,9cyclictest0-21swapper/308:38:453
1091499100,9cyclictest0-21swapper/308:04:353
1091499100,9cyclictest0-21swapper/307:44:393
1090499108,1cyclictest0-21swapper/211:22:402
1090499108,1cyclictest0-21swapper/210:48:402
1090499108,1cyclictest0-21swapper/210:48:402
1090499100,9cyclictest0-21swapper/210:36:502
1089699108,1cyclictest0-21swapper/112:14:441
1089699108,1cyclictest0-21swapper/111:13:021
1089699108,1cyclictest0-21swapper/110:32:531
1089699108,1cyclictest0-21swapper/109:46:341
1089699108,1cyclictest0-21swapper/108:45:141
1089699101,8cyclictest0-21swapper/112:23:371
1089699101,7cyclictest22377-21df_inode10:20:161
1089699101,3cyclictest24015-21df09:50:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional