You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-25 - 14:41

x86 Intel Core i7-2600K @3400 MHz, Linux 5.9.0-rc2-rt1 (Profile)

Latency plot of system in rack #4, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h200 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Sat Mar 25, 2023 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22529521170,113pltrace0-21swapper/207:05:402
647152880,4pltrace2566399cyclictest08:28:427
2533652880,84pltrace0-21swapper/107:09:131
2103352800,1pltrace0-21swapper/510:38:425
2252952790,70pltrace0-21swapper/607:08:096
2538927862,12sleep70-21swapper/707:09:257
66652770,0pltrace0-21swapper/010:18:420
557952770,4pltrace5584-21pltrace-run09:18:425
2521327660,12sleep50-21swapper/507:07:155
1279852760,0pltrace0-21swapper/709:58:427
2541127458,12sleep00-21swapper/007:09:430
2513727348,12sleep30-21swapper/307:06:093
1354052700,0pltrace13545-21pltrace-run07:43:424
78952660,0pltrace0-21swapper/308:18:423
78952660,0pltrace0-21swapper/308:18:423
305152660,0sleep30-21swapper/310:47:283
318132620,0sleep50-21swapper/512:25:125
2525126045,11sleep40-21swapper/407:07:464
2565699230,22cyclictest0-21swapper/608:40:006
60352210,1sleep30-21swapper/308:25:233
232892210,0sleep10-21swapper/112:15:231
102082210,1sleep60-21swapper/611:00:136
3233852200,1pltrace19-21ksoftirqd/111:58:311
1024752200,1pltrace19-21ksoftirqd/109:26:511
3233852190,1pltrace0-21swapper/211:58:312
313452190,0pltrace0-21swapper/512:32:415
2564699190,2cyclictest0-21swapper/410:49:424
137862190,3sleep225-21ksoftirqd/207:45:002
791452180,1pltrace43-21ksoftirqd/507:35:095
2805652180,1pltrace19-21ksoftirqd/110:15:501
25656991814,1cyclictest0-21swapper/609:04:296
2564699181,16cyclictest0-21swapper/409:30:014
2564699181,15cyclictest0-21swapper/412:40:004
25639991812,3cyclictest0-21swapper/311:48:283
2415552180,1pltrace0-21swapper/211:15:122
1975652180,0pltrace0-21swapper/109:37:581
190952180,2pltrace111rcu_preempt11:25:461
1521652180,1pltrace19-21ksoftirqd/112:09:581
1402452180,1pltrace43-21ksoftirqd/511:04:115
113562180,0sleep50-21swapper/509:25:195
1088352180,1pltrace43-21ksoftirqd/510:29:035
77402170,0sleep60-21swapper/608:30:156
322552170,0sleep70-21swapper/708:15:187
322552170,0sleep70-21swapper/708:15:187
2805652170,1pltrace31570-21ssh10:15:493
2805652170,1pltrace31569-21ssh10:15:496
2566399170,7cyclictest29947-21sh11:20:017
2565699171,8cyclictest0-21swapper/612:25:296
25646991715,1cyclictest0-21swapper/411:59:064
25646991711,3cyclictest0-21swapper/412:20:354
2564699171,2cyclictest0-21swapper/408:35:004
25633991712,2cyclictest0-21swapper/207:33:232
225742170,0sleep00-21swapper/010:08:000
190952170,0pltrace0-21swapper/011:25:340
150112170,0sleep2241rcuc/209:28:592
1088352170,1pltrace11046-21ssh10:29:011
3119452160,2pltrace49-21ksoftirqd/610:52:526
3119452160,2pltrace111rcu_preempt10:52:565
3119452160,1pltrace19-21ksoftirqd/110:52:531
264652160,2pltrace111rcu_preempt09:50:331
25663991612,2cyclictest0-21swapper/710:10:017
25663991611,3cyclictest0-21swapper/712:28:447
25663991611,3cyclictest0-21swapper/710:59:577
25663991611,3cyclictest0-21swapper/710:02:167
25663991611,3cyclictest0-21swapper/707:38:217
25663991611,2cyclictest0-21swapper/708:37:047
25656991611,3cyclictest0-21swapper/610:46:116
2565699160,8cyclictest0-21swapper/607:30:006
2565099160,4cyclictest0-21swapper/508:43:145
2564699163,12cyclictest26382-21iostat_ios08:05:184
2564699163,12cyclictest26382-21iostat_ios08:05:174
25646991612,3cyclictest21517-21ssh10:06:014
25646991611,3cyclictest0-21swapper/408:48:354
25646991611,3cyclictest0-21swapper/408:20:044
25646991611,2cyclictest0-21swapper/411:23:434
25646991611,2cyclictest0-21swapper/407:21:484
2564699161,14cyclictest0-21swapper/409:05:014
2564699160,14cyclictest0-21swapper/408:00:004
2564699160,1cyclictest0-21swapper/409:21:584
25639991611,3cyclictest0-21swapper/310:06:223
25639991611,2cyclictest0-21swapper/312:37:273
25639991611,2cyclictest0-21swapper/310:55:573
25633991611,2cyclictest0-21swapper/211:53:192
25633991611,2cyclictest0-21swapper/210:53:272
2563399161,12cyclictest0-21swapper/210:40:002
2563399160,9cyclictest0-21swapper/212:32:332
2563399160,15cyclictest0-21swapper/208:13:042
2563399160,15cyclictest0-21swapper/208:13:042
2562699166,8cyclictest0-21swapper/107:43:151
25621991611,2cyclictest0-21swapper/011:44:020
25621991610,3cyclictest0-21swapper/007:25:380
2562199161,8cyclictest0-21swapper/009:31:480
2562199161,3cyclictest0-21swapper/007:57:510
2103352160,2pltrace111rcu_preempt10:41:135
2103352160,1pltrace24729-21ssh10:41:092
1207552160,1pltrace19-21ksoftirqd/111:36:301
1088352160,1pltrace10886-21grep10:28:420
2805652150,1pltrace43-21ksoftirqd/510:15:505
25663991513,1cyclictest0-21swapper/711:20:167
2566399150,14cyclictest0-21swapper/710:25:267
25656991511,1cyclictest0-21swapper/607:40:006
25656991510,3cyclictest0-21swapper/611:33:076
25656991510,3cyclictest0-21swapper/609:54:076
2565699150,4cyclictest0-21swapper/610:24:026
2565099151,7cyclictest0-21swapper/507:12:385
2565099150,8cyclictest0-21swapper/509:53:405
2565099150,3cyclictest0-21swapper/509:13:145
2564699157,6cyclictest1135-21snmpd07:18:234
25646991512,2cyclictest0-21swapper/410:38:144
25646991511,3cyclictest0-21swapper/411:54:294
2564699151,13cyclictest0-21swapper/410:42:504
2564699150,2cyclictest6167-21latency_hist09:20:014
2564699150,2cyclictest30262-21cat11:20:004
2564699150,2cyclictest0-21swapper/411:50:004
2564699150,2cyclictest0-21swapper/408:50:154
2564699150,14cyclictest0-21swapper/411:04:244
2564699150,14cyclictest0-21swapper/410:13:364
2563999150,8cyclictest0-21swapper/309:51:283
2563399152,12cyclictest0-21swapper/207:24:072
25633991510,3cyclictest0-21swapper/211:08:482
2563399150,8cyclictest0-21swapper/211:14:102
2563399150,3cyclictest0-21swapper/207:50:162
2562699157,7cyclictest0-21swapper/109:43:301
25626991513,1cyclictest0-21swapper/110:40:481
25626991513,1cyclictest0-21swapper/107:29:231
2562699151,13cyclictest0-21swapper/111:49:071
2562699150,3cyclictest0-21swapper/112:01:361
2562699150,3cyclictest0-21swapper/108:52:281
2562199159,3cyclictest0-21swapper/010:40:050
2562199159,3cyclictest0-21swapper/008:42:290
25621991510,3cyclictest0-21swapper/009:40:180
25621991510,3cyclictest0-21swapper/007:30:140
2562199151,3cyclictest0-21swapper/009:07:080
2562199150,8cyclictest0-21swapper/009:13:290
2217652150,11pltrace0-21swapper/611:47:366
2197052150,1pltrace19-21ksoftirqd/107:58:481
1279852150,0pltrace0-21swapper/010:03:140
2566399149,2cyclictest0-21swapper/708:02:157
25663991412,1cyclictest0-21swapper/712:14:017
25663991412,1cyclictest0-21swapper/710:43:017
25663991412,1cyclictest0-21swapper/709:33:347
2566399140,7cyclictest0-21swapper/709:05:507
2566399140,13cyclictest0-21swapper/712:30:127
2566399140,13cyclictest0-21swapper/711:49:347
2566399140,13cyclictest0-21swapper/711:42:277
2566399140,13cyclictest0-21swapper/710:30:247
2566399140,1cyclictest0-21swapper/710:39:287
2566399140,1cyclictest0-21swapper/709:38:457
2565699148,3cyclictest0-21swapper/610:01:556
25656991412,1cyclictest0-21swapper/611:21:446
25656991412,1cyclictest0-21swapper/611:17:226
25656991412,1cyclictest0-21swapper/611:13:006
2565699141,12cyclictest0-21swapper/608:24:566
2565699140,7cyclictest0-21swapper/607:15:596
2565699140,13cyclictest0-21swapper/612:13:436
2565699140,13cyclictest0-21swapper/612:07:256
2565699140,13cyclictest0-21swapper/612:01:056
2565699140,13cyclictest0-21swapper/610:59:056
2565699140,13cyclictest0-21swapper/610:38:016
2565699140,13cyclictest0-21swapper/610:30:556
2565699140,13cyclictest0-21swapper/609:43:476
2565699140,12cyclictest0-21swapper/607:10:516
2565699140,1cyclictest0-21swapper/609:11:246
25650991412,1cyclictest0-21swapper/512:35:225
25650991412,1cyclictest0-21swapper/510:09:205
25650991412,1cyclictest0-21swapper/510:00:165
25650991411,2cyclictest0-21swapper/508:26:195
2565099140,7cyclictest0-21swapper/511:22:075
2565099140,7cyclictest0-21swapper/509:06:445
2565099140,7cyclictest0-21swapper/507:59:395
2565099140,13cyclictest0-21swapper/512:23:045
2565099140,13cyclictest0-21swapper/511:53:085
2565099140,13cyclictest0-21swapper/511:43:535
2565099140,13cyclictest0-21swapper/510:32:485
2565099140,13cyclictest0-21swapper/509:31:275
2565099140,13cyclictest0-21swapper/509:22:135
2565099140,12cyclictest0-21swapper/508:55:145
2565099140,1cyclictest0-21swapper/511:56:115
2564699149,4cyclictest0-21swapper/410:55:434
2564699149,3cyclictest0-21swapper/410:51:124
2564699148,5cyclictest1135-21snmpd09:56:284
25646991412,1cyclictest0-21swapper/412:13:354
25646991412,1cyclictest0-21swapper/410:17:024
2564699141,8cyclictest0-21swapper/407:10:114
2564699140,7cyclictest0-21swapper/411:06:334
2564699140,3cyclictest0-21swapper/410:29:114
2564699140,2cyclictest9264-21latency_hist12:35:004
2564699140,2cyclictest24184-21ssh11:13:454
2564699140,13cyclictest0-21swapper/411:42:404
2564699140,13cyclictest0-21swapper/409:11:564
2564699140,1cyclictest0-21swapper/412:29:234
2564699140,1cyclictest0-21swapper/412:18:204
2564699140,1cyclictest0-21swapper/412:03:434
2564699140,1cyclictest0-21swapper/410:02:274
2564699140,1cyclictest0-21swapper/409:32:014
2563999149,2cyclictest0-21swapper/307:37:063
25639991412,1cyclictest14783-21ssh12:07:563
25639991412,1cyclictest0-21swapper/312:21:223
25639991412,1cyclictest0-21swapper/311:02:513
25639991412,1cyclictest0-21swapper/309:26:313
2563999140,13cyclictest0-21swapper/312:01:373
2563999140,13cyclictest0-21swapper/311:58:033
2563999140,13cyclictest0-21swapper/311:32:393
2563999140,13cyclictest0-21swapper/311:19:313
2563999140,13cyclictest0-21swapper/310:26:253
2563999140,1cyclictest0-21swapper/312:29:183
2563999140,1cyclictest0-21swapper/311:28:163
25633991412,1cyclictest0-21swapper/212:08:112
25633991412,1cyclictest0-21swapper/211:46:002
25633991412,1cyclictest0-21swapper/207:35:122
25633991411,1cyclictest0-21swapper/209:51:312
2563399141,12cyclictest0-21swapper/211:43:432
2563399141,12cyclictest0-21swapper/209:23:452
2563399140,7cyclictest0-21swapper/209:00:572
2563399140,13cyclictest0-21swapper/211:03:052
2563399140,13cyclictest0-21swapper/210:25:512
2563399140,13cyclictest0-21swapper/210:17:162
2563399140,13cyclictest0-21swapper/209:16:432
2563399140,1cyclictest0-21swapper/208:15:122
2563399140,1cyclictest0-21swapper/208:15:122
25626991412,1cyclictest0-21swapper/112:32:401
25626991412,1cyclictest0-21swapper/111:01:501
25626991412,1cyclictest0-21swapper/110:37:141
25626991412,1cyclictest0-21swapper/110:33:401
25626991412,1cyclictest0-21swapper/110:01:081
25626991412,1cyclictest0-21swapper/109:23:531
25626991412,1cyclictest0-21swapper/107:31:531
25626991411,2cyclictest4441-21ssh09:16:181
2562699141,6cyclictest0-21swapper/110:12:131
2562699140,7cyclictest0-21swapper/111:50:351
2562699140,2cyclictest0-21swapper/107:38:531
2562699140,13cyclictest0-21swapper/111:10:351
2562699140,13cyclictest0-21swapper/110:21:321
25621991411,2cyclictest1135-21snmpd11:11:090
25621991411,2cyclictest0-21swapper/011:39:550
25621991411,2cyclictest0-21swapper/010:22:380
2562199141,1cyclictest0-21swapper/011:45:260
2562199141,1cyclictest0-21swapper/011:20:220
2562199140,7cyclictest0-21swapper/012:30:170
2562199140,7cyclictest0-21swapper/008:06:540
2562199140,7cyclictest0-21swapper/008:06:540
2562199140,3cyclictest0-21swapper/007:39:100
2217652140,1pltrace43-21ksoftirqd/511:47:295
1490452140,5pltrace0-21swapper/508:48:275
3233852130,10pltrace0-21swapper/011:53:560
25663991311,1cyclictest0-21swapper/712:37:207
25663991311,1cyclictest0-21swapper/712:20:207
25663991311,1cyclictest0-21swapper/712:16:467
25663991311,1cyclictest0-21swapper/712:08:307
25663991311,1cyclictest0-21swapper/712:00:157
25663991311,1cyclictest0-21swapper/710:53:427
25663991311,1cyclictest0-21swapper/710:20:417
25663991311,1cyclictest0-21swapper/710:18:447
25663991311,1cyclictest0-21swapper/708:55:217
2566399130,12cyclictest0-21swapper/711:56:417
2566399130,12cyclictest0-21swapper/711:13:087
2566399130,12cyclictest0-21swapper/711:09:347
2566399130,12cyclictest0-21swapper/710:49:207
2566399130,1cyclictest0-21swapper/711:53:087
2566399130,1cyclictest0-21swapper/711:01:387
2566399130,1cyclictest0-21swapper/709:43:277
2566399130,1cyclictest0-21swapper/709:26:467
2565699138,3cyclictest0-21swapper/612:24:026
25656991311,1cyclictest0-21swapper/612:33:096
25656991311,1cyclictest0-21swapper/611:57:316
25656991311,1cyclictest0-21swapper/611:44:246
25656991311,1cyclictest0-21swapper/611:08:376
2565699130,12cyclictest0-21swapper/609:46:336
2565699130,12cyclictest0-21swapper/609:38:456
2565699130,1cyclictest0-21swapper/612:36:426
2565699130,1cyclictest0-21swapper/610:41:356
2565699130,1cyclictest0-21swapper/610:25:216
2565699130,1cyclictest0-21swapper/610:10:116
25650991311,1cyclictest0-21swapper/512:00:335
25650991311,1cyclictest0-21swapper/511:31:355
25650991311,1cyclictest0-21swapper/511:09:435
25650991311,1cyclictest0-21swapper/510:55:485
25650991311,1cyclictest0-21swapper/510:14:505
25650991311,1cyclictest0-21swapper/508:20:435
25650991311,1cyclictest0-21swapper/508:10:255
25650991311,1cyclictest0-21swapper/508:10:255
2565099130,12cyclictest0-21swapper/512:12:235
2565099130,12cyclictest0-21swapper/511:26:465
2565099130,12cyclictest0-21swapper/509:56:495
2565099130,12cyclictest0-21swapper/509:45:455
2565099130,12cyclictest0-21swapper/508:00:365
2565099130,12cyclictest0-21swapper/507:25:055
2564699135,7cyclictest1135-21snmpd10:22:414
2564699135,6cyclictest1186-21runrttasks08:39:484
2564699134,3cyclictest17827-21idleruntime07:50:174
25646991311,1cyclictest0-21swapper/412:07:174
25646991311,1cyclictest0-21swapper/411:33:554
25646991311,1cyclictest0-21swapper/409:50:394
25646991311,1cyclictest0-21swapper/409:39:374
2564699131,2cyclictest0-21swapper/408:10:134
2564699131,2cyclictest0-21swapper/408:10:134
2564699131,11cyclictest0-21swapper/408:16:394
2564699131,11cyclictest0-21swapper/408:16:384
2564699130,12cyclictest0-21swapper/408:03:424
2564699130,12cyclictest0-21swapper/407:35:004
2564699130,11cyclictest17816-21latency_hist11:40:004
2564699130,1cyclictest0-21swapper/408:40:224
2563999138,3cyclictest0-21swapper/307:51:333
25639991311,1cyclictest0-21swapper/312:33:413
25639991311,1cyclictest0-21swapper/312:14:153
25639991311,1cyclictest0-21swapper/308:00:133
2563999130,12cyclictest0-21swapper/311:23:543
2563999130,12cyclictest0-21swapper/310:38:143
2563999130,12cyclictest0-21swapper/308:43:503
2563999130,1cyclictest0-21swapper/312:17:003
2563999130,1cyclictest0-21swapper/311:37:013
2563999130,1cyclictest0-21swapper/310:34:413
2563999130,1cyclictest0-21swapper/309:17:173
2563399135,7cyclictest0-21swapper/212:13:062
25633991311,1cyclictest0-21swapper/212:25:592
25633991311,1cyclictest0-21swapper/210:58:432
25633991311,1cyclictest0-21swapper/210:30:332
25633991311,1cyclictest0-21swapper/210:23:552
25633991310,1cyclictest0-21swapper/212:01:062
25633991310,1cyclictest0-21swapper/210:13:012
2563399130,12cyclictest0-21swapper/211:28:312
2563399130,12cyclictest0-21swapper/211:24:082
2563399130,12cyclictest0-21swapper/209:31:562
2563399130,1cyclictest0-21swapper/211:37:152
2563399130,1cyclictest0-21swapper/210:00:452
2563399130,1cyclictest0-21swapper/209:41:292
25626991311,1cyclictest0-21swapper/112:24:441
25626991311,1cyclictest0-21swapper/112:10:291
25626991311,1cyclictest0-21swapper/111:41:111
25626991311,1cyclictest0-21swapper/111:32:261
25626991311,1cyclictest0-21swapper/111:19:201
25626991311,1cyclictest0-21swapper/108:42:571
2562699130,12cyclictest0-21swapper/112:36:141
2562699130,12cyclictest0-21swapper/109:31:301
2562699130,12cyclictest0-21swapper/107:22:271
2562699130,1cyclictest0-21swapper/112:28:191
2562199138,3cyclictest0-21swapper/007:41:420
2562199132,9cyclictest0-21swapper/008:39:160
25621991311,1cyclictest0-21swapper/011:01:040
2562199130,12cyclictest0-21swapper/007:18:260
2562199130,1cyclictest0-21swapper/010:45:320
928652120,0pltrace0-21swapper/108:35:291
2566399127,3cyclictest0-21swapper/708:33:117
25663991210,1cyclictest0-21swapper/711:38:537
25663991210,1cyclictest0-21swapper/707:25:597
2566399121,10cyclictest0-21swapper/708:42:147
2566399120,11cyclictest0-21swapper/707:59:497
2566399120,1cyclictest0-21swapper/711:30:577
2565699129,2cyclictest1264-21lldpd08:05:256
2565699129,2cyclictest1264-21lldpd08:05:256
25656991210,1cyclictest0-21swapper/610:07:256
25656991210,1cyclictest0-21swapper/608:02:466
25656991210,1cyclictest0-21swapper/607:47:426
2565699120,11cyclictest0-21swapper/607:40:346
2565699120,1cyclictest0-21swapper/608:58:166
2565699120,1cyclictest0-21swapper/607:30:326
25650991210,1cyclictest0-21swapper/512:09:375
25650991210,1cyclictest0-21swapper/511:18:285
25650991210,1cyclictest0-21swapper/511:14:055
25650991210,1cyclictest0-21swapper/510:23:255
25650991210,1cyclictest0-21swapper/507:47:405
25650991210,1cyclictest0-21swapper/507:33:255
2565099120,5cyclictest0-21swapper/509:37:005
2565099120,11cyclictest0-21swapper/508:35:065
2564699126,5cyclictest1135-21snmpd07:39:484
25646991210,1cyclictest0-21swapper/409:42:224
2564699121,10cyclictest0-21swapper/409:47:324
2564699121,10cyclictest0-21swapper/407:25:434
2564699120,11cyclictest0-21swapper/409:07:344
2564699120,11cyclictest0-21swapper/408:28:334
2564699120,11cyclictest0-21swapper/407:45:544
2563999129,1cyclictest0-21swapper/310:43:343
2563999128,3cyclictest1135-21snmpd09:05:473
25639991210,1cyclictest0-21swapper/310:24:293
25639991210,1cyclictest0-21swapper/310:01:193
25639991210,1cyclictest0-21swapper/309:34:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional