You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-14 - 22:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Wed May 14, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24920521200,112pltrace0-21swapper/607:05:166
24920521060,63pltrace0-21swapper/707:05:117
27704521050,96pltrace0-21swapper/107:09:361
24920521040,98pltrace0-21swapper/207:05:182
2492052900,81pltrace0-21swapper/507:05:205
2492052820,73pltrace0-21swapper/307:05:133
3073052780,0pltrace0-21swapper/509:08:325
2775327241,18sleep40-21swapper/407:09:094
2750225945,10sleep00-21swapper/007:05:540
324212320,1sleep30-21swapper/311:10:163
2512852320,6pltrace0-21swapper/709:00:147
1506952320,3pltrace0-21swapper/709:40:197
1063452320,1pltrace0-21swapper/112:28:341
1014452320,0pltrace0-21swapper/707:35:217
1341952260,1pltrace13424-21pltrace-run12:33:334
1334552230,3pltrace0-21swapper/710:34:117
2770452200,14pltrace0-21swapper/007:10:130
563352190,2pltrace0-21swapper/708:25:527
27995991813,3cyclictest0-21swapper/712:21:517
2798399181,2cyclictest0-21swapper/511:18:475
2770452180,13pltrace29701-21ntp_states07:10:195
3009052170,4pltrace0-21swapper/711:05:467
2799599171,5cyclictest0-21swapper/712:12:187
2799599171,15cyclictest0-21swapper/711:14:257
27991991711,5cyclictest0-21swapper/607:42:196
2799199171,8cyclictest0-21swapper/607:56:016
2799199170,15cyclictest0-21swapper/610:02:446
27983991713,3cyclictest0-21swapper/512:39:555
27963991714,2cyclictest0-21swapper/010:42:480
27963991712,3cyclictest0-21swapper/012:37:430
27963991711,5cyclictest0-21swapper/008:14:230
841452160,0pltrace0-21swapper/708:30:177
3056052160,8pltrace54-21ksoftirqd/407:15:024
3056052160,11pltrace0-21swapper/707:15:147
2799599161,4cyclictest0-21swapper/712:29:397
2799599161,4cyclictest0-21swapper/707:29:277
2799599161,14cyclictest0-21swapper/710:01:037
2799199161,8cyclictest0-21swapper/611:26:436
2799199161,14cyclictest0-21swapper/607:17:086
2799199160,8cyclictest0-21swapper/609:44:296
2799199160,8cyclictest0-21swapper/607:23:096
2798399161,7cyclictest0-21swapper/508:33:415
2798399161,4cyclictest0-21swapper/508:55:425
2798199161,4cyclictest0-21swapper/408:04:394
2798199160,15cyclictest0-21swapper/412:35:104
2798199160,15cyclictest0-21swapper/410:05:214
2797799161,4cyclictest0-21swapper/309:53:533
2797799161,4cyclictest0-21swapper/309:53:533
2797799160,4cyclictest0-21swapper/310:52:443
2797799160,15cyclictest0-21swapper/307:55:323
27972991611,3cyclictest0-21swapper/211:23:562
27972991611,2cyclictest0-21swapper/211:16:592
2797299161,8cyclictest0-21swapper/207:57:202
2797299161,8cyclictest0-21swapper/207:22:042
2797299161,7cyclictest0-21swapper/208:49:402
2797299161,14cyclictest0-21swapper/210:01:352
2797299160,4cyclictest0-21swapper/210:57:142
27968991613,2cyclictest7246-21timerandwakeup10:20:221
27968991611,3cyclictest0-21swapper/109:15:561
2796899161,8cyclictest0-21swapper/109:13:461
2796899161,13cyclictest0-21swapper/107:27:191
2796899160,8cyclictest0-21swapper/107:12:311
2796899160,15cyclictest0-21swapper/108:57:461
27963991611,4cyclictest0-21swapper/012:29:090
27963991611,4cyclictest0-21swapper/010:37:000
27963991610,3cyclictest0-21swapper/008:35:030
2793852160,10pltrace54-21ksoftirqd/409:04:154
2770452160,10pltrace29416-21latency07:10:167
499152150,1pltrace0-21swapper/710:20:247
3056052150,10pltrace477-21if_err_eth007:15:175
2799599151,8cyclictest0-21swapper/709:30:017
2799599150,14cyclictest0-21swapper/709:34:047
2799199159,5cyclictest0-21swapper/610:40:556
2799199150,8cyclictest0-21swapper/609:11:106
2799199150,7cyclictest0-21swapper/611:32:046
2799199150,4cyclictest0-21swapper/608:50:376
2799199150,14cyclictest0-21swapper/611:35:286
2799199150,14cyclictest0-21swapper/608:16:086
2798399159,5cyclictest0-21swapper/511:26:195
2798399159,5cyclictest0-21swapper/508:41:365
2798399150,8cyclictest0-21swapper/509:18:315
2798399150,7cyclictest0-21swapper/508:16:205
2798399150,14cyclictest0-21swapper/510:20:185
2798399150,14cyclictest0-21swapper/508:20:135
2798399150,14cyclictest0-21swapper/507:40:345
27981991510,3cyclictest0-21swapper/412:13:114
2798199151,7cyclictest0-21swapper/410:15:164
2798199150,4cyclictest0-21swapper/407:11:314
2798199150,14cyclictest0-21swapper/411:34:054
2798199150,14cyclictest0-21swapper/410:52:134
2798199150,14cyclictest0-21swapper/410:26:274
2798199150,14cyclictest0-21swapper/408:40:004
2797799150,8cyclictest0-21swapper/307:43:503
2797799150,7cyclictest0-21swapper/307:30:483
2797799150,4cyclictest0-21swapper/309:10:093
2797799150,14cyclictest0-21swapper/311:54:143
2797799150,14cyclictest0-21swapper/311:33:473
2797799150,14cyclictest0-21swapper/310:29:163
2797799150,14cyclictest0-21swapper/309:37:493
2797799150,14cyclictest0-21swapper/309:31:003
2797299159,5cyclictest0-21swapper/209:14:372
2797299159,5cyclictest0-21swapper/209:07:122
2797299158,4cyclictest30851-21missed_timers10:05:172
27972991510,3cyclictest0-21swapper/210:18:442
27972991510,3cyclictest0-21swapper/208:58:422
2797299150,8cyclictest0-21swapper/212:22:352
2797299150,8cyclictest0-21swapper/211:00:162
2797299150,7cyclictest0-21swapper/211:25:592
2797299150,14cyclictest0-21swapper/208:20:212
27968991513,1cyclictest9563-21fschecks_count08:30:151
27968991513,1cyclictest15601-21timerandwakeup10:35:231
27968991512,2cyclictest8211-21cstates07:30:161
27968991512,2cyclictest4783-21sshd10:15:511
27968991512,2cyclictest30773-21latency12:05:171
27968991512,2cyclictest0-21swapper/108:35:151
27968991510,3cyclictest0-21swapper/108:54:191
27968991510,3cyclictest0-21swapper/108:11:091
2796899150,8cyclictest0-21swapper/111:47:501
2796899150,4cyclictest0-21swapper/112:04:021
2796899150,14cyclictest0-21swapper/112:33:421
2796899150,14cyclictest0-21swapper/111:22:201
27963991510,3cyclictest0-21swapper/007:57:010
2770452150,3pltrace30290-21awk07:10:216
2128652150,1pltrace0-21swapper/107:53:331
3056052140,1pltrace31400-21cat07:15:010
27995991410,3cyclictest1065-17auditd11:50:007
27995991410,2cyclictest0-21swapper/712:18:157
27995991410,2cyclictest0-21swapper/708:40:017
2799599141,12cyclictest0-21swapper/711:40:047
2799599140,7cyclictest0-21swapper/712:00:487
2799599140,7cyclictest0-21swapper/710:45:377
2799599140,13cyclictest0-21swapper/710:50:097
2799599140,13cyclictest0-21swapper/708:15:007
2799599140,1cyclictest0-21swapper/709:05:137
27991991411,2cyclictest0-21swapper/611:51:596
2799199140,7cyclictest0-21swapper/611:45:586
2799199140,7cyclictest0-21swapper/610:55:426
2799199140,7cyclictest0-21swapper/608:02:526
2799199140,13cyclictest0-21swapper/612:30:226
2799199140,13cyclictest0-21swapper/608:20:246
2798399149,3cyclictest0-21swapper/507:31:315
2798399148,5cyclictest0-21swapper/507:47:105
27983991410,3cyclictest1132-21snmpd10:18:575
2798399140,7cyclictest0-21swapper/510:06:425
2798399140,13cyclictest0-21swapper/509:40:265
27981991410,2cyclictest0-21swapper/407:24:114
2798199141,12cyclictest0-21swapper/409:56:544
2798199140,13cyclictest0-21swapper/411:20:134
2798199140,13cyclictest0-21swapper/408:20:164
2798199140,1cyclictest0-21swapper/411:35:174
2797799149,3cyclictest0-21swapper/311:58:053
2797799149,3cyclictest0-21swapper/309:01:483
2797799148,5cyclictest0-21swapper/307:39:293
27977991412,1cyclictest0-21swapper/309:25:243
2797799140,8cyclictest0-21swapper/307:46:503
2797799140,7cyclictest0-21swapper/310:16:273
2797799140,3cyclictest0-21swapper/311:03:173
2797299149,2cyclictest0-21swapper/209:22:312
2797299148,5cyclictest0-21swapper/208:26:452
2797299146,4cyclictest0-21swapper/209:40:232
2797299143,4cyclictest38-21ksoftirqd/210:12:592
2797299140,6cyclictest0-21swapper/208:35:242
27968991413,1cyclictest0-21swapper/109:20:191
27968991412,1cyclictest25703-21latency08:00:171
27968991412,1cyclictest1166-21runrttasks11:58:521
27968991411,2cyclictest1584-21if_err_eth008:15:171
2796899140,13cyclictest0-21swapper/112:37:311
2796899140,13cyclictest0-21swapper/108:20:151
27963991412,1cyclictest0-21swapper/008:15:250
1785552140,4pltrace54-21ksoftirqd/409:46:004
1785552140,4pltrace54-21ksoftirqd/409:46:004
1676852140,9pltrace78-21ksoftirqd/708:45:247
2799599138,3cyclictest0-21swapper/711:36:347
2799599130,12cyclictest0-21swapper/711:27:527
2799199130,12cyclictest0-21swapper/609:58:246
2799199130,1cyclictest0-21swapper/611:43:336
2799199130,1cyclictest0-21swapper/608:35:226
2798399138,3cyclictest0-21swapper/510:00:185
2798399138,2cyclictest0-21swapper/509:52:075
2798399138,2cyclictest0-21swapper/509:52:075
2798399130,7cyclictest0-21swapper/511:24:505
2798399130,12cyclictest0-21swapper/509:57:485
2798399130,12cyclictest0-21swapper/509:00:395
2798199136,6cyclictest0-21swapper/411:54:584
2798199130,12cyclictest0-21swapper/409:43:554
2798199130,12cyclictest0-21swapper/407:56:084
27977991312,1cyclictest0-21swapper/309:45:283
27977991312,1cyclictest0-21swapper/309:45:283
2797799130,12cyclictest0-21swapper/309:09:453
2797799130,12cyclictest0-21swapper/307:28:013
2797799130,1cyclictest0-21swapper/311:08:523
2797299138,3cyclictest0-21swapper/207:51:172
27972991311,1cyclictest1132-21snmpd10:25:322
2797299130,12cyclictest0-21swapper/211:56:452
2796899137,5cyclictest0-21swapper/108:44:311
2796899137,3cyclictest3431-21timerandwakeup11:15:211
2796899137,3cyclictest0-21swapper/110:54:151
2796899133,8cyclictest0-21swapper/109:49:051
2796899133,8cyclictest0-21swapper/109:49:051
27968991312,1cyclictest14624-21timerwakeupswit09:35:241
27968991312,1cyclictest0-21swapper/110:05:211
27968991310,2cyclictest8351-21latency09:25:171
27968991310,2cyclictest19752-21cstates10:45:141
27968991310,1cyclictest191rcu_preempt07:41:031
2796899131,11cyclictest0-21swapper/110:26:471
2796899130,1cyclictest0-21swapper/107:20:171
2796399139,3cyclictest0-21swapper/007:40:560
2796399137,5cyclictest0-21swapper/008:43:100
27963991312,1cyclictest0-21swapper/007:26:010
2796399130,13cyclictest0-21swapper/012:18:360
2796399130,1cyclictest0-21swapper/010:30:250
2770452130,5pltrace0-21swapper/307:10:173
2770452130,5pltrace0-21swapper/207:10:132
2687952130,5pltrace0-21swapper/708:06:287
2169952130,1pltrace0-21swapper/610:50:146
1849652130,4pltrace0-21swapper/607:51:376
1785552130,1pltrace1166-21runrttasks09:46:170
1785552130,1pltrace1166-21runrttasks09:46:170
1676852130,5pltrace0-21swapper/608:45:466
1620152130,0pltrace0-21swapper/712:39:247
1334552130,2pltrace15965-21taskset10:36:285
1014452130,1pltrace0-21swapper/207:35:392
784852120,0pltrace0-21swapper/312:27:193
671752120,5pltrace0-21swapper/009:26:320
563352120,5pltrace0-21swapper/308:26:053
2910352120,6pltrace0-21swapper/712:07:357
2799599127,3cyclictest0-21swapper/707:50:407
2799599122,9cyclictest9985-21sed10:25:347
2799199127,4cyclictest14588-21grep12:35:116
2799199127,3cyclictest0-21swapper/612:02:146
2799199120,11cyclictest0-21swapper/610:08:156
2798399127,2cyclictest0-21swapper/511:50:045
2798399126,5cyclictest0-21swapper/509:12:035
2798399126,5cyclictest0-21swapper/508:35:265
2798399123,8cyclictest0-21swapper/509:39:475
2798399122,8cyclictest0-21swapper/511:09:375
2798399120,11cyclictest0-21swapper/507:53:545
2798199128,3cyclictest54-21ksoftirqd/412:02:384
2798199126,5cyclictest0-21swapper/410:35:404
27981991210,1cyclictest0-21swapper/411:08:054
2798199120,5cyclictest0-21swapper/408:43:324
2797799129,2cyclictest1065-17auditd10:10:003
2797799126,5cyclictest0-21swapper/308:41:153
2797799126,5cyclictest0-21swapper/308:08:333
2797799122,3cyclictest13457-21crond10:35:013
2797799120,11cyclictest0-21swapper/311:43:573
2797799120,11cyclictest0-21swapper/310:48:573
2797299129,2cyclictest17292-21fschecks_count10:40:142
2797299129,2cyclictest15121-21sed10:35:182
2797299129,2cyclictest12652-21sensors10:30:232
2797299129,2cyclictest1132-21snmpd11:09:552
2797299126,5cyclictest0-21swapper/212:08:232
2797299126,5cyclictest0-21swapper/211:12:372
2797299126,5cyclictest0-21swapper/209:34:312
2797299126,3cyclictest38-21ksoftirqd/207:40:182
2797299126,2cyclictest38-21ksoftirqd/209:55:112
2797299126,2cyclictest3058-21cpuspeed_turbos12:15:132
2797299125,2cyclictest0-21swapper/207:26:042
27972991211,1cyclictest0-21swapper/208:05:142
2797299121,10cyclictest21840-21cut11:50:142
2797299120,11cyclictest0-21swapper/209:54:292
2797299120,11cyclictest0-21swapper/209:54:292
2796899129,2cyclictest8711-21ntpq11:25:201
2796899129,2cyclictest631-21timerandwakeup11:10:231
2796899129,2cyclictest5150-21runrttasks12:19:451
2796899129,2cyclictest27835-21aten_r4power_po11:00:121
2796899129,2cyclictest23047-21unixbench-2d09:50:211
2796899129,2cyclictest23047-21unixbench-2d09:50:211
2796899129,2cyclictest10736-21fschecks_count11:30:171
2796899126,5cyclictest0-21swapper/109:31:261
2796899126,5cyclictest0-21swapper/107:45:021
27968991211,1cyclictest0-21swapper/111:40:141
27968991211,1cyclictest0-21swapper/110:00:211
27968991211,1cyclictest0-21swapper/108:05:231
27968991210,1cyclictest21907-21fschecks_count11:50:151
2796899120,6cyclictest13073-21ls07:40:011
2796899120,11cyclictest0-21swapper/108:26:111
2796399128,2cyclictest0-21swapper/009:20:580
27963991211,1cyclictest0-21swapper/009:06:150
27963991210,1cyclictest0-21swapper/012:09:220
27963991210,1cyclictest0-21swapper/011:52:220
2796399120,12cyclictest0-21swapper/009:30:320
2796399120,11cyclictest0-21swapper/009:42:060
2796399120,1cyclictest0-21swapper/011:15:550
2621252120,9pltrace15-21ksoftirqd/010:02:200
2350152120,0pltrace0-21swapper/711:57:577
2234352120,9pltrace24192-21strings08:55:183
2071252120,5pltrace0-21swapper/011:48:350
1236452120,4pltrace14792-21unixbench_multi11:35:252
1014452120,5pltrace0-21swapper/507:35:415
671752110,1pltrace0-21swapper/609:26:306
499152110,7pltrace54-21ksoftirqd/410:22:494
400752110,6pltrace1700-21lldpd11:18:504
2799599116,3cyclictest0-21swapper/709:12:587
2799599112,8cyclictest0-21swapper/708:53:497
27995991110,1cyclictest0-21swapper/712:31:007
2799599110,11cyclictest0-21swapper/707:58:497
2799599110,11cyclictest0-21swapper/707:21:257
2799599110,1cyclictest0-21swapper/711:32:127
2799599110,1cyclictest0-21swapper/708:58:057
2799599110,1cyclictest0-21swapper/708:21:097
2799199117,3cyclictest0-21swapper/612:09:386
2799199116,3cyclictest0-21swapper/607:48:076
2799199115,5cyclictest0-21swapper/607:37:496
27991991110,1cyclictest0-21swapper/610:48:516
27991991110,1cyclictest0-21swapper/608:32:506
2799199110,11cyclictest0-21swapper/611:11:206
2799199110,1cyclictest0-21swapper/608:40:266
2798399118,2cyclictest5177-21latency_hist12:20:005
2798399117,3cyclictest1132-21snmpd10:10:515
2798399117,3cyclictest1132-21snmpd09:32:225
2798399115,3cyclictest0-21swapper/508:09:295
27983991110,1cyclictest0-21swapper/510:45:225
27983991110,1cyclictest0-21swapper/510:43:555
2798399110,11cyclictest0-21swapper/512:13:365
2798399110,10cyclictest0-21swapper/511:40:055
2798399110,10cyclictest0-21swapper/511:34:445
2798399110,1cyclictest0-21swapper/512:00:395
2798399110,1cyclictest0-21swapper/511:59:215
2798399110,1cyclictest0-21swapper/510:32:255
2798399110,1cyclictest0-21swapper/507:20:035
2798199119,1cyclictest0-21swapper/407:49:454
2798199118,2cyclictest26398-21latency_hist12:00:004
2798199117,4cyclictest0-21swapper/411:49:404
2798199115,5cyclictest0-21swapper/412:05:474
2798199115,5cyclictest0-21swapper/411:44:194
2798199114,5cyclictest54-21ksoftirqd/408:15:244
27981991110,1cyclictest0-21swapper/410:12:424
27981991110,1cyclictest0-21swapper/408:10:084
27981991110,1cyclictest0-21swapper/407:41:094
2798199110,11cyclictest0-21swapper/412:24:484
2798199110,11cyclictest0-21swapper/409:06:284
2798199110,1cyclictest0-21swapper/409:30:264
2798199110,1cyclictest0-21swapper/409:24:464
2797799117,3cyclictest1132-21snmpd11:27:413
2797799117,2cyclictest5179-21sadc12:19:593
27977991110,1cyclictest0-21swapper/310:56:533
2797799110,11cyclictest0-21swapper/312:32:533
2797799110,11cyclictest0-21swapper/312:23:003
2797799110,11cyclictest0-21swapper/311:19:433
2797799110,11cyclictest0-21swapper/310:10:353
2797799110,1cyclictest0-21swapper/312:35:583
2797799110,1cyclictest0-21swapper/311:37:323
2797799110,1cyclictest0-21swapper/310:21:553
2797799110,1cyclictest0-21swapper/308:52:123
2797799110,1cyclictest0-21swapper/307:23:013
2797299119,1cyclictest0-21swapper/209:00:552
2797299118,2cyclictest7753-21diskstats09:25:132
2797299118,2cyclictest32756-21cstates12:10:152
2797299118,2cyclictest27137-21cstates12:00:152
2797299118,2cyclictest2521-21grep08:15:252
2797299118,2cyclictest22660-21ls10:50:122
2797299118,2cyclictest21058-21idleruntime08:50:162
2797299118,2cyclictest20827-21awk10:45:192
2797299118,2cyclictest19847-21grep09:45:232
2797299118,2cyclictest19847-21grep09:45:232
2797299118,2cyclictest17863-21sed07:45:232
2797299118,2cyclictest16313-21fschecks_count11:40:162
2797299118,2cyclictest14650-21cat08:40:132
2797299118,2cyclictest12968-21cpuspeed_turbos09:35:142
2797299118,2cyclictest11940-21sort12:30:132
2797299118,2cyclictest1166-21runrttasks09:16:282
2797299116,2cyclictest38-21ksoftirqd/208:10:162
2797299115,2cyclictest0-21swapper/212:35:092
2797299115,2cyclictest0-21swapper/207:15:172
2797299114,4cyclictest38-21ksoftirqd/211:45:132
2797299114,2cyclictest0-21swapper/210:20:142
27972991110,1cyclictest0-21swapper/208:31:282
27972991110,1cyclictest0-21swapper/207:32:222
2796899119,1cyclictest0-21swapper/109:00:181
2796899119,1cyclictest0-21swapper/108:45:191
2796899118,2cyclictest29490-21if_err_eth009:05:161
2796899118,2cyclictest21782-21aten_r4power_cu07:55:141
27968991110,1cyclictest0-21swapper/112:20:171
27968991110,1cyclictest0-21swapper/112:10:151
27968991110,1cyclictest0-21swapper/111:35:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional