You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-22 - 06:39
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Fri Sep 22, 2023 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2370199521518,2cyclictest55-21ksoftirqd/719:10:067
2366699521518,2cyclictest19-21ksoftirqd/119:10:071
2370199515513,1cyclictest55-21ksoftirqd/719:10:067
20419521080,103pltrace0-21swapper/219:06:412
20419521040,99pltrace0-21swapper/619:05:276
2323052960,93pltrace0-21swapper/319:10:063
2332728064,12sleep10-21swapper/119:08:071
2344527256,12sleep50-21swapper/519:09:465
397552680,0pltrace0-21swapper/223:36:502
3034252680,1pltrace30346-21pltrace-run21:11:490
207742680,1sleep20-21swapper/200:30:222
318032640,0sleep00-21swapper/023:32:100
81362630,0sleep50-21swapper/500:15:275
2326026246,12sleep00-21swapper/019:07:100
2321026245,12sleep40-21swapper/419:06:494
190092610,0sleep00-21swapper/019:55:230
35502540,0sleep20-21swapper/221:53:342
860552530,7pltrace2367899cyclictest19:36:493
860552530,7pltrace2367899cyclictest19:36:493
292032280,3chrt2367399cyclictest22:20:262
2369099251,13cyclictest0-21swapper/521:46:345
323782240,1sleep20-21swapper/223:00:012
2369099230,16cyclictest0-21swapper/520:15:515
2366199200,19cyclictest0-21swapper/023:50:320
23696991812,3cyclictest0-21swapper/620:49:326
23683991810,7cyclictest0-21swapper/423:27:094
727152170,2pltrace111rcu_preempt21:23:395
23696991711,3cyclictest0-21swapper/622:57:376
23696991711,3cyclictest0-21swapper/619:54:236
23690991712,3cyclictest0-21swapper/519:30:055
23690991712,3cyclictest0-21swapper/519:30:045
23690991712,2cyclictest0-21swapper/521:36:165
23678991711,3cyclictest0-21swapper/323:41:473
23673991714,1cyclictest0-21swapper/222:44:332
2367399171,4cyclictest0-21swapper/220:06:212
1190452170,0pltrace11908-21pltrace-run21:26:506
793752160,2pltrace111rcu_preempt23:08:255
793752160,1pltrace8606-21ssh23:08:161
488152160,0pltrace0-21swapper/100:15:271
3049352160,2pltrace111rcu_preempt21:49:371
2814652160,1pltrace19-21ksoftirqd/100:02:431
2617052160,1pltrace43-21ksoftirqd/522:54:585
23701991611,3cyclictest0-21swapper/722:41:257
23701991611,2cyclictest0-21swapper/723:49:397
23701991611,2cyclictest0-21swapper/720:38:157
2370199160,4cyclictest0-21swapper/722:47:457
2369699161,14cyclictest0-21swapper/619:20:266
23690991611,3cyclictest0-21swapper/522:13:085
23690991610,3cyclictest0-21swapper/521:10:195
2369099161,8cyclictest0-21swapper/519:35:505
2369099161,8cyclictest0-21swapper/519:35:495
2369099161,7cyclictest0-21swapper/523:12:565
2369099160,15cyclictest0-21swapper/522:37:555
2368399161,8cyclictest0-21swapper/422:23:034
2368399160,14cyclictest32675-21ssh00:06:504
23678991611,3cyclictest0-21swapper/320:44:143
23678991611,3cyclictest0-21swapper/320:06:493
23678991611,2cyclictest0-21swapper/323:56:183
23678991611,2cyclictest0-21swapper/321:28:313
23678991610,3cyclictest0-21swapper/323:08:133
2367899161,8cyclictest0-21swapper/321:31:523
23673991613,2cyclictest1135-21snmpd21:47:502
2367399161,8cyclictest0-21swapper/221:02:272
23666991610,3cyclictest0-21swapper/122:07:121
23666991610,3cyclictest0-21swapper/120:00:081
2366699161,8cyclictest0-21swapper/100:32:251
2366199161,4cyclictest0-21swapper/021:52:110
1874852160,2pltrace19211-21ssh00:27:521
1360052160,1pltrace49-21ksoftirqd/623:49:306
1209452160,2pltrace0-21swapper/222:05:302
740352150,2pltrace11951-21ssh22:01:345
727152150,1pltrace19-21ksoftirqd/121:23:401
48852150,1sleep0101rcuc/023:38:430
434952150,2pltrace7125-21chrt20:31:381
320872150,1sleep60-21swapper/622:24:546
3163352150,2pltrace3607-21ssh23:36:041
2814652150,9pltrace28519-21sh00:02:422
2687152150,2pltrace27473-21sh21:10:011
2687152150,1pltrace27258-21date21:10:001
2617052150,1pltrace27598-21ssh22:54:541
2370199159,3cyclictest0-21swapper/700:29:467
23701991512,2cyclictest0-21swapper/720:16:327
23701991510,3cyclictest0-21swapper/722:01:027
2370199150,14cyclictest0-21swapper/722:14:247
23696991512,2cyclictest0-21swapper/621:50:516
23696991510,4cyclictest0-21swapper/623:32:566
23696991510,3cyclictest0-21swapper/620:04:046
2369699151,7cyclictest0-21swapper/600:13:196
2369699151,7cyclictest0-21swapper/600:07:346
2369699151,13cyclictest0-21swapper/600:31:176
2369699150,11cyclictest0-21swapper/623:15:196
23690991511,1cyclictest0-21swapper/500:05:155
2369099150,8cyclictest0-21swapper/519:47:345
23683991510,2cyclictest0-21swapper/420:28:364
23683991510,2cyclictest0-21swapper/420:13:474
2367899159,4cyclictest0-21swapper/319:18:073
2367899159,3cyclictest0-21swapper/321:09:363
2367899159,3cyclictest0-21swapper/320:10:593
23678991512,2cyclictest0-21swapper/300:36:013
23678991512,2cyclictest0-21swapper/300:36:013
23678991510,3cyclictest0-21swapper/322:29:523
23678991510,3cyclictest0-21swapper/320:52:433
23678991510,2cyclictest0-21swapper/320:25:363
23678991510,2cyclictest0-21swapper/319:28:353
23678991510,2cyclictest0-21swapper/319:28:353
2367899150,8cyclictest0-21swapper/322:20:333
2367899150,4cyclictest28667-21ssh00:03:043
2367899150,14cyclictest0-21swapper/320:33:173
2367399159,3cyclictest0-21swapper/220:23:592
2367399152,12cyclictest0-21swapper/200:23:212
2367399151,13cyclictest0-21swapper/223:24:582
2367399150,8cyclictest0-21swapper/220:27:272
2366699159,3cyclictest0-21swapper/122:29:191
2366699152,12cyclictest0-21swapper/122:20:141
2366699150,3cyclictest0-21swapper/100:10:191
2366199152,12cyclictest0-21swapper/022:22:130
23661991512,2cyclictest22374-21df_inode21:00:190
23661991510,3cyclictest0-21swapper/020:41:440
23661991510,3cyclictest0-21swapper/000:23:300
23661991510,2cyclictest0-21swapper/022:04:020
2366199150,8cyclictest0-21swapper/020:20:410
2366199150,4cyclictest0-21swapper/022:41:500
2366199150,4cyclictest0-21swapper/021:39:100
2366199150,14cyclictest0-21swapper/021:20:270
2366199150,14cyclictest0-21swapper/019:35:270
2366199150,14cyclictest0-21swapper/019:35:270
2138852150,1pltrace23228-21ssh22:15:081
1704252150,1pltrace43-21ksoftirqd/519:55:155
1684552150,0pltrace0-21swapper/122:42:141
1360052150,1pltrace14887-21ssh23:49:231
793752140,10pltrace8633-21ssh23:08:206
2370199149,3cyclictest0-21swapper/722:35:347
2370199149,3cyclictest0-21swapper/721:25:147
2370199149,2cyclictest0-21swapper/719:36:347
2370199149,2cyclictest0-21swapper/719:36:347
23701991412,1cyclictest0-21swapper/723:25:107
23701991412,1cyclictest0-21swapper/722:50:597
23701991412,1cyclictest0-21swapper/721:34:537
23701991412,1cyclictest0-21swapper/721:19:017
23701991412,1cyclictest0-21swapper/700:07:447
23701991411,2cyclictest0-21swapper/721:55:287
23701991411,2cyclictest0-21swapper/721:49:267
23701991411,2cyclictest0-21swapper/721:20:207
2370199140,13cyclictest0-21swapper/723:52:407
2370199140,13cyclictest0-21swapper/723:21:267
2370199140,13cyclictest0-21swapper/723:14:477
2370199140,13cyclictest0-21swapper/721:53:417
2370199140,13cyclictest0-21swapper/700:37:127
2370199140,13cyclictest0-21swapper/700:37:127
2370199140,13cyclictest0-21swapper/700:33:007
2370199140,13cyclictest0-21swapper/700:02:427
2369699149,3cyclictest0-21swapper/620:32:096
2369699149,3cyclictest0-21swapper/600:01:486
2369699149,2cyclictest0-21swapper/621:11:006
2369699148,3cyclictest0-21swapper/622:11:516
23696991412,1cyclictest0-21swapper/621:19:166
23696991412,1cyclictest0-21swapper/600:24:506
23696991411,2cyclictest0-21swapper/619:40:206
2369699140,7cyclictest0-21swapper/621:31:146
2369699140,7cyclictest0-21swapper/619:47:496
2369699140,13cyclictest0-21swapper/620:10:286
2369099149,3cyclictest0-21swapper/523:47:245
2369099149,3cyclictest0-21swapper/521:55:115
23690991412,1cyclictest0-21swapper/523:23:425
23690991412,1cyclictest0-21swapper/522:43:155
23690991412,1cyclictest0-21swapper/522:21:235
23690991412,1cyclictest0-21swapper/522:18:295
23690991410,3cyclictest0-21swapper/519:12:215
2369099140,7cyclictest0-21swapper/523:56:455
2369099140,13cyclictest0-21swapper/523:39:545
2369099140,13cyclictest0-21swapper/522:49:345
2369099140,13cyclictest0-21swapper/522:26:245
2369099140,13cyclictest0-21swapper/500:31:235
2369099140,1cyclictest0-21swapper/521:54:025
2369099140,1cyclictest0-21swapper/500:11:395
2368399149,3cyclictest0-21swapper/419:45:464
2368399149,2cyclictest0-21swapper/419:53:054
2368399148,3cyclictest0-21swapper/421:16:454
2368399148,3cyclictest0-21swapper/419:29:024
2368399148,3cyclictest0-21swapper/419:29:014
2368399147,1cyclictest0-21swapper/423:07:034
23683991412,1cyclictest0-21swapper/423:49:404
23683991412,1cyclictest0-21swapper/423:30:544
23683991412,1cyclictest0-21swapper/421:57:004
23683991412,1cyclictest0-21swapper/421:29:374
23683991412,1cyclictest0-21swapper/400:01:484
2368399140,7cyclictest0-21swapper/423:40:104
2368399140,13cyclictest0-21swapper/423:54:424
2368399140,13cyclictest0-21swapper/423:39:394
2368399140,13cyclictest0-21swapper/422:55:064
2368399140,13cyclictest0-21swapper/422:25:104
2368399140,13cyclictest0-21swapper/421:30:564
2368399140,13cyclictest0-21swapper/400:27:534
2368399140,1cyclictest0-21swapper/421:40:184
2368399140,1cyclictest0-21swapper/421:35:374
2367899148,3cyclictest0-21swapper/320:20:003
2367899148,3cyclictest0-21swapper/319:43:173
23678991412,1cyclictest0-21swapper/323:17:213
23678991412,1cyclictest0-21swapper/322:45:083
23678991412,1cyclictest0-21swapper/322:19:513
23678991412,1cyclictest0-21swapper/319:57:203
23678991412,1cyclictest0-21swapper/300:06:333
2367899140,13cyclictest0-21swapper/323:23:583
2367899140,1cyclictest0-21swapper/323:49:233
2367899140,1cyclictest0-21swapper/323:33:323
2367399148,4cyclictest0-21swapper/219:40:442
2367399146,7cyclictest0-21swapper/222:25:152
23673991412,1cyclictest0-21swapper/222:03:512
23673991412,1cyclictest0-21swapper/200:36:312
23673991412,1cyclictest0-21swapper/200:36:312
23673991411,2cyclictest0-21swapper/221:27:222
23673991411,2cyclictest0-21swapper/221:05:032
2367399140,7cyclictest0-21swapper/220:32:212
2367399140,2cyclictest0-21swapper/200:15:122
2367399140,13cyclictest0-21swapper/223:53:172
2367399140,13cyclictest0-21swapper/200:28:072
2366699148,3cyclictest0-21swapper/119:58:281
23666991412,1cyclictest31719-21ssh22:58:391
23666991412,1cyclictest0-21swapper/123:42:411
23666991412,1cyclictest0-21swapper/122:46:491
23666991412,1cyclictest0-21swapper/121:54:131
23666991412,1cyclictest0-21swapper/121:40:081
2366699140,7cyclictest0-21swapper/123:55:261
2366699140,2cyclictest0-21swapper/100:36:201
2366699140,2cyclictest0-21swapper/100:36:191
2366699140,13cyclictest0-21swapper/123:50:361
2366699140,13cyclictest0-21swapper/123:34:441
2366699140,13cyclictest0-21swapper/123:25:111
2366699140,13cyclictest0-21swapper/122:12:501
2366699140,13cyclictest0-21swapper/121:58:531
2366699140,1cyclictest0-21swapper/122:30:481
2366699140,1cyclictest0-21swapper/100:21:121
2366199149,2cyclictest0-21swapper/020:56:430
23661991412,1cyclictest0-21swapper/021:57:460
23661991412,1cyclictest0-21swapper/000:17:200
23661991411,2cyclictest0-21swapper/000:25:240
2366199141,12cyclictest0-21swapper/022:28:260
2366199141,1cyclictest0-21swapper/021:40:290
2366199140,13cyclictest0-21swapper/023:49:200
2366199140,13cyclictest0-21swapper/022:55:450
2366199140,12cyclictest0-21swapper/023:42:300
2212752140,1pltrace49-21ksoftirqd/623:22:186
2212752140,1pltrace22320-21ssh23:22:130
1684552140,1pltrace111rcu_preempt22:42:206
1360052140,1pltrace25-21ksoftirqd/223:49:302
1142252140,1pltrace0-21swapper/519:43:145
3163352130,0pltrace0-21swapper/623:36:076
2976452130,5pltrace0-21swapper/119:17:241
2976452130,5pltrace0-21swapper/019:17:220
2617052130,5pltrace27617-21sh22:54:572
2370199138,3cyclictest0-21swapper/721:38:437
2370199138,3cyclictest0-21swapper/720:40:537
2370199138,2cyclictest0-21swapper/723:35:387
23701991311,1cyclictest0-21swapper/722:58:367
23701991311,1cyclictest0-21swapper/722:22:197
23701991311,1cyclictest0-21swapper/722:05:407
23701991311,1cyclictest0-21swapper/721:44:167
23701991311,1cyclictest0-21swapper/719:51:187
23701991311,1cyclictest0-21swapper/719:42:327
2370199130,12cyclictest0-21swapper/723:15:267
2370199130,12cyclictest0-21swapper/719:16:147
2370199130,1cyclictest0-21swapper/723:57:417
2370199130,1cyclictest0-21swapper/719:26:507
2370199130,1cyclictest0-21swapper/719:26:497
2369699138,2cyclictest0-21swapper/622:25:506
23696991311,1cyclictest0-21swapper/623:56:586
23696991311,1cyclictest0-21swapper/623:11:186
23696991311,1cyclictest0-21swapper/620:40:256
23696991310,2cyclictest1264-21lldpd21:44:036
2369699130,12cyclictest0-21swapper/622:16:076
2369099133,8cyclictest0-21swapper/521:16:245
23690991311,1cyclictest0-21swapper/522:30:365
23690991311,1cyclictest0-21swapper/500:03:425
23690991310,2cyclictest0-21swapper/523:15:105
23690991310,1cyclictest0-21swapper/519:22:495
2369099130,2cyclictest0-21swapper/521:40:465
2369099130,12cyclictest0-21swapper/522:09:435
2369099130,12cyclictest0-21swapper/500:35:375
2369099130,12cyclictest0-21swapper/500:35:365
2369099130,1cyclictest0-21swapper/521:33:085
23683991311,1cyclictest0-21swapper/423:21:204
23683991311,1cyclictest0-21swapper/423:11:484
23683991311,1cyclictest0-21swapper/421:46:184
23683991311,1cyclictest0-21swapper/400:21:354
23683991310,1cyclictest0-21swapper/420:55:514
2368399130,12cyclictest0-21swapper/419:41:184
2368399130,11cyclictest0-21swapper/420:00:274
2368399130,1cyclictest0-21swapper/422:45:244
2368399130,1cyclictest0-21swapper/421:11:114
23678991311,1cyclictest0-21swapper/322:52:443
23678991311,1cyclictest0-21swapper/321:50:433
2367899130,12cyclictest0-21swapper/322:01:253
2367899130,1cyclictest0-21swapper/323:36:273
2367899130,1cyclictest0-21swapper/322:33:193
2367899130,1cyclictest0-21swapper/321:55:263
2367899130,1cyclictest0-21swapper/321:44:443
2367899130,1cyclictest0-21swapper/321:15:363
2367399138,2cyclictest0-21swapper/223:40:032
2367399133,8cyclictest0-21swapper/220:36:202
23673991311,1cyclictest0-21swapper/223:34:302
23673991311,1cyclictest0-21swapper/222:49:592
23673991311,1cyclictest0-21swapper/222:30:172
23673991311,1cyclictest0-21swapper/222:13:532
2367399130,12cyclictest0-21swapper/221:35:412
2367399130,12cyclictest0-21swapper/220:01:112
2367399130,12cyclictest0-21swapper/200:08:222
2367399130,1cyclictest0-21swapper/221:57:512
2367399130,1cyclictest0-21swapper/221:22:252
2367399130,1cyclictest0-21swapper/221:15:072
23666991311,1cyclictest0-21swapper/122:04:531
23666991311,1cyclictest0-21swapper/120:25:291
23666991311,1cyclictest0-21swapper/100:05:411
2366699131,5cyclictest0-21swapper/123:04:531
2366699130,12cyclictest0-21swapper/121:26:011
2366699130,12cyclictest0-21swapper/119:39:201
2366699130,12cyclictest0-21swapper/119:39:191
2366699130,12cyclictest0-21swapper/119:28:191
2366699130,12cyclictest0-21swapper/119:28:181
2366699130,1cyclictest0-21swapper/123:18:341
23661991311,1cyclictest0-21swapper/023:11:070
23661991311,1cyclictest0-21swapper/023:07:230
23661991311,1cyclictest0-21swapper/023:01:320
23661991311,1cyclictest0-21swapper/022:51:310
23661991311,1cyclictest0-21swapper/021:49:410
23661991311,1cyclictest0-21swapper/021:34:470
23661991311,1cyclictest0-21swapper/000:02:160
23661991310,2cyclictest0-21swapper/000:14:070
2366199131,5cyclictest0-21swapper/020:50:290
2366199130,12cyclictest0-21swapper/022:19:270
2366199130,1cyclictest0-21swapper/023:57:150
2366199130,1cyclictest0-21swapper/020:15:180
2366199130,1cyclictest0-21swapper/000:32:530
2138852130,5pltrace0-21swapper/722:15:117
2118252130,4pltrace0-21swapper/621:37:016
1278552130,1pltrace0-21swapper/520:44:115
740352120,0pltrace0-21swapper/622:01:356
488152120,4pltrace0-21swapper/600:15:336
2370199128,1cyclictest0-21swapper/723:03:147
23701991210,1cyclictest0-21swapper/722:34:587
23701991210,1cyclictest0-21swapper/720:02:557
23701991210,1cyclictest0-21swapper/700:20:217
2370199120,5cyclictest0-21swapper/723:33:097
2370199120,11cyclictest0-21swapper/722:29:467
2369699129,2cyclictest13785-21kworker/u16:023:51:096
2369699127,3cyclictest0-21swapper/620:50:176
2369699127,2cyclictest0-21swapper/622:34:306
23696991210,1cyclictest0-21swapper/623:26:136
23696991210,1cyclictest0-21swapper/623:03:526
23696991210,1cyclictest0-21swapper/622:36:116
23696991210,1cyclictest0-21swapper/600:35:316
23696991210,1cyclictest0-21swapper/600:35:316
2369699120,11cyclictest0-21swapper/622:05:446
2369699120,1cyclictest0-21swapper/623:43:126
2369099127,3cyclictest0-21swapper/500:28:015
2369099127,2cyclictest0-21swapper/523:28:445
23690991210,1cyclictest0-21swapper/522:59:165
23690991210,1cyclictest0-21swapper/520:22:525
23690991210,1cyclictest0-21swapper/519:28:085
23690991210,1cyclictest0-21swapper/519:28:085
2369099120,11cyclictest0-21swapper/520:01:345
2368399127,3cyclictest0-21swapper/422:05:534
23683991210,1cyclictest0-21swapper/422:50:554
23683991210,1cyclictest0-21swapper/422:02:594
23683991210,1cyclictest0-21swapper/421:22:204
23683991210,1cyclictest0-21swapper/420:31:304
23683991210,1cyclictest0-21swapper/420:09:284
23683991210,1cyclictest0-21swapper/419:55:534
23683991210,1cyclictest0-21swapper/400:19:284
2368399121,10cyclictest0-21swapper/422:42:114
2368399120,11cyclictest0-21swapper/419:24:004
23678991210,1cyclictest0-21swapper/323:26:533
23678991210,1cyclictest0-21swapper/323:10:433
23678991210,1cyclictest0-21swapper/322:43:013
2367899120,11cyclictest0-21swapper/320:01:273
2367399129,2cyclictest1135-21snmpd21:31:102
23673991210,1cyclictest0-21swapper/223:55:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional