You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 14:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Sun Apr 28, 2024 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26071521100,106pltrace0-21swapper/719:07:407
26071521040,100pltrace0-21swapper/319:10:013
2328352920,88pltrace0-21swapper/619:05:166
2328352900,86pltrace0-21swapper/219:05:212
2328352790,74pltrace0-21swapper/519:05:015
2626426753,10sleep10-21swapper/119:09:321
2595425945,10sleep40-21swapper/419:05:344
2620025844,10sleep00-21swapper/019:08:410
3121152300,1pltrace0-21swapper/700:15:177
295852280,5pltrace0-21swapper/719:23:497
2649099250,4cyclictest0-21swapper/423:08:504
1397152250,0pltrace0-21swapper/720:50:197
2842052240,0pltrace0-21swapper/700:10:267
3182252230,2pltrace0-21swapper/722:20:187
2649099230,8cyclictest0-21swapper/420:18:494
2648799230,22cyclictest0-21swapper/321:58:403
2647499205,3cyclictest0-21swapper/020:43:410
2562352190,0pltrace0-21swapper/700:05:147
1443752190,9pltrace16135-21irqstats23:45:145
1443752190,13pltrace15089-21wget23:45:093
574252180,6pltrace0-21swapper/719:31:157
574252180,6pltrace0-21swapper/719:31:157
2893852180,3pltrace0-21swapper/719:12:167
1443752180,12pltrace0-21swapper/123:45:131
43452170,1pltrace436-21tee23:16:580
2650299171,8cyclictest0-21swapper/720:28:157
26499991713,2cyclictest0-21swapper/619:58:426
2649099171,8cyclictest0-21swapper/422:04:394
2648799171,4cyclictest0-21swapper/320:11:133
1954752170,0pltrace0-21swapper/721:00:197
1893652170,14pltrace54-21ksoftirqd/422:53:224
500052160,0pltrace0-21swapper/722:30:387
2650299169,6cyclictest9723-21turbostat21:40:007
26502991611,3cyclictest0-21swapper/722:39:227
26502991611,3cyclictest0-21swapper/722:39:227
2650299160,15cyclictest0-21swapper/722:41:497
2650299160,15cyclictest0-21swapper/722:41:497
2650299160,15cyclictest0-21swapper/721:45:237
26490991613,2cyclictest2009-21latency19:20:204
26490991611,4cyclictest14439-21pltrace-run23:41:574
26490991611,3cyclictest0-21swapper/423:14:234
2649099161,8cyclictest0-21swapper/420:33:114
2648799167,8cyclictest0-21swapper/322:24:203
2648799160,8cyclictest0-21swapper/320:24:433
26483991614,1cyclictest0-21swapper/223:15:472
26483991610,5cyclictest0-21swapper/223:56:482
26474991611,3cyclictest0-21swapper/021:21:320
2647499161,7cyclictest0-21swapper/020:00:090
2647499160,4cyclictest0-21swapper/023:29:420
2647499160,15cyclictest0-21swapper/020:28:000
2562352160,3pltrace0-21swapper/700:02:297
2514452160,4pltrace0-21swapper/721:07:067
574252150,4pltrace5900-21taskset19:29:062
26502991511,3cyclictest0-21swapper/723:57:377
2650299150,8cyclictest0-21swapper/723:29:367
2650299150,2cyclictest0-21swapper/723:40:177
2650299150,2cyclictest0-21swapper/722:45:257
2650299150,2cyclictest0-21swapper/722:45:247
2650299150,2cyclictest0-21swapper/719:25:187
26499991511,3cyclictest0-21swapper/623:57:346
2649999150,8cyclictest0-21swapper/622:06:576
2649999150,8cyclictest0-21swapper/600:08:476
2649999150,4cyclictest0-21swapper/623:44:456
2649999150,14cyclictest0-21swapper/621:08:166
2649999150,14cyclictest0-21swapper/619:36:276
2649999150,14cyclictest0-21swapper/619:36:266
26490991513,1cyclictest15488-21cstates23:45:114
26490991512,2cyclictest1166-21runrttasks19:36:084
26490991512,2cyclictest1166-21runrttasks19:36:084
26490991512,2cyclictest0-21swapper/423:20:594
26490991511,2cyclictest0-21swapper/421:31:444
2649099150,7cyclictest0-21swapper/422:55:374
2649099150,7cyclictest0-21swapper/419:51:124
2649099150,4cyclictest0-21swapper/421:07:264
2649099150,2cyclictest0-21swapper/422:45:164
2649099150,2cyclictest0-21swapper/422:45:164
2649099150,14cyclictest0-21swapper/422:30:464
26487991510,3cyclictest0-21swapper/320:58:053
2648799150,8cyclictest0-21swapper/319:12:203
2648799150,7cyclictest0-21swapper/320:25:283
2648799150,7cyclictest0-21swapper/300:31:183
2648799150,4cyclictest0-21swapper/321:48:443
2648799150,13cyclictest0-21swapper/322:39:473
2648799150,13cyclictest0-21swapper/322:39:473
2648399150,7cyclictest0-21swapper/222:47:082
2648399150,7cyclictest0-21swapper/222:47:082
2648399150,14cyclictest0-21swapper/219:49:382
2648399150,13cyclictest0-21swapper/219:52:112
2648399150,13cyclictest0-21swapper/200:19:512
26479991511,3cyclictest7258-21gltestperf19:30:171
26479991511,3cyclictest7258-21gltestperf19:30:171
2647499150,8cyclictest0-21swapper/022:12:070
2647499150,8cyclictest0-21swapper/020:22:330
2647499150,14cyclictest0-21swapper/023:12:390
2647499150,13cyclictest0-21swapper/019:29:080
1486152150,1pltrace0-21swapper/719:50:047
1335752150,1pltrace13360-21pltrace-run22:41:574
1335752150,1pltrace13360-21pltrace-run22:41:574
949152140,8pltrace9666-21runrttasks21:39:221
3242752140,0pltrace0-21swapper/720:23:217
2650299149,3cyclictest0-21swapper/723:45:187
2650299149,3cyclictest0-21swapper/721:13:447
2650299141,6cyclictest0-21swapper/720:12:517
2650299140,7cyclictest0-21swapper/720:10:007
2649999140,7cyclictest0-21swapper/623:21:156
2649999140,7cyclictest0-21swapper/600:34:056
2649999140,3cyclictest0-21swapper/623:46:176
2649999140,13cyclictest0-21swapper/621:45:166
26494991412,1cyclictest0-21swapper/521:50:005
26490991413,1cyclictest0-21swapper/421:25:214
26490991411,2cyclictest54-21ksoftirqd/420:05:204
26490991410,2cyclictest0-21swapper/423:27:354
2649099141,12cyclictest0-21swapper/423:34:594
2649099140,7cyclictest0-21swapper/421:58:444
2649099140,2cyclictest0-21swapper/419:25:224
2649099140,13cyclictest0-21swapper/421:46:404
2649099140,13cyclictest0-21swapper/420:01:584
2649099140,1cyclictest0-21swapper/419:10:124
2648799140,8cyclictest0-21swapper/300:07:173
2648799140,6cyclictest0-21swapper/300:15:173
2648799140,13cyclictest0-21swapper/323:50:113
2648399148,3cyclictest0-21swapper/219:56:542
26483991410,2cyclictest0-21swapper/223:53:502
2648399140,13cyclictest0-21swapper/223:20:182
26479991411,2cyclictest1132-21snmpd00:33:131
26479991410,3cyclictest2670-21gltestperf21:25:141
2647499149,3cyclictest0-21swapper/020:19:240
2647499148,5cyclictest0-21swapper/020:31:220
26474991412,1cyclictest0-21swapper/020:35:180
2647499140,7cyclictest0-21swapper/021:17:020
1785252140,2pltrace0-21swapper/721:55:307
1443752140,3pltrace15038-21/usr/sbin/munin23:45:092
438752130,3pltrace4571-21chrt00:24:317
2795852130,9pltrace78-21ksoftirqd/721:15:087
2795852130,9pltrace30706-21runrttasks21:16:411
2795852130,3pltrace30700-21runrttasks21:16:362
2650299131,5cyclictest0-21swapper/721:33:387
2650299130,12cyclictest0-21swapper/700:38:327
2650299130,1cyclictest0-21swapper/700:30:237
2649999137,1cyclictest0-21swapper/622:40:006
2649999137,1cyclictest0-21swapper/622:40:006
26499991310,2cyclictest5375-21sed19:25:246
2649999130,13cyclictest0-21swapper/620:45:246
2649999130,13cyclictest0-21swapper/620:25:296
2649999130,13cyclictest0-21swapper/620:00:136
2649999130,13cyclictest0-21swapper/619:46:006
2649999130,12cyclictest0-21swapper/620:09:416
2649999130,1cyclictest0-21swapper/622:45:166
2649999130,1cyclictest0-21swapper/622:45:166
2649999130,1cyclictest0-21swapper/619:50:016
2649999130,1cyclictest0-21swapper/600:29:596
2649499130,13cyclictest0-21swapper/500:25:275
2649099139,1cyclictest191rcu_preempt22:37:404
2649099139,1cyclictest191rcu_preempt22:37:404
2649099137,5cyclictest0-21swapper/400:05:174
2649099133,6cyclictest5452ksoftirqd/422:19:014
26490991312,1cyclictest3065-21fschecks_time00:20:114
26490991312,1cyclictest0-21swapper/420:55:314
26490991312,1cyclictest0-21swapper/419:30:224
26490991312,1cyclictest0-21swapper/419:30:224
26490991311,1cyclictest0-21swapper/422:21:004
2649099130,13cyclictest0-21swapper/420:25:154
2649099130,13cyclictest0-21swapper/419:15:184
2649099130,1cyclictest0-21swapper/419:42:404
26487991312,1cyclictest0-21swapper/321:27:583
26487991310,2cyclictest0-21swapper/321:02:133
2648799130,12cyclictest0-21swapper/321:16:303
2648799130,1cyclictest0-21swapper/319:55:123
26483991312,1cyclictest0-21swapper/221:35:142
26483991310,2cyclictest1132-21snmpd23:12:122
2648399130,12cyclictest0-21swapper/221:46:062
2648399130,12cyclictest0-21swapper/220:36:292
2648399130,12cyclictest0-21swapper/220:23:432
2648399130,1cyclictest0-21swapper/219:10:192
26479991310,2cyclictest7393-21crond00:29:591
2647499138,3cyclictest0-21swapper/019:20:500
26474991312,1cyclictest0-21swapper/021:50:260
2647499130,12cyclictest0-21swapper/021:59:250
2562352130,2pltrace25831-21taskset00:04:505
561452120,3pltrace0-21swapper/720:36:257
438752120,0pltrace0-21swapper/700:25:127
2795852120,1pltrace0-21swapper/621:16:336
2732752120,7pltrace0-21swapper/223:09:542
2650299126,5cyclictest0-21swapper/722:11:367
2650299126,5cyclictest0-21swapper/720:40:347
26502991210,1cyclictest0-21swapper/721:41:597
2650299120,12cyclictest0-21swapper/722:05:577
2649999129,2cyclictest31869-21cstates21:20:096
2649999129,2cyclictest10988-21grep21:40:136
2649999126,5cyclictest0-21swapper/623:07:386
2649999126,5cyclictest0-21swapper/621:26:206
2649999126,5cyclictest0-21swapper/620:51:226
2649999125,4cyclictest0-21swapper/622:10:136
2649999120,11cyclictest0-21swapper/619:16:426
2649499129,2cyclictest0-21swapper/519:35:545
2649499129,2cyclictest0-21swapper/519:35:545
2649499129,2cyclictest0-21swapper/500:10:485
2649499120,11cyclictest0-21swapper/520:34:045
2649099125,4cyclictest0-21swapper/400:15:194
2649099123,8cyclictest0-21swapper/420:35:234
26490991211,1cyclictest0-21swapper/421:50:184
26490991211,1cyclictest0-21swapper/420:40:194
26490991211,1cyclictest0-21swapper/400:27:464
26490991210,1cyclictest0-21swapper/423:16:324
26490991210,1cyclictest0-21swapper/400:36:414
2649099121,4cyclictest29705-21expr21:15:174
2649099120,1cyclictest0-21swapper/421:02:394
2649099120,1cyclictest0-21swapper/419:57:144
2649099120,1cyclictest0-21swapper/400:03:484
2648799126,5cyclictest0-21swapper/322:59:013
26487991211,1cyclictest0-21swapper/323:20:483
2648399127,4cyclictest1132-21snmpd21:06:352
2648399120,11cyclictest0-21swapper/223:38:502
2648399120,11cyclictest0-21swapper/222:28:162
2647999129,2cyclictest3557-21tr00:20:161
2647999127,4cyclictest12893-21tr19:45:081
2647999120,12cyclictest0-21swapper/120:55:371
2647999120,1cyclictest0-21swapper/121:55:311
2647499129,2cyclictest1129-21modem-manager00:10:000
2647499126,5cyclictest0-21swapper/020:50:000
2647499126,5cyclictest0-21swapper/020:10:330
26474991211,1cyclictest0-21swapper/021:44:570
26474991211,1cyclictest0-21swapper/021:00:140
26474991210,1cyclictest0-21swapper/023:36:470
2647499120,1cyclictest0-21swapper/019:53:520
2562352120,1pltrace0-21swapper/100:04:481
1614552120,8pltrace16377-21munin-run22:50:011
1614552120,8pltrace16377-21munin-run22:50:011
1614552120,3pltrace18333-21taskset22:50:225
1397152120,1pltrace0-21swapper/520:51:455
1274252120,0pltrace0-21swapper/000:39:080
1057152120,2pltrace1132-21snmpd22:41:163
1057152120,2pltrace1132-21snmpd22:41:163
949152110,1pltrace0-21swapper/021:39:270
561452110,1pltrace0-21swapper/220:33:472
438752110,1pltrace46-21ksoftirqd/300:24:423
3182252110,2pltrace1166-21runrttasks22:19:042
2963852110,7pltrace54-21ksoftirqd/420:20:154
2684352110,2pltrace26998-21chrt20:13:572
2650299113,7cyclictest1028-13auditd19:20:007
26502991110,1cyclictest0-21swapper/722:15:217
26502991110,1cyclictest0-21swapper/720:55:417
26502991110,1cyclictest0-21swapper/720:45:587
2650299110,11cyclictest0-21swapper/723:02:487
2650299110,11cyclictest0-21swapper/722:59:137
2650299110,10cyclictest0-21swapper/722:02:147
2650299110,1cyclictest0-21swapper/723:06:227
2650299110,1cyclictest0-21swapper/719:42:307
2650299110,1cyclictest0-21swapper/719:36:317
2650299110,1cyclictest0-21swapper/719:36:317
2649999119,1cyclictest0-21swapper/600:04:556
2649999118,2cyclictest7440-21cat20:35:206
2649999118,2cyclictest6621-21basename22:30:156
2649999118,2cyclictest6003-21expr21:30:196
2649999118,2cyclictest453-21expr22:20:116
2649999118,2cyclictest31259-21ls20:20:166
2649999118,2cyclictest1697-21lldpd23:02:456
2649999118,2cyclictest11440-21awk23:35:236
2649999114,5cyclictest1028-13auditd22:30:006
2649999112,8cyclictest0-21swapper/620:31:526
2649999112,8cyclictest0-21swapper/619:10:566
26499991110,1cyclictest0-21swapper/619:40:156
26499991110,1cyclictest0-21swapper/600:38:436
2649999110,11cyclictest0-21swapper/622:41:196
2649999110,11cyclictest0-21swapper/622:41:196
2649999110,11cyclictest0-21swapper/622:16:436
2649999110,11cyclictest0-21swapper/621:59:526
2649999110,1cyclictest0-21swapper/623:31:026
2649999110,1cyclictest0-21swapper/620:55:076
2649499119,1cyclictest0-21swapper/520:25:125
26494991110,1cyclictest0-21swapper/522:23:025
26494991110,1cyclictest0-21swapper/521:21:305
2649499110,11cyclictest0-21swapper/522:06:315
2649499110,11cyclictest0-21swapper/521:55:305
2649499110,11cyclictest0-21swapper/520:01:235
2649499110,10cyclictest0-21swapper/500:20:065
2649499110,1cyclictest0-21swapper/523:07:435
2649499110,1cyclictest0-21swapper/520:36:005
2649099118,2cyclictest18828-21idleruntime23:50:124
2649099118,2cyclictest12836-21if_err_eth020:45:144
2649099116,3cyclictest0-21swapper/422:26:464
26490991110,1cyclictest0-21swapper/423:58:184
26490991110,1cyclictest0-21swapper/421:40:254
26490991110,1cyclictest0-21swapper/421:10:184
26490991110,1cyclictest0-21swapper/420:13:164
26490991110,1cyclictest0-21swapper/419:45:134
26490991110,1cyclictest0-21swapper/400:13:024
2649099111,5cyclictest54-21ksoftirqd/421:35:284
2649099111,3cyclictest31367-21cut21:20:094
2649099110,11cyclictest0-21swapper/422:07:334
2649099110,11cyclictest0-21swapper/420:53:454
2648799112,8cyclictest0-21swapper/323:04:523
26487991110,1cyclictest0-21swapper/322:30:463
26487991110,1cyclictest0-21swapper/320:46:443
26487991110,1cyclictest0-21swapper/319:29:083
2648799110,11cyclictest0-21swapper/319:24:153
2648799110,1cyclictest0-21swapper/323:26:093
2648799110,1cyclictest0-21swapper/321:44:103
2648799110,1cyclictest0-21swapper/320:18:403
2648799110,1cyclictest0-21swapper/319:43:033
2648399117,2cyclictest0-21swapper/221:03:532
2648399117,2cyclictest0-21swapper/220:15:342
2648399116,3cyclictest0-21swapper/219:40:002
2648399116,3cyclictest0-21swapper/219:40:002
2648399116,2cyclictest0-21swapper/220:53:142
2648399110,5cyclictest0-21swapper/221:50:522
2648399110,11cyclictest0-21swapper/222:00:402
2648399110,10cyclictest0-21swapper/221:26:202
2648399110,10cyclictest0-21swapper/219:30:042
2648399110,10cyclictest0-21swapper/219:30:042
2648399110,1cyclictest0-21swapper/223:40:142
2648399110,1cyclictest0-21swapper/222:33:022
2648399110,1cyclictest0-21swapper/220:41:002
2648399110,1cyclictest0-21swapper/219:15:302
2648399110,1cyclictest0-21swapper/200:20:412
2648399110,1cyclictest0-21swapper/200:04:302
2647999117,3cyclictest1132-21snmpd19:35:531
2647999117,3cyclictest1132-21snmpd19:35:531
26479991110,1cyclictest0-21swapper/122:33:051
26479991110,1cyclictest0-21swapper/121:11:151
2647999110,11cyclictest0-21swapper/123:04:011
2647999110,11cyclictest0-21swapper/122:06:321
2647999110,1cyclictest0-21swapper/123:33:581
2647999110,1cyclictest0-21swapper/123:18:061
2647999110,1cyclictest0-21swapper/100:13:441
2647499119,1cyclictest0-21swapper/021:30:320
2647499117,3cyclictest0-21swapper/000:11:220
2647499112,8cyclictest0-21swapper/023:57:430
2647499112,8cyclictest0-21swapper/019:17:050
26474991110,1cyclictest0-21swapper/000:22:380
2647499110,11cyclictest0-21swapper/000:02:440
2647499110,10cyclictest0-21swapper/022:16:300
2647499110,1cyclictest0-21swapper/023:48:190
2647499110,1cyclictest0-21swapper/023:30:400
2063952110,1pltrace0-21swapper/221:59:372
1506552110,7pltrace78-21ksoftirqd/721:50:227
1506552110,6pltrace15873-21cpuspeed_turbos21:50:113
2842052100,7pltrace30603-21sendmail_mailst00:10:223
2650299107,2cyclictest0-21swapper/720:15:127
2650299103,6cyclictest0-21swapper/723:12:477
2650299101,8cyclictest0-21swapper/721:25:097
2650299100,9cyclictest17825-21/usr/sbin/munin23:50:177
2650299100,5cyclictest1132-21snmpd22:50:257
2650299100,10cyclictest0-21swapper/719:46:337
2649999109,1cyclictest0-21swapper/600:15:446
2649999108,1cyclictest0-21swapper/623:25:126
2649999107,2cyclictest31162-21expr23:15:106
2649999107,2cyclictest22878-21switchtime22:00:236
2649999107,2cyclictest2218-21open_files19:20:236
2649999107,2cyclictest20621-21cat21:00:106
2649999107,2cyclictest20567-21cat22:55:146
2649999107,2cyclictest18934-21pltrace-run22:51:576
2649999107,2cyclictest10162-21latency20:40:176
2649999106,2cyclictest0-21swapper/623:52:186
2649999104,2cyclictest0-21swapper/600:20:106
2649499109,1cyclictest0-21swapper/523:39:275
2649499109,1cyclictest0-21swapper/522:58:585
2649499109,1cyclictest0-21swapper/522:40:025
2649499109,1cyclictest0-21swapper/522:40:025
2649499109,1cyclictest0-21swapper/522:01:015
2649499109,1cyclictest0-21swapper/521:09:125
2649499109,1cyclictest0-21swapper/520:10:565
2649499109,1cyclictest0-21swapper/519:33:325
2649499109,1cyclictest0-21swapper/519:33:325
2649499109,1cyclictest0-21swapper/519:23:495
2649499108,1cyclictest0-21swapper/519:47:275
2649499107,2cyclictest1700-21lldpd20:05:435
2649499107,2cyclictest1132-21snmpd23:55:025
2649499107,2cyclictest1132-21snmpd20:22:465
2649499107,2cyclictest0-21swapper/523:42:175
2649499107,2cyclictest0-21swapper/521:17:355
2649499105,4cyclictest9828-21open_files22:35:225
2649499105,4cyclictest9828-21open_files22:35:225
2649499100,10cyclictest0-21swapper/520:18:325
2649499100,10cyclictest0-21swapper/519:59:265
2649499100,10cyclictest0-21swapper/500:05:505
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional