You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 23:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Sat Jun 28, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1713352980,95pltrace0-21swapper/307:10:013
1434552900,85pltrace0-21swapper/507:05:215
1713352890,56pltrace0-21swapper/707:08:397
1434552880,80pltrace0-21swapper/107:05:181
1708628549,10sleep20-21swapper/207:07:242
1705427850,10sleep60-21swapper/607:06:566
1700126147,10sleep00-21swapper/007:06:110
1714225743,10sleep40-21swapper/407:07:504
277652280,0sleep50-21swapper/509:20:195
563552270,5pltrace0-21swapper/710:40:157
2239152270,1pltrace0-21swapper/311:07:473
1746499200,8cyclictest0-21swapper/507:52:245
624152190,4pltrace0-21swapper/708:43:437
2967552190,16pltrace54-21ksoftirqd/410:23:244
2749252190,0pltrace0-21swapper/708:24:437
3183552180,4pltrace1942-21unixbench_multi12:25:260
2361952180,11pltrace54-21ksoftirqd/407:20:214
1746499180,14cyclictest0-21swapper/508:58:375
2191352170,10pltrace54-21ksoftirqd/408:13:214
17469991713,3cyclictest8655-21cut09:45:166
17460991711,5cyclictest0-21swapper/411:02:554
1746099171,8cyclictest0-21swapper/410:11:224
1744399171,12cyclictest0-21swapper/011:32:440
1740852170,11pltrace18337-21expr09:05:150
1713352170,2pltrace0-21swapper/707:10:257
1291852170,11pltrace54-21ksoftirqd/409:55:204
842252160,4pltrace9166-21tr10:45:157
794352160,4pltrace0-21swapper/607:50:226
455552160,3pltrace0-21swapper/709:39:257
455552160,3pltrace0-21swapper/709:39:257
3138152160,8pltrace32286-21cstates09:30:172
3075552160,4pltrace1166-21runrttasks11:24:450
17475991611,3cyclictest0-21swapper/710:02:467
1747599161,8cyclictest0-21swapper/710:20:477
1747599160,15cyclictest0-21swapper/710:13:147
1746999161,8cyclictest0-21swapper/607:14:386
1746999161,14cyclictest0-21swapper/609:39:046
1746999161,14cyclictest0-21swapper/609:39:046
1746999160,8cyclictest0-21swapper/611:39:366
17464991610,5cyclictest0-21swapper/509:52:235
1746499161,7cyclictest0-21swapper/509:40:045
1746499161,7cyclictest0-21swapper/509:40:045
1746499161,14cyclictest0-21swapper/512:13:435
1746499161,14cyclictest0-21swapper/509:17:105
1746499161,14cyclictest0-21swapper/508:11:495
1746499160,4cyclictest0-21swapper/510:44:415
1746499160,4cyclictest0-21swapper/510:15:525
1746499160,15cyclictest0-21swapper/512:27:595
1746499160,15cyclictest0-21swapper/510:29:365
17455991611,3cyclictest0-21swapper/310:28:123
1745599161,14cyclictest0-21swapper/308:17:133
1745599160,2cyclictest0-21swapper/311:00:523
1745299162,11cyclictest1378-21cat08:35:172
1745299161,7cyclictest0-21swapper/210:29:572
1745299161,7cyclictest0-21swapper/209:51:332
1745299161,7cyclictest0-21swapper/208:55:482
1745299160,3cyclictest0-21swapper/207:34:222
1745299160,15cyclictest0-21swapper/210:50:062
1745299160,15cyclictest0-21swapper/207:25:212
1745299160,1cyclictest0-21swapper/209:24:392
17447991612,3cyclictest5936-21latency_hist10:40:011
17447991611,3cyclictest0-21swapper/111:18:041
1744799161,3cyclictest0-21swapper/110:58:491
1744399161,8cyclictest0-21swapper/009:57:160
1744399161,8cyclictest0-21swapper/007:59:470
1744399161,4cyclictest0-21swapper/009:33:280
1744399161,4cyclictest0-21swapper/008:14:320
1744399160,4cyclictest0-21swapper/011:01:530
1744399160,15cyclictest0-21swapper/011:38:560
171022160,0sleep70-21swapper/709:00:257
842252150,3pltrace10325-21needreboot10:45:192
345352150,7pltrace54-21ksoftirqd/408:41:064
3138152150,9pltrace31545-21crond09:30:000
3138152150,11pltrace54-21ksoftirqd/409:30:104
3027752150,12pltrace54-21ksoftirqd/408:27:594
2688552150,3pltrace28908-21flock10:20:212
2688552150,3pltrace27672-21cut10:20:146
2239152150,7pltrace22567-21idleruntime-cro11:09:592
2239152150,13pltrace0-21swapper/211:11:422
2191352150,8pltrace22108-21latency_hist08:15:002
1787552150,1pltrace0-21swapper/612:00:206
1747599151,8cyclictest0-21swapper/712:38:467
1747599151,8cyclictest0-21swapper/712:38:467
1747599150,8cyclictest0-21swapper/712:12:247
1747599150,7cyclictest0-21swapper/710:52:267
1747599150,7cyclictest0-21swapper/707:54:507
1747599150,5cyclictest0-21swapper/709:05:037
1747599150,4cyclictest0-21swapper/708:31:437
1747599150,4cyclictest0-21swapper/707:33:227
1747599150,14cyclictest0-21swapper/711:28:357
17469991510,3cyclictest0-21swapper/609:50:326
1746999151,7cyclictest0-21swapper/609:34:436
1746999150,8cyclictest0-21swapper/611:20:366
1746999150,8cyclictest0-21swapper/609:01:266
1746999150,8cyclictest0-21swapper/608:37:096
1746999150,2cyclictest0-21swapper/608:20:226
1746999150,14cyclictest0-21swapper/608:29:456
1746499159,5cyclictest0-21swapper/511:20:175
17464991510,2cyclictest0-21swapper/510:23:005
1746499151,7cyclictest0-21swapper/507:16:055
1746499151,3cyclictest0-21swapper/508:50:245
1746499150,2cyclictest0-21swapper/508:20:205
1746499150,14cyclictest0-21swapper/511:02:075
1746499150,14cyclictest0-21swapper/509:30:375
1746499150,14cyclictest0-21swapper/507:31:155
17460991511,3cyclictest0-21swapper/409:14:224
17460991510,3cyclictest0-21swapper/407:28:274
1746099150,8cyclictest0-21swapper/412:00:004
1746099150,8cyclictest0-21swapper/410:38:424
1746099150,4cyclictest0-21swapper/407:17:354
1746099150,14cyclictest0-21swapper/412:08:334
1746099150,13cyclictest0-21swapper/409:52:464
1745599150,8cyclictest0-21swapper/308:32:233
1745599150,8cyclictest0-21swapper/308:29:323
1745599150,7cyclictest0-21swapper/310:54:133
1745599150,7cyclictest0-21swapper/308:37:423
1745599150,7cyclictest0-21swapper/308:04:523
1745599150,14cyclictest0-21swapper/310:05:143
1745599150,14cyclictest0-21swapper/309:50:413
1745599150,14cyclictest0-21swapper/309:29:103
1745599150,14cyclictest0-21swapper/308:07:193
1745599150,14cyclictest0-21swapper/307:39:093
1745599150,14cyclictest0-21swapper/307:30:273
1745299157,7cyclictest1-21systemd08:45:002
1745299150,8cyclictest0-21swapper/208:53:542
1745299150,4cyclictest0-21swapper/212:05:572
1745299150,2cyclictest0-21swapper/208:20:182
1745299150,14cyclictest0-21swapper/211:17:302
1745299150,14cyclictest0-21swapper/208:05:162
1744799157,7cyclictest0-21swapper/112:39:101
1744799157,7cyclictest0-21swapper/112:39:101
17447991510,3cyclictest0-21swapper/108:41:321
1744799150,4cyclictest0-21swapper/107:32:071
1744399150,14cyclictest0-21swapper/010:26:070
1744399150,14cyclictest0-21swapper/010:19:380
1744399150,14cyclictest0-21swapper/008:59:330
1740852150,5pltrace0-21swapper/209:05:172
842252140,1pltrace0-21swapper/610:45:196
455552140,6pltrace0-21swapper/709:42:197
455552140,6pltrace0-21swapper/709:42:197
455552140,5pltrace0-21swapper/309:40:013
455552140,5pltrace0-21swapper/309:40:013
3246052140,1pltrace1583-21hwlatdetect10:30:152
3027752140,10pltrace0-21swapper/008:28:120
222352140,0pltrace0-21swapper/712:29:427
1747599141,12cyclictest0-21swapper/711:10:247
1747599141,12cyclictest0-21swapper/709:15:097
1747599140,8cyclictest0-21swapper/711:35:097
1747599140,7cyclictest0-21swapper/709:48:557
1747599140,7cyclictest0-21swapper/708:19:087
1747599140,1cyclictest0-21swapper/707:40:237
1746999149,3cyclictest0-21swapper/609:55:566
1746999148,5cyclictest0-21swapper/610:08:596
1746999140,8cyclictest0-21swapper/607:57:196
1746999140,7cyclictest0-21swapper/608:57:166
1746999140,13cyclictest0-21swapper/612:35:406
1746999140,13cyclictest0-21swapper/612:35:406
1746999140,1cyclictest0-21swapper/608:45:236
1746499149,3cyclictest0-21swapper/508:27:465
1746499141,12cyclictest0-21swapper/511:59:435
1746499140,13cyclictest0-21swapper/508:05:175
17460991412,1cyclictest0-21swapper/410:15:164
17460991411,2cyclictest0-21swapper/412:23:414
17460991410,1cyclictest0-21swapper/410:33:344
1746099140,7cyclictest0-21swapper/410:43:544
1746099140,2cyclictest0-21swapper/408:20:224
1745599148,5cyclictest0-21swapper/308:40:503
1745599140,6cyclictest0-21swapper/310:34:393
1745599140,6cyclictest0-21swapper/308:59:553
1745599140,13cyclictest0-21swapper/312:21:303
1745599140,13cyclictest0-21swapper/311:23:143
1745299149,3cyclictest0-21swapper/211:02:462
1745299148,3cyclictest0-21swapper/209:11:552
17452991412,1cyclictest0-21swapper/211:35:202
17452991410,3cyclictest0-21swapper/211:43:532
1745299141,12cyclictest0-21swapper/211:28:022
1744799140,7cyclictest0-21swapper/109:00:511
1744799140,7cyclictest0-21swapper/108:25:081
1744399149,3cyclictest0-21swapper/007:25:390
1744399140,13cyclictest0-21swapper/008:05:210
1744399140,12cyclictest0-21swapper/009:43:430
1744399140,12cyclictest0-21swapper/009:43:430
1740852140,9pltrace19013-21if_err_eth009:05:195
1740852140,4pltrace0-21swapper/309:05:153
1678452140,1pltrace16788-21pltrace-run10:57:487
393052130,3pltrace0-21swapper/111:37:021
345352130,1pltrace0-21swapper/008:40:270
3138152130,4pltrace0-21swapper/309:30:183
3075552130,6pltrace0-21swapper/211:24:542
2626052130,1pltrace0-21swapper/012:14:460
2130852130,4pltrace23575-21chrt10:10:232
1747599138,3cyclictest0-21swapper/711:34:457
1747599133,8cyclictest0-21swapper/707:35:587
1747599132,9cyclictest0-21swapper/711:57:307
1747599130,12cyclictest0-21swapper/708:05:157
1747599130,1cyclictest0-21swapper/711:03:347
1746999138,3cyclictest0-21swapper/611:15:206
17469991310,2cyclictest3667-21unixbench_multi11:30:246
1746999130,13cyclictest0-21swapper/611:06:336
1746999130,12cyclictest0-21swapper/612:06:106
1746999130,12cyclictest0-21swapper/608:11:316
17464991310,2cyclictest1132-21snmpd08:35:035
17464991310,2cyclictest0-21swapper/512:34:115
1746499130,9cyclictest0-21swapper/511:37:505
1746499130,1cyclictest0-21swapper/508:30:225
1746099130,12cyclictest0-21swapper/412:29:174
1746099130,12cyclictest0-21swapper/407:53:184
1745599139,2cyclictest0-21swapper/311:41:353
1745599138,4cyclictest0-21swapper/307:15:143
1745599138,3cyclictest0-21swapper/309:59:083
1745599130,13cyclictest0-21swapper/310:40:323
1745299137,5cyclictest0-21swapper/211:47:412
17452991311,1cyclictest0-21swapper/209:25:292
1745299130,13cyclictest0-21swapper/209:55:182
1745299130,12cyclictest0-21swapper/210:07:272
1744799138,3cyclictest0-21swapper/107:11:241
1744799137,5cyclictest24876-21munin-run08:20:001
1744799130,6cyclictest0-21swapper/110:49:431
1744799130,13cyclictest0-21swapper/109:50:191
1744799130,12cyclictest0-21swapper/110:09:031
1744799130,1cyclictest0-21swapper/108:20:201
1744399130,12cyclictest0-21swapper/009:49:220
1744399130,12cyclictest0-21swapper/007:21:350
1744399130,12cyclictest0-21swapper/007:19:110
1740852130,3pltrace17888-21ls09:05:000
1570252130,4pltrace17150-21expr10:00:201
1181652130,5pltrace0-21swapper/308:53:443
794352120,8pltrace54-21ksoftirqd/407:50:004
794352120,5pltrace0-21swapper/207:51:152
455552120,1pltrace0-21swapper/309:40:173
455552120,1pltrace0-21swapper/309:40:173
3138152120,9pltrace54-21ksoftirqd/409:28:374
3075552120,7pltrace54-21ksoftirqd/411:23:324
2904752120,3pltrace31747-21sshd12:21:487
2580552120,5pltrace0-21swapper/009:18:070
1999252120,4pltrace20776-21timerandwakeup07:15:021
1787552120,3pltrace20223-21timerandwakeup12:00:242
1747599129,2cyclictest14032-21latency11:50:217
1747599122,8cyclictest0-21swapper/708:30:017
17475991210,1cyclictest0-21swapper/708:48:127
1747599120,12cyclictest0-21swapper/708:13:347
1746999129,2cyclictest14769-21grep10:55:146
1746999129,2cyclictest1069-17sedispatch10:51:256
1746999127,4cyclictest6013-21tail09:40:186
1746999127,4cyclictest6013-21tail09:40:186
1746999126,5cyclictest0-21swapper/611:41:366
1746999126,5cyclictest0-21swapper/611:10:436
1746999123,8cyclictest0-21swapper/612:15:526
17469991211,1cyclictest0-21swapper/609:26:106
17469991210,1cyclictest0-21swapper/610:40:096
17469991210,1cyclictest0-21swapper/607:15:016
1746999120,11cyclictest0-21swapper/608:05:236
1746499127,3cyclictest0-21swapper/510:50:095
1746499127,3cyclictest0-21swapper/507:21:595
1746099127,4cyclictest0-21swapper/411:16:134
1746099126,5cyclictest0-21swapper/411:40:384
1746099120,11cyclictest0-21swapper/412:19:574
1745599129,2cyclictest1065-17auditd08:50:003
17455991210,1cyclictest0-21swapper/312:00:213
1745599120,11cyclictest0-21swapper/312:32:353
1745599120,11cyclictest0-21swapper/312:19:333
1745599120,11cyclictest0-21swapper/309:22:493
1745599120,1cyclictest0-21swapper/307:21:333
1745299129,2cyclictest16747-21smart_sda09:00:242
1745299129,2cyclictest1132-21snmpd12:15:242
1745299129,2cyclictest1069-17sedispatch09:50:012
17452991211,1cyclictest0-21swapper/212:30:232
17452991210,1cyclictest0-21swapper/210:43:512
1744799127,4cyclictest27050-21tr12:15:161
1744799126,5cyclictest0-21swapper/111:50:271
1744799126,5cyclictest0-21swapper/111:30:441
1744799126,5cyclictest0-21swapper/108:47:231
1744799122,8cyclictest0-21swapper/109:33:461
17447991211,1cyclictest0-21swapper/109:15:001
1744799120,12cyclictest0-21swapper/109:42:321
1744799120,12cyclictest0-21swapper/109:42:321
1744399127,3cyclictest0-21swapper/007:51:040
1744399126,5cyclictest0-21swapper/011:14:140
1744399120,1cyclictest0-21swapper/008:20:180
1740852120,4pltrace0-21swapper/109:05:191
1678452120,1pltrace0-21swapper/310:59:403
1632652120,4pltrace0-21swapper/208:03:162
1291852120,0pltrace0-21swapper/009:55:000
1228952120,5pltrace0-21swapper/011:49:160
794352110,4pltrace0-21swapper/307:51:173
515852110,7pltrace54-21ksoftirqd/407:44:594
2580552110,7pltrace54-21ksoftirqd/409:17:474
1747599118,2cyclictest12941-21switchtime07:55:237
1747599118,2cyclictest10048-21/usr/sbin/munin11:45:177
1747599116,4cyclictest78-21ksoftirqd/710:15:227
1747599116,4cyclictest32228-21tr09:30:177
1747599116,3cyclictest0-21swapper/712:00:037
1747599116,2cyclictest0-21swapper/709:00:007
17475991110,1cyclictest0-21swapper/711:41:087
17475991110,1cyclictest0-21swapper/708:53:237
1747599110,11cyclictest0-21swapper/712:15:367
1747599110,1cyclictest0-21swapper/710:32:187
1747599110,1cyclictest0-21swapper/710:28:167
1747599110,1cyclictest0-21swapper/709:21:247
1747599110,1cyclictest0-21swapper/708:39:097
1747599110,1cyclictest0-21swapper/708:02:247
1747599110,1cyclictest0-21swapper/707:24:597
1746999118,2cyclictest27444-21iostat_ios09:20:166
1746999118,2cyclictest0-21swapper/607:25:476
1746999116,3cyclictest0-21swapper/612:26:016
1746999116,3cyclictest0-21swapper/610:03:476
1746999115,3cyclictest0-21swapper/612:31:376
1746999112,8cyclictest0-21swapper/607:38:186
17469991110,1cyclictest0-21swapper/611:45:356
17469991110,1cyclictest0-21swapper/610:34:006
17469991110,1cyclictest0-21swapper/607:40:556
17469991110,1cyclictest0-21swapper/607:22:286
1746999110,11cyclictest0-21swapper/610:26:046
1746999110,11cyclictest0-21swapper/608:16:526
1746999110,1cyclictest0-21swapper/608:01:206
1746499117,3cyclictest1132-21snmpd10:10:505
1746499112,8cyclictest0-21swapper/511:50:015
17464991110,1cyclictest0-21swapper/507:45:025
1746499111,5cyclictest0-21swapper/512:20:015
1746499110,1cyclictest0-21swapper/510:58:115
1746499110,1cyclictest0-21swapper/509:12:295
1746499110,1cyclictest0-21swapper/508:40:065
1746099118,2cyclictest16391-21cpu10:00:174
1746099116,3cyclictest0-21swapper/409:47:344
1746099116,3cyclictest0-21swapper/409:44:554
1746099116,3cyclictest0-21swapper/409:44:554
1746099115,5cyclictest0-21swapper/408:53:094
1746099114,6cyclictest1028-13auditd11:07:324
17460991110,1cyclictest0-21swapper/408:36:394
1746099110,9cyclictest0-21swapper/408:07:464
1746099110,11cyclictest0-21swapper/412:14:454
1746099110,11cyclictest0-21swapper/410:07:194
1746099110,1cyclictest0-21swapper/412:32:154
1746099110,1cyclictest0-21swapper/409:37:044
1746099110,1cyclictest0-21swapper/409:37:044
1745599118,2cyclictest1132-21snmpd12:10:333
1745599116,3cyclictest0-21swapper/310:37:063
17455991110,1cyclictest0-21swapper/311:36:433
1745599110,11cyclictest0-21swapper/310:03:573
1745599110,11cyclictest0-21swapper/307:11:413
1745599110,1cyclictest0-21swapper/307:57:103
1745299116,3cyclictest0-21swapper/211:33:442
17452991110,1cyclictest0-21swapper/208:45:492
1745299111,2cyclictest0-21swapper/209:35:252
1745299111,2cyclictest0-21swapper/209:35:242
1745299110,11cyclictest0-21swapper/212:29:052
1745299110,11cyclictest0-21swapper/207:39:172
1745299110,10cyclictest0-21swapper/207:19:232
1745299110,1cyclictest0-21swapper/210:16:302
1745299110,1cyclictest0-21swapper/210:04:322
1745299110,1cyclictest0-21swapper/209:42:112
1745299110,1cyclictest0-21swapper/209:42:112
1745299110,1cyclictest0-21swapper/207:59:502
1744799118,2cyclictest0-21swapper/107:55:231
1744799117,3cyclictest9189-21crond08:54:551
1744799117,3cyclictest1132-21snmpd07:39:341
17447991110,1cyclictest0-21swapper/111:46:331
17447991110,1cyclictest0-21swapper/109:35:441
17447991110,1cyclictest0-21swapper/109:35:441
17447991110,1cyclictest0-21swapper/109:20:311
17447991110,1cyclictest0-21swapper/108:35:301
1744799111,8cyclictest0-21swapper/112:34:581
1744799110,9cyclictest0-21swapper/111:14:101
1744799110,11cyclictest0-21swapper/107:25:441
1744799110,11cyclictest0-21swapper/107:20:251
1744799110,1cyclictest0-21swapper/111:59:401
1744799110,1cyclictest0-21swapper/110:20:361
1744799110,1cyclictest0-21swapper/109:46:261
1744399118,2cyclictest9672-21latency_hist11:45:000
1744399118,2cyclictest1700-21lldpd10:36:400
1744399118,2cyclictest0-21swapper/008:30:230
1744399117,3cyclictest1132-21snmpd07:37:460
1744399116,3cyclictest0-21swapper/010:48:310
1744399116,3cyclictest0-21swapper/008:01:470
1744399112,8cyclictest0-21swapper/009:35:580
1744399112,8cyclictest0-21swapper/009:35:580
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional