You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-03 - 18:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Wed Sep 03, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5002521160,109pltrace0-21swapper/107:05:141
5002521080,91pltrace0-21swapper/507:05:155
781152950,90pltrace0-21swapper/607:06:446
790626551,10sleep20-21swapper/207:06:522
805026450,10sleep30-21swapper/307:08:533
792926450,10sleep70-21swapper/707:07:137
790425945,10sleep00-21swapper/007:06:510
779125642,10sleep40-21swapper/407:05:334
532052360,6pltrace78-21ksoftirqd/708:56:037
365152310,3pltrace0-21swapper/709:55:217
833999241,16cyclictest0-21swapper/607:59:466
1594952230,1pltrace78-21ksoftirqd/711:15:167
8344992217,3cyclictest0-21swapper/707:24:387
833199220,16cyclictest0-21swapper/407:29:164
3268252220,2pltrace2290-21cat11:45:157
2377652210,0pltrace0-21swapper/708:35:227
3268252200,0pltrace0-21swapper/711:44:227
80152190,13pltrace54-21ksoftirqd/409:50:014
2098752190,14pltrace0-21swapper/608:28:366
1034752190,12pltrace1132-21snmpd11:03:220
8327991811,6cyclictest1070-13audispd10:10:013
2989152180,0pltrace0-21swapper/711:36:297
2656252180,13pltrace27823-21/usr/sbin/munin08:40:166
834499171,15cyclictest0-21swapper/711:13:187
8339991711,5cyclictest0-21swapper/611:15:486
833999171,3cyclictest0-21swapper/609:30:306
833699171,15cyclictest0-21swapper/509:24:335
833699170,2cyclictest0-21swapper/507:50:175
8331991711,5cyclictest0-21swapper/407:17:354
833199171,8cyclictest0-21swapper/412:13:444
833199171,8cyclictest0-21swapper/411:35:584
833199171,8cyclictest0-21swapper/410:45:164
832799171,8cyclictest0-21swapper/307:50:123
8322991711,5cyclictest0-21swapper/210:48:102
8319991711,5cyclictest0-21swapper/109:52:121
2767952170,12pltrace54-21ksoftirqd/409:39:044
2767952170,12pltrace54-21ksoftirqd/409:39:044
2320752170,12pltrace54-21ksoftirqd/410:28:194
924452160,12pltrace0-21swapper/010:03:560
8344991611,3cyclictest0-21swapper/711:32:517
834499160,15cyclictest0-21swapper/710:59:127
8339991610,5cyclictest0-21swapper/609:00:576
8336991611,3cyclictest0-21swapper/509:29:015
833699161,14cyclictest0-21swapper/507:48:575
8331991611,4cyclictest0-21swapper/411:56:374
833199161,4cyclictest0-21swapper/408:34:374
832799160,15cyclictest0-21swapper/307:28:383
8322991611,2cyclictest0-21swapper/207:45:182
8319991610,3cyclictest0-21swapper/112:15:371
831999161,4cyclictest0-21swapper/108:25:521
831999161,14cyclictest0-21swapper/107:32:471
831999160,8cyclictest0-21swapper/111:33:541
831999160,8cyclictest0-21swapper/109:18:081
831699161,4cyclictest0-21swapper/012:09:160
831699160,15cyclictest0-21swapper/008:24:530
811252160,0pltrace0-21swapper/709:03:017
2878552160,11pltrace1132-21snmpd10:39:074
2320752160,12pltrace0-21swapper/010:28:320
8344991511,3cyclictest0-21swapper/707:35:517
834499151,7cyclictest0-21swapper/708:31:327
834499151,3cyclictest0-21swapper/707:15:237
834499150,7cyclictest0-21swapper/710:37:237
8339991510,3cyclictest0-21swapper/610:22:316
8339991510,3cyclictest0-21swapper/608:21:056
833999150,8cyclictest0-21swapper/610:09:136
833999150,2cyclictest0-21swapper/607:50:126
833999150,14cyclictest0-21swapper/611:43:236
8336991511,3cyclictest0-21swapper/509:54:035
8336991510,3cyclictest0-21swapper/511:01:005
833699150,8cyclictest0-21swapper/511:42:135
833699150,8cyclictest0-21swapper/510:46:255
833699150,3cyclictest0-21swapper/508:01:505
833199151,7cyclictest0-21swapper/412:16:414
833199150,7cyclictest0-21swapper/411:41:074
833199150,14cyclictest0-21swapper/408:55:414
833199150,14cyclictest0-21swapper/408:43:594
833199150,14cyclictest0-21swapper/407:40:484
832799157,7cyclictest22992-21crond07:35:013
8327991510,4cyclictest0-21swapper/308:16:073
8327991510,3cyclictest0-21swapper/311:48:343
8327991510,3cyclictest0-21swapper/309:52:553
832799151,8cyclictest0-21swapper/310:02:103
832799150,7cyclictest0-21swapper/309:40:463
832799150,4cyclictest0-21swapper/312:16:563
832299159,5cyclictest0-21swapper/208:53:262
832299159,3cyclictest0-21swapper/208:42:442
832299150,7cyclictest0-21swapper/212:39:302
832299150,7cyclictest0-21swapper/211:25:202
832299150,3cyclictest0-21swapper/211:03:232
832299150,14cyclictest0-21swapper/207:59:122
8319991510,2cyclictest0-21swapper/107:50:461
831999150,8cyclictest0-21swapper/109:23:411
831999150,7cyclictest0-21swapper/111:54:541
831999150,14cyclictest0-21swapper/111:27:501
831999150,14cyclictest0-21swapper/107:39:461
831999150,13cyclictest0-21swapper/110:36:561
831999150,13cyclictest0-21swapper/109:00:171
831699159,5cyclictest0-21swapper/009:27:150
831699159,5cyclictest0-21swapper/009:08:590
831699151,7cyclictest0-21swapper/007:16:590
831699150,8cyclictest0-21swapper/009:20:170
831699150,4cyclictest0-21swapper/010:35:500
831699150,14cyclictest0-21swapper/011:11:470
831699150,14cyclictest0-21swapper/010:07:150
831699150,14cyclictest0-21swapper/008:49:460
831699150,14cyclictest0-21swapper/007:59:490
1988152150,11pltrace0-21swapper/607:29:556
1931952150,11pltrace54-21ksoftirqd/409:23:464
1373652150,8pltrace14108-21cut09:15:000
1093152150,3pltrace0-21swapper/709:07:247
8344991410,3cyclictest84-21kauditd10:55:017
834499140,1cyclictest0-21swapper/707:50:147
833999149,2cyclictest0-21swapper/612:31:296
8339991412,1cyclictest0-21swapper/607:35:196
8339991411,2cyclictest15757-21latency_hist08:20:016
833999140,7cyclictest0-21swapper/610:34:076
833999140,7cyclictest0-21swapper/608:45:546
833699149,3cyclictest0-21swapper/510:06:455
833699149,3cyclictest0-21swapper/508:27:475
833699148,5cyclictest0-21swapper/508:41:025
833699140,13cyclictest0-21swapper/508:30:055
833199141,12cyclictest0-21swapper/408:02:214
833199140,1cyclictest0-21swapper/411:00:244
832799143,9cyclictest4581-21expr09:55:183
832799143,4cyclictest17234-21cut09:20:113
832799142,6cyclictest1028-13auditd08:55:003
8327991410,3cyclictest0-21swapper/310:32:403
832799140,14cyclictest0-21swapper/310:38:343
832799140,14cyclictest0-21swapper/308:46:453
832799140,13cyclictest0-21swapper/310:17:063
832799140,13cyclictest0-21swapper/308:22:433
832799140,1cyclictest0-21swapper/307:56:053
832299140,7cyclictest0-21swapper/212:25:592
832299140,7cyclictest0-21swapper/210:16:462
832299140,3cyclictest0-21swapper/210:12:022
831999140,7cyclictest0-21swapper/110:42:381
831999140,7cyclictest0-21swapper/108:47:591
831999140,13cyclictest0-21swapper/112:20:191
831999140,13cyclictest0-21swapper/111:21:481
831999140,13cyclictest0-21swapper/109:55:541
831999140,13cyclictest0-21swapper/109:10:441
831699149,3cyclictest0-21swapper/011:47:450
831699148,3cyclictest0-21swapper/009:44:030
8316991412,1cyclictest0-21swapper/009:45:140
831699141,12cyclictest0-21swapper/011:25:220
831699140,7cyclictest0-21swapper/010:32:180
831699140,13cyclictest0-21swapper/010:15:130
475852140,11pltrace0-21swapper/610:51:406
365152140,8pltrace0-21swapper/609:51:286
2656252140,8pltrace0-21swapper/008:40:180
2656252140,10pltrace28698-21ntp_states08:40:201
1988152140,11pltrace0-21swapper/207:27:092
1540552140,10pltrace0-21swapper/108:16:421
924452130,6pltrace0-21swapper/510:03:565
834499137,3cyclictest0-21swapper/709:45:187
834499133,8cyclictest0-21swapper/710:20:007
834499130,12cyclictest0-21swapper/709:50:577
834499130,1cyclictest0-21swapper/708:20:157
833999139,3cyclictest23137-21open_files08:30:296
833999139,2cyclictest0-21swapper/609:38:276
833999139,2cyclictest0-21swapper/609:38:276
833999139,2cyclictest0-21swapper/609:05:486
8339991310,2cyclictest0-21swapper/608:35:226
833999130,13cyclictest0-21swapper/611:25:246
833999130,1cyclictest0-21swapper/610:55:226
833699130,13cyclictest0-21swapper/508:05:245
833699130,12cyclictest0-21swapper/511:34:375
833699130,12cyclictest0-21swapper/508:37:175
833199139,2cyclictest0-21swapper/410:42:294
833199137,3cyclictest0-21swapper/411:54:164
833199136,6cyclictest0-21swapper/410:59:594
833199133,8cyclictest0-21swapper/410:18:254
8331991312,1cyclictest0-21swapper/411:25:014
8331991312,1cyclictest0-21swapper/410:20:144
8331991311,1cyclictest0-21swapper/408:50:234
833199130,13cyclictest0-21swapper/411:30:324
833199130,12cyclictest0-21swapper/410:33:584
833199130,12cyclictest0-21swapper/408:49:324
833199130,12cyclictest0-21swapper/407:49:164
833199130,1cyclictest0-21swapper/412:08:344
832799134,7cyclictest13287-21kernelversion12:05:173
832799132,4cyclictest7999-21sed09:00:233
832799132,4cyclictest7506-21vmstat10:55:233
832799132,4cyclictest24644-21sed09:30:223
832799132,4cyclictest21705-21aten_r4power_cu08:30:163
832799132,4cyclictest1697-21lldpd10:51:403
832799131,6cyclictest1028-13auditd10:15:013
832799131,5cyclictest1038-21systemd-logind10:25:003
832799131,2cyclictest0-21swapper/312:38:393
832799130,6cyclictest1127-21polkitd09:40:013
832799130,6cyclictest1127-21polkitd09:40:013
832799130,5cyclictest7109-21chrt08:01:403
832799130,12cyclictest0-21swapper/311:31:213
832799130,12cyclictest0-21swapper/307:37:303
832799130,11cyclictest22982-21cat12:25:013
832299139,3cyclictest1132-21snmpd09:34:452
832299138,3cyclictest0-21swapper/209:58:162
832299138,2cyclictest0-21swapper/208:55:382
8322991312,1cyclictest0-21swapper/209:50:122
832299130,2cyclictest0-21swapper/207:50:082
832299130,12cyclictest0-21swapper/211:17:022
832299130,12cyclictest0-21swapper/209:23:202
831999131,11cyclictest0-21swapper/111:11:591
831999131,11cyclictest0-21swapper/108:05:451
831999130,7cyclictest0-21swapper/111:58:251
831699139,2cyclictest0-21swapper/008:15:230
831699130,7cyclictest0-21swapper/010:52:420
831699130,2cyclictest0-21swapper/010:42:010
831699130,12cyclictest0-21swapper/010:57:380
831699130,12cyclictest0-21swapper/007:22:080
303882130,0sleep50-21swapper/510:40:225
2824752130,5pltrace0-21swapper/607:41:346
2210352130,4pltrace22224-21taskset09:27:131
2152552130,0pltrace0-21swapper/711:23:467
1988152130,0pltrace771ktimers/707:30:187
1705852130,5pltrace0-21swapper/712:13:507
1315652130,11pltrace0-21swapper/711:06:127
924452120,3pltrace54-21ksoftirqd/410:03:404
866652120,3pltrace54-21ksoftirqd/412:00:144
834499129,2cyclictest1700-21lldpd12:15:487
834499129,2cyclictest1065-17auditd08:50:017
834499129,2cyclictest0-21swapper/707:14:357
834499127,3cyclictest0-21swapper/710:30:597
834499126,5cyclictest0-21swapper/709:16:057
8344991210,1cyclictest0-21swapper/710:43:327
8344991210,1cyclictest0-21swapper/710:05:097
8344991210,1cyclictest0-21swapper/708:08:537
834499120,11cyclictest0-21swapper/708:53:077
834499120,11cyclictest0-21swapper/708:15:537
833999129,2cyclictest0-21swapper/612:10:146
833999129,2cyclictest0-21swapper/609:10:226
833999128,3cyclictest0-21swapper/612:37:196
833999127,3cyclictest0-21swapper/607:10:116
833999126,5cyclictest0-21swapper/608:01:436
8339991211,1cyclictest0-21swapper/611:10:206
8339991210,1cyclictest0-21swapper/609:57:376
833999120,12cyclictest0-21swapper/612:15:236
833999120,11cyclictest0-21swapper/612:05:036
833999120,11cyclictest0-21swapper/611:22:396
833699127,3cyclictest0-21swapper/509:43:335
833699126,5cyclictest0-21swapper/512:23:165
833699126,5cyclictest0-21swapper/508:20:585
8336991210,1cyclictest0-21swapper/508:55:365
833699120,11cyclictest0-21swapper/510:13:015
833199129,2cyclictest21382-21grep07:30:154
833199129,2cyclictest0-21swapper/410:00:014
833199127,3cyclictest0-21swapper/409:41:494
833199126,5cyclictest0-21swapper/412:27:534
832799126,5cyclictest0-21swapper/311:41:343
832799125,6cyclictest0-21swapper/307:20:013
832799124,7cyclictest9003-21cpuspeed_turbos09:05:143
832799124,7cyclictest6923-21cstates11:55:143
832799124,7cyclictest23523-21cpuspeed_turbos12:25:143
832799124,7cyclictest1132-21snmpd10:26:103
832799123,8cyclictest11014-21expr08:10:123
832799122,4cyclictest30164-21cat07:45:163
832799122,3cyclictest29957-21cat10:40:203
832799120,6cyclictest3883-21wget11:50:143
832799120,5cyclictest8601-21basename07:10:123
832799120,5cyclictest5343-21chrt08:55:483
832799120,5cyclictest31412-21tr09:45:153
832799120,5cyclictest27670-21cstates08:40:143
832799120,5cyclictest27545-21ntp_offset12:30:183
832799120,5cyclictest26378-21expr07:40:133
832799120,5cyclictest24487-21cut08:35:113
832799120,5cyclictest22606-21cat11:25:133
832799120,5cyclictest21420-21unixbench_singl11:20:213
832799120,5cyclictest20070-21aten_r4power_en09:25:133
832799120,5cyclictest17033-21expr11:15:123
832799120,5cyclictest16122-21kernelversion07:20:163
832799120,5cyclictest15674-21/usr/sbin/munin12:10:153
832799120,5cyclictest14713-21expr09:15:133
832799120,5cyclictest14104-21cut11:10:123
832799120,5cyclictest13453-21taskset11:09:413
832799120,5cyclictest12299-21grep09:10:153
832799120,5cyclictest10529-21cat12:00:183
832799120,4cyclictest1166-21runrttasks08:26:143
832799120,4cyclictest1132-21snmpd10:45:103
832299129,2cyclictest8399-21fschecks_count08:05:142
832299129,2cyclictest2884-21cpuspeed_turbos10:50:102
832299129,2cyclictest10915-21gltestperf10:05:142
832299126,5cyclictest0-21swapper/212:06:592
832299126,5cyclictest0-21swapper/207:42:132
832299120,12cyclictest0-21swapper/207:23:172
832299120,11cyclictest0-21swapper/211:55:402
831999123,8cyclictest0-21swapper/110:53:351
831999122,8cyclictest0-21swapper/110:59:351
831999120,11cyclictest0-21swapper/109:44:571
831999120,11cyclictest0-21swapper/108:59:291
831699129,2cyclictest0-21swapper/012:14:560
831699128,3cyclictest1132-21snmpd12:18:580
831699126,3cyclictest0-21swapper/010:45:350
831699122,8cyclictest0-21swapper/008:25:250
8316991210,1cyclictest0-21swapper/009:15:080
831699120,11cyclictest0-21swapper/007:54:210
811252120,6pltrace0-21swapper/009:03:170
3157752120,4pltrace32331-21aten_r4power_po10:45:126
3099852120,5pltrace0-21swapper/712:37:077
2541952120,5pltrace0-21swapper/712:25:427
2320752120,4pltrace0-21swapper/110:28:321
2263152120,8pltrace54-21ksoftirqd/412:24:354
2263152120,1pltrace0-21swapper/112:25:151
1931952120,9pltrace20997-21http_loadtime09:25:156
1594952120,8pltrace0-21swapper/511:15:125
1594952120,8pltrace0-21swapper/011:15:110
1594952120,4pltrace0-21swapper/111:15:111
1373652120,8pltrace14337-21cut09:15:015
1373652120,8pltrace0-21swapper/509:15:055
834499119,1cyclictest0-21swapper/712:05:127
834499119,1cyclictest0-21swapper/709:22:157
834499117,3cyclictest12211-21forks09:10:157
834499117,3cyclictest1132-21snmpd10:10:077
834499117,3cyclictest11133-21timerandwakeup12:00:227
834499115,5cyclictest0-21swapper/712:24:077
8344991110,1cyclictest0-21swapper/709:26:277
8344991110,1cyclictest0-21swapper/708:10:037
834499110,11cyclictest0-21swapper/710:01:067
834499110,1cyclictest0-21swapper/712:32:587
833999118,2cyclictest11209-21munin-node11:00:076
833999118,2cyclictest0-21swapper/610:25:206
833999117,3cyclictest11882-21fschecks_time11:05:166
833999117,3cyclictest1132-21snmpd09:42:456
833999117,3cyclictest1132-21snmpd07:45:576
8339991110,1cyclictest0-21swapper/611:38:506
8339991110,1cyclictest0-21swapper/610:16:346
8339991110,1cyclictest0-21swapper/608:59:506
8339991110,1cyclictest0-21swapper/608:07:526
833999110,11cyclictest0-21swapper/611:46:456
833999110,11cyclictest0-21swapper/610:35:506
833999110,11cyclictest0-21swapper/607:23:206
833999110,10cyclictest0-21swapper/609:24:166
833999110,1cyclictest0-21swapper/609:45:296
833699115,5cyclictest0-21swapper/510:21:015
833699110,9cyclictest0-21swapper/510:19:105
833699110,11cyclictest0-21swapper/512:19:555
833699110,10cyclictest0-21swapper/510:30:085
833699110,1cyclictest0-21swapper/512:25:065
833699110,1cyclictest0-21swapper/511:35:445
833699110,1cyclictest0-21swapper/510:38:245
833699110,1cyclictest0-21swapper/510:27:235
833699110,1cyclictest0-21swapper/509:57:365
833699110,1cyclictest0-21swapper/508:48:095
833199118,2cyclictest29338-21ls12:35:124
8331991110,1cyclictest0-21swapper/409:26:404
8331991110,1cyclictest0-21swapper/408:35:294
833199110,2cyclictest0-21swapper/407:50:144
833199110,11cyclictest0-21swapper/407:37:514
833199110,1cyclictest0-21swapper/411:22:164
833199110,1cyclictest0-21swapper/409:18:154
833199110,1cyclictest0-21swapper/409:11:174
832799111,2cyclictest0-21swapper/311:35:173
832799110,4cyclictest8372-21egrep11:00:133
832799110,4cyclictest7744-21basename08:05:113
832299119,1cyclictest0-21swapper/209:19:352
832299119,1cyclictest0-21swapper/207:30:152
832299118,2cyclictest28508-21cat10:35:182
832299118,2cyclictest2562-21sendmail_mailtr11:45:182
832299118,2cyclictest12400-21cat11:05:182
832299118,2cyclictest1090-21dbus-daemon09:50:012
832299117,3cyclictest1132-21snmpd09:37:432
832299117,3cyclictest1132-21snmpd09:37:432
832299116,2cyclictest0-21swapper/210:34:202
832299115,5cyclictest0-21swapper/210:03:372
832299115,5cyclictest0-21swapper/209:26:022
832299115,5cyclictest0-21swapper/209:02:382
832299112,8cyclictest0-21swapper/208:46:232
8322991110,1cyclictest0-21swapper/212:20:082
832299110,1cyclictest0-21swapper/211:51:472
832299110,1cyclictest0-21swapper/211:22:382
832299110,1cyclictest0-21swapper/208:32:482
831999118,2cyclictest1069-17sedispatch07:15:011
831999117,3cyclictest1132-21snmpd07:21:541
831999115,5cyclictest0-21swapper/107:44:581
831999110,11cyclictest0-21swapper/112:01:041
831999110,11cyclictest0-21swapper/111:45:321
831999110,1cyclictest0-21swapper/107:48:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional