You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-10 - 18:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Thu Jul 10, 2025 12:50:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
495229488,3sleep50-21swapper/507:05:135
469229075,10sleep30-21swapper/307:05:123
106092860,0sleep20-21swapper/208:05:162
725928249,29sleep70-21swapper/707:09:417
727628147,29sleep60-21swapper/607:09:566
698528066,10sleep40-21swapper/407:05:444
714927743,30sleep10-21swapper/107:08:061
701127742,30sleep20-21swapper/207:06:072
700927642,30sleep00-21swapper/007:06:040
243122530,0sleep50-21swapper/511:19:105
141822490,0sleep40-21swapper/412:00:324
56862480,0sleep00-21swapper/010:22:290
168812480,0sleep50-21swapper/508:15:195
97612440,0sleep20-21swapper/211:11:232
69352320,0sleep30-21swapper/309:20:193
7715992929,0cyclictest0-21swapper/412:29:004
7718992510,6cyclictest0-21swapper/708:50:217
7711992525,0cyclictest0-21swapper/011:11:270
7716992424,0cyclictest0-21swapper/510:48:075
7718992216,6cyclictest10875-21munin-node11:23:157
7715992222,0cyclictest69450irq/124-eth011:51:124
771399220,0cyclictest0-21swapper/210:37:402
7714992112,2cyclictest121rcu_preempt12:01:163
7715992020,0cyclictest69450irq/124-eth011:43:404
7714992020,0cyclictest25557-21ssh09:30:043
7712992020,0cyclictest0-21swapper/108:50:181
7711992020,0cyclictest0-21swapper/011:54:410
7711992011,4cyclictest0-21swapper/011:50:000
7711991919,0cyclictest0-21swapper/008:50:170
7717991810,3cyclictest0-21swapper/610:34:336
7717991810,3cyclictest0-21swapper/610:21:066
7715991818,0cyclictest69450irq/124-eth012:35:434
771499182,12cyclictest5324-21ssh12:12:453
7714991810,1cyclictest121rcu_preempt10:05:193
7711991818,0cyclictest0-21swapper/012:10:190
7711991810,0cyclictest0-21swapper/009:38:170
7718991717,0cyclictest0-21swapper/711:41:057
7718991715,1cyclictest721ktimersoftd/707:35:147
7718991712,1cyclictest73-21ksoftirqd/707:15:017
7717991710,4cyclictest0-21swapper/610:55:016
7717991710,3cyclictest0-21swapper/610:36:426
7714991713,4cyclictest1559-21snmpd10:10:423
7714991711,4cyclictest121rcu_preempt12:29:023
7714991710,5cyclictest121rcu_preempt12:20:163
7714991710,5cyclictest121rcu_preempt11:37:563
7714991710,4cyclictest121rcu_preempt11:32:513
7714991710,3cyclictest0-21swapper/310:25:513
7714991710,3cyclictest0-21swapper/310:00:193
7714991710,2cyclictest121rcu_preempt11:50:113
7712991712,5cyclictest6004-21kworker/u16:112:00:161
7712991710,0cyclictest0-21swapper/109:49:101
651621710,3sleep40-21swapper/410:55:014
2949621710,6sleep60-21swapper/611:52:506
7718991615,1cyclictest6075-21ssh09:20:137
7718991615,1cyclictest20524-21ssh09:58:457
7718991615,1cyclictest1082-21ssh11:07:267
7718991610,0cyclictest0-21swapper/712:30:557
7718991610,0cyclictest0-21swapper/712:20:587
7717991610,6cyclictest0-21swapper/612:35:216
7717991610,6cyclictest0-21swapper/611:05:186
7717991610,5cyclictest0-21swapper/611:38:556
7717991610,4cyclictest0-21swapper/611:26:536
7717991610,3cyclictest0-21swapper/611:20:346
7717991610,3cyclictest0-21swapper/611:02:146
7717991610,3cyclictest0-21swapper/609:55:226
7717991610,3cyclictest0-21swapper/609:35:216
7717991610,3cyclictest0-21swapper/609:20:296
7717991610,2cyclictest0-21swapper/612:18:176
7717991610,1cyclictest0-21swapper/610:50:206
7717991610,1cyclictest0-21swapper/609:12:146
7716991615,1cyclictest31898-21ssh11:22:175
7716991610,4cyclictest0-21swapper/507:10:125
7716991610,2cyclictest0-21swapper/509:23:295
7716991610,0cyclictest0-21swapper/511:55:055
7716991610,0cyclictest0-21swapper/511:48:205
7715991615,1cyclictest69450irq/124-eth010:26:124
7715991615,1cyclictest641-21nscd07:45:144
7715991615,0cyclictest1642-21ntpd09:11:024
7715991613,3cyclictest69450irq/124-eth010:40:344
7715991612,3cyclictest662-21ptp4l10:01:124
7715991610,4cyclictest0-21swapper/412:13:444
7715991610,3cyclictest0-21swapper/410:17:444
7715991610,2cyclictest0-21swapper/411:57:414
7715991610,0cyclictest0-21swapper/410:32:284
7715991610,0cyclictest0-21swapper/409:15:294
7715991610,0cyclictest0-21swapper/408:59:544
7714991615,1cyclictest13640-21kworker/u16:011:10:113
7714991612,1cyclictest121rcu_preempt12:34:023
7714991611,2cyclictest121rcu_preempt11:45:063
7714991611,2cyclictest121rcu_preempt10:50:133
7714991611,2cyclictest121rcu_preempt09:55:193
7714991610,4cyclictest121rcu_preempt07:40:123
7714991610,3cyclictest13640-21kworker/u16:011:07:483
7714991610,3cyclictest0-21swapper/312:35:553
7714991610,2cyclictest121rcu_preempt12:17:273
7714991610,2cyclictest121rcu_preempt12:05:213
7714991610,2cyclictest121rcu_preempt11:59:363
7714991610,2cyclictest121rcu_preempt11:26:263
7714991610,2cyclictest121rcu_preempt11:20:183
7714991610,2cyclictest121rcu_preempt11:15:523
7714991610,2cyclictest121rcu_preempt11:03:543
7714991610,2cyclictest121rcu_preempt10:48:353
7714991610,2cyclictest121rcu_preempt10:43:343
7714991610,2cyclictest121rcu_preempt10:35:163
7714991610,2cyclictest121rcu_preempt10:20:153
7714991610,2cyclictest121rcu_preempt09:40:303
7714991610,2cyclictest121rcu_preempt09:26:463
7714991610,2cyclictest121rcu_preempt09:16:363
7714991610,2cyclictest121rcu_preempt09:10:033
7714991610,2cyclictest121rcu_preempt08:45:133
7714991610,2cyclictest121rcu_preempt08:15:143
7714991610,2cyclictest121rcu_preempt08:05:203
7714991610,2cyclictest121rcu_preempt07:25:213
7714991610,2cyclictest0-21swapper/310:32:443
7714991610,2cyclictest0-21swapper/310:16:473
7714991610,0cyclictest0-21swapper/310:55:213
7714991610,0cyclictest0-21swapper/308:55:233
7713991610,4cyclictest0-21swapper/212:05:182
7713991610,3cyclictest0-21swapper/208:50:222
7711991614,1cyclictest30142-21sh10:03:160
2656521610,5sleep626591-21ssh10:00:576
1974721610,0sleep20-21swapper/212:04:162
771899157,6cyclictest121rcu_preempt12:05:437
7718991514,1cyclictest18067-21ntpq09:10:177
7718991514,1cyclictest13419-21ssh10:25:477
7718991514,0cyclictest31051-21ls12:25:167
7718991513,1cyclictest721ktimersoftd/711:35:147
7718991513,1cyclictest721ktimersoftd/707:15:247
7718991510,0cyclictest0-21swapper/711:27:457
7717991510,5cyclictest0-21swapper/612:20:526
7717991510,5cyclictest0-21swapper/612:11:046
7717991510,5cyclictest0-21swapper/611:46:586
7717991510,5cyclictest0-21swapper/610:26:406
7717991510,3cyclictest0-21swapper/612:26:406
7717991510,3cyclictest0-21swapper/611:16:096
7717991510,3cyclictest0-21swapper/609:48:556
7717991510,2cyclictest0-21swapper/612:30:276
7717991510,2cyclictest0-21swapper/611:57:046
7717991510,2cyclictest0-21swapper/608:50:206
7717991510,1cyclictest0-21swapper/612:08:086
7716991513,1cyclictest6828-21sh11:41:255
7716991512,3cyclictest18400-21nscd10:20:405
7716991510,5cyclictest0-21swapper/509:41:135
7716991510,3cyclictest0-21swapper/511:55:005
7716991510,3cyclictest0-21swapper/508:50:225
7716991510,2cyclictest0-21swapper/512:28:255
7716991510,1cyclictest0-21swapper/511:30:135
7716991510,0cyclictest0-21swapper/512:39:085
7716991510,0cyclictest0-21swapper/512:34:595
7716991510,0cyclictest0-21swapper/512:17:405
7716991510,0cyclictest0-21swapper/512:09:265
7716991510,0cyclictest0-21swapper/511:36:065
7716991510,0cyclictest0-21swapper/509:10:155
7715991515,0cyclictest1642-21ntpd09:40:084
7715991514,1cyclictest0-21swapper/407:27:414
7715991510,4cyclictest0-21swapper/412:21:154
7715991510,4cyclictest0-21swapper/411:37:224
7715991510,3cyclictest0-21swapper/411:45:214
7715991510,3cyclictest0-21swapper/411:22:194
7715991510,3cyclictest0-21swapper/411:17:144
7715991510,3cyclictest0-21swapper/411:01:174
7715991510,3cyclictest0-21swapper/407:23:184
7715991510,2cyclictest0-21swapper/407:57:234
7715991510,1cyclictest69450irq/124-eth011:06:054
7715991510,1cyclictest0-21swapper/410:07:104
7715991510,1cyclictest0-21swapper/409:39:454
7715991510,0cyclictest0-21swapper/412:30:524
7714991515,0cyclictest0-21swapper/309:45:133
7714991510,5cyclictest0-21swapper/309:01:583
7714991510,4cyclictest121rcu_preempt08:00:123
7714991510,3cyclictest0-21swapper/308:30:123
7714991510,2cyclictest5628-21kworker/u16:311:40:523
7714991510,2cyclictest121rcu_preempt09:35:163
7714991510,2cyclictest121rcu_preempt08:40:183
7714991510,2cyclictest121rcu_preempt08:35:133
7714991510,2cyclictest121rcu_preempt08:25:163
7714991510,2cyclictest121rcu_preempt08:10:143
7714991510,2cyclictest121rcu_preempt07:45:123
7714991510,2cyclictest121rcu_preempt07:35:153
7714991510,2cyclictest121rcu_preempt07:20:133
7714991510,2cyclictest121rcu_preempt07:15:153
7714991510,2cyclictest0-21swapper/309:52:043
7714991510,2cyclictest0-21swapper/308:50:203
7714991510,1cyclictest121rcu_preempt07:35:013
7713991514,1cyclictest2642-21ssh11:24:202
7713991510,5cyclictest0-21swapper/212:19:332
7713991510,5cyclictest0-21swapper/210:00:042
7713991510,0cyclictest0-21swapper/212:38:342
7712991514,1cyclictest1642-21ntpd11:24:451
7712991510,5cyclictest0-21swapper/111:49:421
7712991510,5cyclictest0-21swapper/111:37:351
7712991510,3cyclictest0-21swapper/112:10:181
7712991510,1cyclictest0-21swapper/108:55:131
7712991510,0cyclictest0-21swapper/112:06:051
7712991510,0cyclictest0-21swapper/111:51:131
7712991510,0cyclictest0-21swapper/111:17:141
7712991510,0cyclictest0-21swapper/111:10:441
7712991510,0cyclictest0-21swapper/110:51:461
7712991510,0cyclictest0-21swapper/109:55:121
7711991514,1cyclictest11007-21awk12:15:170
7711991510,5cyclictest0-21swapper/012:30:090
7711991510,5cyclictest0-21swapper/011:09:030
7711991510,5cyclictest0-21swapper/009:55:200
7711991510,1cyclictest0-21swapper/012:04:060
7711991510,0cyclictest0-21swapper/012:37:470
7711991510,0cyclictest0-21swapper/011:34:340
7711991510,0cyclictest0-21swapper/011:19:040
7711991510,0cyclictest0-21swapper/010:50:540
7711991510,0cyclictest0-21swapper/010:44:240
7711991510,0cyclictest0-21swapper/010:27:360
7711991510,0cyclictest0-21swapper/009:15:290
257032150,0sleep10-21swapper/112:39:131
2453421510,1sleep00-21swapper/012:05:420
771899149,1cyclictest73-21ksoftirqd/712:04:507
771899145,7cyclictest73-21ksoftirqd/712:35:117
771899144,5cyclictest121rcu_preempt10:38:287
7718991413,1cyclictest9188-21systemctl10:55:197
7718991413,1cyclictest721ktimersoftd/710:05:187
7718991413,1cyclictest721ktimersoftd/709:45:147
7718991413,1cyclictest721ktimersoftd/709:34:017
7718991413,1cyclictest27327-21ssh10:49:097
7718991413,1cyclictest26154-21ssh11:04:217
7718991413,1cyclictest23838-21ssh10:30:537
7718991413,1cyclictest15739-21ssh12:17:577
7718991412,1cyclictest721ktimersoftd/710:17:577
7718991412,1cyclictest21789-21sh09:43:397
7718991410,4cyclictest0-21swapper/711:59:027
7718991410,4cyclictest0-21swapper/711:33:177
7718991410,1cyclictest721ktimersoftd/710:52:507
7718991410,0cyclictest0-21swapper/710:40:447
7718991410,0cyclictest0-21swapper/709:50:087
7718991410,0cyclictest0-21swapper/709:35:117
7718991410,0cyclictest0-21swapper/709:15:177
7717991410,4cyclictest0-21swapper/611:44:496
7717991410,4cyclictest0-21swapper/609:40:376
7717991410,4cyclictest0-21swapper/609:30:526
7717991410,4cyclictest0-21swapper/609:28:136
7717991410,4cyclictest0-21swapper/609:15:416
7717991410,2cyclictest0-21swapper/612:00:196
7717991410,1cyclictest0-21swapper/608:40:216
7717991410,1cyclictest0-21swapper/607:15:176
7717991410,0cyclictest0-21swapper/611:10:136
7717991410,0cyclictest0-21swapper/610:43:446
7717991410,0cyclictest0-21swapper/609:52:546
7717991410,0cyclictest0-21swapper/609:00:216
7716991414,0cyclictest0-21swapper/509:30:095
7716991413,1cyclictest14465-21ls09:05:165
7716991412,1cyclictest641-21nscd10:10:015
7716991412,1cyclictest15733-21diskmemload10:39:545
7716991410,4cyclictest0-21swapper/511:05:205
7716991410,4cyclictest0-21swapper/511:01:345
7716991410,0cyclictest0-21swapper/512:01:315
7716991410,0cyclictest0-21swapper/511:25:455
7716991410,0cyclictest0-21swapper/511:14:005
7716991410,0cyclictest0-21swapper/510:55:165
7716991410,0cyclictest0-21swapper/510:51:235
7716991410,0cyclictest0-21swapper/510:42:165
7716991410,0cyclictest0-21swapper/510:26:015
7716991410,0cyclictest0-21swapper/510:05:205
7716991410,0cyclictest0-21swapper/509:55:015
7716991410,0cyclictest0-21swapper/509:45:545
7716991410,0cyclictest0-21swapper/509:37:105
7716991410,0cyclictest0-21swapper/508:04:215
7715991413,0cyclictest0-21swapper/408:35:124
7715991410,4cyclictest69450irq/124-eth010:57:334
7715991410,4cyclictest0-21swapper/412:19:524
7715991410,4cyclictest0-21swapper/412:05:564
7715991410,4cyclictest0-21swapper/411:10:034
7715991410,4cyclictest0-21swapper/409:52:204
7715991410,3cyclictest0-21swapper/411:25:404
7715991410,3cyclictest0-21swapper/410:20:014
7715991410,3cyclictest0-21swapper/410:13:004
7715991410,3cyclictest0-21swapper/409:28:504
7715991410,2cyclictest0-21swapper/408:50:184
7715991410,0cyclictest0-21swapper/410:35:024
7715991410,0cyclictest0-21swapper/409:45:084
7715991410,0cyclictest0-21swapper/409:30:024
7714991410,2cyclictest121rcu_preempt07:10:183
7714991410,1cyclictest121rcu_preempt09:10:003
7714991410,1cyclictest121rcu_preempt08:20:503
7714991410,1cyclictest121rcu_preempt07:55:233
7713991413,1cyclictest989-21ssh09:49:292
7713991413,1cyclictest3739-21ssh09:19:522
7713991413,1cyclictest26797-21ssh11:51:032
7713991413,1cyclictest22783-21ssh09:12:562
7713991413,1cyclictest21715-21ssh09:27:502
7713991413,1cyclictest14274-21ls09:55:172
7713991413,0cyclictest22687-21ls10:30:222
7713991410,4cyclictest0-21swapper/211:45:152
7713991410,1cyclictest0-21swapper/207:10:132
7713991410,0cyclictest0-21swapper/212:11:012
7713991410,0cyclictest0-21swapper/210:56:052
7713991410,0cyclictest0-21swapper/210:53:182
7712991414,0cyclictest0-21swapper/109:32:001
7712991412,1cyclictest641-21nscd09:43:091
7712991412,0cyclictest99-21kswapd010:25:181
7712991410,4cyclictest0-21swapper/112:30:261
7712991410,4cyclictest0-21swapper/111:30:431
7712991410,4cyclictest0-21swapper/110:23:161
7712991410,3cyclictest0-21swapper/107:35:121
7712991410,0cyclictest0-21swapper/112:27:011
7712991410,0cyclictest0-21swapper/112:21:321
7712991410,0cyclictest0-21swapper/112:15:011
7712991410,0cyclictest0-21swapper/111:55:181
7712991410,0cyclictest0-21swapper/111:07:011
7712991410,0cyclictest0-21swapper/111:04:061
7712991410,0cyclictest0-21swapper/110:55:181
7712991410,0cyclictest0-21swapper/110:45:171
7712991410,0cyclictest0-21swapper/110:40:241
7712991410,0cyclictest0-21swapper/110:09:531
7712991410,0cyclictest0-21swapper/110:00:401
7712991410,0cyclictest0-21swapper/107:55:181
7712991410,0cyclictest0-21swapper/107:30:141
7712991410,0cyclictest0-21swapper/107:10:001
7711991414,0cyclictest0-21swapper/011:42:590
7711991414,0cyclictest0-21swapper/009:12:070
7711991413,1cyclictest26565-21ssh10:32:380
7711991410,4cyclictest0-21swapper/012:20:480
7711991410,4cyclictest0-21swapper/009:22:040
7711991410,0cyclictest0-21swapper/011:59:070
7711991410,0cyclictest0-21swapper/011:39:270
7711991410,0cyclictest0-21swapper/011:25:180
7711991410,0cyclictest0-21swapper/011:21:370
7711991410,0cyclictest0-21swapper/010:45:000
7711991410,0cyclictest0-21swapper/010:17:420
7711991410,0cyclictest0-21swapper/010:13:310
7711991410,0cyclictest0-21swapper/009:51:150
7711991410,0cyclictest0-21swapper/009:30:080
7711991410,0cyclictest0-21swapper/009:27:240
7711991410,0cyclictest0-21swapper/007:17:230
771899136,1cyclictest27018-21turbostat08:35:017
7718991312,1cyclictest721ktimersoftd/712:10:187
7718991312,1cyclictest721ktimersoftd/711:17:057
7718991312,1cyclictest721ktimersoftd/710:10:057
7718991312,1cyclictest721ktimersoftd/709:25:227
7718991312,1cyclictest178072sleep708:19:477
7718991312,1cyclictest16933-21ssh11:46:207
7718991311,1cyclictest8072-21ls11:10:227
7718991311,1cyclictest1706-21sendmail-mta11:53:127
7718991310,0cyclictest0-21swapper/710:21:447
7718991310,0cyclictest0-21swapper/710:00:027
7718991310,0cyclictest0-21swapper/708:25:247
7717991311,1cyclictest2804-21switchtime08:45:186
7717991310,3cyclictest0-21swapper/610:46:566
7717991310,3cyclictest0-21swapper/610:10:026
7717991310,3cyclictest0-21swapper/608:55:186
7717991310,3cyclictest0-21swapper/607:35:156
7717991310,1cyclictest0-21swapper/610:16:376
7717991310,1cyclictest0-21swapper/610:05:116
7717991310,0cyclictest0-21swapper/611:30:176
7717991310,0cyclictest0-21swapper/607:40:586
7716991312,1cyclictest24403-21/usr/sbin/munin08:30:125
7716991312,1cyclictest19467-21ssh12:20:115
7716991311,0cyclictest19687-21ls07:25:225
7716991310,0cyclictest0-21swapper/512:10:065
7716991310,0cyclictest0-21swapper/510:34:275
7716991310,0cyclictest0-21swapper/510:15:375
7716991310,0cyclictest0-21swapper/510:00:445
7716991310,0cyclictest0-21swapper/509:50:325
7716991310,0cyclictest0-21swapper/509:26:395
7716991310,0cyclictest0-21swapper/509:17:305
7716991310,0cyclictest0-21swapper/509:03:185
7716991310,0cyclictest0-21swapper/508:58:285
7716991310,0cyclictest0-21swapper/508:28:435
7716991310,0cyclictest0-21swapper/508:24:015
7716991310,0cyclictest0-21swapper/508:05:015
7716991310,0cyclictest0-21swapper/507:54:055
7716991310,0cyclictest0-21swapper/507:45:575
7716991310,0cyclictest0-21swapper/507:41:225
7716991310,0cyclictest0-21swapper/507:24:575
7716991310,0cyclictest0-21swapper/507:15:185
7715991313,0cyclictest69450irq/124-eth011:30:144
7715991313,0cyclictest69450irq/124-eth007:44:434
7715991313,0cyclictest0-21swapper/410:46:224
7715991312,1cyclictest69450irq/124-eth009:59:064
7715991310,2cyclictest0-21swapper/409:05:194
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional