You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 21:26
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Fri Nov 28, 2025 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3445394996156,4cyclictest3812395-21latency_hist18:23:077
344499226151,7sleep40-21swapper/415:21:494
344486925623,29sleep70-21swapper/715:20:077
3445388995249,2cyclictest3751682-21kworker/u16:0+flush-8:018:48:075
344502425242,7sleep60-21swapper/615:22:156
344496325141,7sleep50-21swapper/515:21:265
3445383995046,3cyclictest3691715-21kworker/u16:3+flush-8:019:48:224
3445394994136,4cyclictest3539897-21latency_hist16:08:077
3445388994139,2cyclictest3981081-21kworker/u16:0+flush-8:020:08:075
3445388994139,2cyclictest3981081-21kworker/u16:0+flush-8:020:08:075
3445394994035,4cyclictest3862754-21latency_hist18:48:077
3445388993735,2cyclictest2778414-21kworker/u16:7+flush-8:015:53:355
3445388993634,2cyclictest3953781-21kworker/u16:5+events_unbound19:43:365
3445389993432,2cyclictest2778414-21kworker/u16:7+flush-8:015:53:396
3445394993227,4cyclictest3792234-21latency_hist18:13:077
3445388993230,2cyclictest3981081-21kworker/u16:0+flush-8:020:13:295
3445383993127,3cyclictest3539649-21kworker/u16:2+flush-8:016:08:184
3445394993025,4cyclictest3923152-21latency_hist19:18:077
3445388993027,2cyclictest3943279-21kworker/u16:2+events_unbound19:28:255
3445389992927,2cyclictest3640651-21kworker/u16:5+events_unbound17:03:356
3445388992927,2cyclictest3691715-21kworker/u16:3+events_unbound17:38:165
3445388992927,2cyclictest3640651-21kworker/u16:5+events_unbound17:13:525
3445388992926,2cyclictest3415314-21kworker/u16:0+flush-8:015:25:195
3445394992823,4cyclictest3661008-21latency_hist17:08:077
3445383992826,2cyclictest3691715-21kworker/u16:3+flush-8:018:33:314
3445383992826,2cyclictest3691715-21kworker/u16:3+flush-8:018:33:314
3445394992725,1cyclictest0-21swapper/717:53:357
3445388992725,2cyclictest3981081-21kworker/u16:0+events_unbound20:28:355
3445388992725,2cyclictest3691715-21kworker/u16:3+events_unbound19:23:195
3445388992725,2cyclictest3404956-21kworker/u16:1+flush-8:015:28:125
3445388992724,2cyclictest3404956-21kworker/u16:1+events_unbound16:03:215
3445394992621,4cyclictest3549996-21latency_hist16:13:077
344539499260,24cyclictest3741804-21latency_hist17:48:077
3445388992623,2cyclictest3893276-21kworker/u16:4+flush-8:019:08:245
3445389992522,2cyclictest3691715-21kworker/u16:3+flush-8:017:48:326
3445388992523,2cyclictest3751682-21kworker/u16:0+events_unbound18:30:485
3445388992522,2cyclictest3751682-21kworker/u16:0+flush-8:018:08:515
3445394992423,0cyclictest0-21swapper/716:18:227
3445389992422,2cyclictest3751682-21kworker/u16:0+events_unbound18:43:326
3445389992421,3cyclictest3893276-21kworker/u16:4+flush-8:019:08:356
3445388992422,2cyclictest4004107-21kworker/u16:1+events_unbound19:58:315
3445388992422,2cyclictest3802692-21kworker/u16:2+events_unbound18:26:485
3445388992422,2cyclictest3691715-21kworker/u16:3+events_unbound18:20:485
3445388992422,2cyclictest3640639-21kworker/u16:3+flush-8:017:08:255
3445388992422,2cyclictest3562072-21kworker/u16:1+events_unbound16:35:485
3445388992422,2cyclictest3469028-21kworker/u16:3+flush-8:015:48:245
3445394992317,5cyclictest3469370-21latency_hist15:33:077
3445389992321,2cyclictest3832408-21kworker/u16:1+flush-8:018:38:326
3445389992321,2cyclictest3404956-21kworker/u16:1+events_unbound15:58:366
3445388992320,2cyclictest3691715-21kworker/u16:3+events_unbound19:33:225
3445388992220,2cyclictest3943279-21kworker/u16:2+events_unbound19:53:285
3445388992220,2cyclictest3600753-21kworker/u16:4+events_unbound16:38:305
3445388992220,2cyclictest3580134-21kworker/u16:2+flush-8:016:51:175
3445389992118,2cyclictest3691715-21kworker/u16:3+events_unbound19:38:256
3445388992119,2cyclictest3691715-21kworker/u16:3+events_unbound18:34:475
3445388992119,2cyclictest3691715-21kworker/u16:3+events_unbound18:34:475
3445388992119,2cyclictest3469028-21kworker/u16:3+flush-8:016:08:345
3445388992118,2cyclictest3539649-21kworker/u16:2+flush-8:016:13:355
3445388992118,2cyclictest2778414-21kworker/u16:7+flush-8:016:03:085
3445394992015,4cyclictest4114683-21latency_hist20:53:077
3445394992015,4cyclictest4114683-21latency_hist20:53:077
344539499200,18cyclictest4064307-21latency_hist20:28:077
3445389992017,2cyclictest3469028-21kworker/u16:3+flush-8:015:40:076
3445388992018,2cyclictest4075579-21kworker/u16:2+flush-8:020:48:025
3445388992018,2cyclictest3832408-21kworker/u16:1+events_unbound18:55:475
3445388992018,2cyclictest3691715-21kworker/u16:3+events_unbound20:18:315
3445388992018,2cyclictest3549756-21kworker/u16:0+events_unbound16:25:485
3445389991916,2cyclictest3981081-21kworker/u16:0+flush-8:020:33:316
3445389991915,3cyclictest3549756-21kworker/u16:0+flush-8:016:13:316
3445388991917,2cyclictest4054190-21kworker/u16:1+flush-8:020:23:335
3445388991917,2cyclictest2778414-21kworker/u16:7+events_unbound15:33:275
3445389991816,2cyclictest3562072-21kworker/u16:1+flush-8:016:48:076
3445389991815,2cyclictest3691715-21kworker/u16:3+flush-8:017:33:316
3445389991814,3cyclictest3882748-21kworker/u16:2+flush-8:018:58:326
3445388991816,2cyclictest3882748-21kworker/u16:2+flush-8:018:58:265
3445388991816,2cyclictest3832408-21kworker/u16:1+events_unbound19:18:315
3445388991816,2cyclictest3832408-21kworker/u16:1+events_unbound19:18:315
3445388991816,2cyclictest3681020-21kworker/u16:1+flush-8:017:18:335
3445388991816,2cyclictest3640651-21kworker/u16:5+events_unbound17:00:475
3445388991816,2cyclictest3600753-21kworker/u16:4+flush-8:016:53:335
3445388991816,2cyclictest3562072-21kworker/u16:1+events_unbound16:43:485
3445388991815,2cyclictest4075579-21kworker/u16:2+flush-8:020:38:195
3445394991714,2cyclictest0-21swapper/719:53:077
3445394991714,2cyclictest0-21swapper/718:23:077
3445394991712,4cyclictest3913064-21latency_hist19:13:077
3445394991711,5cyclictest3590305-21latency_hist16:33:087
344539499170,15cyclictest3973656-21latency_hist19:43:077
3445389991715,2cyclictest3580134-21kworker/u16:2+events_unbound17:53:316
3445389991715,2cyclictest3469028-21kworker/u16:3+events_unbound15:48:226
3445388991715,2cyclictest3792063-21kworker/u16:1+flush-8:018:13:315
3445388991715,2cyclictest3701118-21kworker/u16:4+events_unbound17:50:485
3445388991715,2cyclictest3681020-21kworker/u16:1+flush-8:017:23:335
344539499168,7cyclictest27924-21gmain15:43:317
344538999160,1cyclictest0-21swapper/619:58:336
3445388991614,2cyclictest3832408-21kworker/u16:1+events_unbound18:48:335
3445388991614,2cyclictest3580134-21kworker/u16:2+flush-8:017:03:385
3445388991613,2cyclictest3691715-21kworker/u16:3+flush-8:017:28:235
3445388991613,2cyclictest3469028-21kworker/u16:3+flush-8:015:38:235
344538399160,1cyclictest4093801-21switchtime20:38:324
3445394991513,1cyclictest0-21swapper/717:28:257
3445394991512,2cyclictest4175659-21snmpd19:26:117
3445394991512,2cyclictest4175659-21snmpd16:14:147
3445394991511,3cyclictest4175659-21snmpd15:54:597
344539499150,5cyclictest3501247-21kthreadcore15:48:257
344539499150,14cyclictest4175659-21snmpd17:41:537
344539499150,13cyclictest4014121-21cut20:03:077
344539499150,13cyclictest0-21swapper/718:58:327
3445389991513,2cyclictest3943279-21kworker/u16:2+flush-8:019:28:536
3445389991513,2cyclictest3751682-21kworker/u16:0+events_unbound18:08:336
3445389991512,2cyclictest3404956-21kworker/u16:1+flush-8:015:28:336
3445388991513,2cyclictest4104366-21kworker/u16:0+flush-8:020:48:285
3445388991513,2cyclictest4104366-21kworker/u16:0+flush-8:020:48:285
3445388991513,2cyclictest3469028-21kworker/u16:3+flush-8:016:18:485
344539499149,4cyclictest3852704-21latency_hist18:43:077
344539499146,7cyclictest0-21swapper/716:47:197
344539499145,7cyclictest27763-21gvfs-afc-volume19:48:327
3445394991411,2cyclictest4175659-21snmpd20:13:007
3445394991411,2cyclictest0-21swapper/718:38:077
3445394991411,2cyclictest0-21swapper/718:38:077
3445394991411,2cyclictest0-21swapper/716:53:077
3445394991410,3cyclictest4175659-21snmpd20:42:337
3445394991410,3cyclictest3782208-21idleruntime-cro18:08:077
344539499140,1cyclictest0-21swapper/718:13:327
344539499140,12cyclictest4064703-21/usr/sbin/munin20:28:307
3445389991413,1cyclictest0-21swapper/620:23:346
3445389991412,2cyclictest3691715-21kworker/u16:3+events_unbound19:33:306
3445389991410,3cyclictest3691715-21kworker/u16:3+flush-8:019:23:366
344538999140,1cyclictest0-21swapper/616:53:316
344538999140,1cyclictest0-21swapper/616:03:086
344538999140,13cyclictest0-21swapper/615:33:216
3445388991412,2cyclictest3751682-21kworker/u16:0+flush-8:019:03:385
3445388991412,2cyclictest3691715-21kworker/u16:3+events_unbound17:44:475
3445388991412,2cyclictest3562072-21kworker/u16:1+events_unbound16:29:485
3445388991410,3cyclictest3751682-21kworker/u16:0+flush-8:017:58:285
344538399141,6cyclictest0-21swapper/415:53:244
3445383991412,2cyclictest0-21swapper/420:12:314
344538399140,6cyclictest0-21swapper/419:03:234
344539499139,3cyclictest4175659-21snmpd19:20:117
344539499139,3cyclictest4175659-21snmpd19:20:117
344539499137,5cyclictest589235-21systemd-journal17:23:087
3445394991312,0cyclictest0-21swapper/720:13:217
3445394991310,2cyclictest4175659-21snmpd16:00:417
344539499130,13cyclictest0-21swapper/720:03:307
344539499130,13cyclictest0-21swapper/720:03:307
344539499130,11cyclictest3899501-21cut19:03:287
3445389991310,2cyclictest4050724-21kthreadcore20:18:296
3445389991310,2cyclictest3691715-21kworker/u16:3+flush-8:018:03:336
3445389991310,2cyclictest3561434-21irqcore16:18:236
3445388991310,2cyclictest3953781-21kworker/u16:5+flush-8:019:48:205
344538399138,4cyclictest3531032-21cat16:03:244
344538399135,7cyclictest4095684-21cat20:43:254
3445383991312,1cyclictest0-21swapper/419:08:414
3445383991312,1cyclictest0-21swapper/418:43:084
3445383991312,1cyclictest0-21swapper/417:28:154
3445383991312,1cyclictest0-21swapper/417:24:474
3445383991312,1cyclictest0-21swapper/416:41:174
3445383991311,2cyclictest3640651-21kworker/u16:5+flush-8:017:13:224
344538399130,13cyclictest0-21swapper/420:05:164
344538399130,13cyclictest0-21swapper/420:05:164
344538399130,13cyclictest0-21swapper/418:39:504
344538399130,12cyclictest36000962sleep416:34:404
344538399130,12cyclictest0-21swapper/420:18:124
344539499129,2cyclictest4175659-21snmpd19:43:597
344539499129,2cyclictest4175659-21snmpd17:13:077
344539499129,2cyclictest4175659-21snmpd15:27:377
344539499124,7cyclictest27904-21gmain15:33:317
344539499124,6cyclictest27082-21ssh-agent15:38:307
344539499120,11cyclictest4175659-21snmpd18:01:477
344539499120,11cyclictest3691449-21cat17:23:087
344538999120,1cyclictest0-21swapper/615:26:536
344538899129,2cyclictest3751682-21kworker/u16:0+flush-8:018:38:255
344538899129,2cyclictest2778414-21kworker/u16:7+flush-8:015:45:185
3445388991211,1cyclictest0-21swapper/518:04:425
344538899120,12cyclictest0-21swapper/517:56:115
344538399128,3cyclictest3562072-21kworker/u16:1+flush-8:016:28:364
344538399124,7cyclictest3690521-21sendmail_mailst17:18:354
344538399124,7cyclictest3509433-21chrt15:49:204
344538399124,7cyclictest3497379-21cut15:43:284
344538399124,7cyclictest3465299-21kthreadcore15:28:264
344538399123,8cyclictest3763106-21fschecks_time17:58:224
344538399123,8cyclictest3734836-21cut17:43:254
344538399123,8cyclictest3664827-21cut17:08:254
344538399123,8cyclictest3576851-21cut16:23:254
344538399122,5cyclictest0-21swapper/417:53:264
3445383991211,1cyclictest0-21swapper/417:48:554
3445383991211,1cyclictest0-21swapper/417:02:344
3445383991211,1cyclictest0-21swapper/416:02:114
3445383991210,2cyclictest0-21swapper/419:25:334
3445383991210,2cyclictest0-21swapper/418:57:104
3445383991210,2cyclictest0-21swapper/417:37:434
3445383991210,1cyclictest0-21swapper/416:13:354
344538399120,4cyclictest3923127-21taskset19:17:534
344538399120,12cyclictest0-21swapper/419:37:414
344538399120,12cyclictest0-21swapper/416:48:514
344539499118,2cyclictest4175659-21snmpd20:47:427
344539499118,2cyclictest4175659-21snmpd20:35:307
344539499118,2cyclictest4175659-21snmpd18:31:217
344539499118,2cyclictest4175659-21snmpd17:01:347
344539499117,3cyclictest27763-21gvfs-afc-volume18:48:177
344539499114,6cyclictest0-21swapper/718:57:467
344539499114,6cyclictest0-21swapper/716:50:447
344539499111,9cyclictest3600368-21munin-run16:38:077
344539499110,9cyclictest0-21swapper/716:23:287
344539499110,10cyclictest4175659-21snmpd17:33:567
344539499110,10cyclictest3751583-21sshd-session17:48:477
344538999118,2cyclictest2778414-21kworker/u16:7+flush-8:016:08:256
344538999117,3cyclictest3894065-21cat19:03:236
3445389991110,1cyclictest0-21swapper/620:17:546
3445389991110,1cyclictest0-21swapper/618:13:126
3445389991110,1cyclictest0-21swapper/616:24:356
344538999110,11cyclictest0-21swapper/618:34:106
344538999110,11cyclictest0-21swapper/618:34:106
344538999110,11cyclictest0-21swapper/618:20:466
344538999110,0cyclictest0-21swapper/619:47:286
344538999110,0cyclictest0-21swapper/617:10:276
344538899119,2cyclictest3981081-21kworker/u16:0+flush-8:020:33:285
344538899119,2cyclictest3681020-21kworker/u16:1+events_unbound17:33:235
3445388991110,1cyclictest0-21swapper/519:42:555
344538399117,3cyclictest3974827-21cat19:43:224
344538399117,3cyclictest3446426-21cat15:23:234
344538399114,6cyclictest4064703-21/usr/sbin/munin20:28:214
344538399114,6cyclictest4034837-21cat20:13:194
344538399113,7cyclictest4111594-21grep20:48:294
344538399113,7cyclictest4111594-21grep20:48:294
344538399113,7cyclictest3968531-21grep19:38:284
344538399113,7cyclictest3889213-21cut18:58:284
344538399113,7cyclictest3784813-21kthreadcore18:08:264
344538399113,7cyclictest3724361-21cut17:38:244
344538399113,7cyclictest3612499-21cut16:43:234
344538399112,8cyclictest4079058-21cut20:33:264
344538399112,8cyclictest3996945-21cut19:53:264
344538399112,8cyclictest3944896-21cut19:28:234
344538399112,8cyclictest3825579-21cat18:28:244
344538399112,8cyclictest3794857-21cut18:13:254
344538399112,8cyclictest3483124-21cut15:38:254
344538399112,8cyclictest3474313-21cut15:33:284
344538399112,7cyclictest3560585-21cpuspeed_turbos16:18:194
344538399111,4cyclictest3818435-21kthreadcore18:23:284
344538399110,4cyclictest4013883-21chrt20:02:004
344538399110,4cyclictest3923536-21chrt19:18:094
344538399110,4cyclictest3923536-21chrt19:18:094
344538399110,4cyclictest3812164-21taskset18:20:004
344538399110,4cyclictest37762921grep18:03:264
344538399110,4cyclictest3640546-21taskset16:55:474
344539499107,2cyclictest4175659-21snmpd16:42:307
344539499100,9cyclictest3678754-21kthreadcore17:13:307
344539499100,2cyclictest0-21swapper/719:28:317
344538999109,1cyclictest0-21swapper/620:32:246
344538999109,1cyclictest0-21swapper/619:53:096
344538999109,1cyclictest0-21swapper/617:26:326
344538999109,1cyclictest0-21swapper/617:20:336
344538999109,1cyclictest0-21swapper/616:36:536
344538999109,1cyclictest0-21swapper/616:32:476
344538999108,2cyclictest3832408-21kworker/u16:1+events_unbound18:53:326
344538999106,3cyclictest3641924-21cat16:58:226
344538399102,7cyclictest3651427-21cpuspeed_turbos17:03:204
344538399100,4cyclictest3863461-21cat18:48:174
344538399100,3cyclictest4055117-21expr20:23:194
34453949996,2cyclictest0-21swapper/719:33:307
34453949994,4cyclictest4048059-21cat20:18:277
34453899998,1cyclictest0-21swapper/620:47:416
34453899998,1cyclictest0-21swapper/617:38:316
34453899998,1cyclictest0-21swapper/617:28:166
34453899998,1cyclictest0-21swapper/616:40:596
34453899995,3cyclictest4025195-21cat20:08:216
34453889997,2cyclictest3981081-21kworker/u16:0+flush-8:020:08:225
34453889997,2cyclictest3691715-21kworker/u16:3+events_unbound19:15:475
34453899987,1cyclictest0-21swapper/617:45:186
34453899987,1cyclictest0-21swapper/615:44:136
34453899985,2cyclictest0-21swapper/618:23:256
34453899984,3cyclictest3630276-21users16:48:346
34453899983,4cyclictest4105993-21sed20:48:256
34453899983,4cyclictest4105993-21sed20:48:256
34453899983,4cyclictest4023567-21grep20:03:356
34453899983,4cyclictest4023567-21grep20:03:356
34453899983,4cyclictest3932427-21sendmail_mailqu19:18:346
34453899983,4cyclictest3932427-21sendmail_mailqu19:18:346
34453899983,4cyclictest3672349-21cat17:13:236
34453899975,1cyclictest0-21swapper/619:48:256
34453899975,1cyclictest0-21swapper/619:13:216
34453899974,2cyclictest3863306-21cpuspeed_turbos18:48:176
34453899973,3cyclictest3823048-21grep18:28:186
34453899973,3cyclictest3762690-21cat17:58:206
34453899970,6cyclictest0-21swapper/620:38:186
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional