You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 20:28
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Tue Dec 09, 2025 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
385288621000,1sleep73852887-21sed18:32:407
346461127362,7sleep50-21swapper/515:22:095
3464949996052,7cyclictest589235-21systemd-journal17:02:237
3464949995853,4cyclictest3710692-21latency_hist17:22:237
3464949995449,4cyclictest4103713-21latency_hist20:37:237
3464949995448,5cyclictest3841582-21latency_hist18:27:237
3464949995447,5cyclictest589235-21systemd-journal18:57:237
346457824737,7sleep60-21swapper/615:21:466
3464940994341,2cyclictest3760857-21kworker/u16:2+flush-8:018:02:515
3464939994341,2cyclictest3528975-21kworker/u16:0+events_unbound15:55:544
3464939994341,2cyclictest3463840-21kworker/u16:3+events_unbound16:26:544
346494999420,40cyclictest589235-21systemd-journal19:37:227
346461924131,7sleep40-21swapper/415:22:164
34646132415,32sleep70-21swapper/715:22:117
3464940994038,2cyclictest3463840-21kworker/u16:3+flush-8:016:17:505
3464939994038,2cyclictest4043451-21kworker/u16:2+events_unbound20:10:344
3464939994038,2cyclictest3463840-21kworker/u16:3+events_unbound16:17:534
346494999390,37cyclictest3519204-21latency_hist15:47:237
3464949993833,4cyclictest3781225-21latency_hist17:57:247
3464949993833,4cyclictest3700612-21latency_hist17:17:237
3464939993836,2cyclictest3882259-21kworker/u16:0+events_unbound19:06:544
3464939993836,2cyclictest3489362-21kworker/u16:5+events_unbound15:46:564
346494999370,35cyclictest3640181-21latency_hist16:47:237
3464939993735,2cyclictest4043451-21kworker/u16:2+events_unbound20:33:344
3464939993735,2cyclictest3972431-21kworker/u16:5+events_unbound20:06:184
3464939993735,2cyclictest3740015-21kworker/u16:4+events_unbound19:10:334
3464940993634,2cyclictest3740015-21kworker/u16:4+flush-8:019:12:495
3464939993634,2cyclictest3463840-21kworker/u16:3+events_unbound17:09:584
3464949993531,3cyclictest3992921-21latency_hist19:42:237
3464940993533,2cyclictest4063860-21kworker/u16:0+events_unbound20:18:335
3464940993533,2cyclictest3489362-21kworker/u16:5+events_unbound15:43:275
3464939993533,2cyclictest4024354-21kworker/u16:0+events_unbound20:01:334
3464939993533,2cyclictest3952809-21kworker/u16:3+events_unbound20:19:554
3464939993533,2cyclictest3892365-21kworker/u16:1+events_unbound18:53:344
3464939993533,2cyclictest3760857-21kworker/u16:2+flush-8:018:05:544
3464939993533,2cyclictest3740015-21kworker/u16:4+events_unbound18:50:544
3464939993533,2cyclictest3689511-21kworker/u16:0+events_unbound17:28:344
3464939993533,2cyclictest3689511-21kworker/u16:0+events_unbound17:28:344
3464939993533,2cyclictest3475100-21kworker/u16:0+events_unbound15:32:524
3464939993533,2cyclictest3463840-21kworker/u16:3+events_unbound17:04:534
3464940993432,2cyclictest3463840-21kworker/u16:3+events_unbound15:54:265
3464940993431,3cyclictest3489362-21kworker/u16:5+flush-8:015:32:555
3464939993432,2cyclictest3882259-21kworker/u16:0+events_unbound19:21:554
3464939993432,2cyclictest3760857-21kworker/u16:2+events_unbound17:45:344
3464939993432,2cyclictest3590046-21kworker/u16:1+events_unbound16:54:534
3464939993432,2cyclictest3463840-21kworker/u16:3+flush-8:016:27:474
3464939993431,2cyclictest3730565-21kworker/u16:1+events_unbound18:17:334
3464949993327,5cyclictest589235-21systemd-journal18:47:537
3464940993331,2cyclictest3760857-21kworker/u16:2+flush-8:018:32:175
3464940993331,2cyclictest3730565-21kworker/u16:1+events_unbound17:42:185
3464940993330,2cyclictest3590046-21kworker/u16:1+flush-8:016:29:495
3464939993331,2cyclictest3952809-21kworker/u16:3+events_unbound19:49:554
3464939993331,2cyclictest3942754-21kworker/u16:2+events_unbound19:56:344
3464949993230,1cyclictest0-21swapper/717:22:547
3464949993227,4cyclictest589235-21systemd-journal19:43:537
3464949993227,4cyclictest589235-21systemd-journal19:43:537
3464940993230,2cyclictest3463840-21kworker/u16:3+flush-8:017:10:575
3464940993229,2cyclictest3882259-21kworker/u16:0+events_unbound18:47:295
3464939993230,2cyclictest3569498-21kworker/u16:2+events_unbound17:34:144
3464939993229,2cyclictest3740015-21kworker/u16:4+events_unbound18:27:334
3464939993228,3cyclictest3730565-21kworker/u16:1+flush-8:018:22:304
3464940993129,2cyclictest4063860-21kworker/u16:0+flush-8:020:24:495
3464940993129,2cyclictest3882259-21kworker/u16:0+events_unbound19:26:015
3464940993129,2cyclictest3590046-21kworker/u16:1+events_unbound16:49:295
3464939993129,2cyclictest4043451-21kworker/u16:2+flush-8:020:15:554
3464939993129,2cyclictest3942754-21kworker/u16:2+events_unbound19:29:574
3464939993129,2cyclictest3740015-21kworker/u16:4+flush-8:017:58:134
3464939993129,2cyclictest3590046-21kworker/u16:1+flush-8:017:02:084
3464939993129,2cyclictest3569498-21kworker/u16:2+flush-8:016:42:044
3464949993025,4cyclictest3589792-21latency_hist16:22:247
3464940993028,2cyclictest3972431-21kworker/u16:5+flush-8:019:36:095
3464940993028,2cyclictest3911773-21kworker/u16:2+events_unbound19:02:225
3464940993028,2cyclictest3911773-21kworker/u16:2+events_unbound19:02:225
3464940993028,2cyclictest3866686-21kworker/u16:3+flush-8:018:39:015
3464940993028,2cyclictest3740015-21kworker/u16:4+events_unbound18:24:375
3464940993028,2cyclictest3730565-21kworker/u16:1+flush-8:018:15:585
3464940993028,2cyclictest3730565-21kworker/u16:1+flush-8:018:15:585
3464940993028,2cyclictest3569498-21kworker/u16:2+events_unbound17:30:175
3464940993028,2cyclictest3569498-21kworker/u16:2+events_unbound17:30:175
3464940993028,2cyclictest3463840-21kworker/u16:3+events_unbound15:49:175
3464939993028,2cyclictest3760857-21kworker/u16:2+events_unbound17:52:384
3464939993028,2cyclictest3730565-21kworker/u16:1+flush-8:017:38:304
3464949992927,1cyclictest0-21swapper/716:12:497
3464949992924,4cyclictest4073537-21latency_hist20:22:237
346494999290,28cyclictest3972672-21latency_hist19:32:237
3464940992927,2cyclictest3952809-21kworker/u16:3+events_unbound20:41:415
3464940992927,2cyclictest3590046-21kworker/u16:1+events_unbound16:55:385
3464940992927,2cyclictest3569498-21kworker/u16:2+events_unbound16:34:395
3464940992927,2cyclictest3463840-21kworker/u16:3+events_unbound17:15:425
3464939992927,2cyclictest3660102-21kworker/u16:4+events_unbound17:20:544
3464939992927,2cyclictest3569498-21kworker/u16:2+events_unbound16:12:344
3464939992927,2cyclictest3463840-21kworker/u16:3+flush-8:015:50:344
3464939992927,2cyclictest3463840-21kworker/u16:3+events_unbound16:00:064
3464939992927,2cyclictest3463840-21kworker/u16:3+events_unbound16:00:064
3464949992827,0cyclictest0-21swapper/718:37:397
3464949992824,3cyclictest589235-21systemd-journal17:36:537
3464940992826,2cyclictest4043451-21kworker/u16:2+flush-8:020:15:245
3464940992826,2cyclictest4033002-21kworker/u16:1+flush-8:020:08:575
3464940992826,2cyclictest3952809-21kworker/u16:3+events_unbound19:55:025
3464940992826,2cyclictest3882259-21kworker/u16:0+events_unbound19:09:265
3464940992826,2cyclictest3861411-21kworker/u16:1+events_unbound18:36:015
3464940992826,2cyclictest3790477-21kworker/u16:0+events_unbound18:21:455
3464940992826,2cyclictest3559489-21kworker/u16:1+flush-8:016:15:295
3464940992826,2cyclictest3463840-21kworker/u16:3+flush-8:017:21:585
3464940992826,2cyclictest3463840-21kworker/u16:3+flush-8:015:30:305
3464940992826,2cyclictest3463840-21kworker/u16:3+events_unbound17:23:005
3464940992826,2cyclictest3463840-21kworker/u16:3+events_unbound16:06:455
3464939992826,2cyclictest3408283-21kworker/u16:1+flush-8:015:23:104
3464949992722,4cyclictest589235-21systemd-journal16:37:027
3464940992725,2cyclictest4094021-21kworker/u16:4+flush-8:020:36:055
3464940992725,2cyclictest3952809-21kworker/u16:3+flush-8:019:47:575
3464940992725,2cyclictest3892365-21kworker/u16:1+events_unbound19:06:225
3464940992725,2cyclictest3882259-21kworker/u16:0+events_unbound19:18:355
3464940992725,2cyclictest3740015-21kworker/u16:4+events_unbound18:46:155
3464940992725,2cyclictest3528975-21kworker/u16:0+flush-8:016:41:305
3464939992725,2cyclictest4094021-21kworker/u16:4+flush-8:020:48:574
3464939992725,2cyclictest3952809-21kworker/u16:3+flush-8:019:38:454
3464939992725,2cyclictest3952809-21kworker/u16:3+events_unbound19:35:574
3464939992725,2cyclictest3740015-21kworker/u16:4+events_unbound17:48:344
3464939992725,2cyclictest3730565-21kworker/u16:1+events_unbound18:13:274
3464939992725,2cyclictest3730565-21kworker/u16:1+events_unbound18:13:274
3464939992725,2cyclictest3590046-21kworker/u16:1+flush-8:016:37:224
3464940992624,2cyclictest4104107-21kworker/u16:0+events_unbound20:50:505
3464940992624,2cyclictest3730565-21kworker/u16:1+events_unbound18:01:225
3464940992624,2cyclictest3730565-21kworker/u16:1+events_unbound17:44:185
3464940992624,2cyclictest3569498-21kworker/u16:2+flush-8:016:58:095
3464940992624,2cyclictest3569498-21kworker/u16:2+events_unbound16:24:495
3464940992624,2cyclictest3489362-21kworker/u16:5+events_unbound15:59:505
3464940992624,2cyclictest3489362-21kworker/u16:5+events_unbound15:59:505
3464940992624,2cyclictest3463840-21kworker/u16:3+events_unbound17:03:455
3464940992624,2cyclictest3408283-21kworker/u16:1+events_unbound15:24:225
3464939992624,2cyclictest3942754-21kworker/u16:2+flush-8:019:25:574
3464939992624,2cyclictest3475100-21kworker/u16:0+flush-8:015:37:464
3464939992624,2cyclictest3463840-21kworker/u16:3+flush-8:017:15:234
3464949992519,5cyclictest589235-21systemd-journal20:32:237
3464949992519,5cyclictest589235-21systemd-journal20:32:237
3464940992523,2cyclictest4094021-21kworker/u16:4+flush-8:020:45:245
3464940992523,2cyclictest4043451-21kworker/u16:2+events_unbound20:28:295
3464940992523,2cyclictest4043451-21kworker/u16:2+events_unbound20:28:295
3464940992523,2cyclictest3972431-21kworker/u16:5+flush-8:020:05:535
3464940992523,2cyclictest3972431-21kworker/u16:5+flush-8:019:45:345
3464940992523,2cyclictest3972431-21kworker/u16:5+flush-8:019:45:345
3464940992523,2cyclictest3952809-21kworker/u16:3+flush-8:019:38:535
3464940992523,2cyclictest3760857-21kworker/u16:2+events_unbound17:48:455
3464940992523,2cyclictest3740015-21kworker/u16:4+events_unbound17:56:175
3464940992523,2cyclictest3528975-21kworker/u16:0+events_unbound16:11:305
3464939992523,2cyclictest3952809-21kworker/u16:3+events_unbound19:44:564
3464939992523,2cyclictest3952809-21kworker/u16:3+events_unbound19:44:564
3464939992523,2cyclictest3911773-21kworker/u16:2+events_unbound19:01:314
3464939992523,2cyclictest3911773-21kworker/u16:2+events_unbound19:01:314
3464939992523,2cyclictest3760857-21kworker/u16:2+events_unbound18:39:054
3464939992523,2cyclictest3740015-21kworker/u16:4+events_unbound19:15:244
3464939992523,2cyclictest3408283-21kworker/u16:1+events_unbound15:28:224
3464949992423,0cyclictest0-21swapper/716:07:577
3464949992419,4cyclictest3962500-21latency_hist19:27:237
3464940992422,2cyclictest4024354-21kworker/u16:0+events_unbound19:58:475
3464940992422,2cyclictest3942754-21kworker/u16:2+events_unbound19:27:335
3464940992422,2cyclictest3463840-21kworker/u16:3+events_unbound16:44:455
3464940992422,2cyclictest3463840-21kworker/u16:3+events_unbound15:39:485
3464939992422,2cyclictest3866686-21kworker/u16:3+events_unbound18:43:254
3464939992422,2cyclictest3760857-21kworker/u16:2+flush-8:018:08:174
3464939992422,2cyclictest3569498-21kworker/u16:2+events_unbound16:49:344
3464939992421,2cyclictest4043451-21kworker/u16:2+flush-8:020:22:244
3464939992321,2cyclictest3952809-21kworker/u16:3+events_unbound20:38:414
3464939992321,2cyclictest3590046-21kworker/u16:1+flush-8:016:42:484
3464939992321,2cyclictest3559489-21kworker/u16:1+events_unbound16:09:584
3464939992321,2cyclictest3528975-21kworker/u16:0+events_unbound16:04:474
3464939992321,2cyclictest3463840-21kworker/u16:3+events_unbound17:22:584
3464949992221,0cyclictest0-21swapper/715:22:397
3464949992217,4cyclictest589235-21systemd-journal16:02:247
3464949992216,5cyclictest4063482-21latency_hist20:17:227
3464949992216,5cyclictest3690550-21latency_hist17:12:247
3464939992220,2cyclictest4043451-21kworker/u16:2+events_unbound20:29:184
3464939992220,2cyclictest4043451-21kworker/u16:2+events_unbound20:29:184
3464939992219,3cyclictest3760857-21kworker/u16:2+flush-8:018:32:484
3464949992116,4cyclictest3912032-21latency_hist19:02:237
3464949992116,4cyclictest3912032-21latency_hist19:02:237
346494999210,19cyclictest4133895-21latency_hist20:52:237
3464940992119,2cyclictest3882259-21kworker/u16:0+flush-8:018:57:165
3464940992119,2cyclictest3569498-21kworker/u16:2+flush-8:017:32:405
3464949992019,0cyclictest0-21swapper/719:57:257
3464949992019,0cyclictest0-21swapper/715:27:397
3464949992015,4cyclictest589235-21systemd-journal18:07:537
3464940992018,2cyclictest3730565-21kworker/u16:1+flush-8:018:11:405
3464949991913,5cyclictest3751043-21latency_hist17:42:237
3464949991912,5cyclictest589235-21systemd-journal18:17:237
3464949991912,5cyclictest589235-21systemd-journal18:17:237
3464939991917,2cyclictest4094021-21kworker/u16:4+events_unbound20:43:324
3464949991813,4cyclictest3509129-21latency_hist15:42:247
3464949991812,5cyclictest4013051-21latency_hist19:52:237
346494999180,16cyclictest4123870-21ls20:47:237
3464949991712,4cyclictest3529305-21latency_hist15:52:237
346494999170,16cyclictest4175659-21snmpd16:40:277
3464949991614,1cyclictest96-21ksoftirqd/720:22:507
3464944991615,1cyclictest3912950-21/usr/sbin/munin19:02:376
3464944991614,2cyclictest3740015-21kworker/u16:4+events_unbound18:30:016
3464944991614,2cyclictest3489362-21kworker/u16:5+events_unbound15:51:016
3464944991613,2cyclictest3463840-21kworker/u16:3+events_unbound16:42:266
346494499160,15cyclictest3685664-21cut17:07:466
3464949991515,0cyclictest0-21swapper/718:42:517
3464949991513,1cyclictest3851812-21cat18:32:237
346494999150,13cyclictest589235-21systemd-journal17:42:537
3464944991513,2cyclictest3740015-21kworker/u16:4+events_unbound17:41:026
3464944991513,2cyclictest3463840-21kworker/u16:3+flush-8:015:27:246
3464944991513,2cyclictest3463840-21kworker/u16:3+events_unbound17:06:376
3464944991513,1cyclictest4095036-21irqcore20:32:396
3464944991512,2cyclictest3952809-21kworker/u16:3+events_unbound19:42:306
3464944991512,2cyclictest3952809-21kworker/u16:3+events_unbound19:42:306
3464949991411,2cyclictest0-21swapper/717:57:537
3464949991410,3cyclictest96-21ksoftirqd/719:07:397
346494999140,12cyclictest3722435-21taskset17:27:407
346494999140,12cyclictest3722435-21taskset17:27:407
3464944991413,1cyclictest4105261-21sed20:37:406
3464944991413,1cyclictest3873381-21sed18:42:396
3464944991413,1cyclictest3860739-21ntpq18:32:486
3464944991413,1cyclictest3615320-21grep16:32:426
3464944991413,1cyclictest3601176-21irqcore16:27:406
3464944991413,1cyclictest3592112-21kthreadcore16:22:426
3464944991413,1cyclictest0-21swapper/615:32:436
3464944991412,2cyclictest3952809-21kworker/u16:3+flush-8:019:52:236
3464944991412,2cyclictest3869553-21kthreadcore18:37:466
3464944991412,2cyclictest3590046-21kworker/u16:1+events_unbound16:40:196
3464944991412,1cyclictest3780433-21sendmail_mailtr17:52:516
3464944991412,1cyclictest3719850-21awk17:22:486
3464944991412,1cyclictest3534030-21kthreadcore15:52:476
346494499140,14cyclictest86-21ksoftirqd/616:12:516
346494999137,5cyclictest3499074-21latency_hist15:37:247
3464944991312,1cyclictest4184980-17auditd18:48:536
3464944991312,1cyclictest4127059-21grep20:47:406
3464944991312,1cyclictest4041140-21cut20:02:466
3464944991312,1cyclictest3986522-21kthreadcore19:37:436
3464944991312,1cyclictest3928393-21cut19:07:466
3464944991312,1cyclictest3563644-21kthreadcore16:07:426
3464944991312,1cyclictest3550605-21hostname16:02:376
3464944991312,1cyclictest3516188-21tr15:42:436
3464944991312,1cyclictest0-21swapper/619:22:376
3464944991312,1cyclictest0-21swapper/618:17:456
3464944991311,2cyclictest3569498-21kworker/u16:2+events_unbound17:35:446
3464944991311,1cyclictest4048023-21kthreadcore20:07:436
3464944991311,1cyclictest4032236-21ntpq19:57:466
3464944991311,1cyclictest3967461-21kthreadcore19:27:456
3464944991311,1cyclictest3941379-21ntpq19:12:486
3464944991311,1cyclictest3640728-21cpuspeed_turbos16:47:356
3464944991311,1cyclictest3479232-21ntpq15:27:286
3464944991311,1cyclictest0-21swapper/620:42:436
3464944991311,1cyclictest0-21swapper/618:07:436
346494499130,13cyclictest0-21swapper/618:02:476
346494499130,13cyclictest0-21swapper/617:12:476
346494999129,2cyclictest4175659-21snmpd17:48:587
346494999129,2cyclictest4175659-21snmpd16:26:007
346494999128,2cyclictest0-21swapper/715:52:507
346494999126,5cyclictest27174-21gnome-shell19:13:227
346494999120,10cyclictest4043544-21date20:07:237
346494499122,6cyclictest86-21ksoftirqd/617:47:366
346494499121,6cyclictest86-21ksoftirqd/615:57:366
346494499121,6cyclictest86-21ksoftirqd/615:57:366
3464944991211,1cyclictest3943194-21cstates19:17:366
3464944991211,1cyclictest3722427-21kthreadcore17:27:416
3464944991211,1cyclictest3722427-21kthreadcore17:27:416
3464944991211,1cyclictest0-21swapper/620:27:436
3464944991211,1cyclictest0-21swapper/620:27:436
3464944991211,1cyclictest0-21swapper/620:17:366
3464944991211,1cyclictest0-21swapper/618:57:516
3464944991211,1cyclictest0-21swapper/618:57:516
3464944991211,1cyclictest0-21swapper/617:57:546
3464944991210,1cyclictest3669438-21ntpq16:57:496
3464944991210,1cyclictest0-21swapper/620:12:406
3464944991210,1cyclictest0-21swapper/618:22:386
346494999118,2cyclictest4175659-21snmpd17:05:137
346494999117,3cyclictest921-21dbus-daemon18:22:237
346494999117,3cyclictest27642-21gmain16:52:377
346494999117,3cyclictest27204-21gmain15:57:537
346494999117,3cyclictest27204-21gmain15:57:537
346494999114,6cyclictest4023127-21latency_hist19:57:237
346494999113,7cyclictest0-21swapper/719:02:297
346494999112,8cyclictest4175659-21snmpd20:07:407
3464949991110,0cyclictest0-21swapper/718:02:287
346494999110,2cyclictest0-21swapper/719:17:427
346494499119,1cyclictest0-21swapper/618:12:366
346494499119,1cyclictest0-21swapper/618:12:366
346494499113,4cyclictest4015199-21kthreadcore19:52:426
346494499112,5cyclictest86-21ksoftirqd/616:52:446
346494499111,6cyclictest86-21ksoftirqd/619:32:376
346494499111,6cyclictest86-21ksoftirqd/615:37:436
346494499111,5cyclictest86-21ksoftirqd/617:17:376
3464944991110,1cyclictest3893555-21kthreadcore18:52:416
3464944991110,1cyclictest0-21swapper/620:22:486
3464944991110,1cyclictest0-21swapper/616:17:366
346494499110,4cyclictest3758597-21cut17:42:466
346494999107,2cyclictest4175659-21snmpd20:42:087
346494999105,4cyclictest3608982-21ntpq16:27:487
346494999103,6cyclictest0-21swapper/716:47:547
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional