You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 17:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Mon Dec 29, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3898018994538,5cyclictest3927980-21ntpq07:35:213
3898006994037,2cyclictest3996726-21kworker/u8:0+events_unbound10:25:180
3898006994037,2cyclictest3944916-21kworker/u8:1+events_unbound08:20:150
3898006993936,2cyclictest3897987-21kworker/u8:3+events_unbound07:35:150
3898014993834,3cyclictest3996726-21kworker/u8:0+flush-259:008:50:002
3898018993733,3cyclictest3849870-21kworker/u8:0+events_unbound08:20:003
3898011993631,3cyclictest3996726-21kworker/u8:0+flush-259:009:35:001
3898006993632,3cyclictest4126784-21/usr/sbin/munin10:50:130
3898006993530,3cyclictest41091-21latency_hist12:35:000
3898011993431,2cyclictest4007114-21kworker/u8:4+flush-259:009:00:001
3898011993427,5cyclictest3960150-21latency_hist08:10:001
3898006993430,2cyclictest1-21systemd09:24:590
3898014993329,3cyclictest4100585-21apt-get10:25:142
3898006993330,2cyclictest4162172-21kworker/u8:0+events_unbound11:40:290
3898006993330,2cyclictest3881867-21kworker/u8:1+events_unbound07:30:230
3898006993330,2cyclictest3881867-21kworker/u8:1+events_unbound07:25:240
3898006993327,4cyclictest30813-21latency_hist12:25:010
3898006993229,2cyclictest15393-21kworker/u8:0+events_unbound12:25:230
3898006993229,2cyclictest15393-21kworker/u8:0+events_unbound12:25:230
3898014993127,3cyclictest4059125-21kworker/u8:3+flush-259:011:30:232
3898011993125,4cyclictest252-21systemd-journal09:05:301
3898006993128,2cyclictest3849870-21kworker/u8:0+events_unbound07:15:200
3898018993023,4cyclictest4173265-21latency_hist11:35:013
3898014993027,2cyclictest4059125-21kworker/u8:3+events_unbound10:36:312
3898006993027,2cyclictest4121593-21kworker/u8:2+events_unbound11:03:320
3898014992925,3cyclictest15393-21kworker/u8:0+flush-259:012:25:212
3898014992925,3cyclictest15393-21kworker/u8:0+flush-259:012:25:212
3898011992925,3cyclictest3932834-21kworker/u8:2+flush-259:007:55:161
3898011992923,4cyclictest252-21systemd-journal11:25:401
3898006992925,3cyclictest4173309-21kworker/u8:1+events_unbound11:45:180
3898014992824,3cyclictest4059125-21kworker/u8:3+flush-259:011:55:202
3898014992824,3cyclictest4059125-21kworker/u8:3+flush-259:010:05:172
3898014992824,3cyclictest4059125-21kworker/u8:3+flush-259:010:00:162
3898018992722,4cyclictest3996726-21kworker/u8:0+flush-259:010:25:183
3898014992722,4cyclictest4152330-21kworker/u8:1+flush-259:011:15:202
3898011992720,5cyclictest252-21systemd-journal10:00:001
3898006992725,2cyclictest3996726-21kworker/u8:0+events_unbound09:50:250
3898006992724,2cyclictest4007114-21kworker/u8:4+events_unbound09:27:330
3898006992724,2cyclictest3932834-21kworker/u8:2+events_unbound07:45:110
3898006992724,2cyclictest3849870-21kworker/u8:0+events_unbound07:20:140
3898018992621,4cyclictest3913614-21/usr/sbin/munin07:25:253
3898014992623,2cyclictest4059125-21kworker/u8:3+events_unbound09:55:172
3898014992623,2cyclictest3944916-21kworker/u8:1+flush-259:008:25:272
3898014992621,4cyclictest4007114-21kworker/u8:4+flush-259:009:35:222
3898006992623,2cyclictest4059125-21kworker/u8:3+events_unbound12:10:230
3898006992623,2cyclictest4007114-21kworker/u8:4+events_unbound09:48:320
3898006992621,4cyclictest3897987-21kworker/u8:3+events_unbound07:40:010
3898006992621,4cyclictest3897987-21kworker/u8:3+events_unbound07:40:010
3898018992521,3cyclictest3986016-21kworker/u8:3+flush-259:008:50:253
3898018992520,3cyclictest4007114-21kworker/u8:4+flush-259:009:25:203
3898014992522,2cyclictest4059125-21kworker/u8:3+flush-259:010:20:012
3898014992521,3cyclictest3897987-21kworker/u8:3+events_unbound07:40:252
3898014992521,3cyclictest3897987-21kworker/u8:3+events_unbound07:40:252
3898014992517,3cyclictest3881867-21kworker/u8:1+flush-259:007:25:232
3898011992520,3cyclictest4059125-21kworker/u8:3+flush-259:011:55:001
3898011992519,4cyclictest4038446-21latency_hist09:25:001
3898011992519,4cyclictest252-21systemd-journal12:25:001
3898006992522,2cyclictest4059125-21kworker/u8:3+events_unbound10:15:190
3898006992522,2cyclictest3849870-21kworker/u8:0+events_unbound08:25:120
3898006992521,3cyclictest3932834-21kworker/u8:2+flush-259:008:15:000
3898018992417,5cyclictest4094291-21ntpq10:15:213
3898014992420,3cyclictest3897987-21kworker/u8:3+events_unbound07:45:222
3898011992417,5cyclictest4272-21ntpq11:55:221
389801199241,21cyclictest1-21systemd07:45:011
389801199241,21cyclictest1-21systemd07:45:001
3898011992410,12cyclictest252-21systemd-journal08:25:011
3898006992421,2cyclictest4121593-21kworker/u8:2+events_unbound11:10:010
3898018992320,2cyclictest3932834-21kworker/u8:2+flush-259:008:30:173
3898018992319,3cyclictest3944916-21kworker/u8:1+events_unbound08:40:193
3898014992320,2cyclictest3897987-21kworker/u8:3+flush-259:007:35:162
3898014992320,2cyclictest3849870-21kworker/u8:0+flush-259:007:10:202
3898014992320,2cyclictest3849870-21kworker/u8:0+events_unbound07:15:132
3898014992319,3cyclictest4121593-21kworker/u8:2+flush-259:010:55:212
3898014992319,3cyclictest3996726-21kworker/u8:0+flush-259:009:10:162
3898014992319,3cyclictest3932834-21kworker/u8:2+events_unbound08:00:152
3898011992317,4cyclictest4183585-21apt11:45:001
3898006992320,3cyclictest4162172-21kworker/u8:0+events_unbound11:33:330
3898006992320,2cyclictest4111293-21kworker/u8:4+events_unbound10:41:320
3898006992319,3cyclictest3996726-21kworker/u8:0+events_unbound09:40:270
3898006992319,3cyclictest3996726-21kworker/u8:0+events_unbound09:19:330
3898006992319,3cyclictest3932834-21kworker/u8:2+events_unbound08:35:130
3898006992317,4cyclictest3944916-21kworker/u8:1+flush-259:008:45:120
3898018992216,4cyclictest4188989-21apt-get11:50:133
3898014992219,2cyclictest4059125-21kworker/u8:3+events_unbound12:30:142
3898014992219,2cyclictest3932834-21kworker/u8:2+flush-259:007:50:242
3898014992218,3cyclictest4059125-21kworker/u8:3+flush-259:011:50:162
3898014992218,3cyclictest3996726-21kworker/u8:0+events_unbound09:50:192
3898014992218,3cyclictest3996726-21kworker/u8:0+events_unbound09:20:172
3898014992218,3cyclictest3996726-21kworker/u8:0+events_unbound08:50:152
389801199229,2cyclictest43642-21kthreadcore12:35:181
389801199229,2cyclictest0-21swapper/108:10:261
3898011992216,4cyclictest4152330-21kworker/u8:1+flush-259:011:15:151
3898011992216,4cyclictest3986673-21apt-get08:35:131
3898006992220,1cyclictest4059125-21kworker/u8:3+events_unbound10:20:160
3898006992219,2cyclictest4007114-21kworker/u8:4+events_unbound09:33:310
3898018992116,3cyclictest4162172-21kworker/u8:0+flush-259:011:40:203
3898018992115,4cyclictest4111293-21kworker/u8:4+flush-259:011:00:013
3898018992115,4cyclictest3849870-21kworker/u8:0+flush-259:008:25:323
3898014992118,2cyclictest4162172-21kworker/u8:0+events_unbound11:40:262
3898014992118,2cyclictest25673-21kworker/u8:2+flush-259:012:20:242
3898014992117,3cyclictest4121593-21kworker/u8:2+flush-259:010:50:172
3898014992117,3cyclictest4059125-21kworker/u8:3+events_unbound11:35:222
3898014992116,4cyclictest3986648-21/usr/sbin/munin08:35:162
3898014992115,2cyclictest3932834-21kworker/u8:2+events_unbound08:10:202
3898006992118,2cyclictest4111293-21kworker/u8:4+events_unbound10:37:320
3898006992118,2cyclictest3932834-21kworker/u8:2+events_unbound08:30:180
3898006992118,2cyclictest25673-21kworker/u8:2+events_unbound12:36:330
3898006992117,3cyclictest4173309-21kworker/u8:1+events_unbound12:15:410
3898006992116,4cyclictest3849870-21kworker/u8:0+flush-259:008:05:180
3898018992016,3cyclictest4007114-21kworker/u8:4+flush-259:009:35:283
3898018992016,3cyclictest3932834-21kworker/u8:2+flush-259:008:35:213
3898018992016,3cyclictest3849870-21kworker/u8:0+flush-259:008:05:203
3898018992015,3cyclictest4059125-21kworker/u8:3+flush-259:012:05:203
3898014992017,2cyclictest4111293-21kworker/u8:4+flush-259:010:45:212
3898014992016,3cyclictest4163231-21apt-get11:25:132
3898011992015,4cyclictest507-21in:imuxsock07:15:001
3898011992012,5cyclictest509-21rs:main1
3898006992017,2cyclictest4121593-21kworker/u8:2+events_unbound11:05:380
3898006992017,2cyclictest3996726-21kworker/u8:0+events_unbound09:10:210
3898006992015,3cyclictest4007289-21gpgv08:55:000
3898006992014,5cyclictest4173309-21kworker/u8:1+flush-259:012:00:200
3898006992013,5cyclictest3948963-21ntpq07:55:220
3898018991916,2cyclictest4059125-21kworker/u8:3+flush-259:011:10:213
3898018991915,3cyclictest3932834-21kworker/u8:2+flush-259:008:05:003
3898018991915,3cyclictest3849870-21kworker/u8:0+flush-259:007:35:003
3898018991914,4cyclictest252-21systemd-journal09:10:013
3898018991914,3cyclictest3996726-21kworker/u8:0+events_unbound08:55:233
3898018991914,3cyclictest252-21systemd-journal11:25:003
3898018991913,4cyclictest4126547-21latency_hist10:50:013
3898014991916,2cyclictest4173309-21kworker/u8:1+events_unbound12:00:252
3898014991916,2cyclictest3932834-21kworker/u8:2+flush-259:008:20:212
3898014991915,3cyclictest4152330-21kworker/u8:1+flush-259:011:20:162
3898014991915,3cyclictest4059125-21kworker/u8:3+flush-259:010:10:252
3898014991915,3cyclictest4007114-21kworker/u8:4+events_unbound09:34:322
3898011991912,5cyclictest3907404-21ntpq07:15:221
3898006991916,2cyclictest3944916-21kworker/u8:1+events_unbound08:40:200
3898018991814,3cyclictest3897987-21kworker/u8:3+flush-259:007:50:173
3898018991812,4cyclictest4059125-21kworker/u8:3+flush-259:010:05:213
3898014991815,2cyclictest3932834-21kworker/u8:2+flush-259:008:15:132
3898014991814,3cyclictest4173309-21kworker/u8:1+events_unbound11:45:112
3898014991814,3cyclictest3932834-21kworker/u8:2+flush-259:008:30:262
3898014991814,3cyclictest3932834-21kworker/u8:2+events_unbound07:55:212
3898006991815,2cyclictest4111293-21kworker/u8:4+events_unbound10:46:290
3898006991815,2cyclictest4059125-21kworker/u8:3+events_unbound12:05:320
3898006991815,2cyclictest4007114-21kworker/u8:4+events_unbound09:07:100
3898006991815,2cyclictest3996726-21kworker/u8:0+flush-259:010:14:470
3898006991814,3cyclictest4121593-21kworker/u8:2+events_unbound11:19:200
3898006991813,3cyclictest4100341-21kworker/u8:2+events_unbound10:34:220
3898018991714,2cyclictest4121593-21kworker/u8:2+events_unbound11:05:223
3898018991714,2cyclictest4059125-21kworker/u8:3+flush-259:010:00:243
3898014991713,3cyclictest4173309-21kworker/u8:1+events_unbound12:19:572
3898014991713,3cyclictest4121593-21kworker/u8:2+events_unbound11:01:332
3898014991712,3cyclictest4059125-21kworker/u8:3+events_unbound10:30:172
3898014991712,3cyclictest3944916-21kworker/u8:1+flush-259:008:44:362
3898011991711,5cyclictest252-21systemd-journal12:20:001
3898011991711,4cyclictest252-21systemd-journal10:05:311
3898011991710,4cyclictest4121365-21latency_hist10:45:011
3898006991714,2cyclictest60350irq/132-enp0s29f1:rx-010:08:300
3898006991714,2cyclictest4059125-21kworker/u8:3+events_unbound11:56:330
3898006991713,3cyclictest4152330-21kworker/u8:1+events_unbound11:20:330
3898006991713,3cyclictest3897987-21kworker/u8:3+events_unbound07:50:030
389801899169,5cyclictest4001913-21latency_hist08:50:013
3898018991612,3cyclictest4059125-21kworker/u8:3+flush-259:012:30:203
3898018991611,4cyclictest3849870-21kworker/u8:0+flush-259:007:15:173
3898014991613,2cyclictest4059125-21kworker/u8:3+events_unbound10:20:112
3898014991612,3cyclictest4007114-21kworker/u8:4+flush-259:009:45:162
3898014991612,3cyclictest4007114-21kworker/u8:4+flush-259:009:25:302
3898011991613,2cyclictest4187851-21ntp_kernel_pll_11:45:201
3898011991610,4cyclictest3932834-21kworker/u8:2+flush-259:008:00:151
389800699169,5cyclictest4192938-21ntpq11:50:220
3898006991614,1cyclictest60350irq/132-enp0s29f1:rx-011:35:250
3898006991614,1cyclictest60350irq/132-enp0s29f1:rx-009:55:180
3898006991613,2cyclictest60350irq/132-enp0s29f1:rx-009:35:230
3898006991612,3cyclictest3996726-21kworker/u8:0+events_unbound08:57:060
3898006991612,2cyclictest60350irq/132-enp0s29f1:rx-008:15:120
389800699161,1cyclictest191rcu_preempt10:00:120
389801899159,4cyclictest4059125-21kworker/u8:3+events_unbound10:10:203
3898018991511,3cyclictest4059125-21kworker/u8:3+flush-259:012:35:173
3898018991511,3cyclictest4007114-21kworker/u8:4+events_unbound09:30:153
3898018991511,3cyclictest3996726-21kworker/u8:0+flush-259:009:50:253
3898018991511,3cyclictest3996726-21kworker/u8:0+flush-259:009:40:253
3898018991510,4cyclictest25673-21kworker/u8:2+events_unbound12:20:193
389801499158,5cyclictest4152276-21awk11:15:012
3898014991512,2cyclictest4059125-21kworker/u8:3+events_unbound12:07:332
3898014991511,3cyclictest4012914-21kworker/u8:1+flush-259:009:15:272
3898014991511,3cyclictest3849870-21kworker/u8:0+flush-259:007:30:152
3898014991510,4cyclictest4012683-21latency_hist09:00:012
389801199159,4cyclictest4100560-21/usr/sbin/munin10:25:251
3898011991512,2cyclictest4146321-21ntp_offset11:05:211
3898011991512,2cyclictest4063329-21needreboot09:45:201
3898011991512,2cyclictest0-21swapper/111:25:011
3898011991511,3cyclictest4162172-21kworker/u8:0+flush-259:011:35:001
3898011991511,3cyclictest4059125-21kworker/u8:3+events_unbound12:10:171
3898011991511,3cyclictest3932834-21kworker/u8:2+flush-259:008:30:271
3898011991510,4cyclictest4059125-21kworker/u8:3+flush-259:011:10:171
3898011991510,4cyclictest3944267-21apt-get07:55:001
3898006991513,2cyclictest4059125-21kworker/u8:3+events_unbound11:28:290
3898006991513,1cyclictest60350irq/132-enp0s29f1:rx-007:10:220
3898006991512,2cyclictest4121593-21kworker/u8:2+events_unbound10:55:380
389801899149,4cyclictest252-21systemd-journal09:25:003
3898018991411,2cyclictest4110633-21unixbench_singl10:30:283
3898018991410,2cyclictest3996726-21kworker/u8:0+events_unbound09:10:183
389801899140,12cyclictest4168153-21munin-run11:30:013
3898014991411,2cyclictest4007114-21kworker/u8:4+flush-259:009:40:302
3898014991410,3cyclictest3996726-21kworker/u8:0+flush-259:009:05:322
3898014991410,3cyclictest3849870-21kworker/u8:0+events_unbound08:07:482
3898014991410,3cyclictest3849870-21kworker/u8:0+events_unbound07:23:162
389801199149,3cyclictest4173529-21apt-get11:35:131
389801199148,4cyclictest36001-21kworker/u8:1+flush-259:012:30:051
3898006991412,1cyclictest60350irq/132-enp0s29f1:rx-009:00:120
389801899139,3cyclictest4059125-21kworker/u8:3+flush-259:010:20:163
389801899139,3cyclictest3996726-21kworker/u8:0+flush-259:009:15:253
389801899138,3cyclictest4173309-21kworker/u8:1+flush-259:012:05:003
389801899138,3cyclictest4152548-21apt-get11:15:113
3898018991310,2cyclictest4131673-21journal-offline10:55:003
389801499139,3cyclictest4121593-21kworker/u8:2+events_unbound11:05:102
389801199137,4cyclictest252-21systemd-journal10:55:011
3898011991310,2cyclictest5534-21fschecks_count12:00:161
3898006991311,1cyclictest60350irq/132-enp0s29f1:rx-008:00:060
389801899129,2cyclictest4605-21unixbench_multi11:55:253
389801899129,2cyclictest3947248-21kthreadcore07:55:183
389801899129,2cyclictest15393-21kworker/u8:0+events_unbound12:15:003
389801899128,3cyclictest3881867-21kworker/u8:1+flush-259:007:40:173
389801899128,3cyclictest3881867-21kworker/u8:1+flush-259:007:40:173
389801499129,2cyclictest4059125-21kworker/u8:3+events_unbound12:10:162
389801499129,2cyclictest25673-21kworker/u8:2+flush-259:012:37:562
389801199129,2cyclictest4058687-21unixbench_singl09:40:261
389801199129,2cyclictest0-21swapper/110:00:251
389801199128,2cyclictest4094379-21sshd10:15:221
389801199128,2cyclictest0-21swapper/112:30:011
389801199128,2cyclictest0-21swapper/112:30:001
389801199126,4cyclictest4007114-21kworker/u8:4+flush-259:009:35:171
389801899119,1cyclictest0-21swapper/307:25:003
389801899118,2cyclictest4183914-21cpuspeed_turbos11:45:133
389801899117,3cyclictest4173309-21kworker/u8:1+flush-259:012:15:193
389801899117,3cyclictest4111293-21kworker/u8:4+events_unbound10:40:173
389801899117,3cyclictest3897987-21kworker/u8:3+flush-259:007:45:253
389801899115,5cyclictest4175842-21sed11:35:183
389801899115,4cyclictest4064293-21latency_hist09:50:013
389801899115,4cyclictest3902042-21memory07:10:193
389801499117,3cyclictest4012914-21kworker/u8:1+events_unbound09:00:232
389801499117,3cyclictest3996726-21kworker/u8:0+events_unbound10:43:112
389801199118,2cyclictest4106415-21fschecks_time10:30:171
389801199118,2cyclictest4028454-21cpuspeed_turbos09:15:141
389801199118,2cyclictest0-21swapper/111:00:151
389801199115,4cyclictest4069453-21latency_hist09:55:001
389801199111,8cyclictest1-21systemd10:23:301
389801899108,1cyclictest0-21swapper/309:00:133
389801899108,1cyclictest0-21swapper/308:20:173
389801899104,4cyclictest3960416-21apt-get08:10:153
389801199104,4cyclictest3927905-21ntpq07:35:191
389801199104,4cyclictest252-21systemd-journal10:39:311
389801199100,2cyclictest3897923-21latency_hist07:10:011
38980119997,1cyclictest0-21swapper/107:45:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional