You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 10:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Jan 12, 2026 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3274894994426,13cyclictest3373012-21ntpq20:40:232
3274888994441,2cyclictest3504060-21kworker/u8:0+events_unbound23:00:130
3274888994036,3cyclictest3545562-21kworker/u8:3+flush-259:023:45:220
3274888993834,3cyclictest3509545-21kworker/u8:2+events_unbound23:55:210
3274888993734,2cyclictest3321597-21kworker/u8:2+flush-259:020:10:240
3274888993733,3cyclictest3264846-21kworker/u8:0+flush-259:019:55:150
3274888993733,3cyclictest3264846-21kworker/u8:0+events_unbound19:10:140
3274894993626,9cyclictest3414200-21ntpq21:20:232
3274888993532,2cyclictest3264846-21kworker/u8:0+events_unbound19:20:250
3274888993330,2cyclictest3613652-21kworker/u8:2+events_unbound00:35:120
3274888993327,4cyclictest3597505-21latency_hist00:20:010
3274888993229,2cyclictest3363200-21kworker/u8:0+events_unbound21:30:130
3274898993127,3cyclictest3504060-21kworker/u8:0+flush-259:022:50:013
3274898993126,4cyclictest3342153-21kworker/u8:3+flush-259:020:25:163
3274894993125,4cyclictest3509521-21latency_hist22:55:002
3274888993128,2cyclictest3321597-21kworker/u8:2+events_unbound21:15:240
3274888993128,2cyclictest3264846-21kworker/u8:0+events_unbound20:15:150
3274898993027,2cyclictest3545562-21kworker/u8:3+flush-259:023:35:153
3274898993023,3cyclictest3363200-21kworker/u8:0+events_unbound21:20:003
3274894993024,4cyclictest3456739-21ntpq22:00:232
327488899300,28cyclictest3462767-21true22:10:010
3274898992926,2cyclictest3254325-21kworker/u8:1+events_unbound19:10:003
3274898992925,3cyclictest3468117-21kworker/u8:3+flush-259:022:45:003
3274888992925,3cyclictest3509545-21kworker/u8:2+events_unbound23:15:140
3274898992825,2cyclictest3509545-21kworker/u8:2+flush-259:000:15:233
3274898992824,3cyclictest3509545-21kworker/u8:2+flush-259:023:55:203
3274891992823,3cyclictest252-21systemd-journal21:34:591
3274888992825,2cyclictest3509545-21kworker/u8:2+events_unbound23:20:180
3274888992825,2cyclictest3357655-21kworker/u8:1+events_unbound20:35:250
3274888992824,3cyclictest3576467-21kworker/u8:1+events_unbound00:20:120
3274888992824,3cyclictest3306157-21kworker/u8:3+events_unbound19:50:230
3274888992824,3cyclictest3306157-21kworker/u8:3+events_unbound19:50:230
3274898992724,2cyclictest3545562-21kworker/u8:3+events_unbound23:40:133
3274888992724,2cyclictest3478326-21kworker/u8:2+events_unbound22:45:240
3274888992723,3cyclictest3602695-21kworker/u8:3+events_unbound00:25:190
3274888992723,3cyclictest3602695-21kworker/u8:3+events_unbound00:25:180
3274888992721,4cyclictest3388468-21ntpq20:55:220
3274888992720,5cyclictest3467883-21latency_hist22:15:000
3274888992720,5cyclictest3467883-21latency_hist22:15:000
3274898992622,3cyclictest3306157-21kworker/u8:3+events_unbound19:50:133
3274898992622,3cyclictest3306157-21kworker/u8:3+events_unbound19:50:133
3274898992620,4cyclictest3296279-21diskstats19:30:133
3274894992519,4cyclictest3570453-21ntpq23:50:202
3274894992519,4cyclictest3357998-21latency_hist20:30:002
3274891992520,4cyclictest3514681-21latency_hist23:00:011
3274888992522,2cyclictest3488911-21kworker/u8:1+events_unbound23:25:220
3274888992522,2cyclictest3264846-21kworker/u8:0+events_unbound20:05:000
3274888992522,2cyclictest3254325-21kworker/u8:1+events_unbound19:30:220
3274888992521,3cyclictest3306395-21apt-get19:40:100
3274898992419,4cyclictest3468117-21kworker/u8:3+flush-259:022:30:003
3274891992422,1cyclictest3519802-21turbostat.cron23:05:011
3274891992419,4cyclictest252-21systemd-journal21:13:451
3274888992422,2cyclictest3478326-21kworker/u8:2+events_unbound22:26:470
3274888992421,2cyclictest3488911-21kworker/u8:1+events_unbound23:05:240
3274888992421,2cyclictest3321597-21kworker/u8:2+events_unbound21:45:240
3274888992421,2cyclictest3264846-21kworker/u8:0+flush-259:019:25:190
3274898992319,3cyclictest3321597-21kworker/u8:2+events_unbound21:50:123
3274891992313,9cyclictest3285369-21/usr/sbin/munin19:20:161
3274888992320,2cyclictest3478326-21kworker/u8:2+flush-259:022:35:150
3274888992320,2cyclictest3478326-21kworker/u8:2+events_unbound22:30:240
3274888992320,2cyclictest3321597-21kworker/u8:2+flush-259:021:10:230
3274888992319,3cyclictest3363200-21kworker/u8:0+events_unbound21:00:480
3274898992219,2cyclictest3473093-21apt-key22:20:003
3274898992218,3cyclictest3363200-21kworker/u8:0+events_unbound21:04:443
3274898992218,2cyclictest3321597-21kworker/u8:2+flush-259:021:55:203
3274898992215,5cyclictest3457568-21latency_hist22:05:013
3274894992216,4cyclictest3290270-21latency_hist19:25:002
3274888992220,2cyclictest3264846-21kworker/u8:0+events_unbound19:35:240
3274888992219,2cyclictest3404768-21kworker/u8:1+events_unbound22:20:100
3274888992219,2cyclictest3404768-21kworker/u8:1+events_unbound21:35:210
3274888992219,2cyclictest3264846-21kworker/u8:0+events_unbound20:20:190
3274888992218,3cyclictest3306157-21kworker/u8:3+flush-259:019:45:230
327488899220,20cyclictest1-21systemd22:56:450
3274898992118,2cyclictest3602695-21kworker/u8:3+events_unbound00:25:233
3274898992118,2cyclictest3602695-21kworker/u8:3+events_unbound00:25:223
3274898992118,2cyclictest3488911-21kworker/u8:1+flush-259:023:10:113
3274898992117,3cyclictest3254325-21kworker/u8:1+events_unbound19:40:123
3274898992114,5cyclictest3346410-21ntpq20:15:213
327489499219,6cyclictest3336972-21apt20:10:002
3274894992116,3cyclictest3404742-21latency_hist21:15:012
3274891992118,1cyclictest0-21swapper/100:14:591
3274891992117,3cyclictest3581855-21/usr/sbin/munin00:05:151
3274888992118,2cyclictest3545562-21kworker/u8:3+events_unbound23:35:090
3274888992118,2cyclictest3363200-21kworker/u8:0+flush-259:021:40:250
3274888992117,3cyclictest3509545-21kworker/u8:2+events_unbound00:14:470
3274898992017,2cyclictest3478326-21kworker/u8:2+flush-259:022:30:253
327489499208,11cyclictest252-21systemd-journal21:49:592
327489499201,9cyclictest3498264-21ntp_22:40:192
3274894992014,5cyclictest252-21systemd-journal22:15:002
3274894992014,5cyclictest252-21systemd-journal22:14:592
3274891992017,2cyclictest252-21systemd-journal20:15:001
3274891992016,3cyclictest3561270-21/usr/sbin/munin23:45:141
3274891992016,3cyclictest3473019-21sh22:20:011
327489199201,3cyclictest3368369-21pluginstate20:40:001
3274888992017,2cyclictest3509545-21kworker/u8:2+events_unbound23:10:230
3274888992017,2cyclictest3321597-21kworker/u8:2+events_unbound20:41:250
3274888992016,3cyclictest3363200-21kworker/u8:0+events_unbound21:06:310
3274888992016,3cyclictest3254325-21kworker/u8:1+events_unbound19:15:290
3274898991918,1cyclictest3545562-21kworker/u8:3+events_unbound23:50:133
3274898991916,2cyclictest3602695-21kworker/u8:3+events_unbound00:33:433
3274898991916,2cyclictest3509545-21kworker/u8:2+events_unbound00:05:223
3274898991916,2cyclictest3404768-21kworker/u8:1+events_unbound22:20:133
3274898991916,2cyclictest3254325-21kworker/u8:1+events_unbound19:21:133
3274898991915,3cyclictest3613652-21kworker/u8:2+events_unbound00:37:463
3274898991915,3cyclictest3363200-21kworker/u8:0+events_unbound21:21:353
3274898991915,3cyclictest3254325-21kworker/u8:1+events_unbound19:37:463
3274898991914,4cyclictest3489139-21apt-get22:35:153
3274898991913,4cyclictest252-21systemd-journal20:01:453
3274894991915,3cyclictest3306157-21kworker/u8:3+flush-259:019:45:202
3274894991911,7cyclictest3298409-21kthreadcore19:30:172
3274888991916,2cyclictest3504060-21kworker/u8:0+events_unbound22:50:230
3274888991916,2cyclictest3306157-21kworker/u8:3+flush-259:020:05:460
3274898991816,1cyclictest0-21swapper/323:00:003
3274898991815,2cyclictest3488911-21kworker/u8:1+flush-259:023:00:213
3274898991815,2cyclictest3321597-21kworker/u8:2+events_unbound21:05:163
3274898991814,3cyclictest3509545-21kworker/u8:2+events_unbound23:20:233
3274898991814,3cyclictest3357655-21kworker/u8:1+events_unbound20:50:173
3274894991815,2cyclictest3591901-21latency_hist00:15:012
3274894991810,7cyclictest3605224-21kthreadcore00:25:192
3274894991810,7cyclictest3605224-21kthreadcore00:25:182
3274891991816,1cyclictest0-21swapper/100:00:001
327489199181,5cyclictest252-21systemd-journal20:50:001
327489199181,4cyclictest3529696-21unixbench_singl23:10:241
327489199181,4cyclictest3352453-21awk20:25:011
3274891991814,3cyclictest3504060-21kworker/u8:0+flush-259:022:55:001
3274891991810,7cyclictest3318353-21kthreadcore19:50:171
3274891991810,7cyclictest3318353-21kthreadcore19:50:171
327489199180,2cyclictest3560655-21runrttasks23:40:371
3274888991815,2cyclictest3602695-21kworker/u8:3+events_unbound00:31:460
3274888991815,2cyclictest3545562-21kworker/u8:3+events_unbound23:41:460
3274888991815,2cyclictest3545562-21kworker/u8:3+events_unbound23:32:000
3274888991815,2cyclictest3509545-21kworker/u8:2+flush-259:023:52:470
3274888991815,2cyclictest3468117-21kworker/u8:3+events_unbound22:40:170
3274898991715,1cyclictest0-21swapper/319:30:003
3274898991714,2cyclictest3404768-21kworker/u8:1+events_unbound21:25:263
3274898991713,3cyclictest3446843-21kworker/u8:4+flush-259:022:10:223
3274898991713,3cyclictest3446843-21kworker/u8:4+flush-259:022:10:223
3274898991713,3cyclictest3254325-21kworker/u8:1+events_unbound19:16:453
327489499179,7cyclictest0-21swapper/220:35:232
327489499178,7cyclictest0-21swapper/221:30:182
327489499177,8cyclictest3405423-21irqcore21:15:182
3274894991712,3cyclictest252-21systemd-journal20:05:002
327489499170,9cyclictest3535680-21df23:20:122
327489199175,10cyclictest3478596-21turbostat22:25:001
327489199173,2cyclictest3555829-21sshd23:39:451
327489199172,13cyclictest0-21swapper/123:30:011
327489199171,6cyclictest3607812-21latency_hist00:30:001
327489199171,6cyclictest3607812-21latency_hist00:29:591
327489199171,5cyclictest3499190-21awk22:45:001
327489199171,5cyclictest3316584-21cat19:50:001
327489199171,4cyclictest3483730-21latency_hist22:30:001
327489199171,4cyclictest3452426-21cron22:00:001
327489199171,4cyclictest3435119-21needreboot21:40:201
327489199171,4cyclictest3296093-21/usr/sbin/munin19:30:251
327489199171,3cyclictest3441234-21latency_hist21:50:001
327489199171,3cyclictest3415067-21munin-run21:25:001
327489199171,3cyclictest3389807-21grep21:00:131
327489199171,3cyclictest3379738-21ps20:50:161
327489199171,3cyclictest3279984-21idleruntime-cro19:15:001
327489199171,2cyclictest3285124-21awk19:20:001
327489199170,4cyclictest3384865-21sed20:55:171
327489199170,3cyclictest3618781-21cat00:40:001
327489199170,3cyclictest3530565-21sort23:15:131
327489199170,3cyclictest3488872-21latency_hist22:35:001
3274888991714,2cyclictest60350irq/132-enp0s29f1:rx-022:15:170
3274888991714,2cyclictest3404768-21kworker/u8:1+events_unbound21:56:450
3274888991714,2cyclictest3342153-21kworker/u8:3+events_unbound20:25:300
3274898991613,2cyclictest3509545-21kworker/u8:2+flush-259:000:00:153
3274898991613,2cyclictest3321597-21kworker/u8:2+events_unbound20:41:463
327489499165,9cyclictest3377264-21tr20:45:172
327489499165,9cyclictest3319693-21tr19:50:182
327489499165,9cyclictest3319693-21tr19:50:182
3274894991613,2cyclictest3548124-21kthreadcore23:30:192
3274894991612,3cyclictest3608330-21base6400:30:012
3274894991612,3cyclictest252-21systemd-journal20:31:452
327489499160,8cyclictest3493703-21taskset22:35:522
327489499160,14cyclictest3530191-21cat23:15:012
327489499160,14cyclictest3473022-21idleruntime-cro22:20:002
327489499160,14cyclictest3306135-21turbostat.cron19:40:002
327489499160,13cyclictest3352755-21grep20:25:002
327489499160,13cyclictest3347295-21idleruntime-cro20:20:002
327489199161,4cyclictest3409910-21awk21:20:001
327489199161,4cyclictest3346861-21unixbench_singl20:15:251
327489199161,3cyclictest3581617-21dump-pmu-power00:04:591
327489199161,3cyclictest3550729-21basename23:35:001
327489199161,3cyclictest3523995-21strings23:05:201
327489199161,3cyclictest3467078-21grep22:10:231
327489199161,3cyclictest3467078-21grep22:10:221
327489199161,3cyclictest3394726-21apt-get21:05:111
327489199161,3cyclictest3310865-21head19:40:241
327489199161,3cyclictest3306134-21idleruntime-cro19:40:001
327489199161,3cyclictest252-21systemd-journal00:25:001
327489199161,3cyclictest1-21systemd20:30:001
327489199161,2cyclictest3570407-21ntpq23:50:201
327489199161,2cyclictest3325861-21ntpq19:55:221
327489199161,2cyclictest3290532-21apt-get19:25:121
3274891991612,3cyclictest3607811-21apt-get00:30:021
327489199160,5cyclictest3442336-21sed21:50:141
327489199160,4cyclictest3420628-21munin-run21:30:001
327489199160,4cyclictest3362176-21tail20:30:191
327489199160,3cyclictest3592728-21cpuspeed_turbos00:15:121
327489199160,3cyclictest252-21systemd-journal22:04:591
3274888991613,2cyclictest3509545-21kworker/u8:2+flush-259:000:00:460
3274888991612,3cyclictest3509545-21kworker/u8:2+events_unbound00:09:470
3274888991612,3cyclictest3383284-21ntp_kernel_pll_20:50:210
3274888991612,3cyclictest3363200-21kworker/u8:0+events_unbound21:21:320
3274888991612,3cyclictest3357655-21kworker/u8:1+events_unbound20:31:210
327489899159,5cyclictest3591896-21apt-get00:15:013
327489899159,4cyclictest3311558-21apt-get19:45:103
3274898991512,2cyclictest3363200-21kworker/u8:0+events_unbound21:30:163
3274898991512,2cyclictest252-21systemd-journal19:15:003
3274898991511,3cyclictest3509545-21kworker/u8:2+events_unbound23:16:463
3274898991511,3cyclictest3509545-21kworker/u8:2+events_unbound00:20:463
3274898991511,3cyclictest3321597-21kworker/u8:2+events_unbound21:12:463
327489499157,7cyclictest3337320-21cpuspeed_turbos20:10:142
327489499156,8cyclictest3509904-21tr22:55:112
327489499156,7cyclictest3394469-21latency_hist21:05:012
327489499155,9cyclictest3586872-21cat00:10:012
327489499155,9cyclictest3501529-21cat22:45:202
327489499155,8cyclictest3483082-21threads22:25:252
327489499153,10cyclictest3284005-21latency19:15:312
3274894991511,3cyclictest3389346-21sh21:00:002
327489499151,12cyclictest252-21systemd-journal23:45:002
327489499150,13cyclictest3446816-21awk21:54:592
327489499150,12cyclictest3566166-21idleruntime-cro23:49:592
327489499150,12cyclictest3520201-21cpuspeed_turbos23:05:132
327489499150,12cyclictest3321990-21df_inode19:55:142
327489199151,4cyclictest3458234-21grep22:05:151
327489199151,3cyclictest3499465-21taskset22:45:151
327489199151,3cyclictest3484108-21cpuspeed_turbos22:30:151
327489199151,3cyclictest3274959-21latency_hist19:10:001
327489199151,2cyclictest3336965-21turbostat.cron20:10:001
327489199151,2cyclictest3327387-21grep20:00:171
327489199151,2cyclictest0-21swapper/120:40:181
327489199150,5cyclictest3538426-21grep23:20:171
327489199150,3cyclictest3426452-21cut21:35:161
3274888991513,1cyclictest60350irq/132-enp0s29f1:rx-020:46:060
3274888991512,2cyclictest60350irq/132-enp0s29f1:rx-022:00:190
3274898991412,1cyclictest3342118-21idleruntime-cro20:15:013
3274898991411,2cyclictest3365722-21kthreadcore20:35:193
3274898991410,3cyclictest3363200-21kworker/u8:0+events_unbound20:55:183
327489499148,5cyclictest0-21swapper/219:10:172
327489499144,8cyclictest3429946-21netstat21:35:192
327489499142,11cyclictest3293752-21cut19:25:182
327489499142,10cyclictest3576939-21grep00:00:142
3274894991411,2cyclictest3431427-21fschecks_time21:40:142
327489499141,11cyclictest3310386-21ntpq19:40:192
327489499141,10cyclictest0-21swapper/223:55:142
327489499140,13cyclictest3478582-21sh22:25:012
327489499140,13cyclictest3420632-21awk21:29:592
327489499140,12cyclictest3592989-21tail00:15:152
327489499140,12cyclictest3555987-21latency_hist23:40:002
327489499140,12cyclictest3540777-21tr23:25:142
327489499140,12cyclictest3519784-21cron23:05:012
327489499140,12cyclictest3484403-21ps22:30:182
327489499140,12cyclictest3462276-21grep22:05:242
327489499140,12cyclictest3447618-21cut21:55:122
327489499140,11cyclictest3614094-21grep00:35:172
327489499140,11cyclictest3379747-21sed20:50:172
3274888991411,2cyclictest60350irq/132-enp0s29f1:rx-021:50:160
327489899139,3cyclictest3545562-21kworker/u8:3+events_unbound23:45:183
327489899139,3cyclictest3509545-21kworker/u8:2+events_unbound23:30:293
3274898991310,2cyclictest3321597-21kworker/u8:2+events_unbound20:45:453
3274898991310,2cyclictest3321597-21kworker/u8:2+events_unbound20:30:203
3274898991310,2cyclictest3306157-21kworker/u8:3+events_unbound20:05:203
3274898991310,2cyclictest3264846-21kworker/u8:0+events_unbound20:20:113
327489499130,11cyclictest3597914-21grep00:20:132
327489899129,2cyclictest3431235-21cpuspeed_turbos21:40:123
327489899129,2cyclictest3306157-21kworker/u8:3+events_unbound19:59:463
3274898991210,1cyclictest0-21swapper/322:05:213
3274898991210,1cyclictest0-21swapper/321:50:003
327489899117,3cyclictest3488911-21kworker/u8:1+events_unbound23:06:473
327489899116,4cyclictest273-21systemd-udevd22:53:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional