You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 00:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Wed Nov 05, 2025 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29963372990,1chrt2996335-21irqrtprio08:06:310
32180662890,2chrt481ktimers/311:46:423
2938780996862,5cyclictest3113678-21kworker/u8:0+flush-8:010:01:391
2938780996862,5cyclictest3113678-21kworker/u8:0+flush-8:010:01:391
29912532630,4chrt0-21swapper/108:01:311
2938778996155,4cyclictest2975734-21kworker/u8:0+events_unbound07:56:440
2938778996153,5cyclictest2970509-21latency_hist07:40:010
2938780996051,7cyclictest3144139-21kworker/u8:0+events_unbound10:51:421
2938780995948,8cyclictest3049098-21ntpq08:56:341
30173012580,1sleep13017302-21grep08:26:311
2938778995749,5cyclictest3203793-21latency_hist11:35:000
2938778995748,6cyclictest3080260-21diskstats09:31:290
2938780995647,6cyclictest3139295-21latency_hist10:30:011
2938778995546,7cyclictest3025264-21/usr/sbin/munin08:36:340
2938778995544,8cyclictest3063995-21ntpq09:11:370
2938780995449,4cyclictest3099948-21kworker/u8:3+flush-8:009:56:401
2938780995449,4cyclictest3069812-21kworker/u8:0+flush-8:009:36:411
2938778995443,9cyclictest3158526-21ntpq10:47:070
2938780995348,4cyclictest2975734-21kworker/u8:0+flush-8:008:06:421
2938780995347,3cyclictest3198574-21kworker/u8:0+flush-8:011:41:271
2938778995342,8cyclictest3179049-21latency_hist11:10:010
2938780995248,3cyclictest2960641-21kworker/u8:1+events_unbound09:20:001
2938780995246,4cyclictest2946032-21kworker/u8:2+events_unbound07:21:251
2938780995245,5cyclictest3154309-21gpgv10:45:011
2938780995045,3cyclictest2975734-21kworker/u8:0+flush-8:008:16:411
2938780995044,4cyclictest3050204-21kworker/u8:3+flush-8:009:21:261
2938780994943,4cyclictest3129632-21kworker/u8:2+events_unbound10:31:381
2938780994942,5cyclictest2975734-21kworker/u8:0+flush-8:007:56:381
2938778994942,5cyclictest2943642-21apt-get07:15:030
2938780994843,3cyclictest2960641-21kworker/u8:1+flush-8:009:51:331
2938780994842,4cyclictest2946032-21kworker/u8:2+events_unbound07:51:271
2938778994842,4cyclictest2960641-21kworker/u8:1+flush-8:010:21:230
2938780994741,4cyclictest2960641-21kworker/u8:1+flush-8:010:21:261
2938780994741,4cyclictest2887145-21kworker/u8:3+events_unbound07:26:291
2938780994740,5cyclictest2975734-21kworker/u8:0+flush-8:008:46:361
2938778994743,3cyclictest3227943-21kworker/u8:1+flush-8:012:01:430
2938778994736,8cyclictest3109585-21latency_hist10:00:000
2938780994642,3cyclictest3164246-21kworker/u8:2+flush-8:011:11:231
2938780994641,3cyclictest2946032-21kworker/u8:2+flush-8:007:37:041
2938780994640,5cyclictest3113678-21kworker/u8:0+events_unbound10:06:401
2938780994640,4cyclictest2946032-21kworker/u8:2+flush-8:007:31:221
2938780994638,6cyclictest2960641-21kworker/u8:1+flush-8:009:01:341
2938778994640,4cyclictest2960641-21kworker/u8:1+flush-8:009:46:280
2938778994640,4cyclictest2960641-21kworker/u8:1+flush-8:009:16:420
2938780994541,3cyclictest3144139-21kworker/u8:0+events_unbound10:36:421
2938780994540,4cyclictest2946032-21kworker/u8:2+flush-8:007:16:271
2938780994540,3cyclictest2960641-21kworker/u8:1+events_unbound10:16:261
2938778994540,3cyclictest3243191-21kworker/u8:0+events_unbound12:26:420
2938778994540,3cyclictest2887145-21kworker/u8:3+flush-8:007:26:340
2938778994538,5cyclictest3113678-21kworker/u8:0+flush-8:010:11:360
2938778994538,4cyclictest3227943-21kworker/u8:1+events_unbound12:00:010
2938778994535,7cyclictest3015619-21diskstats08:26:250
2938780994440,3cyclictest3164246-21kworker/u8:2+events_unbound11:40:001
2938780994440,3cyclictest3164246-21kworker/u8:2+events_unbound11:40:001
2938780994439,3cyclictest3144139-21kworker/u8:0+flush-8:010:47:011
2938780994435,6cyclictest3188951-21latency_hist11:20:011
2938778994439,3cyclictest3184001-21kworker/u8:3+flush-8:011:51:410
2938778994436,6cyclictest282-21systemd-journal11:30:000
2938780994339,3cyclictest2887145-21kworker/u8:3+flush-8:007:41:391
2938780994338,4cyclictest3014277-21ntpq08:21:381
2938780994337,5cyclictest3164246-21kworker/u8:2+flush-8:011:56:351
2938780994337,4cyclictest3243191-21kworker/u8:0+flush-8:012:21:351
2938780994335,6cyclictest3020282-21apt-get08:31:201
2938778994238,3cyclictest2960641-21kworker/u8:1+flush-8:011:21:380
2938778994237,3cyclictest2946032-21kworker/u8:2+flush-8:007:31:410
2938778994236,5cyclictest2975734-21kworker/u8:0+events_unbound07:50:010
2938780994137,3cyclictest2960641-21kworker/u8:1+events_unbound09:46:261
2938780994135,4cyclictest3079903-21kworker/u8:2+flush-8:009:31:401
2938778994134,5cyclictest3009177-21ntpq08:16:350
2938778994133,6cyclictest2946032-21kworker/u8:2+flush-8:007:21:350
2938780994036,3cyclictest2960641-21kworker/u8:1+flush-8:008:36:391
2938780994035,4cyclictest3184001-21kworker/u8:3+flush-8:011:51:371
2938780994034,4cyclictest3227943-21kworker/u8:1+events_unbound12:06:221
2938778994031,6cyclictest3248594-21latency_hist12:20:000
2938778994030,8cyclictest3113698-21ntpq10:01:350
2938778994030,8cyclictest3113698-21ntpq10:01:350
29385302403,31sleep30-21swapper/307:09:333
29383842394,10sleep10-21swapper/107:07:381
2938780993833,4cyclictest3184001-21kworker/u8:3+flush-8:011:46:311
2938780993830,6cyclictest3060245-21apt-get09:11:211
2938780993731,5cyclictest3243191-21kworker/u8:0+flush-8:012:26:381
2938780993731,5cyclictest3227943-21kworker/u8:1+flush-8:012:31:381
2938780993731,5cyclictest3164246-21kworker/u8:2+flush-8:012:36:351
2938780993728,6cyclictest282-21systemd-journal07:11:521
2938780993726,9cyclictest3090103-21/usr/sbin/munin09:41:361
2938778993732,3cyclictest2902167-21kworker/u8:0+flush-8:007:11:380
2938778993729,5cyclictest3213726-21latency_hist11:45:000
2938780993631,4cyclictest2975734-21kworker/u8:0+events_unbound07:46:381
2938780993629,5cyclictest2960641-21kworker/u8:1+flush-8:008:51:361
2938778993628,6cyclictest3159623-21/usr/sbin/munin10:51:350
2938780993531,3cyclictest3164246-21kworker/u8:2+events_unbound10:56:321
2938780993530,4cyclictest2960641-21kworker/u8:1+events_unbound08:41:391
2938780993527,6cyclictest3144139-21kworker/u8:0+flush-8:011:01:361
2938780993524,8cyclictest3005109-21latency_hist08:15:011
2938780993428,4cyclictest2960641-21kworker/u8:1+events_unbound11:21:341
2938780993425,6cyclictest3203439-21grep11:31:451
32367742330,4sleep3481ktimers/312:06:293
2938780993328,4cyclictest2960641-21kworker/u8:1+events_unbound11:26:331
2938780993326,5cyclictest3050204-21kworker/u8:3+flush-8:009:26:351
2938783993222,8cyclictest282-21systemd-journal09:35:002
2938780993227,3cyclictest3144139-21kworker/u8:0+events_unbound11:06:461
2938780993226,4cyclictest2960641-21kworker/u8:1+flush-8:009:06:341
2938778993227,4cyclictest2960641-21kworker/u8:1+flush-8:008:31:400
2938778993227,3cyclictest2960641-21kworker/u8:1+flush-8:009:41:390
2938780993125,4cyclictest3113678-21kworker/u8:0+flush-8:010:11:371
2938778993127,3cyclictest2960641-21kworker/u8:1+flush-8:008:11:400
2938778993125,4cyclictest2960641-21kworker/u8:1+events_unbound08:41:270
2938778993124,5cyclictest587-21runrttasks11:01:310
2938778993124,5cyclictest3243260-21grep12:11:420
2938778993124,5cyclictest2960641-21kworker/u8:1+events_unbound11:20:000
2938778993121,7cyclictest2938579-21latency_hist07:10:010
2938780993023,5cyclictest3227943-21kworker/u8:1+flush-8:012:01:391
2938778993026,2cyclictest3074764-21systemd-run09:25:000
2938778993024,4cyclictest282-21systemd-journal10:05:010
2938778993021,7cyclictest3238465-21latency_hist12:10:010
2938783992924,3cyclictest3065397-21tr09:16:312
2938778992919,8cyclictest3207936-21ntpq11:36:340
2938778992919,8cyclictest3207936-21ntpq11:36:340
2938778992919,7cyclictest3079657-21latency_hist09:30:000
2938783992823,3cyclictest2995674-21aten2_r5power_e08:06:262
2938783992822,4cyclictest3039828-21apt-get08:50:002
2938778992823,3cyclictest2975734-21kworker/u8:0+events_unbound08:46:400
2938778992822,4cyclictest2975734-21kworker/u8:0+events_unbound08:01:370
2938778992821,6cyclictest2960641-21kworker/u8:1+events_unbound10:20:010
2938783992724,2cyclictest3168804-21unixbench_multi10:56:422
2938778992722,4cyclictest2946032-21kworker/u8:2+flush-8:007:51:400
2938778992721,4cyclictest3049981-21sendmail09:00:000
2938783992623,2cyclictest3095138-21aten2_r5power_c09:46:252
2938783992623,2cyclictest3074375-21unixbench_multi09:21:412
2938783992622,2cyclictest2985063-21grep07:51:422
2938783992621,3cyclictest2939351-21fschecks_time07:11:262
2938783992618,6cyclictest282-21systemd-journal10:40:012
293877899266,19cyclictest2928860-21kworker/0:1+events11:48:580
2938778992621,4cyclictest3164246-21kworker/u8:2+events_unbound12:36:410
2938778992621,3cyclictest2960641-21kworker/u8:1+flush-8:009:51:370
2938783992521,2cyclictest2956167-21aten2_r5power_v07:26:232
2938783992519,5cyclictest3124417-21kworker/u8:3+flush-8:010:41:402
2938783992516,7cyclictest282-21systemd-journal08:56:522
2938778992519,5cyclictest3129632-21kworker/u8:2+flush-8:010:26:260
2938783992421,2cyclictest3019607-21unixbench_singl08:26:422
2938783992420,3cyclictest3228502-21cron12:00:002
2938783992420,2cyclictest3218626-21cron11:50:002
2938783992419,3cyclictest3089401-21grep09:36:432
2938783992418,4cyclictest2944092-21cp07:15:012
2938780992420,3cyclictest3164246-21kworker/u8:2+flush-8:012:16:351
2938780992419,3cyclictest3242833-21ntpq12:11:361
293877899242,21cyclictest2928860-21kworker/0:1+events09:01:250
293877899242,21cyclictest2928860-21kworker/0:1+events08:50:020
2938778992420,3cyclictest2960641-21kworker/u8:1+events_unbound09:06:270
2938778992419,3cyclictest3164246-21kworker/u8:2+flush-8:010:56:310
2938778992418,4cyclictest2970966-21aten2_r5power_p07:41:240
2938783992319,3cyclictest3229148-21fschecks_count12:01:282
2938783992319,2cyclictest3025520-21tr08:36:252
2938783992319,2cyclictest0-21swapper/207:40:002
293877899234,18cyclictest2928860-21kworker/0:1+events11:13:570
2938778992319,3cyclictest0-21swapper/012:25:000
2938783992218,2cyclictest3158981-21unixbench_multi10:47:172
2938783992218,2cyclictest3100122-21aten2_r5power_p09:51:222
2938783992218,2cyclictest0-21swapper/210:30:002
2938783992218,2cyclictest0-21swapper/210:00:002
2938783992216,4cyclictest3113678-21kworker/u8:0+flush-8:010:01:382
2938783992216,4cyclictest3113678-21kworker/u8:0+flush-8:010:01:382
2938783992213,7cyclictest2953335-21tr07:21:312
2938778992219,2cyclictest833-21mta-sts-daemon10:31:350
2938778992218,2cyclictest0-21swapper/008:21:390
2938778992217,3cyclictest3124417-21kworker/u8:3+flush-8:010:36:300
2938778992216,4cyclictest3144139-21kworker/u8:0+flush-8:010:41:360
293877899221,10cyclictest587-21runrttasks12:31:250
293878399216,3cyclictest32080262ld11:36:372
293878399216,3cyclictest32080262ld11:36:372
2938783992118,2cyclictest0-21swapper/212:10:012
2938783992117,2cyclictest3183152-21ntp_kernel_pll_11:11:322
2938783992117,2cyclictest0-21swapper/212:11:332
2938783992113,6cyclictest282-21systemd-journal11:05:002
2938783992113,6cyclictest282-21systemd-journal09:14:522
293877899215,15cyclictest2928860-21kworker/0:1+events09:36:020
293878399207,9cyclictest2960963-21apt-get07:31:222
2938783992017,2cyclictest3125067-21fschecks_time10:16:262
2938783992016,2cyclictest3159762-21aten2_r5power_v10:51:252
2938783992016,2cyclictest3040636-21cpuspeed_turbos08:51:232
2938783992016,2cyclictest0-21swapper/207:41:332
2938783992015,4cyclictest3198511-21unixbench_singl11:26:412
2938783992015,3cyclictest3189537-21cpuspeed_turbos11:21:232
2938783992012,6cyclictest3250221-21tr12:21:292
2938783992012,6cyclictest2995251-21cron08:05:002
2938783992011,7cyclictest3114912-21cut10:06:242
293878399200,8cyclictest313012350tune2fs10:21:262
29337602204,10sleep20-21swapper/207:05:022
293878399199,8cyclictest3002384-21tr08:11:322
293878399194,3cyclictest32474902cut12:16:332
2938783991915,2cyclictest0-21swapper/210:31:262
2938783991914,3cyclictest3091034-21sed09:41:322
293878399191,13cyclictest2985754-21apt-get07:56:262
2938783991814,2cyclictest0-21swapper/212:36:422
2938783991814,2cyclictest0-21swapper/209:26:332
2938783991814,2cyclictest0-21swapper/208:31:392
2938783991810,6cyclictest3256708-21tr12:26:342
29461162175,6sleep30-21swapper/307:16:233
293878399177,8cyclictest3260500-21tr12:31:322
2938783991713,2cyclictest3034540-21grep08:41:432
2938783991713,2cyclictest0-21swapper/211:51:352
2938783991713,2cyclictest0-21swapper/209:06:252
2938783991712,3cyclictest0-21swapper/211:31:332
2938783991712,3cyclictest0-21swapper/208:16:432
293878399171,12cyclictest3184338-21apt-get11:16:232
293878399168,6cyclictest2978480-21tr07:46:312
293878399167,7cyclictest3053254-21tr09:01:292
2938783991613,2cyclictest0-21swapper/211:06:262
2938783991612,2cyclictest0-21swapper/211:42:162
2938783991612,2cyclictest0-21swapper/208:20:522
2938783991611,3cyclictest0-21swapper/210:11:322
31615322144,7sleep30-21swapper/310:51:333
32585202132,6sleep30-21swapper/312:30:013
32029292132,7sleep30-21swapper/311:31:383
31491772132,6sleep30-21swapper/310:40:013
30399772132,6sleep30-21swapper/308:50:013
30257582133,7sleep30-21swapper/308:36:293
30203252132,8sleep30-21swapper/308:31:233
32641582122,6sleep30-21swapper/312:36:273
32439172122,6sleep30-21swapper/312:15:533
32390502122,6sleep30-21swapper/312:11:233
32334032122,6sleep30-21swapper/312:03:533
32191032123,6sleep30-21swapper/311:51:253
32088162122,6sleep30-21swapper/311:40:013
32042042122,6sleep30-21swapper/311:36:243
32042042122,6sleep30-21swapper/311:36:243
31982482122,6sleep30-21swapper/311:26:413
31930922122,6sleep30-21swapper/311:21:353
31839862122,6sleep30-21swapper/311:15:013
31488892122,6sleep30-21swapper/310:36:443
31396532122,6sleep30-21swapper/310:31:273
31367942122,6sleep30-21swapper/310:26:333
31301352122,6sleep30-21swapper/310:21:273
31235612122,6sleep30-21swapper/310:11:363
31190562122,6sleep30-21swapper/310:06:433
30997092122,6sleep30-21swapper/309:50:013
30905072122,6sleep30-21swapper/309:41:303
30877262122,6sleep30-21swapper/309:36:333
30747582122,5sleep30-21swapper/309:25:013
30651592122,6sleep30-21swapper/309:16:223
30646232122,6sleep30-21swapper/309:12:533
30552492123,5sleep30-21swapper/309:06:153
30397262122,6sleep30-21swapper/308:48:533
30150442123,6sleep30-21swapper/308:25:013
30005522122,6sleep30-21swapper/308:11:253
30001832122,6sleep30-21swapper/308:10:003
29903202122,6sleep30-21swapper/308:00:013
29712522122,6sleep30-21swapper/307:41:273
29668472123,5sleep30-21swapper/307:36:543
29603262122,6sleep30-21swapper/307:26:423
29511042122,6sleep30-21swapper/307:21:253
32584312112,6sleep30-21swapper/312:29:013
32489642112,5sleep30-21swapper/312:21:223
32239482112,5sleep30-21swapper/311:56:223
31837252112,5sleep30-21swapper/311:11:523
31750302112,5sleep30-21swapper/311:06:343
31695952112,5sleep30-21swapper/311:01:243
31645362112,6sleep30-21swapper/310:56:073
31546912112,6sleep30-21swapper/310:46:153
31246382112,5sleep30-21swapper/310:15:133
31137362112,5sleep30-21swapper/310:01:373
31137362112,5sleep30-21swapper/310:01:373
31049772112,6sleep30-21swapper/309:56:233
30992512112,5sleep30-21swapper/309:46:443
30800422112,5sleep30-21swapper/309:31:273
30700222112,5sleep30-21swapper/309:20:113
30542152112,5sleep30-21swapper/309:01:373
30473712112,6sleep30-21swapper/308:56:323
30302432111,6sleep30-21swapper/308:41:233
30104192112,5sleep30-21swapper/308:21:243
30054312112,5sleep30-21swapper/308:16:233
29857262112,5sleep30-21swapper/307:56:243
29816792112,5sleep30-21swapper/307:51:303
29757572112,6sleep30-21swapper/307:45:413
29613052112,5sleep30-21swapper/307:31:283
29431312112,6sleep30-21swapper/307:11:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional