You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 14:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Sun Nov 23, 2025 12:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3420303996762,4cyclictest3651069-21kworker/u8:2+flush-8:011:16:300
3420304996153,6cyclictest3446857-21kworker/u8:0+flush-8:007:51:241
3420304996053,5cyclictest3531819-21kworker/u8:0+flush-8:009:31:231
3420304996053,5cyclictest3531819-21kworker/u8:0+flush-8:009:31:231
3420304996053,5cyclictest3531819-21kworker/u8:0+events_unbound09:16:221
3420304995954,3cyclictest3415281-21kworker/u8:1+events_unbound08:01:111
3420304995952,5cyclictest3512025-21kworker/u8:3+events_unbound08:56:421
3420304995952,5cyclictest3467458-21kworker/u8:2+events_unbound08:11:241
3420309995853,4cyclictest3531819-21kworker/u8:0+events_unbound09:26:322
3420304995650,4cyclictest3512025-21kworker/u8:3+events_unbound10:21:321
3420304995649,5cyclictest3512025-21kworker/u8:3+events_unbound09:01:231
3420303995544,9cyclictest3507420-21diskstats08:36:140
3420304995445,6cyclictest3699633-21ntpq11:46:231
3420303995446,6cyclictest3675600-21latency_hist11:25:010
3420304995346,5cyclictest3512025-21kworker/u8:3+events_unbound08:41:251
3420304995345,6cyclictest3492245-21kworker/u8:0+flush-8:008:21:261
3420304995344,7cyclictest3580490-21ntpq09:46:261
3420304995343,8cyclictest3451214-21ntpq07:36:211
3420309995248,3cyclictest3566220-21kworker/u8:2+events_unbound10:36:282
3420309995247,4cyclictest3512025-21kworker/u8:3+events_unbound08:46:262
3420304995248,3cyclictest3512025-21kworker/u8:3+flush-8:010:51:301
3420304995244,6cyclictest3467458-21kworker/u8:2+flush-8:008:06:241
3420304995244,5cyclictest3566220-21kworker/u8:2+flush-8:009:41:241
3420304995243,6cyclictest3665712-21kworker/u8:0+flush-8:011:41:241
3420303995244,7cyclictest3512025-21kworker/u8:3+flush-8:008:56:410
3420309995147,3cyclictest3704852-21kworker/u8:3+events_unbound11:51:302
3420309995146,4cyclictest3651069-21kworker/u8:2+flush-8:011:41:292
3420309995146,4cyclictest3415281-21kworker/u8:1+flush-8:007:31:452
3420303995144,5cyclictest3710610-21latency_hist12:00:010
3420304995044,4cyclictest3715597-21kworker/u8:0+events_unbound12:11:261
3420304995044,4cyclictest3665712-21kworker/u8:0+events_unbound11:21:141
3420304995044,4cyclictest3590399-21kworker/u8:0+events_unbound10:26:251
3420304995044,4cyclictest3427561-21kworker/u8:3+events_unbound08:16:281
3420304995043,5cyclictest3512025-21kworker/u8:3+flush-8:009:11:221
3420309994945,3cyclictest3566220-21kworker/u8:2+events_unbound10:01:292
3420304994942,5cyclictest3566220-21kworker/u8:2+flush-8:009:51:291
3420304994843,4cyclictest3590399-21kworker/u8:0+events_unbound10:31:261
3420304994843,4cyclictest3492245-21kworker/u8:0+events_unbound08:26:231
3420304994843,4cyclictest3492245-21kworker/u8:0+events_unbound08:26:221
3420304994842,4cyclictest3704852-21kworker/u8:3+flush-8:012:31:261
3420304994842,4cyclictest3512025-21kworker/u8:3+flush-8:009:21:241
3420304994839,6cyclictest3655043-21ntpq11:01:281
3420304994839,6cyclictest3415281-21kworker/u8:1+flush-8:007:21:221
3420304994743,3cyclictest3512025-21kworker/u8:3+events_unbound09:56:281
3420304994742,4cyclictest3651069-21kworker/u8:2+flush-8:011:31:171
3420304994742,4cyclictest3600327-21ntpq10:06:251
3420304994740,5cyclictest3651069-21kworker/u8:2+flush-8:011:16:241
3420304994739,5cyclictest3467458-21kworker/u8:2+events_unbound08:51:261
3420304994738,6cyclictest3570605-21ntpq09:36:231
3420304994738,6cyclictest3570605-21ntpq09:36:231
3420303994738,6cyclictest3437266-21latency_hist07:25:010
35821312460,2sleep3586-21kworker/3:2-events09:51:213
3420304994639,5cyclictest3446857-21kworker/u8:0+flush-8:007:41:301
3420304994635,9cyclictest3689571-21ntpq11:36:251
3420303994637,7cyclictest3420515-21/usr/sbin/munin07:11:270
3420309994541,3cyclictest3645014-21kworker/u8:0+events_unbound10:51:262
3420304994539,4cyclictest3531819-21kworker/u8:0+flush-8:009:06:251
3420304994539,4cyclictest3492245-21kworker/u8:0+flush-8:008:36:251
3420304994537,5cyclictest3625726-21kworker/u8:1+events_unbound11:26:261
3420303994535,8cyclictest3729793-21ntpq12:16:240
3420304994439,4cyclictest2583211-21kworker/u8:0+events_unbound07:11:351
3420304994439,3cyclictest3735363-21kworker/u8:0+flush-8:012:36:301
3420304994438,4cyclictest3645014-21kworker/u8:0+events_unbound10:56:311
3420304994438,4cyclictest3415281-21kworker/u8:1+flush-8:007:46:251
3420303994434,8cyclictest3694517-21ntpq11:41:230
3420304994337,4cyclictest3531819-21kworker/u8:0+events_unbound09:26:161
3420304994337,4cyclictest3520878-21kworker/u8:1+flush-8:008:46:301
3420303994335,6cyclictest3437851-21diskstats07:26:130
3420304994238,3cyclictest3625726-21kworker/u8:1+events_unbound10:46:131
3420304994227,10cyclictest282-21systemd-journal10:11:361
3420303994234,6cyclictest3650013-21ntpq10:56:270
3420304994134,5cyclictest3625726-21kworker/u8:1+flush-8:010:36:261
3420309994036,3cyclictest3512025-21kworker/u8:3+flush-8:010:34:372
3420304994035,4cyclictest3590399-21kworker/u8:0+events_unbound10:16:161
3420304994032,6cyclictest3391494-21kworker/u8:2+flush-8:007:26:221
3420303994032,5cyclictest3420131-21latency_hist07:10:010
3420309993934,4cyclictest3467458-21kworker/u8:2+events_unbound07:56:162
3420304993934,4cyclictest3471428-21ntpq07:56:291
3420304993932,6cyclictest3415281-21kworker/u8:1+flush-8:008:31:231
3420304993932,5cyclictest3704852-21kworker/u8:3+events_unbound11:56:231
3420304993932,5cyclictest3625726-21kworker/u8:1+flush-8:012:01:241
3420304993932,5cyclictest3625726-21kworker/u8:1+flush-8:011:06:421
3420309993827,9cyclictest282-21systemd-journal12:33:462
3420304993832,4cyclictest3391494-21kworker/u8:2+events_unbound07:16:311
34198572384,10sleep10-21swapper/107:06:351
3420309993733,3cyclictest3665712-21kworker/u8:0+events_unbound11:37:482
3420304993729,6cyclictest3665683-21latency_hist11:15:001
3420303993731,4cyclictest3621205-21apt-get10:31:100
3420303993727,8cyclictest3556802-21cp09:25:010
3420309993627,7cyclictest3547013-21tr09:16:122
3420304993630,5cyclictest3625726-21kworker/u8:1+events_unbound10:41:161
3420304993630,4cyclictest3625726-21kworker/u8:1+events_unbound12:26:171
3420304993629,5cyclictest3442515-21apt-get07:31:091
3420303993630,4cyclictest3630787-21latency_hist10:40:000
3420303993626,8cyclictest282-21systemd-journal10:50:000
3420303993624,10cyclictest3651069-21kworker/u8:2+flush-8:011:06:400
342030399361,32cyclictest3467036-21munin-run07:55:000
3420303993528,5cyclictest3467458-21kworker/u8:2+flush-8:008:06:250
3420303993526,6cyclictest3526634-21latency_hist08:55:000
3420309993429,4cyclictest3415281-21kworker/u8:1+events_unbound07:15:062
3420304993427,5cyclictest282-21systemd-journal12:19:461
3420303993429,4cyclictest3512025-21kworker/u8:3+events_unbound11:01:220
3420309993329,3cyclictest3415281-21kworker/u8:1+events_unbound08:30:002
3420309993329,3cyclictest3415281-21kworker/u8:1+events_unbound08:30:002
3420303993325,6cyclictest282-21systemd-journal08:35:000
3420309993226,4cyclictest3556426-21gpgv09:25:012
3420303993225,5cyclictest3462033-21latency_hist07:50:010
3420303993224,6cyclictest3415281-21kworker/u8:1+events_unbound07:31:430
3420309993127,3cyclictest3415281-21kworker/u8:1+events_unbound08:10:082
3420309993126,4cyclictest3531819-21kworker/u8:0+flush-8:009:00:392
3420309993126,3cyclictest3655792-21cron11:05:002
3420304993124,5cyclictest3704845-21ntpq11:51:261
3420303993126,3cyclictest3665712-21kworker/u8:0+events_unbound11:46:290
3420303993124,5cyclictest3595357-21ntpq10:01:240
3420303993123,6cyclictest3725503-21kworker/u8:2+events_unbound12:21:300
3420303993123,6cyclictest3725503-21kworker/u8:2+events_unbound12:21:300
3420304993024,4cyclictest3725503-21kworker/u8:2+events_unbound12:21:191
3420304993024,4cyclictest3725503-21kworker/u8:2+events_unbound12:21:191
3420304993023,5cyclictest3719694-21ntpq12:06:241
3420303993020,8cyclictest3631119-21/usr/sbin/munin10:41:230
3420309992925,3cyclictest3716003-21aten2_r5power_p12:06:132
3420309992924,4cyclictest3415281-21kworker/u8:1+events_unbound08:05:452
3420309992916,10cyclictest3611564-21diskstats10:21:162
3420304992924,4cyclictest3590399-21kworker/u8:0+flush-8:010:01:281
3420303992926,2cyclictest3619958-21needreboot10:26:230
3420303992923,5cyclictest3391494-21kworker/u8:2+flush-8:007:20:000
3420303992919,8cyclictest3664825-21ntpq11:11:230
3420309992824,3cyclictest3566220-21kworker/u8:2+events_unbound09:46:352
3420309992824,3cyclictest2583211-21kworker/u8:0+events_unbound07:11:312
3420309992824,2cyclictest3699591-21needreboot11:46:222
3420309992822,4cyclictest3511761-21idleruntime-cro08:40:012
3420303992824,3cyclictest3550467-21needreboot09:16:210
3420303992823,3cyclictest3610600-21unixbench_multi10:16:320
3420309992723,3cyclictest3730665-21sendmail12:20:002
3420309992719,6cyclictest3725566-21cp12:15:002
3420309992719,6cyclictest282-21systemd-journal10:45:002
3420303992720,5cyclictest3566154-21grep09:31:300
3420303992720,5cyclictest3566154-21grep09:31:300
3420309992620,4cyclictest3456744-21unixbench_multi07:41:322
342030399264,20cyclictest3400399-21kworker/0:1+events08:44:340
3420303992623,2cyclictest832-21mta-sts-daemon09:01:150
3420303992621,3cyclictest3605638-21unixbench_multi10:11:320
3420303992621,3cyclictest3585945-21unixbench_multi09:51:350
3420303992620,4cyclictest3625726-21kworker/u8:1+flush-8:010:51:240
3420309992520,4cyclictest3704852-21kworker/u8:3+events_unbound11:55:132
3420309992520,4cyclictest3566220-21kworker/u8:2+events_unbound09:32:282
3420309992520,4cyclictest3566220-21kworker/u8:2+events_unbound09:32:272
3420309992520,3cyclictest3750102-21unixbench_multi12:36:312
342030399252,1cyclictest0-21swapper/011:31:290
3420303992521,2cyclictest3725791-21latency_hist12:15:010
3420303992520,4cyclictest3566220-21kworker/u8:2+events_unbound10:00:010
3420303992520,4cyclictest3415281-21kworker/u8:1+flush-8:007:56:320
3420303992520,3cyclictest0-21swapper/012:40:010
3420309992419,3cyclictest3656277-21aten2_r5power_v11:06:372
3420309992419,3cyclictest3506507-21unixbench_singl08:31:302
3420309992418,4cyclictest3684556-21needreboot11:31:212
342030399242,21cyclictest0-21swapper/009:06:290
3420303992420,3cyclictest3715597-21kworker/u8:0+events_unbound12:10:000
3420303992420,3cyclictest3597100-21sort10:06:160
3420303992420,2cyclictest0-21swapper/011:51:120
3420303992419,4cyclictest3665712-21kworker/u8:0+flush-8:011:39:220
3420303992419,3cyclictest3483354-21kthreadcore08:11:190
3420303992415,6cyclictest3571775-21/usr/sbin/munin09:41:290
34198432244,12sleep20-21swapper/207:06:312
350615722314,5sleep30-21swapper/308:31:283
3420309992320,2cyclictest3526298-21unixbench_singl08:51:292
3420309992318,4cyclictest3735363-21kworker/u8:0+events_unbound12:21:582
3420309992318,4cyclictest3735363-21kworker/u8:0+events_unbound12:21:572
3420303992320,1cyclictest0-21swapper/012:01:310
3420303992319,2cyclictest832-21mta-sts-daemon07:39:370
3420303992319,2cyclictest3512025-21kworker/u8:3+flush-8:009:46:210
3420303992318,4cyclictest3704852-21kworker/u8:3+events_unbound12:31:290
3420303992318,3cyclictest3625726-21kworker/u8:1+events_unbound11:26:190
3420303992318,3cyclictest3566220-21kworker/u8:2+events_unbound09:35:120
3420303992318,3cyclictest3566220-21kworker/u8:2+events_unbound09:35:110
3420303992318,3cyclictest0-21swapper/008:16:210
3420309992218,3cyclictest3566220-21kworker/u8:2+events_unbound09:57:462
3420309992216,4cyclictest3541471-21cron09:10:002
3420309992214,6cyclictest282-21systemd-journal09:42:362
3420303992218,2cyclictest0-21swapper/007:41:160
3420303992217,4cyclictest3615082-21ntp_kernel_pll_10:21:250
3420303992217,3cyclictest0-21swapper/008:46:190
3420303992215,5cyclictest0-21swapper/008:26:220
3420303992215,5cyclictest0-21swapper/008:26:220
3420309992118,2cyclictest3566934-21aten2_r5power_v09:36:122
3420309992118,2cyclictest3566934-21aten2_r5power_v09:36:122
3420309992117,2cyclictest0-21swapper/207:55:012
3420309992112,7cyclictest3581892-21cpuspeed_turbos09:51:182
342030399213,2cyclictest161rcu_preempt12:26:280
3420303992117,2cyclictest0-21swapper/009:11:190
3420303992117,2cyclictest0-21swapper/008:01:130
3420303992116,3cyclictest584-21snmpd08:21:300
342030999202,13cyclictest3541823-21apt-get09:11:112
3420309992016,2cyclictest3432700-21aten2_r5power_c07:21:112
3420309992013,5cyclictest282-21systemd-journal10:20:002
3420303992014,4cyclictest3551975-21cut09:21:140
35306062190,2sleep3481ktimers/308:56:393
342030999192,4cyclictest0-21swapper/210:26:312
342030999192,12cyclictest3601424-21apt-get10:11:142
3420309991916,2cyclictest3512209-21aten2_r5power_p08:41:112
3420309991916,2cyclictest3496584-21unixbench_singl08:21:312
3420309991915,2cyclictest3680162-21grep11:26:312
3420309991914,3cyclictest371rcuc/210:06:192
342030999191,13cyclictest3739719-21ntpq12:26:202
3420309991910,7cyclictest3449796-21seq07:36:182
342030999184,4cyclictest0-21swapper/208:01:112
342030999183,2cyclictest421rcuog/211:11:242
3420309991814,3cyclictest0-21swapper/207:26:292
3420309991814,2cyclictest3491978-21idleruntime-cro08:20:002
3420309991814,2cyclictest0-21swapper/207:46:182
342030999181,12cyclictest3669811-21ntpq11:16:252
34198912183,8sleep30-21swapper/307:07:013
342030999172,4cyclictest0-21swapper/211:21:262
3420309991713,2cyclictest0-21swapper/208:56:482
3420309991712,3cyclictest0-21swapper/212:01:202
342030999170,8cyclictest161rcu_preempt11:00:002
342030999170,4cyclictest3636828-21/usr/sbin/munin10:46:162
37213602156,5sleep30-21swapper/312:11:193
34488192146,5sleep30-21swapper/307:36:183
36697972132,7sleep30-21swapper/311:16:263
36648192132,7sleep30-21swapper/311:11:243
36549292135,5sleep30-21swapper/311:01:243
35754172132,7sleep30-21swapper/309:41:233
35316032132,6sleep30-21swapper/309:00:013
34821282132,6sleep30-21swapper/308:10:013
37259492122,6sleep30-21swapper/312:15:213
37156562122,6sleep30-21swapper/312:05:013
37062512122,6sleep30-21swapper/311:56:153
37049152122,6sleep30-21swapper/311:51:283
36904492121,6sleep30-21swapper/311:40:013
36844772122,7sleep30-21swapper/311:31:233
36709852122,6sleep30-21swapper/311:21:113
36466502122,6sleep30-21swapper/310:56:193
36433392121,6sleep30-21swapper/310:51:223
36409152122,5sleep30-21swapper/310:50:013
36162762122,6sleep30-21swapper/310:26:133
36158862122,6sleep30-21swapper/310:24:373
36051112122,6sleep30-21swapper/310:11:253
35968832122,6sleep30-21swapper/310:06:153
35803272122,6sleep30-21swapper/309:46:253
35630472122,6sleep30-21swapper/309:31:203
35630472122,6sleep30-21swapper/309:31:203
35556042122,6sleep30-21swapper/309:21:293
35208122122,7sleep30-21swapper/308:46:253
35070122122,6sleep30-21swapper/308:35:013
34938532122,6sleep30-21swapper/308:21:213
34775022122,6sleep30-21swapper/308:06:123
34632682122,6sleep30-21swapper/307:51:193
34574042123,5sleep30-21swapper/307:45:463
34376062122,6sleep30-21swapper/307:26:113
34323452122,6sleep30-21swapper/307:20:013
34249862121,7sleep30-21swapper/307:12:353
37457512112,5sleep30-21swapper/312:35:143
37408062112,5sleep30-21swapper/312:30:023
37396882112,5sleep30-21swapper/312:26:203
37309932112,5sleep30-21swapper/312:21:113
37309932112,5sleep30-21swapper/312:21:113
37112042112,5sleep30-21swapper/312:01:163
37001892112,6sleep30-21swapper/311:46:313
36894982112,5sleep30-21swapper/311:36:243
36759112112,6sleep30-21swapper/311:26:093
36560732112,5sleep30-21swapper/311:05:463
36342342112,5sleep30-21swapper/310:41:223
36299102112,5sleep30-21swapper/310:36:253
36211652112,6sleep30-21swapper/310:30:593
36067292112,5sleep30-21swapper/310:16:183
35917922112,5sleep30-21swapper/310:01:133
35877612112,5sleep30-21swapper/309:56:193
35665012112,5sleep30-21swapper/309:35:013
35665012112,5sleep30-21swapper/309:35:013
35578282112,5sleep30-21swapper/309:26:223
35476992112,5sleep30-21swapper/309:16:193
35417872112,6sleep30-21swapper/309:11:013
35380702110,7sleep30-21swapper/309:06:203
35257732112,6sleep30-21swapper/308:51:233
35167262112,5sleep30-21swapper/308:45:013
34971762112,5sleep30-21swapper/308:25:303
34971762112,5sleep30-21swapper/308:25:303
34873122112,5sleep30-21swapper/308:16:073
34728842112,5sleep30-21swapper/308:01:153
34675352112,6sleep30-21swapper/307:56:093
34425942112,6sleep30-21swapper/307:31:293
34279412112,5sleep30-21swapper/307:16:163
34524052102,5sleep30-21swapper/307:40:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional