You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 23:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Fri Nov 21, 2025 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
415346321960,7sleep30-21swapper/312:31:213
393990421880,2sleep3471rcuc/308:56:203
39290442750,3sleep3586-21kworker/3:2+events08:46:173
3832219997443,21cyclictest4068039-21apt-get11:06:100
3832223996760,5cyclictest3832102-21kworker/u8:0+events_unbound08:06:281
3832219996758,6cyclictest4110784-21kworker/u8:0+events_unbound12:40:010
3832223996458,4cyclictest3832102-21kworker/u8:0+events_unbound07:36:301
3832223996457,5cyclictest3913760-21kworker/u8:2+flush-8:008:56:301
3832223996255,5cyclictest3952688-21kworker/u8:3+flush-8:009:56:291
40848072610,5sleep20-21swapper/211:21:222
3832223996153,5cyclictest3853896-21kworker/u8:2+flush-8:008:11:241
3832223996050,7cyclictest3913531-21latency_hist08:30:011
3832225995841,14cyclictest3933536-21gpgv08:50:002
3832223995852,5cyclictest4057574-21kworker/u8:1+events_unbound12:01:151
3832223995852,4cyclictest4007800-21kworker/u8:2+events_unbound10:45:011
3832223995851,5cyclictest3952688-21kworker/u8:3+flush-8:010:01:291
3832223995849,6cyclictest3848943-21latency_hist07:25:001
3832223995849,6cyclictest3848943-21latency_hist07:25:001
3832223995752,4cyclictest3913760-21kworker/u8:2+flush-8:009:46:291
3832223995750,5cyclictest3853896-21kworker/u8:2+events_unbound08:21:311
3832223995649,5cyclictest3963625-21kworker/u8:0+events_unbound09:26:231
3832223995550,4cyclictest4012762-21kworker/u8:1+flush-8:010:21:261
3832223995549,4cyclictest3952688-21kworker/u8:3+events_unbound10:26:291
3832223995549,4cyclictest3832102-21kworker/u8:0+events_unbound07:31:291
3832223995546,6cyclictest4037800-21latency_hist10:35:011
3832219995546,7cyclictest4057868-21/usr/sbin/munin10:56:290
3832223995450,3cyclictest3832102-21kworker/u8:0+events_unbound07:56:261
3832223995449,4cyclictest4052336-21kworker/u8:0+flush-8:011:06:321
3832223995449,4cyclictest3913760-21kworker/u8:2+events_unbound09:21:231
3832223995448,4cyclictest4077271-21kworker/u8:2+events_unbound12:16:141
3832223995448,4cyclictest3913760-21kworker/u8:2+events_unbound08:36:151
3832223995447,5cyclictest4012762-21kworker/u8:1+flush-8:010:11:231
3832223995447,5cyclictest3952688-21kworker/u8:3+flush-8:009:16:251
3832223995446,6cyclictest4052336-21kworker/u8:0+flush-8:011:31:251
3832223995446,5cyclictest3913760-21kworker/u8:2+flush-8:009:31:331
3832223995345,6cyclictest3952688-21kworker/u8:3+flush-8:009:51:291
3832223995245,5cyclictest3952688-21kworker/u8:3+flush-8:010:56:301
3832223995240,10cyclictest282-21systemd-journal11:01:421
3832219995243,6cyclictest3898679-21latency_hist08:15:010
3832219995242,7cyclictest3928450-21latency_hist08:45:010
3832223995146,3cyclictest4110784-21kworker/u8:0+events_unbound11:51:231
3832223995145,4cyclictest3913760-21kworker/u8:2+flush-8:008:41:251
3832223995144,5cyclictest2995847-21kworker/u8:3+flush-8:007:11:261
3832223995144,5cyclictest2995847-21kworker/u8:3+flush-8:007:11:251
3832219995143,5cyclictest3934005-21apt-get08:51:110
3832223995045,4cyclictest3943571-21kworker/u8:1+flush-8:009:11:281
3832223995045,3cyclictest4077271-21kworker/u8:2+flush-8:012:21:211
3832223995044,4cyclictest3982600-21ntpq09:36:271
3832223995042,6cyclictest4127825-21/usr/sbin/munin12:06:311
3832223994945,3cyclictest4077271-21kworker/u8:2+flush-8:011:46:211
3832223994943,4cyclictest4077271-21kworker/u8:2+events_unbound11:56:221
3832223994943,4cyclictest4077271-21kworker/u8:2+events_unbound11:56:221
3832223994943,4cyclictest3952688-21kworker/u8:3+flush-8:009:41:271
3832223994942,5cyclictest4077271-21kworker/u8:2+flush-8:011:41:261
3832223994942,5cyclictest4038113-21/usr/sbin/munin10:36:221
3832219994940,6cyclictest4082512-21latency_hist11:20:010
3832223994843,3cyclictest3933891-21kworker/u8:0+events_unbound09:06:191
3832223994842,4cyclictest3952688-21kworker/u8:3+events_unbound10:06:201
3832223994742,3cyclictest3853896-21kworker/u8:2+flush-8:008:01:201
3832219994737,7cyclictest3853049-21ntpq07:26:220
3832219994736,8cyclictest3992418-21ntpq09:46:250
3832223994642,3cyclictest3832102-21kworker/u8:0+flush-8:007:41:331
3832223994641,3cyclictest4110784-21kworker/u8:0+events_unbound12:35:011
3832223994640,5cyclictest4052336-21kworker/u8:0+flush-8:011:26:301
3832223994638,6cyclictest3839273-21kworker/u8:1+flush-8:007:26:221
3832219994534,8cyclictest4022009-21ntpq10:16:230
3832219994524,9cyclictest4111790-21ntpq11:46:250
3832223994440,3cyclictest3908312-21kworker/u8:3+flush-8:008:31:301
3832223994438,4cyclictest3933891-21kworker/u8:0+flush-8:009:01:261
3832223994437,5cyclictest4057574-21kworker/u8:1+flush-8:012:11:251
3832219994439,4cyclictest3913760-21kworker/u8:2+flush-8:008:36:270
3832219994433,9cyclictest3836313-21ntpq07:11:270
3832219994433,9cyclictest3836313-21ntpq07:11:270
3832223994338,3cyclictest4110784-21kworker/u8:0+events_unbound12:26:311
3832223994337,4cyclictest3913760-21kworker/u8:2+events_unbound08:51:201
3832223994237,4cyclictest4110784-21kworker/u8:0+flush-8:012:36:151
3832223994237,3cyclictest3952688-21kworker/u8:3+events_unbound10:46:211
3832223994236,4cyclictest3923341-21kworker/u8:1+flush-8:008:46:221
3832219994232,8cyclictest3887707-21ntpq08:01:250
3832219994131,7cyclictest4077542-21latency_hist11:15:010
3832223994035,4cyclictest3832102-21kworker/u8:0+flush-8:007:51:341
3832223994035,3cyclictest3853896-21kworker/u8:2+flush-8:007:46:181
3832223993935,3cyclictest4077271-21kworker/u8:2+events_unbound11:21:211
3832223993934,4cyclictest4077271-21kworker/u8:2+events_unbound11:36:161
3832223993833,4cyclictest4077271-21kworker/u8:2+events_unbound11:16:181
3832223993833,3cyclictest4007800-21kworker/u8:2+flush-8:010:16:261
3832223993832,5cyclictest4052336-21kworker/u8:0+flush-8:010:51:281
3832225993725,11cyclictest4068354-21fschecks_time11:06:172
3832225993723,12cyclictest4038357-21cpuspeed_turbos10:36:152
3832219993730,5cyclictest4103113-21apt-get11:41:110
383222599369,4cyclictest0-21swapper/208:21:132
3832223993631,3cyclictest3853896-21kworker/u8:2+events_unbound08:20:011
3832225993526,7cyclictest282-21systemd-journal09:30:002
3832225993523,11cyclictest4078172-21fschecks_count11:16:152
383222599352,13cyclictest3913873-21apt-get08:31:142
3832223993529,5cyclictest4052336-21kworker/u8:0+events_unbound11:11:231
3832219993527,6cyclictest4077271-21kworker/u8:2+events_unbound11:40:000
38318072354,9sleep10-21swapper/107:06:551
3832225993426,6cyclictest282-21systemd-journal12:30:002
3832219993423,8cyclictest3902716-21ntpq08:16:250
383222599339,8cyclictest4024801-21tr10:21:202
3832225993326,5cyclictest3888555-21apt-get08:05:002
3832225993317,14cyclictest4092071-21unixbench_singl11:26:312
3832225993316,15cyclictest3879271-21fschecks_count07:56:142
3832225993313,18cyclictest4118857-21kthreadcore11:56:182
3832225993313,18cyclictest4118857-21kthreadcore11:56:182
3832219993325,6cyclictest3839273-21kworker/u8:1+flush-8:007:16:130
3832219993325,6cyclictest3839273-21kworker/u8:1+flush-8:007:16:130
3832219993324,7cyclictest4126633-21ntpq12:01:240
3832225993222,8cyclictest282-21systemd-journal09:12:422
3832225993219,11cyclictest0-21swapper/211:40:002
3832225993212,8cyclictest4003757-21fschecks_count10:01:152
383222599321,12cyclictest3844346-21apt-get07:21:102
383222599321,12cyclictest3844346-21apt-get07:21:092
3832219993221,9cyclictest282-21systemd-journal12:15:010
383222599311,9cyclictest3923809-21apt-get08:41:112
3832225993113,2cyclictest0-21swapper/209:41:322
3832225993113,16cyclictest3908997-21aten2_r5power_p08:26:132
3832225993112,6cyclictest3949205-21fschecks_time09:06:202
383222599311,11cyclictest4057893-21apt-get10:56:122
3832219993024,4cyclictest3832102-21kworker/u8:0+flush-8:007:56:270
383221999302,26cyclictest3832087-21kworker/0:2+events07:51:320
3832219993018,10cyclictest4007237-21ntpq10:01:240
383221999301,27cyclictest3958429-21kworker/0:0+events10:11:240
383221999301,27cyclictest3832087-21kworker/0:2+events07:21:200
383221999301,27cyclictest3832087-21kworker/0:2+events07:21:200
383222599298,5cyclictest3892329-21tr08:06:182
383222599298,5cyclictest0-21swapper/209:01:212
383222599292,10cyclictest4043994-21ps10:41:222
3832225992912,2cyclictest0-21swapper/210:46:292
3832225992911,16cyclictest3978957-21cpuspeed_turbos09:36:142
383222599291,10cyclictest3999205-21ps09:56:182
3832223992925,3cyclictest3839273-21kworker/u8:1+flush-8:007:16:221
3832223992925,3cyclictest3839273-21kworker/u8:1+flush-8:007:16:221
383221999297,16cyclictest4127528-21kworker/0:1+events12:16:120
3832219992924,3cyclictest3939467-21irqcore08:56:190
3832219992924,3cyclictest3913183-21unixbench_multi08:26:310
3832219992922,5cyclictest3903955-21apt-get08:21:110
383221999291,27cyclictest3958429-21kworker/0:0+events09:31:240
38318832292,3sleep00-21swapper/007:07:550
383222599287,6cyclictest4063076-21sort11:01:252
383222599286,7cyclictest4018115-21grep10:15:012
383222599282,8cyclictest4156340-21ntpq12:31:272
3832225992822,4cyclictest3978163-21cron09:35:002
3832225992822,4cyclictest3963014-21unixbench_singl09:16:302
3832225992813,13cyclictest4137774-21aten2_r5power_e12:16:142
3832225992813,13cyclictest4033380-21cpuspeed_turbos10:31:162
3832225992811,3cyclictest0-21swapper/211:31:192
383222599281,12cyclictest4132781-21apt-get12:11:122
3832225992810,3cyclictest0-21swapper/207:20:002
3832225992810,3cyclictest0-21swapper/207:20:002
383221999289,17cyclictest3832087-21kworker/0:2+events08:31:240
3832219992820,6cyclictest4066630-21ntpq11:01:350
3832219992819,7cyclictest3933433-21latency_hist08:50:010
383222599271,6cyclictest3993556-21apt-get09:51:122
3832225992712,2cyclictest0-21swapper/210:51:252
3832225992712,13cyclictest0-21swapper/211:11:202
3832225992711,3cyclictest0-21swapper/209:21:302
383222599271,10cyclictest3894646-21ps08:11:162
3832225992710,3cyclictest0-21swapper/207:31:202
3832225992710,2cyclictest0-21swapper/211:46:302
383221999279,17cyclictest3958429-21kworker/0:0+events09:56:160
383221999272,24cyclictest0-21swapper/011:51:290
3832219992721,4cyclictest3994089-21irqcore09:51:180
383222599268,3cyclictest0-21swapper/211:45:002
383222599262,5cyclictest3859207-21apt-get07:36:132
3832225992614,10cyclictest40099512chrt10:06:212
3832225992610,3cyclictest0-21swapper/212:01:222
3832225992610,2cyclictest0-21swapper/209:00:002
383221999267,17cyclictest3958429-21kworker/0:0+events10:31:280
3832219992620,4cyclictest4097729-21latency_hist11:35:010
3832219992619,5cyclictest3953472-21latency_hist09:10:010
3832219992619,3cyclictest0-21swapper/009:41:140
3832219992617,6cyclictest3893721-21latency_hist08:10:010
383222599259,3cyclictest0-21swapper/212:36:202
383222599258,4cyclictest0-21swapper/211:51:202
383222599256,3cyclictest0-21swapper/212:25:012
383222599251,9cyclictest3874332-21diskstats07:51:242
383221999258,16cyclictest3958429-21kworker/0:0+events09:26:150
383221999252,17cyclictest3869120-21taskset07:46:100
3832219992518,5cyclictest4008719-21fschecks_time10:06:170
3832219992517,5cyclictest4092396-21latency_hist11:30:010
383222599246,3cyclictest0-21swapper/208:51:212
3832225992421,2cyclictest3919118-21fschecks_count08:36:152
3832225992420,3cyclictest3849336-21aten2_r5power_e07:26:112
3832225992410,3cyclictest0-21swapper/210:26:202
383221999249,13cyclictest3958429-21kworker/0:0+events09:21:120
383221999244,18cyclictest4032839-21kworker/0:2+events10:40:360
383221999242,21cyclictest0-21swapper/007:36:300
3832219992420,3cyclictest4122593-21sh12:00:000
3832219992420,3cyclictest4122593-21sh11:59:590
3832219992420,2cyclictest0-21swapper/012:21:280
3832219992418,4cyclictest0-21swapper/012:35:000
383222599238,3cyclictest3899371-21fschecks_time08:16:172
383222599237,6cyclictest3867898-21awk07:41:262
383222599235,7cyclictest4129015-21tr12:06:222
383222599232,1cyclictest0-21swapper/207:48:152
3832225992317,4cyclictest3988911-21fschecks_count09:46:152
383221999236,15cyclictest4032839-21kworker/0:2+events10:46:120
383221999234,17cyclictest3958429-21kworker/0:0+events10:26:240
383221999232,20cyclictest0-21swapper/011:21:310
383221999232,20cyclictest0-21swapper/010:21:300
3832219992319,3cyclictest0-21swapper/009:15:000
3832219992318,3cyclictest0-21swapper/010:36:290
3832219992219,2cyclictest0-21swapper/009:16:230
3832219992218,2cyclictest0-21swapper/012:26:200
3832219992216,4cyclictest4057251-21unixbench_singl10:51:340
3832219992215,5cyclictest3948162-21grep09:01:310
38319402224,14sleep20-21swapper/207:08:412
3832225992117,2cyclictest3832816-21fschecks_time07:11:172
3832225992117,2cyclictest3832816-21fschecks_time07:11:172
3832219992116,4cyclictest3866339-21kthreadcore07:41:240
3832219992116,3cyclictest0-21swapper/009:40:000
3832219992015,3cyclictest4128235-21idleruntime12:06:190
3832219991915,2cyclictest0-21swapper/007:31:130
38313172193,9sleep30-21swapper/307:06:293
39096232180,2sleep3471rcuc/308:26:183
383222599179,6cyclictest4022886-21basename10:20:002
40666782156,5sleep30-21swapper/311:01:373
39732172155,7sleep30-21swapper/309:30:003
41595252130,2sleep3451irq_work/312:36:193
41464282132,7sleep30-21swapper/312:21:273
41126802132,6sleep30-21swapper/311:50:013
40615812132,7sleep30-21swapper/310:56:243
40517232132,7sleep30-21swapper/310:46:243
40278762132,6sleep30-21swapper/310:25:023
40030962132,6sleep30-21swapper/310:00:013
39741752133,6sleep30-21swapper/309:31:283
39534972132,6sleep30-21swapper/309:10:013
38879562132,7sleep30-21swapper/308:01:313
38825702132,7sleep30-21swapper/307:56:213
38596792132,7sleep30-21swapper/307:36:193
41495172122,6sleep30-21swapper/312:26:213
41380862122,6sleep30-21swapper/312:16:173
41330252121,7sleep30-21swapper/312:11:163
41266082122,6sleep30-21swapper/312:01:253
41017712122,6sleep30-21swapper/311:36:293
40970572122,6sleep30-21swapper/311:31:433
40874632122,6sleep30-21swapper/311:25:013
40784062122,6sleep30-21swapper/311:16:183
40733672123,5sleep30-21swapper/311:11:163
40575152122,6sleep30-21swapper/310:54:433
40381422122,6sleep30-21swapper/310:36:133
40275132122,6sleep30-21swapper/310:21:323
40217292122,6sleep30-21swapper/310:16:233
39971432122,6sleep30-21swapper/309:51:233
39784142122,5sleep30-21swapper/309:35:013
39675822122,6sleep30-21swapper/309:21:293
39345212122,6sleep30-21swapper/308:51:193
39233312122,6sleep30-21swapper/308:38:433
39177552122,6sleep30-21swapper/308:31:293
39029932122,6sleep30-21swapper/308:16:323
38979232122,6sleep30-21swapper/308:11:263
38890992122,6sleep30-21swapper/308:06:093
38691172125,5sleep30-21swapper/307:46:103
38645652122,6sleep30-21swapper/307:41:213
38582692122,6sleep30-21swapper/307:31:293
38396252122,6sleep30-21swapper/307:16:173
38396252122,6sleep30-21swapper/307:16:173
38324702122,6sleep30-21swapper/307:11:153
38324702122,6sleep30-21swapper/307:11:153
41320552112,5sleep30-21swapper/312:06:343
41178952112,6sleep30-21swapper/311:55:593
41178952112,6sleep30-21swapper/311:55:593
41085542112,5sleep30-21swapper/311:46:203
41054742110,7sleep30-21swapper/311:41:223
40873602112,6sleep30-21swapper/311:23:573
40679792112,5sleep30-21swapper/311:05:473
40430032112,5sleep30-21swapper/310:40:433
40331282112,6sleep30-21swapper/310:31:073
40134292112,5sleep30-21swapper/310:11:133
40083472112,6sleep30-21swapper/310:05:473
39988082112,5sleep30-21swapper/309:56:163
39882932112,5sleep30-21swapper/309:45:013
39837212112,5sleep30-21swapper/309:41:153
39586842112,6sleep30-21swapper/309:15:433
39491052112,5sleep30-21swapper/309:06:203
39438652112,5sleep30-21swapper/309:01:093
39240442112,5sleep30-21swapper/308:41:163
39085072112,5sleep30-21swapper/308:24:233
38786432112,6sleep30-21swapper/307:54:533
38491712112,5sleep30-21swapper/307:25:073
38445942112,5sleep30-21swapper/307:21:143
38445942112,5sleep30-21swapper/307:21:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional