You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 01:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Tue Apr 14, 2026 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13028422160,3sleep375-21kworker/3:1+mm_percpu_wq10:56:173
13698521930,2sleep3481ktimers/311:01:223
473332640,2sleep10-21swapper/109:31:241
546202630,6chrt0-21swapper/309:36:323
1698212620,4sleep10-21swapper/111:36:121
4098288995442,9cyclictest280-21systemd-journal07:15:010
4098288995247,4cyclictest55362-21kworker/u8:2+flush-8:010:31:320
4098288995242,8cyclictest280-21systemd-journal08:55:000
4098293994943,4cyclictest4125444-21kworker/u8:3+flush-8:008:01:171
4098293994940,6cyclictest184340-21latency_hist11:50:011
4098293994737,7cyclictest4174632-21latency_hist08:25:011
4098288994737,8cyclictest4139213-21ntpq07:46:270
4098293994637,7cyclictest4125444-21kworker/u8:3+flush-8:008:06:191
1439622460,2sleep3481ktimers/311:06:323
4098288994534,8cyclictest280-21systemd-journal11:35:000
4098288994438,4cyclictest4066514-21kworker/u8:0+flush-8:007:26:270
4098293994337,4cyclictest4159826-21kworker/u8:2+flush-8:009:01:201
4098293994336,5cyclictest46384-21kworker/u8:4+flush-8:009:36:171
4098293994236,4cyclictest4103280-21cp07:15:001
4098293994233,6cyclictest129970-21/usr/sbin/munin10:56:231
4098288994238,3cyclictest134874-21kworker/u8:1+flush-8:011:21:190
409608024230,6sleep20-21swapper/207:06:232
4098293994136,3cyclictest90059-21kworker/u8:1+flush-8:010:31:271
4098293994131,8cyclictest188486-21ntpq11:51:271
4098293994034,5cyclictest4125444-21kworker/u8:3+flush-8:008:41:321
4098288994035,4cyclictest46384-21kworker/u8:4+flush-8:009:36:290
4098288994034,4cyclictest280-21systemd-journal07:50:240
4098288994033,5cyclictest55362-21kworker/u8:2+events_unbound09:51:200
4098293993933,4cyclictest10377-21kworker/u8:0+flush-8:009:21:191
4098293993929,8cyclictest4140350-21/usr/sbin/munin07:51:251
4098288993834,3cyclictest4125444-21kworker/u8:3+flush-8:008:01:300
4098288993833,4cyclictest4125444-21kworker/u8:3+flush-8:008:31:160
4098288993833,4cyclictest4110013-21kworker/u8:1+flush-8:008:16:190
4098295993729,6cyclictest280-21systemd-journal10:05:012
4098293993733,3cyclictest84835-21kworker/u8:0+flush-8:011:16:291
4098293993731,4cyclictest30667-21apt-get09:16:131
4098293993728,7cyclictest10670-21/usr/sbin/munin08:56:261
4098288993733,3cyclictest55362-21kworker/u8:2+flush-8:009:46:200
4098295993626,7cyclictest280-21systemd-journal11:00:002
4098288993631,4cyclictest4066514-21kworker/u8:0+flush-8:007:36:180
4098288993631,4cyclictest4066514-21kworker/u8:0+flush-8:007:36:180
4098288993627,7cyclictest90637-21diskstats10:16:170
4098288993626,8cyclictest140155-21diskstats11:06:170
4098295993526,7cyclictest280-21systemd-journal10:07:102
4098295993525,8cyclictest280-21systemd-journal09:44:092
4098293993530,4cyclictest84835-21kworker/u8:0+events_unbound11:11:161
4098288993531,3cyclictest10377-21kworker/u8:0+flush-8:009:21:310
4098288993525,8cyclictest228021-21ntpq12:31:270
4098295993428,4cyclictest4140689-21fschecks_time07:51:172
4098293993429,3cyclictest175562-21irqrtprio11:41:231
4098293993427,5cyclictest134874-21kworker/u8:1+flush-8:011:21:271
4098295993327,4cyclictest90737-21fschecks_time10:16:182
4098295993327,4cyclictest4159481-21unixbench_singl08:06:342
4098293993327,4cyclictest144738-21latency_hist11:10:011
4098293993325,6cyclictest4179936-21apt-get08:31:111
4098293993324,6cyclictest169496-21latency_hist11:35:001
4098293993323,7cyclictest4164742-21latency_hist08:15:011
4098288993329,3cyclictest84835-21kworker/u8:0+flush-8:010:21:250
4098288993326,6cyclictest4125444-21kworker/u8:3+flush-8:008:37:110
4098288993326,5cyclictest30006-21grep09:11:340
1536372330,4sleep0141ktimers/011:16:250
4098295993226,4cyclictest4102274-21needreboot07:11:252
4098293993227,4cyclictest84835-21kworker/u8:0+events_unbound10:20:011
4098293993226,4cyclictest209258-21kworker/u8:3+flush-8:012:16:201
4098293993225,5cyclictest40512-21/usr/sbin/munin09:26:261
4098293993225,5cyclictest40512-21/usr/sbin/munin09:26:261
4098293993225,4cyclictest4109953-21grep07:16:311
4098293993223,6cyclictest4130487-21apt-get07:41:141
4098288993228,3cyclictest90059-21kworker/u8:1+flush-8:010:36:320
4098288993227,4cyclictest4098432-21kworker/u8:2+flush-8:007:31:180
4098288993225,6cyclictest4125444-21kworker/u8:3+flush-8:009:06:170
4098295993126,3cyclictest218651-21unixbench_singl12:21:342
4098295993125,4cyclictest45848-21fschecks_time09:31:192
4098295993125,4cyclictest150300-21fschecks_count11:16:182
4098295993122,7cyclictest280-21systemd-journal07:47:072
4098293993123,6cyclictest280-21systemd-journal12:24:091
4098293993121,7cyclictest280-21systemd-journal07:29:061
4098293993121,7cyclictest233821-21latency_hist12:40:011
4098288993126,4cyclictest70420-21kworker/u8:3+flush-8:009:58:400
4098288993126,4cyclictest4159826-21kworker/u8:2+flush-8:008:11:300
4098288993126,3cyclictest185409-21irqrtprio11:51:210
4098288993125,5cyclictest4125444-21kworker/u8:3+flush-8:007:56:180
4098288993124,5cyclictest118785-21kworker/u8:2+flush-8:011:01:180
4098288993122,7cyclictest49264-21ntpq09:31:260
4098288993122,7cyclictest118913-21ntpq10:41:260
4098288993121,6cyclictest35258-21latency_hist09:20:010
4098295993025,3cyclictest209663-21fschecks_count12:16:152
4098295993024,4cyclictest224491-21cpuspeed_turbos12:31:162
4098295993020,7cyclictest4160419-21fschecks_count08:11:172
4098293993023,5cyclictest70771-21cpuspeed_turbos09:56:161
4098293993023,5cyclictest4169361-21unixbench_multi08:16:341
4098293993020,8cyclictest280-21systemd-journal12:00:011
4098288993026,3cyclictest4159826-21kworker/u8:2+flush-8:009:01:380
4098288993026,3cyclictest149686-21systemd-run11:15:000
4098288993025,4cyclictest209258-21kworker/u8:3+flush-8:012:15:400
4098288993023,5cyclictest232889-21grep12:36:260
4098288993022,5cyclictest174477-21latency_hist11:40:010
4098295992925,2cyclictest114462-21unixbench_multi10:36:352
4098295992924,3cyclictest160346-21fschecks_time11:26:202
4098295992924,3cyclictest160346-21fschecks_time11:26:192
4098295992923,4cyclictest70802-21fschecks_count09:56:172
4098293992924,4cyclictest4159826-21kworker/u8:2+flush-8:009:05:301
4098293992923,4cyclictest139201-21unixbench_multi11:01:341
4098293992921,6cyclictest4130102-21cron07:40:001
4098293992921,6cyclictest4130102-21cron07:40:001
4098293992920,7cyclictest4174980-21apt-get08:26:141
4098288992925,2cyclictest4488-21needreboot08:46:250
4098288992924,4cyclictest4098432-21kworker/u8:2+flush-8:007:11:310
4098288992924,4cyclictest4066514-21kworker/u8:0+flush-8:007:22:380
4098288992924,4cyclictest109921-21kworker/u8:3+flush-8:010:51:280
4098288992924,3cyclictest15721-21latency_hist09:00:010
4098288992922,5cyclictest184575-21kworker/u8:2+flush-8:012:01:260
4098288992921,6cyclictest123905-21ntpq10:46:280
4098295992824,2cyclictest99981-21turbostat.cron10:25:002
4098295992824,2cyclictest149334-21unixbench_singl11:11:322
4098295992823,3cyclictest15963-21aten2_r5power_e09:01:172
4098295992818,8cyclictest280-21systemd-journal08:39:592
4098293992824,2cyclictest201688-21kthreadcore12:06:221
4098293992823,4cyclictest4110013-21kworker/u8:1+flush-8:007:55:311
4098293992823,3cyclictest84752-21unixbench_multi10:06:341
4098293992822,4cyclictest4110013-21kworker/u8:1+events_unbound08:40:011
4098293992821,6cyclictest104004-21needreboot10:26:231
4098288992824,3cyclictest10377-21kworker/u8:0+events_unbound09:26:330
4098288992824,3cyclictest10377-21kworker/u8:0+events_unbound09:26:330
4098288992824,2cyclictest216534-21kthreadcore12:21:230
4098288992823,4cyclictest70420-21kworker/u8:3+flush-8:010:01:320
4098288992823,4cyclictest4110013-21kworker/u8:1+events_unbound08:41:190
4098288992823,4cyclictest4110013-21kworker/u8:1+events_unbound08:21:210
4098288992821,5cyclictest4159826-21kworker/u8:2+events_unbound08:26:190
4098295992724,2cyclictest4170182-21aten2_r5power_v08:21:172
4098295992724,2cyclictest4115948-21fschecks_time07:26:182
4098295992723,2cyclictest35656-21aten2_r5power_c09:21:162
4098295992721,4cyclictest4190208-21fschecks_time08:41:192
4098295992719,6cyclictest50826-21fschecks_time09:36:172
4098295992718,7cyclictest4125799-21fschecks_count07:36:182
4098295992718,7cyclictest4125799-21fschecks_count07:36:182
4098293992723,3cyclictest4098283-21cyclictest10:51:181
4098293992722,3cyclictest4125444-21kworker/u8:3+events_unbound08:51:161
4098293992721,5cyclictest4098432-21kworker/u8:2+events_unbound07:31:301
4098293992721,5cyclictest134874-21kworker/u8:1+flush-8:012:01:271
4098293992721,3cyclictest0-21swapper/110:21:141
4098288992722,4cyclictest84835-21kworker/u8:0+flush-8:011:45:310
4098288992722,3cyclictest208686-21unixbench_multi12:11:330
4098288992721,4cyclictest184575-21kworker/u8:2+flush-8:011:55:310
4098288992720,5cyclictest84792-21grep10:06:350
4098295992623,2cyclictest194245-21idleruntime-cro12:00:002
4098295992621,4cyclictest20977-21cron09:20:002
4098295992621,4cyclictest144645-21latency_hist11:10:002
4098295992621,4cyclictest1070-21cpuspeed_turbos08:46:172
4098295992620,4cyclictest26108-21fschecks_time09:11:192
4098295992620,4cyclictest175765-21kthreadcore11:41:242
4098295992620,4cyclictest121051-21kthreadcore10:46:222
4098295992620,4cyclictest115416-21cpuspeed_turbos10:41:172
4098295992618,6cyclictest280-21systemd-journal10:54:102
4098293992622,2cyclictest0-21swapper/112:31:191
4098293992622,2cyclictest0-21swapper/109:15:001
4098293992621,4cyclictest55362-21kworker/u8:2+flush-8:009:41:331
4098293992621,3cyclictest85673-21tr10:11:161
4098293992620,4cyclictest204464-21apt-get12:11:141
4098293992619,5cyclictest84835-21kworker/u8:0+flush-8:010:46:191
4098293992619,5cyclictest4580-21ntpq08:46:271
4098288992622,3cyclictest84835-21kworker/u8:0+flush-8:010:11:180
4098288992621,3cyclictest4131018-21irqcore07:41:200
4098295992520,3cyclictest229471-21fschecks_count12:36:192
4098295992520,3cyclictest14941-21unixbench_multi08:56:342
4098295992519,4cyclictest4175331-21fschecks_time08:26:192
4098295992518,5cyclictest280-21systemd-journal12:27:102
4098293992521,2cyclictest0-21swapper/110:40:001
4098293992520,3cyclictest1-21systemd07:47:071
4098293992518,5cyclictest4110709-21aten2_r5power_e07:21:131
4098293992518,5cyclictest115278-21aten2_r5power_p10:41:161
4098293992517,6cyclictest65026-21latency_hist09:50:011
4098288992520,4cyclictest184575-21kworker/u8:2+flush-8:012:26:290
4098288992519,4cyclictest148896-21kworker/u8:4+events_unbound11:26:170
4098288992519,4cyclictest148896-21kworker/u8:4+events_unbound11:26:160
4098288992519,4cyclictest134874-21kworker/u8:1+events_unbound11:45:010
40980462253,3sleep00-21swapper/007:09:060
4098295992420,3cyclictest4150550-21fschecks_count08:01:172
4098295992419,4cyclictest55362-21kworker/u8:2+flush-8:009:51:192
4098295992419,3cyclictest6049-21fschecks_count08:51:172
4098295992419,3cyclictest4130801-21fschecks_time07:41:192
4098295992418,4cyclictest204696-21cpuspeed_turbos12:11:172
4098295992418,3cyclictest184047-21grep11:46:362
4098295992415,7cyclictest4145646-21fschecks_time07:56:172
4098293992418,5cyclictest219370-21aten2_r5power_e12:26:161
4098293992416,6cyclictest75560-21apt-get10:01:131
4098293992416,6cyclictest280-21systemd-journal11:27:091
4098293992416,6cyclictest280-21systemd-journal11:27:091
4098288992419,2cyclictest0-21swapper/010:56:230
4098288992415,7cyclictest59246-21ntpq09:41:280
4098295992320,2cyclictest44782-21unixbench_multi09:26:342
4098295992320,2cyclictest44782-21unixbench_multi09:26:342
4098295992320,2cyclictest4105985-21fschecks_count07:16:172
4098295992319,3cyclictest184908-21cpuspeed_turbos11:51:162
4098295992318,3cyclictest4110870-21cpuspeed_turbos07:21:142
4098295992315,6cyclictest4183616-21needreboot08:31:232
4098295992315,6cyclictest0-21swapper/209:50:012
2099742230,2sleep375-21kworker/3:1+events12:16:193
409829599228,8cyclictest0-21swapper/211:22:452
4098295992218,3cyclictest105547-21fschecks_count10:31:182
4098295992217,4cyclictest84835-21kworker/u8:0+flush-8:010:11:302
409829399224,16cyclictest68760-21tr09:51:241
4098288992216,4cyclictest55362-21kworker/u8:2+flush-8:010:26:200
409828899220,20cyclictest4159575-21sshd08:07:080
40935292224,10sleep10-21swapper/107:05:381
4098295992117,2cyclictest0-21swapper/211:35:002
4098288992119,1cyclictest184575-21kworker/u8:2+events_unbound12:06:080
4098295992016,2cyclictest0-21swapper/211:40:012
4098295992015,3cyclictest0-21swapper/212:06:262
4098295992014,4cyclictest194885-21fschecks_count12:01:172
40935732193,9sleep30-21swapper/307:06:123
41589672188,6sleep30-21swapper/308:06:283
409829599181,12cyclictest100321-21apt-get10:26:142
409829599172,4cyclictest0-21swapper/209:08:362
4098295991714,2cyclictest4120605-21aten2_r5power_e07:31:162
4098295991711,4cyclictest4110013-21kworker/u8:1+events_unbound08:19:232
4098295991711,2cyclictest0-21swapper/211:01:322
1884692142,8sleep30-21swapper/311:51:283
744732130,9sleep30-21swapper/309:56:303
41870442130,4sleep3481ktimers/308:36:243
41845192132,7sleep30-21swapper/308:35:013
41360342131,7sleep30-21swapper/307:46:213
41348122133,7sleep30-21swapper/307:41:363
41022812132,7sleep30-21swapper/307:11:263
359942133,6sleep30-21swapper/309:21:203
2289862132,6sleep30-21swapper/312:35:013
208182132,7sleep30-21swapper/309:06:153
1997702133,7sleep30-21swapper/312:06:173
152922132,6sleep30-21swapper/309:00:013
845272122,6sleep30-21swapper/310:06:343
642722122,6sleep30-21swapper/309:46:323
551452122,6sleep30-21swapper/309:40:023
54632122,6sleep30-21swapper/308:50:023
495542122,6sleep30-21swapper/309:31:353
41746592121,6sleep30-21swapper/308:25:023
41704402122,6sleep30-21swapper/308:21:203
41689432123,5sleep30-21swapper/308:16:313
41499552122,6sleep30-21swapper/308:00:023
41488572122,6sleep30-21swapper/307:56:243
41253322122,6sleep30-21swapper/307:35:023
41253322122,6sleep30-21swapper/307:35:023
41205792122,6sleep30-21swapper/307:31:163
41167672122,6sleep30-21swapper/307:26:233
41144592122,6sleep30-21swapper/307:21:243
304782122,6sleep30-21swapper/309:15:023
257622122,6sleep30-21swapper/309:11:153
2241752122,6sleep30-21swapper/312:30:243
2230552122,6sleep30-21swapper/312:26:273
2041412122,6sleep30-21swapper/312:10:023
1938122122,6sleep30-21swapper/311:56:333
1794232122,5sleep30-21swapper/311:45:013
1757382122,6sleep30-21swapper/311:41:253
1649332122,6sleep30-21swapper/311:31:173
1602372122,6sleep30-21swapper/311:26:203
1602372122,6sleep30-21swapper/311:26:203
159102122,6sleep30-21swapper/309:01:173
1139342122,6sleep30-21swapper/310:36:283
1049602122,6sleep30-21swapper/310:30:023
952662112,5sleep30-21swapper/310:20:023
909092112,5sleep30-21swapper/310:16:213
854552112,5sleep30-21swapper/310:11:163
800682112,5sleep30-21swapper/310:04:093
658412112,6sleep30-21swapper/309:51:213
41897332112,5sleep30-21swapper/308:40:023
41642402112,5sleep30-21swapper/308:11:333
41403792110,6sleep30-21swapper/307:51:143
41045622112,5sleep30-21swapper/307:15:033
409932112,5sleep30-21swapper/309:26:213
409932112,5sleep30-21swapper/309:26:213
2179022112,5sleep30-21swapper/312:21:253
1944732112,5sleep30-21swapper/312:00:033
1698442112,5sleep30-21swapper/311:36:153
1549022112,5sleep30-21swapper/311:20:023
1499172112,6sleep30-21swapper/311:15:243
1449262112,5sleep30-21swapper/311:10:023
1281242112,6sleep30-21swapper/310:51:253
12752112,5sleep30-21swapper/308:46:203
1200062112,5sleep30-21swapper/310:45:023
1153552112,5sleep30-21swapper/310:41:173
1005662112,5sleep30-21swapper/310:26:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional