You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 20:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Mar 30, 2026 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2327951994742,3cyclictest2478939-21fwupd09:35:031
2327949994034,4cyclictest2558120-21diskstats10:50:140
2327954993935,3cyclictest2322946-21kworker/u8:4+flush-259:007:10:302
2327959993835,2cyclictest2442488-21kworker/u8:2+events_unbound09:25:153
2327954993834,3cyclictest2567510-21kworker/u8:1+flush-259:011:15:002
2327959993734,2cyclictest2458296-21kworker/u8:1+events_unbound10:00:223
2327949993735,1cyclictest2478939-21fwupd09:35:020
2327959993533,1cyclictest74-1kworker/3:1H+kblockd09:35:013
2327954993532,2cyclictest2442488-21kworker/u8:2+events_unbound09:20:242
2327951993530,3cyclictest2531312-21apt-get10:25:121
2327959993429,2cyclictest2322946-21kworker/u8:4+events_unbound07:25:133
2327954993431,2cyclictest2661821-21kworker/u8:0+flush-259:012:35:222
2327959993330,2cyclictest2604618-21kworker/u8:3+events_unbound11:45:173
2327954993329,3cyclictest2442488-21kworker/u8:2+flush-259:009:50:212
2327954993329,3cyclictest2359047-21kworker/u8:0+flush-259:008:20:252
2327954993228,3cyclictest2588914-21kworker/u8:2+flush-259:012:15:152
2327954993228,3cyclictest2353665-21kworker/u8:2+events_unbound08:40:152
2327959993129,2cyclictest2604618-21kworker/u8:3+events_unbound12:27:213
2327959993127,3cyclictest2442488-21kworker/u8:2+flush-259:009:10:163
2327954993128,2cyclictest2567510-21kworker/u8:1+events_unbound11:15:252
2327949993128,2cyclictest2353665-21kworker/u8:2+events_unbound07:45:010
2327954993027,2cyclictest2588914-21kworker/u8:2+events_unbound11:50:242
2327959992926,2cyclictest2322946-21kworker/u8:4+events_unbound07:14:443
2327959992826,2cyclictest2588914-21kworker/u8:2+events_unbound11:40:233
2327959992825,2cyclictest2442488-21kworker/u8:2+events_unbound09:40:223
2327959992823,3cyclictest2563557-21/usr/sbin/munin10:55:233
2327959992822,4cyclictest2478914-21latency_hist09:35:013
2327954992825,2cyclictest2588914-21kworker/u8:2+flush-259:011:55:262
2327954992825,2cyclictest2520758-21kworker/u8:2+events_unbound10:45:212
2327954992825,2cyclictest2442488-21kworker/u8:2+flush-259:009:30:232
2327954992824,3cyclictest2458296-21kworker/u8:1+flush-259:010:15:152
2327954992824,3cyclictest2353665-21kworker/u8:2+events_unbound08:10:222
2327954992823,3cyclictest2500202-21cp09:55:012
2327951992823,4cyclictest2567983-21kworker/u8:4+flush-259:011:00:261
2327951992823,4cyclictest252-21systemd-journal11:40:001
2327959992724,2cyclictest2353665-21kworker/u8:2+events_unbound07:50:193
2327954992724,2cyclictest2359047-21kworker/u8:0+flush-259:008:15:182
2327954992724,2cyclictest2322946-21kworker/u8:4+flush-259:007:25:182
2327954992723,3cyclictest2567510-21kworker/u8:1+flush-259:011:00:242
2327954992723,3cyclictest2452977-21kworker/u8:0+events_unbound09:15:202
2327954992723,3cyclictest2359047-21kworker/u8:0+events_unbound07:54:592
2327951992722,3cyclictest2354304-21apt-get07:35:121
2327951992721,5cyclictest509-21rs:main1
2327951992721,4cyclictest252-21systemd-journal09:45:011
2327959992624,2cyclictest2656288-21kworker/u8:4+events_unbound12:30:213
2327959992624,2cyclictest2442488-21kworker/u8:2+events_unbound09:50:083
2327959992621,3cyclictest2588914-21kworker/u8:2+events_unbound12:15:183
2327954992623,2cyclictest2604618-21kworker/u8:3+flush-259:012:05:242
2327954992623,2cyclictest2588914-21kworker/u8:2+flush-259:011:30:162
2327954992623,2cyclictest2353665-21kworker/u8:2+events_unbound08:30:242
2327954992623,2cyclictest2322946-21kworker/u8:4+events_unbound07:35:142
2327949992623,2cyclictest2442488-21kworker/u8:2+flush-259:009:55:230
2327959992523,2cyclictest2520758-21kworker/u8:2+events_unbound10:50:233
2327959992523,2cyclictest2353665-21kworker/u8:2+events_unbound08:32:213
2327959992522,2cyclictest2656288-21kworker/u8:4+events_unbound12:38:233
2327959992522,2cyclictest2442488-21kworker/u8:2+flush-259:009:04:573
2327959992522,2cyclictest2353665-21kworker/u8:2+events_unbound08:22:223
2327954992522,2cyclictest2510398-21kworker/u8:3+events_unbound10:55:022
2327954992522,2cyclictest2359047-21kworker/u8:0+flush-259:008:50:002
2327954992522,2cyclictest2359047-21kworker/u8:0+events_unbound08:55:012
2327954992519,4cyclictest2484125-21latency_hist09:40:002
2327951992517,6cyclictest2593451-21ntpq11:20:201
2327959992422,2cyclictest2567510-21kworker/u8:1+events_unbound11:15:213
2327959992421,2cyclictest2604618-21kworker/u8:3+events_unbound12:10:233
2327954992421,2cyclictest2578851-21kworker/u8:0+events_unbound12:00:022
2327954992420,3cyclictest2333006-21kworker/u8:1+flush-259:007:15:252
2327951992421,2cyclictest2442488-21kworker/u8:2+flush-259:009:10:181
2327951992419,3cyclictest2431677-21ntpq08:45:221
2327949992421,2cyclictest2567983-21kworker/u8:4+events_unbound11:05:000
2327949992421,2cyclictest2353665-21kworker/u8:2+flush-259:008:45:170
2327959992321,2cyclictest2604618-21kworker/u8:3+events_unbound12:05:333
2327959992321,2cyclictest2353665-21kworker/u8:2+events_unbound08:40:083
2327954992320,2cyclictest2604618-21kworker/u8:3+events_unbound12:25:142
2327954992319,3cyclictest2588914-21kworker/u8:2+flush-259:011:40:242
2327951992319,3cyclictest2588914-21kworker/u8:2+flush-259:011:30:231
2327951992319,3cyclictest2359047-21kworker/u8:0+flush-259:008:20:171
2327949992320,2cyclictest2541620-21apt-get10:35:120
2327959992216,5cyclictest2579068-21/usr/sbin/munin11:10:143
2327954992219,2cyclictest2578851-21kworker/u8:0+events_unbound11:20:142
2327954992219,2cyclictest2510398-21kworker/u8:3+events_unbound10:20:232
2327959992119,1cyclictest2458296-21kworker/u8:1+events_unbound10:05:193
2327959992118,2cyclictest490-21dbus-daemon10:40:003
2327954992118,2cyclictest2458296-21kworker/u8:1+events_unbound10:00:242
2327954992118,2cyclictest2432506-21kworker/u8:1+events_unbound08:55:232
2327954992118,2cyclictest2432506-21kworker/u8:1+events_unbound08:55:222
2327954992118,2cyclictest2353665-21kworker/u8:2+flush-259:007:55:232
2327954992118,2cyclictest2307425-21kworker/u8:0+events_unbound07:30:262
2327954992117,3cyclictest2520758-21kworker/u8:2+events_unbound10:50:012
2327954992117,3cyclictest2458296-21kworker/u8:1+flush-259:010:10:212
2327949992117,3cyclictest2453434-21/usr/sbin/munin09:10:150
2327959992018,2cyclictest2432506-21kworker/u8:1+events_unbound08:54:573
2327959992018,2cyclictest2353665-21kworker/u8:2+events_unbound07:55:203
2327959992016,3cyclictest252-21systemd-journal11:50:213
2327959992011,1cyclictest191rcu_preempt11:10:003
2327954992016,3cyclictest2510398-21kworker/u8:3+flush-259:010:25:122
2327951992014,4cyclictest2375088-21latency_hist07:55:011
2327949992014,4cyclictest252-21systemd-journal11:56:210
232794999201,18cyclictest509-21rs:main0
232794999200,18cyclictest1-21systemd08:56:220
232794999200,18cyclictest1-21systemd08:56:220
2327959991917,2cyclictest2359047-21kworker/u8:0+events_unbound08:49:553
2327959991917,2cyclictest2359047-21kworker/u8:0+events_unbound08:25:253
2327959991917,2cyclictest2353665-21kworker/u8:2+events_unbound08:38:233
2327959991916,2cyclictest2520758-21kworker/u8:2+events_unbound10:48:213
2327959991916,2cyclictest2359047-21kworker/u8:0+events_unbound08:55:093
2327959991916,2cyclictest2359047-21kworker/u8:0+events_unbound08:55:083
2327959991915,3cyclictest2458296-21kworker/u8:1+events_unbound10:18:083
2327959991911,6cyclictest2401683-21latency_hist08:20:003
2327954991916,2cyclictest2604618-21kworker/u8:3+flush-259:011:35:252
2327954991916,2cyclictest2442488-21kworker/u8:2+events_unbound09:45:272
2327954991916,2cyclictest2359047-21kworker/u8:0+flush-259:008:25:152
2327954991915,3cyclictest2604618-21kworker/u8:3+flush-259:012:10:252
2327954991915,3cyclictest2452977-21kworker/u8:0+flush-259:009:10:232
2327954991915,3cyclictest2442488-21kworker/u8:2+flush-259:009:25:122
2327954991913,4cyclictest2452311-21ntpq09:05:202
2327954991911,2cyclictest2489335-21apt09:45:002
2327951991914,4cyclictest2391402-21latency_hist08:10:011
2327949991915,3cyclictest252-21systemd-journal11:33:210
2327949991913,4cyclictest252-21systemd-journal08:07:220
232794999190,17cyclictest1-21systemd09:50:000
2327959991816,2cyclictest2588914-21kworker/u8:2+events_unbound11:32:213
2327959991815,2cyclictest2432506-21kworker/u8:1+events_unbound09:05:203
2327959991815,2cyclictest2392690-21kthreadcore08:10:183
2327959991815,2cyclictest2307425-21kworker/u8:0+events_unbound07:22:003
2327959991814,3cyclictest2578851-21kworker/u8:0+events_unbound11:20:583
2327959991814,3cyclictest2458296-21kworker/u8:1+events_unbound10:14:593
2327954991815,2cyclictest2567983-21kworker/u8:4+events_unbound11:05:232
2327954991815,2cyclictest2307425-21kworker/u8:0+events_unbound07:20:192
2327954991814,3cyclictest2578851-21kworker/u8:0+events_unbound11:25:202
2327954991814,3cyclictest2578851-21kworker/u8:0+events_unbound11:25:192
2327951991816,1cyclictest0-21swapper/107:50:001
2327951991815,2cyclictest2327941-21cyclictest08:04:591
2327951991814,3cyclictest2588914-21kworker/u8:2+flush-259:011:40:161
2327951991814,3cyclictest2500343-21apt-get09:55:131
2327951991813,4cyclictest2499654-21latency_hist09:55:011
2327949991816,1cyclictest2588914-21kworker/u8:2+events_unbound11:40:210
2327949991814,3cyclictest2656288-21kworker/u8:4+flush-259:012:35:000
2327949991814,3cyclictest2463559-21kworker/u8:3+flush-259:009:40:160
232795999177,1cyclictest191rcu_preempt08:00:203
2327959991714,2cyclictest2578851-21kworker/u8:0+events_unbound11:26:233
2327959991714,2cyclictest2578851-21kworker/u8:0+events_unbound11:26:233
2327959991714,2cyclictest2510398-21kworker/u8:3+events_unbound10:21:183
2327959991714,2cyclictest2458296-21kworker/u8:1+events_unbound09:47:243
2327959991713,2cyclictest491ktimers/312:05:003
2327959991711,4cyclictest2354050-21latency_hist07:35:013
2327954991714,2cyclictest2604618-21kworker/u8:3+events_unbound11:46:222
2327954991714,2cyclictest2359047-21kworker/u8:0+flush-259:007:45:012
2327954991713,3cyclictest2510398-21kworker/u8:3+events_unbound10:30:172
2327954991712,3cyclictest252-21systemd-journal10:09:222
2327951991715,1cyclictest2578814-21cron11:10:001
2327951991711,4cyclictest2462640-21ntpq09:15:211
2327949991715,1cyclictest0-21swapper/007:50:000
2327949991714,2cyclictest61150irq/133-enp0s29f1:tx-008:20:250
2327949991714,2cyclictest60350irq/132-enp0s29f1:rx-009:30:100
2327949991713,3cyclictest60350irq/132-enp0s29f1:rx-007:55:000
2327949991713,3cyclictest2588914-21kworker/u8:2+flush-259:011:35:200
2327949991710,5cyclictest2400813-21ntpq08:15:220
232795999169,5cyclictest252-21systemd-journal10:00:013
2327959991611,3cyclictest2359441-21/usr/sbin/munin07:40:153
2327954991613,2cyclictest2510398-21kworker/u8:3+events_unbound10:35:242
2327951991613,2cyclictest2588914-21kworker/u8:2+events_unbound11:55:221
2327951991613,2cyclictest2510398-21kworker/u8:3+flush-259:010:40:001
2327951991611,4cyclictest2412251-21apt-get08:30:121
2327949991614,1cyclictest60350irq/132-enp0s29f1:rx-011:25:180
2327949991614,1cyclictest60350irq/132-enp0s29f1:rx-011:25:180
2327949991614,1cyclictest60350irq/132-enp0s29f1:rx-009:00:230
2327949991614,1cyclictest60350irq/132-enp0s29f1:rx-008:43:320
2327949991613,3cyclictest2588914-21kworker/u8:2+events_unbound11:45:280
2327949991613,2cyclictest61150irq/133-enp0s29f1:tx-008:50:230
2327949991613,2cyclictest60350irq/132-enp0s29f1:rx-008:35:220
2327949991613,2cyclictest60350irq/132-enp0s29f1:rx-007:30:110
2327949991613,2cyclictest60350irq/132-enp0s29f1:rx-007:21:180
2327949991613,2cyclictest2510398-21kworker/u8:3+flush-259:010:40:190
2327949991612,2cyclictest60350irq/132-enp0s29f1:rx-007:55:190
2327949991611,4cyclictest2442488-21kworker/u8:2+flush-259:009:50:180
2327949991610,4cyclictest2656596-21latency_hist12:25:010
232795999154,5cyclictest50-21ksoftirqd/309:20:173
2327959991511,3cyclictest2651173-21kworker/u8:0+events_unbound12:20:043
2327949991513,1cyclictest60350irq/132-enp0s29f1:rx-011:50:200
2327949991513,1cyclictest60350irq/132-enp0s29f1:rx-011:11:090
2327949991513,1cyclictest60350irq/132-enp0s29f1:rx-010:46:580
2327949991513,1cyclictest60350irq/132-enp0s29f1:rx-010:20:420
2327949991513,1cyclictest60350irq/132-enp0s29f1:rx-009:16:470
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-012:35:170
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-010:30:110
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-010:05:220
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-009:20:020
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-009:05:320
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-008:30:230
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-008:25:090
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-008:00:450
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-007:15:180
2327949991512,2cyclictest60350irq/132-enp0s29f1:rx-007:12:180
2327949991512,2cyclictest2525892-21idleruntime-cro10:20:010
2327949991511,3cyclictest2604618-21kworker/u8:3+events_unbound12:00:190
2327949991511,2cyclictest0-21swapper/007:40:010
232795999144,2cyclictest191rcu_preempt09:15:133
232795999143,1cyclictest191rcu_preempt11:55:193
232795999142,1cyclictest0-21swapper/307:35:163
232795999141,4cyclictest832-21runrttasks07:15:023
232795999141,2cyclictest481rcuc/310:25:213
2327959991411,2cyclictest2567983-21kworker/u8:4+events_unbound11:03:223
2327959991411,2cyclictest2520758-21kworker/u8:2+events_unbound10:41:513
2327959991411,2cyclictest2510398-21kworker/u8:3+events_unbound10:30:093
2327954991412,2cyclictest2353665-21kworker/u8:2+flush-259:008:05:212
2327954991411,2cyclictest2510398-21kworker/u8:3+flush-259:010:40:262
2327954991411,2cyclictest2381759-21irqcore08:00:152
232795199148,1cyclictest0-21swapper/107:25:001
232795199147,5cyclictest2557374-21cp10:50:001
2327951991412,1cyclictest0-21swapper/110:10:011
2327951991411,2cyclictest2516227-21irqcore10:10:171
2327951991410,3cyclictest2604618-21kworker/u8:3+flush-259:012:05:261
2327949991413,1cyclictest60350irq/132-enp0s29f1:rx-010:25:120
2327949991412,1cyclictest60350irq/132-enp0s29f1:rx-010:55:120
2327949991412,1cyclictest60350irq/132-enp0s29f1:rx-010:02:110
2327949991412,1cyclictest60350irq/132-enp0s29f1:rx-009:25:250
2327949991412,1cyclictest60350irq/132-enp0s29f1:rx-008:11:120
2327949991411,2cyclictest60350irq/132-enp0s29f1:rx-012:08:420
2327959991310,1cyclictest191rcu_preempt08:05:123
2327954991310,2cyclictest2359047-21kworker/u8:0+flush-259:008:35:242
2327954991310,1cyclictest0-21swapper/207:45:192
232795199139,3cyclictest2510398-21kworker/u8:3+flush-259:010:40:191
232795199139,3cyclictest2353665-21kworker/u8:2+events_unbound07:55:181
232795199139,3cyclictest2322946-21kworker/u8:4+flush-259:007:35:001
232795199134,8cyclictest0-21swapper/107:20:001
2327951991310,2cyclictest2437750-21turbostat.cron08:55:001
232794999136,5cyclictest252-21systemd-journal12:20:000
232794999132,3cyclictest2588303-21ntpq11:15:230
2327949991310,2cyclictest60350irq/132-enp0s29f1:rx-011:20:230
232795499122,1cyclictest191rcu_preempt09:00:182
232795199129,2cyclictest0-21swapper/107:15:011
232795199128,3cyclictest2625245-21latency_hist11:55:011
232795199127,4cyclictest2585292-21tr11:15:191
2327951991210,1cyclictest2327941-21cyclictest12:15:141
232794999129,2cyclictest60350irq/132-enp0s29f1:rx-007:25:230
2327949991210,2cyclictest60350irq/132-enp0s29f1:rx-012:27:130
232795199119,1cyclictest0-21swapper/110:55:171
232795199118,2cyclictest2505583-21cpuspeed_turbos10:00:111
232795199116,3cyclictest252-21systemd-journal10:50:211
232795199107,2cyclictest2407504-21irqcore08:25:141
232795199105,3cyclictest252-21systemd-journal10:19:211
232795199100,3cyclictest2653773-21grep12:20:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional