You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 18:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Nov 03, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1834597994643,2cyclictest75-21kswapd011:31:453
1834594993327,4cyclictest1865852-21kworker/u8:1+flush-259:008:00:222
1834587993327,4cyclictest2698871-21systemd-journal10:10:000
1834597993228,3cyclictest1865852-21kworker/u8:1+flush-259:007:55:003
1834592993227,3cyclictest1979457-21kworker/u8:2+flush-259:009:50:011
1834597993027,2cyclictest2005014-21kworker/u8:4+flush-259:009:55:013
1834594993022,6cyclictest2698871-21systemd-journal09:55:012
1834592993025,3cyclictest2140438-21latency_hist12:05:011
1834594992926,2cyclictest2005013-21kworker/u8:3+flush-259:010:20:012
1834594992722,4cyclictest2104291-21latency_hist11:30:002
1834594992721,4cyclictest2005013-21kworker/u8:3+events_unbound10:00:142
1834592992721,4cyclictest2083651-21apt-key11:10:011
1834597992623,2cyclictest1953743-21kworker/u8:3+flush-259:009:25:173
183458799268,17cyclictest1-21systemd09:55:000
1834594992521,3cyclictest1886414-21kworker/u8:3+events_unbound08:05:242
1834594992519,5cyclictest2052421-21latency_hist10:40:002
1834594992519,4cyclictest1938324-21latency_hist08:50:002
1834597992320,2cyclictest1886414-21kworker/u8:3+flush-259:008:30:253
1834594992318,4cyclictest2005014-21kworker/u8:4+flush-259:010:40:162
1834594992317,5cyclictest1922859-21kworker/u8:2+events_unbound09:05:202
1834594992317,5cyclictest1922859-21kworker/u8:2+events_unbound09:05:202
1834587992319,3cyclictest1943060-21kworker/u8:1+flush-259:009:25:000
1834587992317,4cyclictest2030937-21ntpq10:15:230
1834597992219,2cyclictest1892053-21base6408:05:013
1834597992216,4cyclictest1834509-21latency_hist07:10:013
1834594992217,3cyclictest1886414-21kworker/u8:3+flush-259:008:25:252
1834594992216,4cyclictest2088845-21latency_hist11:15:012
1834594992215,5cyclictest1907414-21latency_hist08:20:002
183459499221,19cyclictest1876098-21cron07:50:002
1834592992217,4cyclictest2057782-21/usr/sbin/munin10:45:121
1834597992118,2cyclictest2061869-21kworker/u8:2+flush-259:011:00:193
1834597992118,2cyclictest1886414-21kworker/u8:3+flush-259:008:45:003
1834597992116,4cyclictest2698871-21systemd-journal10:05:003
1834594992117,2cyclictest0-21swapper/209:40:012
1834594992113,6cyclictest2036150-21ntpq10:20:212
1834594992113,5cyclictest2698871-21systemd-journal07:25:002
1834592992117,3cyclictest2005013-21kworker/u8:3+flush-259:009:55:161
1834592992117,3cyclictest1907442-21kworker/u8:1+events_unbound08:35:011
1834592992116,4cyclictest2052673-21/usr/sbin/munin10:40:181
1834594992017,1cyclictest0-21swapper/209:00:002
1834594992016,3cyclictest2109719-21kworker/u8:2+events_unbound11:59:552
1834594992016,3cyclictest1989692-21kworker/u8:0+flush-259:011:15:242
1834594992015,4cyclictest1964045-21kworker/u8:0+events_unbound09:15:192
1834594992014,4cyclictest1953743-21kworker/u8:3+flush-259:009:15:002
1834592992015,4cyclictest3000063-21rs:main1
1834592992014,5cyclictest2698871-21systemd-journal09:20:001
1834592992012,5cyclictest2042138-21latency_hist10:30:021
1834594991916,2cyclictest2173911-21kthreadcore12:35:182
1834594991915,3cyclictest2109719-21kworker/u8:2+events_unbound11:50:192
1834594991915,3cyclictest2109719-21kworker/u8:2+events_unbound11:45:232
1834594991915,3cyclictest2099009-21kworker/u8:0+events_unbound11:31:452
1834594991915,3cyclictest1802211-21kworker/u8:3+events_unbound07:35:132
1834594991914,3cyclictest1886414-21kworker/u8:3+events_unbound08:40:282
1834592991914,3cyclictest2037019-21latency_hist10:25:011
1834592991912,5cyclictest2698871-21systemd-journal11:31:421
1834592991912,5cyclictest2698871-21systemd-journal09:40:001
1834587991915,3cyclictest1948882-21apt-get09:00:130
1834597991815,2cyclictest1943060-21kworker/u8:1+flush-259:009:40:193
1834594991815,2cyclictest2061869-21kworker/u8:2+events_unbound11:20:172
1834594991815,2cyclictest2061869-21kworker/u8:2+events_unbound11:20:162
1834594991815,2cyclictest1989692-21kworker/u8:0+flush-259:010:45:182
1834594991815,2cyclictest1953743-21kworker/u8:3+events_unbound09:25:202
1834594991814,3cyclictest2109719-21kworker/u8:2+flush-259:011:35:162
1834594991814,3cyclictest1844881-21kworker/u8:1+events_unbound07:27:422
1834594991813,4cyclictest2698871-21systemd-journal12:14:422
1834594991813,3cyclictest1979457-21kworker/u8:2+events_unbound09:45:172
1834594991812,4cyclictest1927072-21ntpq08:35:222
183459299183,1cyclictest3002-21cron08:10:001
1834592991814,3cyclictest2021327-21apt-get10:10:121
1834587991816,1cyclictest0-21swapper/012:40:010
1834587991814,3cyclictest1886654-21apt-get08:00:120
1834587991813,3cyclictest1850669-21apt-get07:25:100
1834597991715,1cyclictest0-21swapper/310:44:593
1834597991714,2cyclictest2061869-21kworker/u8:2+flush-259:011:15:003
1834594991714,2cyclictest0-21swapper/212:05:012
1834594991713,3cyclictest1922859-21kworker/u8:2+events_unbound08:50:232
1834594991713,2cyclictest1802211-21kworker/u8:3+events_unbound07:10:222
1834594991712,3cyclictest2061869-21kworker/u8:2+events_unbound11:00:192
1834594991711,5cyclictest1844847-21apt-get07:20:002
1834594991711,4cyclictest2145587-21latency_hist12:10:002
1834594991711,4cyclictest2037446-21diskstats10:25:162
1834592991714,2cyclictest2047581-21apt-get10:35:121
1834592991714,2cyclictest2027957-21kthreadcore10:15:201
1834592991714,2cyclictest1886414-21kworker/u8:3+flush-259:008:40:191
1834592991713,3cyclictest1922859-21kworker/u8:2+flush-259:009:05:191
1834592991713,3cyclictest1922859-21kworker/u8:2+flush-259:009:05:181
1834592991713,3cyclictest1886414-21kworker/u8:3+flush-259:008:45:151
1834592991713,2cyclictest2095750-21kthreadcore11:20:181
1834592991713,2cyclictest2095750-21kthreadcore11:20:171
1834592991713,2cyclictest2068914-21irqcore10:55:181
1834592991712,3cyclictest3000061-21in:imuxsock12:40:011
1834592991712,3cyclictest2005013-21kworker/u8:3+events_unbound10:00:191
1834592991710,5cyclictest2083647-21cat11:10:001
1834587991713,3cyclictest503-21systemd-logind07:15:000
1834587991713,3cyclictest1912822-21apt-get08:25:120
1834597991611,3cyclictest2698871-21systemd-journal12:37:413
1834594991613,2cyclictest1823854-21kworker/u8:2+flush-259:007:40:162
1834594991612,3cyclictest1943060-21kworker/u8:1+events_unbound09:20:252
1834592991613,2cyclictest1974289-21idleruntime-cro09:25:011
1834592991613,2cyclictest1892182-21kworker/u8:2+flush-259:008:15:261
1834592991610,4cyclictest2698871-21systemd-journal11:45:421
183458799169,6cyclictest1906544-21ntpq08:15:210
1834587991614,1cyclictest60650irq/133-enp0s29f1:rx-011:20:210
1834587991614,1cyclictest60650irq/133-enp0s29f1:rx-011:20:200
1834587991614,1cyclictest60650irq/133-enp0s29f1:rx-010:20:250
1834587991614,1cyclictest60650irq/133-enp0s29f1:rx-007:50:150
1834587991614,1cyclictest2166192-21cron12:30:010
1834587991613,2cyclictest61450irq/134-enp0s29f1:tx-008:20:200
1834587991613,2cyclictest60650irq/133-enp0s29f1:rx-012:11:010
1834587991613,2cyclictest60650irq/133-enp0s29f1:rx-011:40:250
1834587991613,2cyclictest3002-21cron07:40:000
1834587991613,2cyclictest2048002-21irqcore10:35:170
1834587991612,3cyclictest2166469-21apt-get12:30:120
1834597991512,2cyclictest2083285-21kworker/u8:1-ext4-rsv-conversion12:05:013
1834597991512,2cyclictest1948192-21unixbench_singl08:55:253
1834597991512,1cyclictest0-21swapper/310:35:013
1834597991511,3cyclictest1802211-21kworker/u8:3+flush-259:007:25:153
1834594991512,2cyclictest1989692-21kworker/u8:0+events_unbound11:05:172
1834594991512,2cyclictest1943060-21kworker/u8:1+flush-259:009:40:222
1834594991511,3cyclictest2129506-21kworker/u8:0+events_unbound12:24:552
1834594991511,3cyclictest2005013-21kworker/u8:3+events_unbound10:33:432
1834594991511,3cyclictest1886414-21kworker/u8:3+events_unbound08:14:422
1834594991511,3cyclictest1865852-21kworker/u8:1+events_unbound07:50:142
1834592991513,1cyclictest0-21swapper/107:35:001
1834592991511,3cyclictest2109719-21kworker/u8:2+flush-259:011:35:171
1834592991511,2cyclictest1886414-21kworker/u8:3+flush-259:008:20:161
1834587991513,1cyclictest61450irq/134-enp0s29f1:tx-011:45:210
1834587991513,1cyclictest60650irq/133-enp0s29f1:rx-009:55:250
1834587991513,1cyclictest60650irq/133-enp0s29f1:rx-008:39:590
1834587991513,1cyclictest60650irq/133-enp0s29f1:rx-007:55:160
1834587991513,1cyclictest60650irq/133-enp0s29f1:rx-007:30:140
1834587991512,2cyclictest61450irq/134-enp0s29f1:tx-008:45:220
1834587991512,2cyclictest60650irq/133-enp0s29f1:rx-012:15:170
1834587991512,2cyclictest60650irq/133-enp0s29f1:rx-011:30:180
1834587991512,2cyclictest60650irq/133-enp0s29f1:rx-010:45:230
1834587991512,2cyclictest60650irq/133-enp0s29f1:rx-009:28:370
1834587991512,2cyclictest60650irq/133-enp0s29f1:rx-008:40:170
1834587991512,2cyclictest60650irq/133-enp0s29f1:rx-007:47:350
1834587991512,2cyclictest2072972-21turbostat11:00:000
1834587991510,4cyclictest2129282-21ntpq11:50:220
1834587991510,4cyclictest1845695-21diskstats07:20:150
1834587991510,1cyclictest60650irq/133-enp0s29f1:rx-009:10:180
1834597991411,2cyclictest2021735-21irqcore10:10:163
1834597991411,2cyclictest1989692-21kworker/u8:0+flush-259:010:35:193
1834597991411,2cyclictest1959241-21cpuspeed_turbos09:10:173
1834597991411,2cyclictest0-21swapper/307:50:003
1834597991410,3cyclictest2698871-21systemd-journal10:15:013
183459499149,3cyclictest2109719-21kworker/u8:2+events_unbound12:15:212
1834594991411,2cyclictest3000061-21in:imuxsock10:54:592
1834594991410,3cyclictest1855748-21kworker/u8:0+events_unbound07:30:272
1834594991410,2cyclictest0-21swapper/210:15:012
1834592991413,1cyclictest1834570-21cyclictest07:20:001
1834592991411,2cyclictest503-21systemd-logind09:45:001
1834592991411,2cyclictest1912787-21kworker/u8:0+flush-259:008:40:001
1834592991411,2cyclictest1837103-21kthreadcore07:10:191
1834592991411,1cyclictest0-21swapper/110:10:001
1834592991410,3cyclictest1865852-21kworker/u8:1+flush-259:007:55:201
183459299140,5cyclictest2078124-21munin-run11:04:591
1834587991413,1cyclictest61450irq/134-enp0s29f1:tx-011:25:220
1834587991412,2cyclictest60650irq/133-enp0s29f1:rx-009:15:240
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-012:23:120
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-012:00:240
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-011:55:030
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-011:05:150
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-011:00:260
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-010:50:230
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-010:13:340
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-010:00:200
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-009:35:250
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-009:32:030
1834587991412,1cyclictest60650irq/133-enp0s29f1:rx-008:10:270
1834587991411,2cyclictest60650irq/133-enp0s29f1:rx-011:35:230
1834587991411,2cyclictest60650irq/133-enp0s29f1:rx-009:40:280
1834587991411,2cyclictest60650irq/133-enp0s29f1:rx-008:55:200
1834587991411,2cyclictest1958908-21turbostat09:10:000
1834587991411,2cyclictest1958908-21turbostat09:10:000
1834597991310,2cyclictest2145923-21cpuspeed_turbos12:10:183
183459499139,3cyclictest2083285-21kworker/u8:1+events_unbound11:40:142
183459499139,3cyclictest2005014-21kworker/u8:4+events_unbound10:55:212
1834594991310,2cyclictest1885384-21needreboot07:55:212
183459299139,3cyclictest2165305-21ntp_kernel_pll_12:25:221
183459299131,1cyclictest21547741kthreadcore12:15:181
1834592991311,1cyclictest3000063-21rs:main1
1834592991311,1cyclictest0-21swapper/107:50:001
1834592991310,2cyclictest1861120-21fschecks_count07:35:141
183458799139,3cyclictest3000063-21rs:main0
1834587991312,1cyclictest60650irq/133-enp0s29f1:rx-012:06:580
1834587991311,1cyclictest60650irq/133-enp0s29f1:rx-011:15:160
1834587991311,1cyclictest60650irq/133-enp0s29f1:rx-008:50:190
1834587991311,1cyclictest60650irq/133-enp0s29f1:rx-007:41:520
1834587991311,1cyclictest2057538-21idleruntime-cro10:45:000
1834587991311,1cyclictest0-21swapper/008:10:000
1834587991310,2cyclictest60650irq/133-enp0s29f1:rx-007:15:210
183459799129,2cyclictest503-21systemd-logind08:55:003
183459799129,2cyclictest2082840-21unixbench_singl11:05:263
183459799129,2cyclictest1989692-21kworker/u8:0+flush-259:010:30:003
183459799129,2cyclictest1979743-21cpuspeed_turbos09:30:133
183459799129,2cyclictest1954054-21cpuspeed_turbos09:05:153
183459799129,2cyclictest1954054-21cpuspeed_turbos09:05:153
183459799129,2cyclictest1953249-21unixbench_multi09:00:263
183459799129,2cyclictest1912887-21cpuspeed_turbos08:25:133
183459799126,4cyclictest2130135-21cron11:55:013
183459499129,2cyclictest2129506-21kworker/u8:0+events_unbound12:25:282
183459499129,2cyclictest2109719-21kworker/u8:2+events_unbound12:30:282
183459499129,2cyclictest2005013-21kworker/u8:3+flush-259:010:05:152
183459499129,2cyclictest1892182-21kworker/u8:2+flush-259:008:20:292
183459499128,3cyclictest1886414-21kworker/u8:3+events_unbound08:30:242
183459299129,2cyclictest2109719-21kworker/u8:2+events_unbound12:05:191
183459299128,3cyclictest2698871-21systemd-journal11:52:411
183459299127,3cyclictest3000063-21rs:main1
1834592991210,1cyclictest0-21swapper/110:30:121
183458799129,2cyclictest60650irq/133-enp0s29f1:rx-008:30:270
1834587991211,1cyclictest60650irq/133-enp0s29f1:rx-010:30:070
1834587991211,1cyclictest60650irq/133-enp0s29f1:rx-009:45:100
183459799119,1cyclictest0-21swapper/310:50:003
183459799119,1cyclictest0-21swapper/308:45:173
183459799118,2cyclictest2166534-21cpuspeed_turbos12:30:153
183459799118,2cyclictest2140920-21fschecks_time12:05:163
183459799118,2cyclictest2130830-21irqcore11:55:163
183459799118,2cyclictest1839225-21unixbench_multi07:10:263
183459799115,4cyclictest2698871-21systemd-journal11:40:003
183459499117,3cyclictest2005013-21kworker/u8:3+events_unbound09:55:442
183459299118,2cyclictest2099009-21kworker/u8:0+events_unbound11:25:181
183459299118,2cyclictest1959241-21cpuspeed_turbos09:10:161
183459299118,2cyclictest1850743-21cpuspeed_turbos07:25:121
183459299116,3cyclictest1974538-21/usr/sbin/munin09:25:211
183459799108,1cyclictest0-21swapper/311:40:123
183459799108,1cyclictest0-21swapper/311:25:173
183459799108,1cyclictest0-21swapper/309:20:143
183459799107,2cyclictest491ktimers/310:20:133
183459299108,1cyclictest0-21swapper/108:00:121
18345929996,2cyclictest2129506-21kworker/u8:0+events_unbound12:10:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional