You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 13:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Sun Dec 07, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1982963993327,4cyclictest2231564-21/usr/sbin/munin23:10:161
1982958993330,2cyclictest2008678-21kworker/u8:2+flush-259:020:05:180
1982963993227,3cyclictest2039590-21kworker/u8:0+flush-259:020:35:011
1982963993125,4cyclictest2039807-21/usr/sbin/munin20:05:161
1982958993128,2cyclictest2138847-21kworker/u8:1+flush-259:021:55:170
1982958993128,2cyclictest2008678-21kworker/u8:2+events_unbound21:05:190
1982967993025,3cyclictest3000063-21rs:main2
1982967993024,4cyclictest1993226-21latency_hist19:20:022
1982963993024,5cyclictest2132594-21ntpq21:30:231
1982958993025,4cyclictest2091455-21latency_hist20:55:000
1982958993025,3cyclictest2190340-21apt-get22:30:120
1982958992923,4cyclictest3000063-21rs:main0
1982958992922,5cyclictest2287661-21ntpq00:00:230
1982967992824,3cyclictest2185129-21kworker/u8:1+flush-259:022:50:162
1982963992824,3cyclictest2133499-21kworker/u8:3+events_unbound21:45:141
1982963992824,3cyclictest2123458-21kworker/u8:0+flush-259:021:40:231
1982958992823,3cyclictest1-21systemd21:00:010
1982958992822,5cyclictest2698871-21systemd-journal00:10:010
1982958992821,5cyclictest2024106-21latency_hist19:50:000
1982963992722,3cyclictest2185129-21kworker/u8:1+events_unbound22:35:171
1982967992620,4cyclictest2200430-21latency_hist22:40:012
1982963992623,2cyclictest2159164-21kworker/u8:0+events_unbound22:00:001
1982963992622,3cyclictest2283568-21kworker/u8:1+events_unbound00:25:011
1982963992522,2cyclictest2304005-21kworker/u8:2+events_unbound00:30:191
1982963992522,2cyclictest2174420-21fwupd22:13:441
1982963992522,2cyclictest1993457-21kworker/u8:4+flush-259:019:40:261
1982958992523,1cyclictest2143603-21kworker/0:0+pm22:13:450
1982967992418,4cyclictest2184898-21latency_hist22:25:002
1982963992418,4cyclictest2698871-21systemd-journal00:00:001
1982958992417,5cyclictest2060577-21latency_hist20:25:000
1982958992319,3cyclictest2309180-21gpgv00:24:590
1982958992319,3cyclictest2070871-21kworker/u8:3+flush-259:021:05:000
1982958992317,4cyclictest1992325-21ntpq19:15:200
1982967992218,3cyclictest1982917-21kworker/u8:1+flush-259:019:25:172
1982963992219,2cyclictest2268154-21apt-get23:45:131
1982963992218,3cyclictest2148918-21kworker/u8:2+events_unbound22:00:231
1982963992218,3cyclictest1993457-21kworker/u8:4+flush-259:020:05:001
1982963992216,4cyclictest2262689-21latency_hist23:40:011
1982958992216,4cyclictest2039590-21kworker/u8:0+flush-259:020:45:000
1982958992215,4cyclictest2169397-21latency_hist22:10:010
198296999216,6cyclictest2135346-21cut21:35:173
1982969992117,3cyclictest2236495-21kworker/u8:0+flush-259:023:45:173
1982967992117,3cyclictest2246807-21latency_hist23:25:012
198296799210,3cyclictest2148857-21taskset21:49:452
1982963992118,2cyclictest2185129-21kworker/u8:1+events_unbound22:40:211
1982963992118,2cyclictest2185129-21kworker/u8:1+events_unbound22:40:201
1982958992118,2cyclictest2267898-21munin-plugin-st23:45:000
1982958992115,4cyclictest2698871-21systemd-journal22:57:450
1982958992114,5cyclictest2185129-21kworker/u8:1+flush-259:022:45:170
1982958992114,5cyclictest2185129-21kworker/u8:1+flush-259:022:45:160
1982969992016,3cyclictest2236495-21kworker/u8:0+flush-259:000:15:163
198296799200,4cyclictest2231115-21chrt23:07:432
198296799200,1cyclictest2225792-21chrt23:00:352
1982963992017,2cyclictest2236495-21kworker/u8:0+events_unbound23:30:011
1982963992017,2cyclictest2236495-21kworker/u8:0+events_unbound00:10:161
1982963992017,2cyclictest2205580-21kworker/u8:3+flush-259:023:15:281
1982963992017,2cyclictest2169625-21kworker/u8:3+events_unbound22:20:241
1982963992017,2cyclictest1967841-21kworker/u8:0+events_unbound19:20:151
1982963992016,3cyclictest2123493-21apt-get21:25:131
1982963992016,3cyclictest2039590-21kworker/u8:0+flush-259:020:40:151
1982958992016,3cyclictest2236495-21kworker/u8:0+flush-259:000:25:280
1982958992015,3cyclictest2008678-21kworker/u8:2+flush-259:021:15:250
198296999199,9cyclictest2070381-21unixbench_multi20:30:273
198296799198,9cyclictest2288753-21apt-get00:05:142
1982967991915,3cyclictest2236495-21kworker/u8:0+flush-259:023:45:162
1982967991915,3cyclictest2008678-21kworker/u8:2+flush-259:021:10:192
1982967991914,4cyclictest2117640-21apt-get21:20:012
1982967991914,3cyclictest2294153-21cut00:10:152
198296799191,16cyclictest1988055-21idleruntime-cro19:15:002
198296799190,3cyclictest2153776-21chrt21:52:062
1982963991916,2cyclictest2262922-21kworker/u8:4+flush-259:000:00:161
1982963991916,2cyclictest1935149-21kworker/u8:2+flush-259:019:10:241
1982963991915,3cyclictest2148918-21kworker/u8:2+flush-259:022:30:451
1982963991915,3cyclictest2008678-21kworker/u8:2+flush-259:021:05:201
1982963991915,2cyclictest2283568-21kworker/u8:1+events_unbound00:20:191
1982963991914,3cyclictest2008678-21kworker/u8:2+events_unbound21:20:121
1982958991916,2cyclictest2049962-21gpgv20:15:000
1982958991913,4cyclictest2071088-21/usr/sbin/munin20:35:200
198296999186,3cyclictest2043808-21ntpq20:05:213
198296999184,4cyclictest1987694-21sshd19:10:433
1982969991814,3cyclictest2096898-21apt-get21:00:143
1982969991813,3cyclictest0-21swapper/322:45:203
1982969991813,3cyclictest0-21swapper/322:45:193
1982967991812,5cyclictest2185166-21apt-get22:25:142
1982967991810,7cyclictest2050794-21fschecks_time20:15:152
198296799180,4cyclictest2278031-21chrt23:53:282
198296799180,3cyclictest2314410-21taskset00:26:172
198296799180,2cyclictest2194969-21chrt22:31:452
198296799180,2cyclictest2086091-21chrt20:47:352
198296799180,16cyclictest2044714-21idleruntime-cro20:10:002
1982963991815,2cyclictest1993457-21kworker/u8:4+events_unbound19:50:151
1982963991814,3cyclictest2205580-21kworker/u8:3+events_unbound23:05:171
1982963991814,3cyclictest2039590-21kworker/u8:0+events_unbound20:50:021
1982963991814,3cyclictest2039590-21kworker/u8:0+events_unbound20:15:241
1982963991814,3cyclictest1988292-21kworker/u8:3+events_unbound19:17:181
1982958991814,3cyclictest2112311-21gpgv21:15:010
1982958991814,2cyclictest2133482-21sh21:35:000
1982958991813,3cyclictest2698871-21systemd-journal00:38:270
1982958991812,5cyclictest2204735-21ntpq22:40:240
1982958991812,5cyclictest2204735-21ntpq22:40:230
198296999177,2cyclictest2075121-21ntpq20:35:223
198296999174,7cyclictest2262955-21apt-get23:40:123
1982969991714,2cyclictest2061258-21irqcore20:25:163
1982969991714,1cyclictest0-21swapper/322:30:003
198296799178,7cyclictest0-21swapper/220:10:282
198296799176,10cyclictest2020151-21sed19:45:182
1982967991714,2cyclictest2309476-21latency_hist00:25:002
1982967991714,2cyclictest2018046-21ntp_kernel_pll_19:40:232
198296799171,14cyclictest2060556-21cron20:25:002
198296799170,3cyclictest2034031-21taskset19:55:392
198296799170,2cyclictest2241589-21chrt23:19:212
198296799170,1cyclictest2303926-21chrt00:19:332
198296799170,1cyclictest22882342sleep200:02:102
198296799170,1cyclictest2267675-21chrt23:42:382
198296799170,1cyclictest2262364-21chrt23:36:132
198296799170,1cyclictest21641792sleep222:03:572
198296799170,1cyclictest21433692sleep221:40:292
198296799170,1cyclictest2117406-21chrt21:17:252
198296799170,1cyclictest20392832sleep220:01:432
198296799170,1cyclictest2029038-21taskset19:52:102
198296799170,1cyclictest0-21swapper/223:25:112
198296799170,1cyclictest0-21swapper/200:34:392
1982963991714,2cyclictest2039590-21kworker/u8:0+events_unbound20:55:161
1982963991713,3cyclictest2185129-21kworker/u8:1+flush-259:022:55:501
1982963991713,3cyclictest2185129-21kworker/u8:1+events_unbound23:00:161
1982963991713,3cyclictest2148918-21kworker/u8:2+events_unbound22:15:241
1982963991713,3cyclictest2081392-21apt-get20:45:151
1982963991713,3cyclictest2039590-21kworker/u8:0+flush-259:020:20:201
1982963991713,3cyclictest1982917-21kworker/u8:1+events_unbound19:49:561
1982963991713,2cyclictest2008686-21true19:35:001
1982963991712,4cyclictest2185129-21kworker/u8:1+flush-259:022:50:211
1982963991711,4cyclictest2698871-21systemd-journal22:25:441
1982958991712,3cyclictest2065999-21apt-get20:30:150
1982958991711,5cyclictest2123458-21kworker/u8:0+flush-259:021:25:260
1982958991711,4cyclictest2241924-21apt-get23:20:130
198296999169,5cyclictest2050225-21apt-key20:15:013
198296999166,9cyclictest2107018-21cat21:10:013
198296999166,2cyclictest2091869-21df_inode20:55:153
198296999165,9cyclictest2086308-21latency_hist20:50:003
198296999165,5cyclictest2080973-21runrttasks20:43:213
1982969991610,4cyclictest2278176-21latency_hist23:55:013
198296799162,12cyclictest0-21swapper/222:50:002
198296799162,12cyclictest0-21swapper/222:50:002
1982967991614,1cyclictest2096650-21turbostat21:00:002
1982967991612,3cyclictest2011396-21runrttasks19:35:182
198296799161,12cyclictest837-21snmpd20:25:122
198296799160,5cyclictest2324236-21ntpq00:35:222
198296799160,4cyclictest835-21runrttasks23:10:412
198296799160,3cyclictest2106588-21chrt21:05:422
198296799160,3cyclictest1998076-21chrt19:21:252
198296799160,2cyclictest2174234-21chrt22:11:352
198296799160,1cyclictest22784062sleep223:55:092
198296799160,1cyclictest20808092sleep220:41:222
198296799160,14cyclictest2154706-21cut21:55:152
198296799160,14cyclictest2133463-21cron21:35:012
198296799160,13cyclictest837-21snmpd21:25:142
1982963991613,2cyclictest2117635-21cron21:20:011
1982963991612,3cyclictest1982917-21kworker/u8:1+events_unbound19:56:451
198295899169,5cyclictest2314972-21/usr/sbin/munin00:30:160
1982958991613,2cyclictest61450irq/134-enp0s29f1:tx-022:50:230
1982958991613,2cyclictest60650irq/133-enp0s29f1:rx-022:20:140
1982958991613,2cyclictest60650irq/133-enp0s29f1:rx-020:16:370
1982958991613,2cyclictest60650irq/133-enp0s29f1:rx-020:03:070
1982958991612,3cyclictest2278184-21turbostat23:55:010
1982958991611,4cyclictest2252431-21kworker/u8:1+flush-259:023:30:250
1982958991611,4cyclictest2185129-21kworker/u8:1+flush-259:022:25:180
198296999159,4cyclictest2059712-21ntpq20:20:213
198296999159,4cyclictest2024484-21tr19:50:143
198296999158,5cyclictest2138983-21tr21:40:133
198296999157,7cyclictest2200432-21awk22:39:593
198296999157,6cyclictest2168456-21netstat22:05:203
198296999157,5cyclictest2232002-21cut23:10:183
198296999156,1cyclictest2240586-21memory23:15:243
198296999155,3cyclictest2148821-21chrt21:49:203
198296999154,3cyclictest2305228-21irqstats00:20:183
198296999154,2cyclictest2184500-21users22:20:263
198296999153,3cyclictest2247495-21apt-get23:25:113
198296999153,3cyclictest2029528-21/usr/sbin/munin19:55:163
1982969991511,3cyclictest2236495-21kworker/u8:0+flush-259:000:05:183
198296799154,10cyclictest2204890-21cat22:40:252
198296799154,10cyclictest2204890-21cat22:40:242
198296799153,10cyclictest2168527-21ntpq22:05:222
198296799151,12cyclictest2257569-21turbostat.cron23:34:592
198296799150,6cyclictest2066003-21taskset20:30:142
198296799150,5cyclictest2097249-21/usr/sbin/munin21:00:182
198296799150,1cyclictest2220623-21grep22:55:252
198296799150,1cyclictest20037632sleep219:30:102
198296399159,4cyclictest2698871-21systemd-journal20:37:351
1982963991512,3cyclictest2236495-21kworker/u8:0+events_unbound00:05:201
1982963991512,2cyclictest2236495-21kworker/u8:0+flush-259:023:50:161
1982963991511,3cyclictest2252431-21kworker/u8:1+flush-259:023:30:161
198295899159,4cyclictest2029528-21/usr/sbin/munin19:55:230
198295899157,6cyclictest1987201-21ntpq19:10:220
198295899156,6cyclictest2064910-21ntpq20:25:220
1982958991513,1cyclictest60650irq/133-enp0s29f1:rx-023:05:140
1982958991512,2cyclictest60650irq/133-enp0s29f1:rx-019:37:010
1982958991512,2cyclictest60650irq/133-enp0s29f1:rx-000:11:330
198296999149,4cyclictest2008653-21cron19:35:013
198296999149,3cyclictest2175232-21grep22:15:183
198296999149,3cyclictest2132573-21ntpq21:30:243
198296999149,3cyclictest2112096-21apt21:15:003
198296999149,3cyclictest2014502-21grep19:40:183
198296999148,5cyclictest2298454-21unixbench_singl00:10:263
198296999148,5cyclictest2284000-21irqcore00:00:163
198296999148,5cyclictest2159558-21cpuspeed_turbos22:00:163
198296999148,4cyclictest2315097-21grep00:30:153
198296999148,4cyclictest2246296-21grep23:20:253
198296999148,4cyclictest2120629-21tr21:20:173
198296999148,4cyclictest2118059-21latency_hist21:20:013
198296999148,4cyclictest2086989-21irqcore20:50:193
198296999147,5cyclictest2322477-21ps00:35:183
198296999147,5cyclictest2314851-21sed00:30:003
198296999147,5cyclictest2282455-21ntpq23:55:213
198296999147,5cyclictest2205583-21systemd-run22:45:013
198296999147,5cyclictest2205583-21systemd-run22:45:013
198296999147,5cyclictest2023321-21runrttasks19:45:243
198296999147,5cyclictest1992456-21ps19:15:243
198296999147,4cyclictest2035088-21irqcore20:00:183
198296999146,6cyclictest2262019-21cat23:35:263
198296999146,6cyclictest2169674-21grep22:10:173
198296999146,6cyclictest2051869-21grep20:15:183
198296999146,6cyclictest1994441-21sed19:20:173
198296999144,2cyclictest2221589-21fw_conntrack23:00:153
198296999144,2cyclictest2153585-21unixbench_singl21:50:253
1982969991411,2cyclictest2154392-21cpuspeed_turbos21:55:113
1982969991411,2cyclictest2123902-21irqcore21:25:173
1982969991411,2cyclictest0-21swapper/322:30:223
1982969991410,3cyclictest2216267-21cut22:55:133
198296799141,1cyclictest21337452sleep221:35:112
198296799140,3cyclictest2175073-21sort22:15:162
198296799140,2cyclictest2086707-21tr20:50:162
198296799140,2cyclictest1983092-21cat19:10:012
1982963991411,2cyclictest2236495-21kworker/u8:0+events_unbound00:35:231
1982963991411,2cyclictest2008678-21kworker/u8:2+flush-259:021:35:191
1982963991411,2cyclictest2008678-21kworker/u8:2+flush-259:021:10:281
1982963991410,3cyclictest2133499-21kworker/u8:3+events_unbound21:52:441
1982958991413,1cyclictest60650irq/133-enp0s29f1:rx-019:40:140
1982958991412,1cyclictest60650irq/133-enp0s29f1:rx-023:55:220
1982958991412,1cyclictest60650irq/133-enp0s29f1:rx-023:45:150
1982958991412,1cyclictest60650irq/133-enp0s29f1:rx-023:12:460
1982958991412,1cyclictest60650irq/133-enp0s29f1:rx-022:35:270
1982958991412,1cyclictest60650irq/133-enp0s29f1:rx-022:00:190
1982958991411,2cyclictest60650irq/133-enp0s29f1:rx-019:50:210
1982958991411,2cyclictest60650irq/133-enp0s29f1:rx-019:20:580
1982958991411,2cyclictest2236495-21kworker/u8:0+flush-259:000:15:200
198295899140,5cyclictest2179744-21awk22:19:590
198296999139,3cyclictest1998747-21grep19:25:143
198296999138,4cyclictest2226854-21grep23:05:163
198296999138,3cyclictest2211104-21grep22:50:123
198296999137,4cyclictest2009034-21cut19:35:133
1982969991310,2cyclictest2252813-21cpuspeed_turbos23:30:133
198296399139,3cyclictest2236495-21kworker/u8:0+events_unbound00:18:461
198296399139,3cyclictest2008678-21kworker/u8:2+flush-259:020:25:261
1982963991310,2cyclictest2263377-21irqcore23:40:151
1982958991312,1cyclictest60650irq/133-enp0s29f1:rx-021:49:080
1982958991312,1cyclictest60650irq/133-enp0s29f1:rx-021:20:310
1982958991311,1cyclictest60650irq/133-enp0s29f1:rx-021:40:350
1982958991311,1cyclictest60650irq/133-enp0s29f1:rx-021:38:350
1982958991311,1cyclictest60650irq/133-enp0s29f1:rx-019:25:150
1982958991310,2cyclictest2225739-21unixbench_singl23:00:240
198296399129,2cyclictest0-21swapper/122:10:001
198296399129,2cyclictest0-21swapper/120:15:001
198295899129,2cyclictest60650irq/133-enp0s29f1:rx-020:49:230
198295899126,4cyclictest2149160-21apt-get21:50:140
198296399118,2cyclictest2097302-21irqcore21:00:191
198295899118,2cyclictest60650irq/133-enp0s29f1:rx-023:39:450
198295899107,2cyclictest2241181-21unixbench_multi23:15:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional