You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 13:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Tue Mar 31, 2026 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1820187994336,4cyclictest1923845-21latency_hist20:50:011
1820192994237,3cyclictest2006854-21kworker/u8:1+flush-259:022:20:152
1820192994135,4cyclictest1882655-21latency_hist20:10:002
1820192993935,3cyclictest2058282-21kworker/u8:3+flush-259:023:05:002
1820196993833,3cyclictest1985839-21/usr/sbin/munin21:50:233
1820187993832,4cyclictest2115306-21apt-get23:55:141
1820192993633,2cyclictest2115063-21kworker/u8:0+flush-259:000:05:012
1820182993632,3cyclictest1856491-21kworker/u8:2+flush-259:020:50:250
1820187993430,3cyclictest2027562-21apt-get22:30:121
1820182993227,4cyclictest2089208-21latency_hist23:30:000
1820192993127,3cyclictest1985301-21kworker/u8:2+flush-259:022:05:012
1820192993125,4cyclictest1825540-21latency_hist19:15:012
1820187993125,4cyclictest1970191-21latency_hist21:35:021
1820187993124,5cyclictest1897233-21ntpq20:20:211
1820187993124,5cyclictest1897233-21ntpq20:20:211
1820182993127,3cyclictest2104886-21apt-get23:45:130
1820196993026,3cyclictest252-21systemd-journal22:35:003
1820196993026,3cyclictest2006854-21kworker/u8:1+flush-259:022:15:003
1820196993025,3cyclictest1851327-21latency_hist19:40:003
1820196992925,3cyclictest1866975-21kworker/u8:4+flush-259:020:00:003
1820187992925,3cyclictest2037643-21kworker/u8:2+flush-259:022:55:011
1820182992925,3cyclictest1929045-21kworker/u8:0+events_unbound21:35:160
1820182992923,5cyclictest2120211-21cp00:00:010
1820182992922,5cyclictest1984691-21ntpq21:45:210
1820196992822,4cyclictest2063868-21diskstats23:05:143
1820187992822,4cyclictest2006854-21kworker/u8:1+flush-259:022:30:011
1820187992821,5cyclictest2005736-21ntpq22:05:211
1820182992820,6cyclictest2022560-21diskstats22:25:140
1820196992723,3cyclictest1866975-21kworker/u8:4+flush-259:020:15:183
1820192992723,3cyclictest2078880-21kworker/u8:3+flush-259:023:20:002
1820192992723,3cyclictest1836123-21kworker/u8:1+flush-259:019:35:252
1820192992723,3cyclictest1819976-21kworker/u8:2+flush-259:019:30:262
1820187992721,4cyclictest1913817-21apt-get20:40:121
1820182992724,2cyclictest2073548-21kworker/u8:1+events_unbound23:15:190
1820192992619,5cyclictest1938384-21ntpq21:00:202
1820187992620,4cyclictest1892951-21latency_hist20:20:001
1820187992615,9cyclictest1820621-21/usr/sbin/munin19:10:141
1820182992622,3cyclictest1866975-21kworker/u8:4+events_unbound20:15:240
1820182992620,4cyclictest1898091-21latency_hist20:25:000
1820182992620,4cyclictest1898091-21latency_hist20:25:000
1820196992520,4cyclictest1820387-21gpgv19:10:013
1820192992522,2cyclictest2078880-21kworker/u8:3+events_unbound23:20:152
1820187992521,3cyclictest252-21systemd-journal21:17:161
1820182992520,3cyclictest1851118-21kworker/u8:3+flush-259:019:40:200
1820182992519,4cyclictest1819927-21latency_hist19:10:010
1820196992420,3cyclictest2027340-21kworker/u8:0+events_unbound22:35:153
1820196992420,3cyclictest1819976-21kworker/u8:2+events_unbound19:30:153
1820196992419,4cyclictest2012074-21apt-get22:15:133
1820196992419,4cyclictest1872592-21apt-get20:00:113
1820192992418,4cyclictest252-21systemd-journal00:26:332
1820187992418,5cyclictest2141505-21apt-get00:20:121
1820187992417,5cyclictest2078087-21ntpq23:15:221
1820182992420,3cyclictest2058282-21kworker/u8:3+flush-259:023:10:140
1820182992420,3cyclictest1819976-21kworker/u8:2+flush-259:019:15:120
1820182992418,4cyclictest1964220-21ntpq21:25:220
1820196992320,2cyclictest1856491-21kworker/u8:2+flush-259:020:20:243
1820196992320,2cyclictest1856491-21kworker/u8:2+flush-259:020:20:243
1820192992320,2cyclictest1856491-21kworker/u8:2+events_unbound21:25:222
1820192992317,4cyclictest2115063-21kworker/u8:0+flush-259:000:20:182
1820192992316,5cyclictest1928075-21ntpq20:50:212
1820182992320,2cyclictest2115063-21kworker/u8:0+flush-259:000:00:270
1820182992320,2cyclictest2042824-21kworker/u8:4+events_unbound22:58:280
1820182992319,3cyclictest2109521-21kworker/u8:4+events_unbound00:05:250
1820182992319,3cyclictest2058282-21kworker/u8:3+events_unbound23:00:240
1820182992319,3cyclictest2006854-21kworker/u8:1+events_unbound22:55:010
1820182992319,3cyclictest1929045-21kworker/u8:0+events_unbound21:05:200
182018299230,21cyclictest1-21systemd22:04:270
1820196992219,2cyclictest1887807-21kworker/u8:0+events_unbound20:30:173
1820196992216,4cyclictest2130086-21ntpq00:05:253
1820192992217,3cyclictest2068848-21apt-get23:10:132
1820187992218,3cyclictest1851118-21kworker/u8:3+flush-259:019:49:561
1820187992217,3cyclictest2151524-21cron00:30:001
1820187992217,3cyclictest1929266-21apt-get20:55:111
1820187992215,5cyclictest252-21systemd-journal00:34:361
1820187992215,4cyclictest252-21systemd-journal20:00:021
1820187992211,8cyclictest252-21systemd-journal20:08:251
1820182992219,2cyclictest1929045-21kworker/u8:0+flush-259:021:40:260
1820182992218,3cyclictest1866975-21kworker/u8:4+events_unbound20:45:240
1820182992218,3cyclictest1851118-21kworker/u8:3+flush-259:019:45:160
1820182992217,4cyclictest2053351-21kworker/u8:0+flush-259:023:30:150
1820196992119,2cyclictest1929045-21kworker/u8:0+events_unbound21:00:163
1820196992117,3cyclictest2141284-21kworker/u8:1+flush-259:000:40:003
1820196992117,3cyclictest2058282-21kworker/u8:3+flush-259:023:00:203
1820196992117,3cyclictest2001510-21kworker/u8:3+events_unbound22:05:183
1820196992117,3cyclictest1970450-21apt-get21:35:123
1820196992117,3cyclictest1918725-21kworker/u8:1+events_unbound20:55:213
1820187992114,5cyclictest2104621-21latency_hist23:45:001
1820187992114,4cyclictest2161822-21latency_hist00:40:001
1820182992118,2cyclictest1929045-21kworker/u8:0+events_unbound21:15:010
1820182992117,3cyclictest2109521-21kworker/u8:4+flush-259:000:35:150
1820196992017,2cyclictest1856491-21kworker/u8:2+flush-259:020:35:253
1820196992016,3cyclictest1856491-21kworker/u8:2+flush-259:020:25:203
1820196992016,3cyclictest1819976-21kworker/u8:2+flush-259:019:30:013
182019299208,5cyclictest1990771-21munin-run21:54:592
1820192992016,2cyclictest1856491-21kworker/u8:2+flush-259:021:25:002
1820192992015,3cyclictest252-21systemd-journal19:42:262
1820192992013,5cyclictest2068580-21latency_hist23:10:012
1820187992016,3cyclictest2079192-21apt-get23:20:091
1820187992016,3cyclictest1985301-21kworker/u8:2+flush-259:021:55:261
1820187992015,3cyclictest252-21systemd-journal21:52:271
1820187992015,3cyclictest1929045-21kworker/u8:0+flush-259:021:15:001
1820187992012,6cyclictest0-21swapper/119:40:001
1820187992011,7cyclictest503-21tpm2-abrmd21:40:011
1820182992016,3cyclictest2109521-21kworker/u8:4+flush-259:000:30:250
1820182992016,3cyclictest1866975-21kworker/u8:4+events_unbound20:05:200
1820182992016,3cyclictest1856491-21kworker/u8:2+events_unbound19:55:230
1820182992016,3cyclictest1819976-21kworker/u8:2+events_unbound19:27:260
1820182992014,4cyclictest1883076-21diskstats20:10:140
1820196991915,3cyclictest2984858-21kworker/u8:3+flush-259:019:20:263
1820196991915,3cyclictest2017181-21kworker/u8:3+events_unbound22:25:183
1820196991914,3cyclictest1856491-21kworker/u8:2+events_unbound20:50:153
1820196991914,3cyclictest1799616-21kworker/u8:0+events_unbound19:40:243
1820196991913,4cyclictest509-21rs:main3
1820192991913,4cyclictest2115042-21latency_hist23:55:022
182018799192,5cyclictest252-21systemd-journal20:00:001
1820187991913,4cyclictest509-21rs:main1
1820187991913,4cyclictest1866975-21kworker/u8:4+flush-259:020:10:261
1820187991911,6cyclictest1979565-21ntpq21:40:211
1820182991915,3cyclictest2109521-21kworker/u8:4+events_unbound00:19:350
1820182991914,4cyclictest2088253-21kworker/u8:1+events_unbound23:38:290
1820196991815,2cyclictest1819976-21kworker/u8:2+flush-259:019:10:213
1820196991814,3cyclictest2088253-21kworker/u8:1+events_unbound23:45:243
1820196991814,3cyclictest2058282-21kworker/u8:3+events_unbound23:10:183
1820196991814,3cyclictest1949629-21kworker/u8:1+flush-259:021:30:213
1820196991813,3cyclictest252-21systemd-journal22:04:283
1820192991815,2cyclictest2115063-21kworker/u8:0+flush-259:000:35:002
1820192991815,2cyclictest2006854-21kworker/u8:1+flush-259:022:40:222
1820192991814,3cyclictest1929045-21kworker/u8:0+flush-259:021:05:152
1820192991812,4cyclictest1835899-21latency_hist19:25:012
182018799185,11cyclictest0-21swapper/119:20:011
1820187991816,1cyclictest2109823-21grep23:50:011
1820187991814,3cyclictest2053351-21kworker/u8:0+flush-259:023:30:261
1820187991814,2cyclictest0-21swapper/122:10:231
1820187991812,4cyclictest2141242-21latency_hist00:20:011
1820187991811,5cyclictest1944456-21latency_hist21:10:011
1820182991815,2cyclictest2006854-21kworker/u8:1+events_unbound22:32:310
1820182991815,2cyclictest1851118-21kworker/u8:3+events_unbound19:51:250
1820182991812,5cyclictest252-21systemd-journal00:24:590
1820182991812,5cyclictest1995057-21ntpq21:55:240
1820196991713,3cyclictest1866975-21kworker/u8:4+events_unbound20:05:183
1820196991712,3cyclictest0-21swapper/322:45:013
1820192991714,2cyclictest1856491-21kworker/u8:2+events_unbound20:35:192
1820192991713,3cyclictest1887807-21kworker/u8:0+flush-259:020:30:002
1820192991712,3cyclictest1985301-21kworker/u8:2+events_unbound21:55:202
1820192991712,3cyclictest1929045-21kworker/u8:0+flush-259:022:15:182
1820187991714,2cyclictest1928991-21idleruntime-cro20:55:001
182018799171,2cyclictest2047062-21ntpq22:45:201
182018799171,14cyclictest252-21systemd-journal23:55:011
182018799171,14cyclictest2058257-21idleruntime-cro23:00:001
1820187991711,4cyclictest2037858-21/usr/sbin/munin22:40:201
182018799170,5cyclictest2089215-21munin-plugin-st23:30:001
1820182991714,2cyclictest2037643-21kworker/u8:2+events_unbound23:20:220
1820182991714,2cyclictest1929045-21kworker/u8:0+events_unbound22:15:210
1820182991714,2cyclictest1866975-21kworker/u8:4+events_unbound20:40:140
1820182991712,4cyclictest2115063-21kworker/u8:0+events_unbound00:12:360
1820182991712,4cyclictest2006893-21apt-get22:10:150
182019699169,5cyclictest2093494-21ntpq23:30:223
1820196991614,1cyclictest1819976-21kworker/u8:2+flush-259:019:15:173
1820196991613,2cyclictest2053351-21kworker/u8:0+events_unbound23:25:233
1820196991612,3cyclictest1929045-21kworker/u8:0+events_unbound21:25:153
1820196991610,4cyclictest2124933-21ntpq00:00:203
182019299169,5cyclictest1865892-21ntpq19:50:212
1820192991613,2cyclictest2136084-21cron00:14:592
1820192991613,2cyclictest1856491-21kworker/u8:2+flush-259:019:55:152
1820192991612,3cyclictest2037643-21kworker/u8:2+flush-259:022:55:002
1820192991612,3cyclictest1929045-21kworker/u8:0+events_unbound22:10:272
1820192991612,3cyclictest1836123-21kworker/u8:1+flush-259:019:25:262
1820192991611,3cyclictest252-21systemd-journal00:36:422
182018799161,5cyclictest2095027-21ps23:35:181
182018799161,5cyclictest2063447-21turbostat.cron23:05:001
182018799161,4cyclictest1981145-21grep21:45:161
182018799161,3cyclictest252-21systemd-journal22:04:591
182018799161,3cyclictest2125853-21awk00:05:011
182018799161,3cyclictest2016948-21sh22:19:591
182018799161,3cyclictest1939304-21idleruntime-cro21:05:011
182018799161,3cyclictest1908781-21tr20:35:131
1820187991613,2cyclictest1903904-21irqcore20:30:161
182018799160,4cyclictest2021367-21ps22:20:221
182018799160,4cyclictest1958899-21grep21:20:201
1820182991614,1cyclictest60350irq/132-enp0s29f1:rx-020:35:240
1820182991614,1cyclictest60350irq/132-enp0s29f1:rx-020:25:260
1820182991613,2cyclictest60350irq/132-enp0s29f1:rx-022:35:160
1820182991612,3cyclictest2006854-21kworker/u8:1+events_unbound22:20:190
1820182991612,3cyclictest1866975-21kworker/u8:4+events_unbound20:04:270
1820182991612,3cyclictest1856491-21kworker/u8:2+events_unbound21:21:280
1820182991612,2cyclictest2104629-21turbostat23:45:000
182019699158,5cyclictest1959905-21latency_hist21:25:023
1820196991511,3cyclictest2115063-21kworker/u8:0+flush-259:000:25:003
1820196991511,3cyclictest2109521-21kworker/u8:4+events_unbound00:10:223
1820196991511,3cyclictest2088253-21kworker/u8:1+events_unbound23:50:233
1820196991510,4cyclictest252-21systemd-journal23:00:013
1820196991510,3cyclictest2006854-21kworker/u8:1+flush-259:022:50:163
1820192991511,3cyclictest2115063-21kworker/u8:0+flush-259:023:55:162
182018799153,4cyclictest2128031-21cat00:05:201
182018799152,2cyclictest0-21swapper/100:10:201
182018799151,3cyclictest2064131-21irqcore23:05:181
182018799151,3cyclictest1898763-21irqcore20:25:181
182018799151,3cyclictest1840571-21grep19:25:251
182018799151,2cyclictest1845655-21grep19:30:251
182018799150,5cyclictest2032839-21grep22:35:151
182018799150,5cyclictest1960326-21taskset21:25:131
182018799150,4cyclictest1851734-21cpuspeed_turbos19:40:131
182018799150,4cyclictest1835456-21grep19:20:241
1820182991513,1cyclictest60350irq/132-enp0s29f1:rx-019:20:240
1820182991512,2cyclictest60350irq/132-enp0s29f1:rx-000:26:200
1820182991512,2cyclictest1903918-21irqcore20:30:160
1820182991512,2cyclictest1819976-21kworker/u8:2+flush-259:019:30:210
1820182991511,3cyclictest60350irq/132-enp0s29f1:rx-019:10:170
1820182991511,3cyclictest1856491-21kworker/u8:2+events_unbound21:18:280
1820182991511,2cyclictest60350irq/132-enp0s29f1:rx-022:05:190
182019699149,3cyclictest2109521-21kworker/u8:4+flush-259:000:25:233
1820196991412,1cyclictest1941221-21kthreadcore21:05:203
1820196991411,2cyclictest2037643-21kworker/u8:2+flush-259:022:45:203
1820196991411,2cyclictest1820177-21cyclictest19:55:003
1820196991410,3cyclictest2109521-21kworker/u8:4+flush-259:000:18:103
1820196991410,2cyclictest1929045-21kworker/u8:0+flush-259:021:15:193
182019299149,3cyclictest252-21systemd-journal20:40:272
182019299148,4cyclictest252-21systemd-journal00:10:002
1820192991411,2cyclictest1919985-21kthreadcore20:45:172
1820192991410,3cyclictest2027340-21kworker/u8:0+flush-259:022:40:002
182018799140,4cyclictest2069075-21uniq23:10:151
1820182991413,1cyclictest60350irq/132-enp0s29f1:rx-022:45:130
1820182991412,1cyclictest60350irq/132-enp0s29f1:rx-021:50:180
1820182991411,2cyclictest60350irq/132-enp0s29f1:rx-023:08:440
1820182991410,3cyclictest60350irq/132-enp0s29f1:rx-022:40:120
1820182991410,3cyclictest1936715-21tr21:00:180
1820182991410,3cyclictest1856491-21kworker/u8:2+events_unbound20:56:280
182019699139,3cyclictest2037643-21kworker/u8:2+flush-259:023:21:343
182019699137,4cyclictest2151962-21diskstats00:30:133
1820196991310,3cyclictest1949629-21kworker/u8:1+flush-259:021:40:163
1820192991310,2cyclictest2095034-21irqcore23:35:182
1820192991310,2cyclictest1984591-21needreboot21:45:202
1820192991310,1cyclictest0-21swapper/223:30:002
182019299130,11cyclictest2032468-21cron22:35:002
1820182991311,1cyclictest60350irq/132-enp0s29f1:rx-019:35:140
182019699129,2cyclictest1916106-21kthreadcore20:40:183
182019699128,3cyclictest1929045-21kworker/u8:0+events_unbound21:45:213
182019299129,2cyclictest1965374-21cpuspeed_turbos21:30:122
182019299129,2cyclictest1918725-21kworker/u8:1+flush-259:020:55:242
182019299129,2cyclictest1883032-21cpuspeed_turbos20:10:132
182019299129,1cyclictest0-21swapper/223:00:002
182019299128,3cyclictest2088253-21kworker/u8:1+flush-259:023:40:162
182019299128,2cyclictest0-21swapper/219:20:012
182019299126,4cyclictest252-21systemd-journal21:15:002
182019299126,4cyclictest2047101-21ntpq22:45:212
1820192991211,1cyclictest1819976-21kworker/u8:2+flush-259:019:10:012
1820192991210,1cyclictest1873008-21irqcore20:00:162
182019699119,1cyclictest0-21swapper/323:40:263
182019699119,1cyclictest0-21swapper/322:20:123
182019699118,2cyclictest2073548-21kworker/u8:1+events_unbound23:15:243
182019699117,3cyclictest1851118-21kworker/u8:3+flush-259:019:45:263
182019299119,1cyclictest0-21swapper/222:05:182
182019299118,2cyclictest1975311-21cron21:40:002
182019299118,2cyclictest1954340-21unixbench_singl21:15:272
182019299108,1cyclictest2089589-21cpuspeed_turbos23:30:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional