You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-31 - 21:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Fri Jul 31, 2026 12:44:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1908632995754,2cyclictest2121968-21kworker/u8:4+events_unbound11:10:163
1908627995146,4cyclictest2152998-21/usr/sbin/munin11:05:151
1908627995146,4cyclictest2152998-21/usr/sbin/munin11:05:151
1908623995144,5cyclictest2157926-21latency_hist11:10:010
1908623995144,5cyclictest2157926-21latency_hist11:10:010
1908627994843,3cyclictest335557-21systemd-journal10:31:151
1908632994744,2cyclictest2121968-21kworker/u8:4+events_unbound11:35:023
1908629994742,3cyclictest509-21rs:main2
1908629994640,4cyclictest2173614-21/usr/sbin/munin11:25:132
1908632994440,3cyclictest335557-21systemd-journal10:44:593
1908632994339,3cyclictest2121968-21kworker/u8:4+events_unbound12:03:153
1908632994238,3cyclictest2230868-21kworker/u8:3+events_unbound12:22:163
1908623994236,4cyclictest1966033-21/usr/sbin/munin08:05:180
1908632994138,2cyclictest1918688-21kworker/u8:2+flush-259:007:20:163
1908632994137,3cyclictest2043590-21kworker/u8:0+events_unbound10:36:163
1908632994037,2cyclictest2090971-21kworker/u8:1+events_unbound10:34:473
1908629993935,3cyclictest2215230-21gpgv12:05:012
1908627993933,4cyclictest2225919-21latency_hist12:15:011
1908627993933,4cyclictest2017350-21latency_hist08:55:011
1908623993932,5cyclictest335557-21systemd-journal09:10:010
1908623993833,3cyclictest509-21rs:main0
1908632993734,2cyclictest2121968-21kworker/u8:4+events_unbound11:57:143
1908632993734,2cyclictest2121968-21kworker/u8:4+events_unbound11:33:153
1908632993734,2cyclictest2106275-21kworker/u8:2+events_unbound10:29:163
1908632993632,3cyclictest2090971-21kworker/u8:1+events_unbound10:24:443
1908627993631,3cyclictest2209665-21cat12:00:011
1908632993532,2cyclictest2043590-21kworker/u8:0+flush-259:010:10:243
1908632993431,2cyclictest2121968-21kworker/u8:4+events_unbound11:20:133
1908632993431,2cyclictest2090971-21kworker/u8:1+events_unbound10:15:133
1908632993431,2cyclictest2043590-21kworker/u8:0+events_unbound11:02:163
1908632993431,2cyclictest2043590-21kworker/u8:0+events_unbound11:02:163
1908632993430,3cyclictest2043590-21kworker/u8:0+events_unbound10:50:203
1908632993430,3cyclictest1934838-21kworker/u8:1+events_unbound07:53:153
1908632993330,2cyclictest1934838-21kworker/u8:1+events_unbound08:24:143
1908632993330,2cyclictest1864138-21kworker/u8:1+events_unbound07:10:233
1908632993327,4cyclictest2178528-21latency_hist11:30:013
1908629993329,3cyclictest335557-21systemd-journal09:20:002
1908623993330,2cyclictest2121968-21kworker/u8:4+events_unbound11:45:000
1908632993228,3cyclictest2241483-21kworker/u8:2+events_unbound12:32:153
1908632993227,3cyclictest2043590-21kworker/u8:0+events_unbound09:20:133
1908627993226,5cyclictest1929250-21apt-get07:30:011
1908623993226,4cyclictest2121741-21latency_hist10:35:010
1908623993225,4cyclictest2090746-21latency_hist10:05:010
1908632993128,2cyclictest1986428-21kworker/u8:2+events_unbound08:49:233
1908632993126,2cyclictest2173062-21kworker/u8:2+events_unbound11:45:223
1908623993127,3cyclictest2220754-21kworker/u8:0+flush-259:012:10:150
1908623993125,4cyclictest1950312-21latency_hist07:50:010
1908632993027,2cyclictest2230868-21kworker/u8:3+events_unbound12:28:153
1908632993027,2cyclictest1986428-21kworker/u8:2+events_unbound08:25:163
1908632993026,3cyclictest1986428-21kworker/u8:2+events_unbound08:35:223
1908632993026,3cyclictest1986428-21kworker/u8:2+events_unbound08:35:223
1908632993023,5cyclictest2080171-21cp09:55:013
1908629993023,5cyclictest1996684-21latency_hist08:35:002
1908627993025,3cyclictest335557-21systemd-journal10:06:151
1908627993024,4cyclictest2147602-21latency_hist11:00:011
1908632992926,2cyclictest1893197-21kworker/u8:4+events_unbound07:55:233
1908623992925,3cyclictest1934838-21kworker/u8:1+events_unbound08:40:150
1908623992925,3cyclictest1934838-21kworker/u8:1+events_unbound08:40:150
1908632992824,3cyclictest1986428-21kworker/u8:2+events_unbound09:35:043
1908632992824,3cyclictest1986428-21kworker/u8:2+events_unbound09:30:183
1908623992825,2cyclictest2209355-21kworker/u8:1+events_unbound12:15:150
1908623992822,4cyclictest335557-21systemd-journal12:00:140
1908632992724,2cyclictest2121968-21kworker/u8:4+flush-259:012:05:063
1908632992724,2cyclictest2059523-21kworker/u8:3+events_unbound09:43:153
1908632992724,2cyclictest2007044-21kworker/u8:3+events_unbound09:15:133
1908629992722,3cyclictest2137262-21latency_hist10:50:012
1908623992723,3cyclictest1893197-21kworker/u8:4+events_unbound07:50:150
1908623992721,4cyclictest335557-21systemd-journal08:45:140
1908623992720,5cyclictest335557-21systemd-journal07:35:000
1908623992720,5cyclictest2182842-21ntpq11:30:200
1908632992623,2cyclictest2220754-21kworker/u8:0+events_unbound12:10:123
1908632992623,2cyclictest2059523-21kworker/u8:3+events_unbound10:05:173
1908632992623,2cyclictest1986428-21kworker/u8:2+events_unbound09:00:143
1908627992620,4cyclictest2236307-21latency_hist12:25:011
1908623992623,2cyclictest2059523-21kworker/u8:3+events_unbound09:45:160
1908632992522,2cyclictest2209355-21kworker/u8:1+events_unbound12:35:283
1908632992522,2cyclictest1893197-21kworker/u8:4+events_unbound08:08:153
1908632992521,3cyclictest2085582-21kworker/u8:2+events_unbound10:00:143
1908627992521,3cyclictest1893197-21kworker/u8:4+flush-259:008:14:591
1908623992521,3cyclictest2022593-21kworker/u8:0+flush-259:009:04:240
1908623992520,4cyclictest2230868-21kworker/u8:3+events_unbound12:27:160
1908632992421,2cyclictest2209355-21kworker/u8:1+events_unbound12:15:233
1908632992421,2cyclictest1934838-21kworker/u8:1+events_unbound08:41:083
1908632992421,2cyclictest1934838-21kworker/u8:1+events_unbound08:41:083
1908629992418,4cyclictest335557-21systemd-journal09:25:012
1908629992418,4cyclictest2007020-21latency_hist08:45:012
1908629992418,4cyclictest2007020-21latency_hist08:45:002
1908627992418,4cyclictest2059055-21cp09:35:001
1908627992410,13cyclictest2079988-21apt-get09:55:011
1908629992320,2cyclictest2215768-21sshd12:05:042
1908629992318,3cyclictest335557-21systemd-journal08:26:152
1908627992318,3cyclictest1986640-21/usr/sbin/munin08:25:131
1908627992316,5cyclictest2011269-21ntpq08:45:221
1908623992320,2cyclictest2168225-21idleruntime-cro11:20:000
1908623992319,3cyclictest2080606-21base6409:55:010
1908623992319,3cyclictest2043590-21kworker/u8:0+flush-259:010:40:150
1908623992318,4cyclictest1914053-21apt-get07:15:110
1908623992317,4cyclictest2043365-21latency_hist09:20:010
1908627992216,4cyclictest2209355-21kworker/u8:1+flush-259:012:15:211
1908623992218,3cyclictest2043590-21kworker/u8:0+events_unbound10:49:160
1908623992217,3cyclictest2220754-21kworker/u8:0+events_unbound12:20:210
1908623992216,3cyclictest1960658-21kworker/u8:0+events_unbound08:15:230
1908632992118,2cyclictest1934838-21kworker/u8:1+events_unbound08:00:133
1908632992118,2cyclictest1908579-21kworker/u8:0+flush-259:007:15:253
1908632992118,2cyclictest1893197-21kworker/u8:4+events_unbound07:48:163
1908632992117,3cyclictest1986428-21kworker/u8:2+events_unbound08:50:233
190862799219,3cyclictest1949828-21unixbench_multi07:45:251
190862799219,2cyclictest0-21swapper/110:10:151
1908627992113,5cyclictest509-21rs:main1
1908623992117,3cyclictest2209355-21kworker/u8:1+flush-259:012:35:160
1908623992117,3cyclictest2173062-21kworker/u8:2+events_unbound11:50:030
1908623992117,3cyclictest2121968-21kworker/u8:4+events_unbound12:05:220
1908632992016,3cyclictest335557-21systemd-journal11:50:143
1908632992014,4cyclictest1933957-21ntpq07:30:233
1908629992014,4cyclictest2204373-21latency_hist11:55:012
1908627992017,2cyclictest2121968-21kworker/u8:4+events_unbound11:30:151
1908627992017,2cyclictest2043590-21kworker/u8:0+flush-259:009:25:011
1908627992016,3cyclictest1960658-21kworker/u8:0+flush-259:008:05:191
1908627992014,5cyclictest61250irq/135-enp0s29f1:tx-110:42:261
1908623992017,2cyclictest2230868-21kworker/u8:3+events_unbound12:35:000
1908623992016,3cyclictest2121968-21kworker/u8:4+events_unbound11:40:000
1908623992015,3cyclictest2090971-21kworker/u8:1+flush-259:010:20:250
1908623992015,3cyclictest2059533-21/usr/sbin/munin09:35:220
1908623992013,5cyclictest1909016-21diskstats07:10:150
1908632991916,2cyclictest2152802-21kworker/u8:1+events_unbound11:16:153
1908632991916,2cyclictest2043590-21kworker/u8:0+flush-259:010:55:193
1908632991916,2cyclictest1893197-21kworker/u8:4+flush-259:007:44:163
1908627991916,2cyclictest2001881-21systemd-run08:40:011
1908627991916,2cyclictest2001881-21systemd-run08:40:001
1908627991912,4cyclictest1940005-21latency_hist07:40:021
1908623991916,2cyclictest1934838-21kworker/u8:1+events_unbound07:40:150
1908623991915,3cyclictest2058023-21ntpq09:30:240
1908623991914,3cyclictest0-21swapper/012:00:010
1908632991815,2cyclictest1986428-21kworker/u8:2+events_unbound09:25:263
1908632991815,2cyclictest1986428-21kworker/u8:2+events_unbound08:57:153
1908629991813,3cyclictest2028016-21apt-get09:05:122
190862799188,9cyclictest2132775-21irqcore10:45:161
1908627991815,2cyclictest2022558-21idleruntime-cro09:00:001
1908627991814,2cyclictest1918688-21kworker/u8:2+flush-259:007:25:191
1908623991815,2cyclictest2059523-21kworker/u8:3+events_unbound09:41:150
1908623991814,3cyclictest2106275-21kworker/u8:2+events_unbound10:25:230
1908623991813,4cyclictest2091005-21apt-get10:05:120
1908632991714,2cyclictest1960658-21kworker/u8:0+events_unbound08:14:163
190862999171,5cyclictest2064681-21cat09:40:012
1908629991714,2cyclictest0-21swapper/207:25:002
1908627991714,1cyclictest0-21swapper/109:10:181
1908627991710,5cyclictest2085802-21/usr/sbin/munin10:00:151
1908623991713,3cyclictest2173062-21kworker/u8:2+events_unbound11:29:570
1908623991713,3cyclictest2059523-21kworker/u8:3+events_unbound09:54:380
1908623991713,3cyclictest2043590-21kworker/u8:0+flush-259:010:55:200
1908623991713,2cyclictest60350irq/132-enp0s29f1:rx-010:15:130
190862999162,2cyclictest2250905-21ntpq12:35:222
1908629991611,4cyclictest2121968-21kworker/u8:4+flush-259:012:15:122
1908627991613,2cyclictest1992844-21kthreadcore08:30:191
1908627991613,2cyclictest0-21swapper/109:10:011
1908627991611,3cyclictest2189347-21grep11:40:181
1908627991610,4cyclictest2163070-21latency_hist11:15:011
1908632991512,2cyclictest2173062-21kworker/u8:2+events_unbound11:40:153
1908632991511,3cyclictest335557-21systemd-journal09:59:383
190862999154,3cyclictest0-21swapper/212:30:012
190862999152,5cyclictest1960582-21runrttasks07:59:502
190862999152,3cyclictest2245198-21grep12:30:182
190862999152,2cyclictest2115679-21ntpq10:25:212
190862999152,2cyclictest2100214-21ntpq10:10:202
190862999151,9cyclictest2052793-21ntpq09:25:212
1908629991512,2cyclictest2127615-21irqcore10:40:162
1908629991512,2cyclictest2085563-21turbostat10:00:012
1908629991511,3cyclictest2209732-21cat12:00:012
1908629991511,3cyclictest1955463-21munin-run07:54:592
190862799159,4cyclictest335557-21systemd-journal07:40:151
1908627991512,2cyclictest1961309-21tr08:00:171
190862799150,7cyclictest1981236-21sh08:19:591
190862399159,4cyclictest335557-21systemd-journal09:12:140
1908623991513,1cyclictest60350irq/132-enp0s29f1:rx-010:35:170
1908623991513,1cyclictest60350irq/132-enp0s29f1:rx-010:10:230
1908623991512,2cyclictest60350irq/132-enp0s29f1:rx-007:20:130
1908623991512,2cyclictest1986428-21kworker/u8:2+events_unbound09:25:210
1908623991512,2cyclictest1893197-21kworker/u8:4+flush-259:008:10:270
1908623991511,3cyclictest2109693-21snmpd11:20:180
190863299149,1cyclictest191rcu_preempt07:30:003
1908632991412,2cyclictest2043590-21kworker/u8:0+flush-259:011:05:263
1908632991412,2cyclictest2043590-21kworker/u8:0+flush-259:011:05:253
1908632991411,2cyclictest2043590-21kworker/u8:0+events_unbound10:45:123
1908632991411,2cyclictest1980283-21needreboot08:15:203
1908632991410,3cyclictest1986428-21kworker/u8:2+events_unbound09:14:293
190862999149,3cyclictest335557-21systemd-journal09:46:142
190862999143,2cyclictest2193216-21ntpq11:40:262
190862999143,2cyclictest2111580-21cat10:25:002
190862999142,3cyclictest2157059-21ntpq11:05:222
190862999142,3cyclictest2157059-21ntpq11:05:222
190862999142,3cyclictest1970027-21ntpq08:05:232
190862999142,2cyclictest2147843-21/usr/sbin/munin11:00:132
190862999142,2cyclictest2147843-21/usr/sbin/munin11:00:132
190862999142,2cyclictest2089921-21ntpq10:00:222
190862999142,2cyclictest2079149-21ld09:50:232
190862999141,4cyclictest2162981-21runrttasks11:14:142
190862999141,4cyclictest2139749-21kthreadcore10:50:182
190862999141,4cyclictest1980336-21modprobe08:15:212
190862999141,4cyclictest1948043-21cut07:45:182
1908629991410,3cyclictest335557-21systemd-journal08:59:552
190862799147,5cyclictest2168652-21diskstats11:20:141
190862799141,11cyclictest1-21systemd09:46:151
1908627991410,3cyclictest2215121-21apt-get12:05:001
1908627991410,3cyclictest1986428-21kworker/u8:2+flush-259:009:25:191
1908627991410,2cyclictest2242151-21irqcore12:30:161
1908623991413,1cyclictest60350irq/132-enp0s29f1:rx-010:50:200
1908623991412,2cyclictest61150irq/133-enp0s29f1:tx-007:35:210
1908623991412,1cyclictest60350irq/132-enp0s29f1:rx-011:45:190
1908623991412,1cyclictest60350irq/132-enp0s29f1:rx-011:10:160
1908623991412,1cyclictest60350irq/132-enp0s29f1:rx-008:56:270
1908623991412,1cyclictest60350irq/132-enp0s29f1:rx-008:26:520
1908623991410,3cyclictest1918688-21kworker/u8:2+events_unbound07:29:020
1908623991410,3cyclictest1893197-21kworker/u8:4+events_unbound08:20:190
1908632991310,2cyclictest1986428-21kworker/u8:2+events_unbound09:08:293
1908632991310,2cyclictest1893197-21kworker/u8:4+events_unbound07:39:073
190862999139,2cyclictest0-21swapper/208:35:182
190862999139,2cyclictest0-21swapper/208:35:182
190862999133,2cyclictest2037383-21/usr/sbin/munin09:10:232
190862999132,3cyclictest2173384-21/usr/sbin/munin11:25:002
190862999132,2cyclictest2069419-21runrttasks09:42:432
190862999131,7cyclictest2122156-21df10:35:152
190862999131,7cyclictest2016456-21ntpq08:50:212
190862999131,3cyclictest1918780-21sshd07:18:142
1908629991310,2cyclictest335557-21systemd-journal10:10:002
1908629991310,2cyclictest2146738-21ntp_kernel_pll_10:55:232
190862999130,8cyclictest2054966-21irqstats09:30:182
190862999130,5cyclictest1875552-21munin-node08:10:122
190862799138,3cyclictest1959702-21ntpq07:55:221
190862799131,3cyclictest2204371-21awk11:54:591
1908627991310,1cyclictest0-21swapper/112:25:251
1908623991311,1cyclictest60350irq/132-enp0s29f1:rx-009:23:050
190863299129,2cyclictest1934838-21kworker/u8:1+flush-259:008:30:193
190862999128,3cyclictest2173062-21kworker/u8:2+flush-259:011:45:242
190862999128,3cyclictest2023017-21fschecks_count09:00:142
190862999127,4cyclictest335557-21systemd-journal11:40:002
190862999126,3cyclictest0-21swapper/208:47:192
190862999126,1cyclictest191rcu_preempt10:30:212
190862999121,6cyclictest1944320-21ntp_states07:40:252
190862999121,5cyclictest2225720-21runrttasks12:12:472
190862999121,2cyclictest2163491-21diskstats11:15:152
190862999121,2cyclictest2101535-21df_inode10:15:142
190862999121,10cyclictest2183730-21munin-run11:35:002
190862799129,2cyclictest2152743-21cron11:05:001
190862799129,2cyclictest2152743-21cron11:04:591
190862799121,9cyclictest490-21dbus-daemon09:20:001
190862799120,11cyclictest2059707-21sshd09:35:141
1908623991210,1cyclictest60350irq/132-enp0s29f1:rx-008:35:230
1908623991210,1cyclictest60350irq/132-enp0s29f1:rx-008:35:230
1908623991210,1cyclictest60350irq/132-enp0s29f1:rx-007:55:200
190862999112,2cyclictest41-21ksoftirqd/212:21:142
190862999112,2cyclictest41-21ksoftirqd/207:10:172
190862999111,3cyclictest1981678-21diskstats08:20:152
190862999111,2cyclictest1965654-21runrttasks08:03:342
190862999110,7cyclictest1935377-21fw_forwarded_lo07:35:152
190862799119,1cyclictest0-21swapper/108:25:001
190862799118,2cyclictest0-21swapper/107:20:191
190862799118,1cyclictest0-21swapper/111:15:251
190862799117,2cyclictest2220742-21latency_hist12:10:011
190862799115,4cyclictest2183983-21apt-get11:35:111
190862799110,8cyclictest2007020-21latency_hist08:45:011
190862799110,8cyclictest2007020-21latency_hist08:45:011
190862399118,2cyclictest60350irq/132-enp0s29f1:rx-008:30:210
190862799107,2cyclictest2106943-21irqcore10:20:171
190862799106,2cyclictest2194765-21sed11:45:161
190862799106,2cyclictest2122075-21cpuspeed_turbos10:35:151
19086279996,2cyclictest2111845-21fschecks_count10:25:131
19086279994,4cyclictest2065481-21cut09:40:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional