You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 04:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon May 25, 2026 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1875731994134,5cyclictest2214572-21ntpq00:30:232
1875726994137,3cyclictest1859714-21kworker/u8:1+flush-259:021:05:171
1875726994037,2cyclictest1859714-21kworker/u8:1+flush-259:020:40:211
1875726993933,4cyclictest2006308-21apt-get21:15:121
1875731993833,4cyclictest1948205-21ntpq20:15:202
1875731993734,2cyclictest2072928-21kworker/u8:0+flush-259:022:45:002
1875736993632,3cyclictest1959500-21kworker/u8:0+flush-259:020:55:213
1875736993618,13cyclictest2032286-21diskstats21:40:123
1875736993529,4cyclictest2194773-21latency_hist00:15:013
1875736993430,3cyclictest2020670-21kworker/u8:2+flush-259:021:50:223
1875726993326,5cyclictest1979276-21ntpq20:45:211
1875719993327,4cyclictest335557-21systemd-journal23:55:150
1875719993326,4cyclictest1984396-21ntpq20:50:200
1875726993229,2cyclictest2042233-21kworker/u8:0+flush-259:022:00:261
1875726993229,2cyclictest1859714-21kworker/u8:1+flush-259:020:20:001
1875726993226,4cyclictest335557-21systemd-journal19:59:161
1875726993127,3cyclictest1859714-21kworker/u8:1+flush-259:021:40:201
1875726993125,5cyclictest1876039-21cat19:10:011
1875719993127,3cyclictest2072928-21kworker/u8:0+flush-259:022:20:160
1875719993126,3cyclictest2110270-21kworker/u8:1+flush-259:023:05:170
1875719993126,3cyclictest2084254-21apt-get22:30:110
1875731993025,4cyclictest2195039-21apt-get00:15:122
1875726993025,3cyclictest335557-21systemd-journal20:54:161
1875719993026,2cyclictest0-21swapper/022:55:170
1875736992925,3cyclictest2141412-21apt-get23:21:163
1875726992925,3cyclictest2031878-21kworker/u8:3+flush-259:022:35:261
1875731992823,4cyclictest335557-21systemd-journal00:22:262
1875726992824,3cyclictest2174272-21kworker/u8:4+events_unbound00:10:221
1875726992823,3cyclictest2105322-21/usr/sbin/munin22:50:221
1875726992823,3cyclictest1859714-21kworker/u8:1+events_unbound20:10:121
1875726992820,5cyclictest509-21rs:main1
1875719992824,3cyclictest2084639-21kworker/u8:2+flush-259:022:35:250
1875736992721,4cyclictest2094557-21/usr/sbin/munin22:40:133
1875726992724,2cyclictest2078880-21kworker/u8:1+events_unbound22:30:001
1875726992724,2cyclictest2072928-21kworker/u8:0+events_unbound22:40:171
1875726992720,5cyclictest2042221-21latency_hist21:50:001
1875736992622,3cyclictest1859714-21kworker/u8:1+flush-259:020:10:163
1875736992621,3cyclictest1901716-21gpgv19:35:003
1875731992622,3cyclictest1891429-21kworker/u8:2+flush-259:020:05:172
187573199260,24cyclictest1-21systemd00:15:002
1875726992622,3cyclictest2084639-21kworker/u8:2+flush-259:023:25:251
1875726992622,3cyclictest1937960-21kworker/u8:3+flush-259:020:24:471
1875726992620,4cyclictest2210265-21latency_hist00:30:001
187571999261,23cyclictest1-21systemd20:25:000
1875736992520,3cyclictest1881348-21apt-get19:15:113
1875736992519,4cyclictest2026733-21latency_hist21:35:013
1875731992520,3cyclictest2052519-21latency_hist22:00:002
1875726992521,3cyclictest1990732-21kworker/u8:3+events_unbound21:00:221
1875726992520,3cyclictest2121278-21apt-get23:05:091
1875726992518,5cyclictest2047606-21/usr/sbin/munin21:55:141
1875726992518,4cyclictest509-21rs:main1
1875719992519,4cyclictest2215395-21latency_hist00:35:010
1875736992417,6cyclictest0-21swapper/322:15:003
1875726992420,3cyclictest1859714-21kworker/u8:1+flush-259:019:45:231
1875726992417,5cyclictest335557-21systemd-journal22:25:001
1875719992421,2cyclictest1896553-21kworker/u8:4+flush-259:019:35:170
187571999241,21cyclictest1-21systemd00:01:260
1875736992317,4cyclictest1974970-21latency_hist20:45:013
1875736992312,10cyclictest1907536-21/usr/sbin/munin19:40:143
1875731992320,2cyclictest1891390-21idleruntime-cro19:25:002
1875731992316,5cyclictest2016431-21latency_hist21:25:002
1875726992320,2cyclictest1896553-21kworker/u8:4+flush-259:019:50:201
1875726992319,3cyclictest1959500-21kworker/u8:0+events_unbound20:30:251
1875726992319,3cyclictest1859714-21kworker/u8:1+events_unbound21:12:181
1875726992318,4cyclictest2174272-21kworker/u8:4+flush-259:000:05:281
1875726992318,3cyclictest2110270-21kworker/u8:1+events_unbound23:00:231
1875726992316,4cyclictest1969778-21latency_hist20:40:011
1875736992217,4cyclictest2120584-21apt-get23:05:003
1875736992217,3cyclictest335557-21systemd-journal23:49:163
1875731992215,5cyclictest2189583-21latency_hist00:10:012
1875726992218,3cyclictest2110506-21apt-get22:55:111
1875726992218,3cyclictest2084639-21kworker/u8:2+events_unbound23:14:181
1875726992218,3cyclictest2020670-21kworker/u8:2+events_unbound21:25:231
1875726992218,3cyclictest1891429-21kworker/u8:2+flush-259:019:40:211
1875726992217,3cyclictest2110270-21kworker/u8:1+flush-259:023:15:201
1875726992216,4cyclictest1886487-21/usr/sbin/munin19:20:141
1875719992219,2cyclictest2031878-21kworker/u8:3+events_unbound22:25:190
1875719992218,3cyclictest1896553-21kworker/u8:4+events_unbound19:50:190
1875719992217,3cyclictest1881315-21kworker/u8:3+flush-259:019:25:250
1875719992216,4cyclictest2199955-21latency_hist00:20:010
1875736992117,3cyclictest2072928-21kworker/u8:0+flush-259:022:50:263
1875736992117,3cyclictest1859714-21kworker/u8:1+flush-259:020:25:163
1875731992118,2cyclictest1985548-21kworker/u8:2+flush-259:021:10:212
1875731992116,3cyclictest1902352-21/usr/sbin/munin19:35:152
1875731992115,5cyclictest1859714-21kworker/u8:1+flush-259:020:30:172
1875731992113,5cyclictest1990492-21latency_hist21:00:002
1875726992119,1cyclictest2157484-21kworker/u8:1+events_unbound23:56:151
1875726992118,2cyclictest2072928-21kworker/u8:0+flush-259:022:32:321
1875726992118,2cyclictest1859714-21kworker/u8:1+events_unbound21:35:181
1875726992118,2cyclictest1859714-21kworker/u8:1+events_unbound21:35:171
1875726992117,3cyclictest2199980-21kworker/u8:0+events_unbound00:20:141
1875726992117,3cyclictest1859714-21kworker/u8:1+flush-259:022:09:471
1875726992116,3cyclictest335557-21systemd-journal00:40:001
1875726992115,4cyclictest335557-21systemd-journal19:34:161
1875719992118,2cyclictest2120623-21apt-key23:04:590
1875719992117,3cyclictest2215665-21apt-get00:35:130
1875719992117,3cyclictest2031878-21kworker/u8:3+flush-259:022:40:160
1875719992117,3cyclictest1964906-21apt-get20:35:130
187573699206,12cyclictest513-21gdbus00:30:003
1875736992014,4cyclictest1990734-21/usr/sbin/munin21:00:143
1875731992016,3cyclictest335557-21systemd-journal22:02:252
1875726992017,2cyclictest1891429-21kworker/u8:2+events_unbound20:05:141
1875726992017,2cyclictest1859714-21kworker/u8:1+flush-259:021:30:241
1875726992016,3cyclictest2157484-21kworker/u8:1+events_unbound23:40:211
1875726992014,4cyclictest1886254-21latency_hist19:20:011
1875719992017,2cyclictest2157484-21kworker/u8:1+events_unbound23:40:160
1875719992017,2cyclictest2072928-21kworker/u8:0+flush-259:022:55:000
1875719992016,3cyclictest2031878-21kworker/u8:3+flush-259:022:45:160
1875719992016,3cyclictest2020670-21kworker/u8:2+events_unbound21:50:220
1875719992016,3cyclictest1985548-21kworker/u8:2+flush-259:021:10:210
1875719992016,3cyclictest1859714-21kworker/u8:1+events_unbound21:05:140
1875719992015,4cyclictest2084639-21kworker/u8:2+flush-259:023:35:260
1875719992015,4cyclictest2072928-21kworker/u8:0+flush-259:022:20:010
1875719992015,3cyclictest1948156-21tail20:15:190
1875736991915,3cyclictest2031878-21kworker/u8:3+flush-259:022:02:213
1875736991914,3cyclictest2037301-21apt-get21:45:093
1875731991913,4cyclictest1881083-21latency_hist19:15:012
1875726991915,3cyclictest2131342-21kworker/u8:3+flush-259:023:20:211
1875726991915,3cyclictest2063060-21kworker/u8:4+events_unbound22:12:281
1875726991915,3cyclictest1896553-21kworker/u8:4+flush-259:019:35:171
1875726991915,2cyclictest2141998-21kworker/u8:0+flush-259:023:35:161
1875726991914,4cyclictest1859714-21kworker/u8:1+flush-259:022:15:241
1875726991914,3cyclictest1985582-21apt-get20:55:111
1875719991915,3cyclictest2063060-21kworker/u8:4+flush-259:022:10:240
1875719991915,3cyclictest2006280-21kworker/u8:0+events_unbound21:15:190
1875719991915,3cyclictest1859714-21kworker/u8:1+flush-259:021:40:180
1875719991915,3cyclictest1859714-21kworker/u8:1+events_unbound21:30:200
1875719991915,3cyclictest1859714-21kworker/u8:1+events_unbound21:30:000
1875719991914,3cyclictest335557-21systemd-journal00:09:260
1875719991914,3cyclictest1954530-21/usr/sbin/munin20:25:140
187573699188,9cyclictest0-21swapper/323:40:003
187573699181,5cyclictest1880219-21ntpq19:10:223
1875736991815,1cyclictest2126829-21irqcore23:10:153
1875736991813,4cyclictest2168310-21apt-get23:50:103
1875736991810,6cyclictest1996005-21cpuspeed_turbos21:05:143
1875731991814,2cyclictest0-21swapper/221:30:002
1875731991813,4cyclictest1891429-21kworker/u8:2+flush-259:019:47:472
1875731991813,3cyclictest509-21rs:main2
1875731991812,4cyclictest2073714-21base6422:20:012
1875731991811,5cyclictest2062835-21latency_hist22:10:012
1875726991815,2cyclictest2006280-21kworker/u8:0+flush-259:021:20:191
1875726991814,3cyclictest2174272-21kworker/u8:4+events_unbound00:00:011
1875726991813,4cyclictest1928342-21kworker/u8:0+events_unbound20:01:181
1875726991813,3cyclictest2157484-21kworker/u8:1+events_unbound00:15:131
1875719991814,3cyclictest2174272-21kworker/u8:4+events_unbound00:10:160
1875719991814,3cyclictest2157484-21kworker/u8:1+flush-259:000:25:170
1875719991814,3cyclictest2131342-21kworker/u8:3+events_unbound23:20:210
1875719991814,3cyclictest1859714-21kworker/u8:1+events_unbound19:20:190
1875719991813,4cyclictest2006280-21kworker/u8:0+flush-259:021:20:160
1875736991712,3cyclictest509-21rs:main3
1875736991712,3cyclictest2068293-21apt-get22:15:113
187573699170,4cyclictest1917692-21cron19:50:003
187573699170,16cyclictest0-21swapper/300:05:163
1875726991714,2cyclictest2157484-21kworker/u8:1+events_unbound00:30:141
1875726991714,2cyclictest2031878-21kworker/u8:3+flush-259:022:45:231
1875719991713,3cyclictest1881315-21kworker/u8:3+flush-259:019:40:240
1875719991713,3cyclictest1859714-21kworker/u8:1+events_unbound19:10:240
187573699161,4cyclictest1938342-21cron20:10:003
1875736991613,2cyclictest1896517-21idleruntime-cro19:29:593
1875736991611,4cyclictest1970012-21kworker/u8:3+flush-259:020:50:263
187573699160,4cyclictest2162439-21turbostat23:45:003
187573699160,4cyclictest2011915-21ps21:20:163
1875731991614,1cyclictest1979904-21fwupd20:47:122
1875731991613,1cyclictest0-21swapper/223:40:002
1875731991612,3cyclictest2094472-21latency_hist22:40:012
1875731991612,3cyclictest1896553-21kworker/u8:4+flush-259:019:57:212
1875731991611,3cyclictest0-21swapper/220:54:592
1875731991610,4cyclictest2173186-21apt-get23:55:002
1875726991613,2cyclictest2084639-21kworker/u8:2+events_unbound23:30:211
1875726991612,3cyclictest2020670-21kworker/u8:2+flush-259:021:52:371
1875719991614,1cyclictest60350irq/132-enp0s29f1:rx-023:25:150
1875719991613,2cyclictest60350irq/132-enp0s29f1:rx-021:58:210
1875719991613,2cyclictest60350irq/132-enp0s29f1:rx-020:46:420
1875719991613,2cyclictest60350irq/132-enp0s29f1:rx-019:45:220
1875719991613,2cyclictest60350irq/132-enp0s29f1:rx-019:15:180
187573699155,2cyclictest2219760-21ntp_offset00:35:243
187573699151,5cyclictest2093399-21ntpq22:35:213
187573699151,3cyclictest335557-21systemd-journal00:20:003
187573699151,3cyclictest2021602-21systemd-run21:30:003
1875736991512,2cyclictest1859714-21kworker/u8:1+flush-259:020:15:173
1875736991511,3cyclictest2199980-21kworker/u8:0+flush-259:000:23:243
187573699150,5cyclictest2115390-21/usr/sbin/munin23:00:003
187573699150,4cyclictest2125313-21/usr/sbin/munin23:05:193
187573699150,4cyclictest2088683-21tail22:30:243
187573699150,4cyclictest1907302-21turbostat19:40:003
187573199153,6cyclictest1969785-21munin-run20:40:002
1875731991512,2cyclictest2167628-21latency_hist23:50:002
1875726991510,3cyclictest1943132-21kworker/u8:4+events_unbound20:26:181
1875719991513,1cyclictest60350irq/132-enp0s29f1:rx-021:45:150
1875719991512,2cyclictest60350irq/132-enp0s29f1:rx-023:50:110
1875719991512,2cyclictest60350irq/132-enp0s29f1:rx-020:33:400
1875719991512,2cyclictest60350irq/132-enp0s29f1:rx-019:55:130
1875719991511,3cyclictest1859714-21kworker/u8:1+events_unbound20:40:140
1875719991510,4cyclictest1859714-21kworker/u8:1+flush-259:020:10:260
187573699148,4cyclictest335557-21systemd-journal22:00:003
187573699144,3cyclictest2152257-21sed23:35:013
187573699143,3cyclictest1923619-21sed19:55:173
187573699141,3cyclictest2074296-21/usr/sbin/munin22:20:173
1875736991411,2cyclictest2083987-21sh22:30:003
187573699140,5cyclictest1954284-21awk20:25:003
187573699140,4cyclictest2210725-21uniq00:30:143
187573699140,4cyclictest1918395-21grep19:50:173
187573699140,4cyclictest1875499-21apt-get19:10:003
187573199147,4cyclictest1959458-21latency_hist20:30:012
1875731991411,2cyclictest2157945-21irqcore23:40:172
1875731991411,2cyclictest2010239-21needreboot21:15:202
1875731991411,2cyclictest2000845-21cron21:10:002
1875731991410,2cyclictest0-21swapper/222:50:012
1875719991413,1cyclictest60350irq/132-enp0s29f1:rx-021:00:230
1875719991412,1cyclictest60350irq/132-enp0s29f1:rx-023:15:140
1875719991412,1cyclictest60350irq/132-enp0s29f1:rx-021:38:050
1875719991412,1cyclictest60350irq/132-enp0s29f1:rx-021:38:050
1875719991412,1cyclictest60350irq/132-enp0s29f1:rx-020:00:240
1875719991412,1cyclictest60350irq/132-enp0s29f1:rx-019:30:150
1875719991412,1cyclictest60350irq/132-enp0s29f1:rx-000:20:250
1875719991411,2cyclictest60350irq/132-enp0s29f1:rx-023:10:150
1875719991411,2cyclictest2166663-21needreboot23:45:180
1875719991410,2cyclictest2152096-21turbostat23:34:590
187573699134,1cyclictest2141999-21/usr/sbin/munin23:25:223
1875736991310,2cyclictest1890954-21unixbench_singl19:20:243
187573699130,6cyclictest1932353-21ntpq20:00:203
187573699130,4cyclictest2004499-21grep21:10:203
187573699130,4cyclictest1960721-21uname20:30:173
187573199139,3cyclictest2078880-21kworker/u8:1+flush-259:022:25:262
187573199139,3cyclictest2042233-21kworker/u8:0+flush-259:021:50:062
1875731991311,1cyclictest0-21swapper/223:15:002
1875731991310,2cyclictest2145989-21ntp_kernel_err23:25:202
1875731991310,2cyclictest2105756-21irqcore22:50:182
1875731991310,2cyclictest2068361-21cpuspeed_turbos22:15:132
1875731991310,2cyclictest2024139-21kthreadcore21:30:172
1875731991310,2cyclictest1901732-21systemd19:35:002
1875731991310,2cyclictest1893946-21kthreadcore19:25:182
1875719991311,1cyclictest60350irq/132-enp0s29f1:rx-020:07:570
187573699123,2cyclictest1965258-21cat20:35:173
187573699120,4cyclictest1979424-21/usr/sbin/munin20:45:243
187573199126,4cyclictest335557-21systemd-journal23:20:002
187573199121,3cyclictest2120609-21awk23:05:002
187573199121,3cyclictest2115384-21awk23:00:002
1875719991210,1cyclictest60350irq/132-enp0s29f1:rx-020:55:200
187573199119,1cyclictest0-21swapper/223:20:132
187573199117,2cyclictest0-21swapper/220:15:012
187573199116,3cyclictest2032098-21/usr/sbin/munin21:40:142
187573199110,6cyclictest1954302-21munin-run20:25:002
187573199107,2cyclictest0-21swapper/200:00:222
187573199105,3cyclictest335557-21systemd-journal22:35:012
187573199100,2cyclictest1974977-21grep20:45:002
18757319986,1cyclictest0-21swapper/221:35:142
18757319986,1cyclictest0-21swapper/221:35:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional