You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 19:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sun Dec 14, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2001295994339,3cyclictest2225265-21kworker/u8:2+events_unbound11:25:222
2001295993631,3cyclictest2173648-21/usr/sbin/munin09:55:232
2001295993427,5cyclictest2070285-21diskstats08:15:152
2001295993229,2cyclictest2158506-21kworker/u8:2+events_unbound10:15:202
2001288993228,3cyclictest2011938-21gpgv07:20:000
2001295993128,2cyclictest2012029-21kworker/u8:0+events_unbound07:30:252
2001293992823,3cyclictest2318467-21latency_hist12:15:001
2001295992622,3cyclictest2110089-21kworker/u8:0+events_unbound08:50:252
2001288992620,4cyclictest2183671-21latency_hist10:05:000
2001299992519,4cyclictest2152356-21apt-get09:35:003
2001299992519,4cyclictest2044063-21kworker/u8:1+flush-259:008:10:213
2001299992518,5cyclictest2125644-21ntpq09:05:223
2001293992520,4cyclictest2023118-21gpgv07:30:011
200128899251,22cyclictest1-21systemd10:55:000
2001295992421,2cyclictest2193682-21kworker/u8:3+events_unbound10:35:152
2001295992420,3cyclictest2023029-21gpgv07:29:592
2001295992417,4cyclictest2277194-21latency_hist11:35:002
2001295992317,4cyclictest2333904-21latency_hist12:30:002
2001293992317,4cyclictest252-21systemd-journal12:05:001
200128899239,5cyclictest2115328-21/usr/sbin/munin08:55:200
2001299992218,3cyclictest2001697-21/usr/sbin/munin07:10:123
2001299992218,2cyclictest2011841-21https07:20:003
2001299992217,3cyclictest2225265-21kworker/u8:2+flush-259:010:50:173
2001299992216,4cyclictest2121397-21latency_hist09:05:003
2001299992214,5cyclictest252-21systemd-journal10:45:133
2001299992214,5cyclictest2059525-21latency_hist08:05:013
2001295992218,3cyclictest2297793-21kworker/u8:0+events_unbound12:05:172
2001295992218,3cyclictest2225265-21kworker/u8:2+flush-259:011:15:012
2001295992218,3cyclictest2193682-21kworker/u8:3+events_unbound10:30:162
2001295992216,4cyclictest2142437-21diskstats09:25:142
2001293992216,4cyclictest2152500-21cat09:35:001
200128899228,4cyclictest2218665-21load10:35:190
200128899225,6cyclictest2339092-21latency_hist12:35:010
2001288992217,4cyclictest2156996-21ntpq09:35:210
2001288992217,4cyclictest2043124-21ntpq07:45:200
2001299992117,2cyclictest0-21swapper/309:15:003
2001299992114,5cyclictest2293081-21diskstats11:50:123
2001295992118,2cyclictest2241217-21kworker/u8:4+flush-259:011:05:002
2001295992117,3cyclictest2225265-21kworker/u8:2+flush-259:011:50:202
2001295992116,3cyclictest2095143-21kworker/u8:2+events_unbound09:30:222
200128899218,8cyclictest2241323-21cut11:00:010
200128899216,6cyclictest2069454-21uptime08:10:260
200128899216,5cyclictest2343293-21netstat12:35:200
2001288992114,5cyclictest252-21systemd-journal08:00:000
2001299992016,2cyclictest0-21swapper/309:50:003
2001299992012,6cyclictest2291834-21ntpq11:45:223
2001295992018,2cyclictest2110089-21kworker/u8:0+events_unbound09:15:212
2001295992017,2cyclictest2044063-21kworker/u8:1+events_unbound08:20:232
2001295992017,2cyclictest1991005-21kworker/u8:3+events_unbound07:45:232
2001295992016,3cyclictest2298038-21apt-get11:55:112
2001295992015,3cyclictest2006881-21/usr/sbin/munin07:15:242
2001295992014,2cyclictest1985681-21kworker/u8:1+events_unbound07:10:172
2001293992016,3cyclictest2158186-21apt-get09:40:121
2001293992016,3cyclictest2017541-21gpgv07:25:011
2001293992014,4cyclictest252-21systemd-journal08:55:011
200128899207,4cyclictest2060112-21ls08:05:150
200128899207,4cyclictest2013137-21cut07:20:180
200128899206,9cyclictest2173275-21chrt09:53:370
200128899206,8cyclictest2225577-21apt-get10:45:150
200128899206,4cyclictest2313548-21https12:10:130
200128899206,4cyclictest2085379-21latency_hist08:30:010
200128899206,4cyclictest2001590-21cp07:10:010
200128899204,6cyclictest2198295-21ntp_kernel_pll_10:15:210
2001288992015,4cyclictest252-21systemd-journal11:34:310
2001288992015,3cyclictest2323608-21latency_hist12:20:010
2001299991915,3cyclictest2059551-21kworker/u8:3+events_unbound08:45:193
2001299991913,4cyclictest2317598-21ntpq12:10:233
2001293991916,2cyclictest2199185-21kworker/u8:0+flush-259:010:30:311
2001293991914,4cyclictest2116528-21apt-get09:00:131
2001293991914,3cyclictest2038909-21awk07:45:001
200128899196,9cyclictest2287462-21taskset11:44:480
200128899196,8cyclictest2168185-21chrt09:49:330
200128899196,7cyclictest2162799-21chrt09:41:270
200128899196,5cyclictest2282434-21cat11:40:010
200128899196,4cyclictest2094873-21ps08:35:240
200128899196,4cyclictest2087379-21cat08:30:180
200128899196,4cyclictest2054593-21https08:00:110
200128899196,3cyclictest2257917-21kthreadcore11:15:180
200128899196,2cyclictest2038023-21ntpq07:40:210
200128899195,5cyclictest2249756-21grep11:05:180
200128899195,5cyclictest2241767-21hddtemp_smartct11:00:160
2001299991815,2cyclictest2277217-21kworker/u8:0+flush-259:011:40:003
2001299991814,3cyclictest2287518-21kworker/u8:1+flush-259:012:00:013
2001295991813,3cyclictest2193682-21kworker/u8:3+flush-259:010:15:002
2001295991812,5cyclictest2236052-21latency_hist10:55:012
2001293991812,4cyclictest252-21systemd-journal11:00:011
200128899188,2cyclictest2209978-21/usr/sbin/munin10:30:150
200128899188,2cyclictest2075283-21apt-get08:20:140
200128899186,5cyclictest2106599-21cat08:50:180
200128899186,4cyclictest2295284-21cut11:50:160
200128899186,4cyclictest2096400-21cut08:40:170
200128899186,3cyclictest0-21swapper/010:25:170
200128899185,5cyclictest2291471-21kthreadcore11:45:180
200128899184,8cyclictest2299021-21irqstats11:55:160
200128899184,8cyclictest2131601-21taskset09:14:210
2001288991816,1cyclictest0-21swapper/007:10:000
2001288991812,4cyclictest2152576-21cat09:35:000
2001299991713,3cyclictest2017423-21apt-get07:25:023
2001299991713,3cyclictest1985681-21kworker/u8:1+flush-259:007:20:103
2001299991711,4cyclictest2209512-21latency_hist10:30:013
2001299991710,5cyclictest252-21systemd-journal11:10:173
2001295991714,2cyclictest2225265-21kworker/u8:2+flush-259:010:55:212
2001295991713,3cyclictest2241217-21kworker/u8:4+flush-259:012:35:162
2001295991713,3cyclictest1991005-21kworker/u8:3+events_unbound07:40:222
200128899179,6cyclictest2256635-21apt11:15:000
200128899175,4cyclictest2184979-21grep10:05:170
200128899175,4cyclictest2184979-21grep10:05:170
200128899175,3cyclictest2224376-21ntp_10:40:200
200128899174,7cyclictest2329180-21df12:25:120
200128899174,7cyclictest2044589-21fw_conntrack07:50:150
200128899174,7cyclictest2029828-21irqstats07:35:170
200128899174,3cyclictest2267493-21fw_forwarded_lo11:25:160
200128899172,10cyclictest2136040-21cut09:15:240
2001288991711,4cyclictest252-21systemd-journal12:10:000
2001295991613,2cyclictest2287518-21kworker/u8:1+flush-259:012:30:242
2001295991613,2cyclictest2044063-21kworker/u8:1+flush-259:008:40:262
2001295991612,3cyclictest2178321-21kworker/u8:1+flush-259:010:00:052
2001295991612,3cyclictest2085355-21kworker/u8:0+events_unbound08:30:162
2001293991613,2cyclictest2174096-21irqcore09:55:171
200129399160,11cyclictest32-21ksoftirqd/109:11:021
200128899169,5cyclictest2101225-21diskstats08:45:150
200128899169,5cyclictest2075008-21idleruntime-cro08:19:590
200128899169,5cyclictest2028563-21cron07:35:000
200128899166,7cyclictest2173640-21chrt09:55:060
200128899164,6cyclictest2120329-21memory09:00:210
200128899164,4cyclictest2263653-21grep11:20:200
200128899164,3cyclictest2190082-21irqstats10:10:160
200128899163,7cyclictest2327972-21ntp_states12:20:290
200128899163,7cyclictest2146120-21memory09:25:190
2001288991610,4cyclictest2023201-21date07:30:010
200129999159,5cyclictest2029008-21diskstats07:35:153
200129999159,4cyclictest2095640-21latency_hist08:40:003
200129999151,6cyclictest2344237-21latency_hist12:40:013
2001299991512,2cyclictest500-21systemd-logind08:10:013
2001299991512,2cyclictest2181092-21kthreadcore10:00:193
200129999151,12cyclictest1-21systemd10:35:003
2001295991513,1cyclictest0-21swapper/211:39:592
2001295991513,1cyclictest0-21swapper/208:55:202
2001295991511,3cyclictest2287518-21kworker/u8:1+events_unbound12:15:172
2001295991511,3cyclictest2110089-21kworker/u8:0+flush-259:009:40:232
2001295991511,3cyclictest2044063-21kworker/u8:1+events_unbound07:55:132
2001295991511,3cyclictest2012029-21kworker/u8:0+events_unbound07:20:192
2001295991511,2cyclictest2060010-21cut08:05:132
2001295991510,3cyclictest2059551-21kworker/u8:3+events_unbound08:50:012
200129399159,1cyclictest191rcu_preempt09:28:081
200129399158,5cyclictest2332990-21ntpq12:25:201
200129399152,4cyclictest2172816-21sed09:50:251
200129399151,4cyclictest2262071-21apt-get11:20:141
2001293991512,1cyclictest191rcu_preempt09:15:471
200128899158,5cyclictest2303660-21cut12:00:160
200128899158,4cyclictest2121668-21chrt09:05:130
200128899154,7cyclictest2141651-21taskset09:21:260
200128899154,2cyclictest2203484-21ntpq10:20:230
200129999149,4cyclictest2095875-21/usr/sbin/munin08:40:153
2001299991412,1cyclictest2106637-21irqcore08:50:193
2001299991410,3cyclictest2135869-21ntp_kernel_err09:15:213
200129599149,4cyclictest2044063-21kworker/u8:1+flush-259:008:10:412
200129599148,4cyclictest252-21systemd-journal09:35:012
2001295991412,2cyclictest2095143-21kworker/u8:2+events_unbound09:00:292
2001295991411,2cyclictest2241217-21kworker/u8:4+events_unbound12:20:142
2001295991410,3cyclictest2110089-21kworker/u8:0+events_unbound09:24:322
200129599140,8cyclictest2126544-21munin-run09:10:002
200129399147,1cyclictest191rcu_preempt08:35:301
200129399146,1cyclictest191rcu_preempt09:07:301
200129399145,2cyclictest0-21swapper/111:15:241
200129399143,3cyclictest32-21ksoftirqd/112:25:001
200129399143,2cyclictest2203390-21ntpq10:20:221
200129399142,8cyclictest32-21ksoftirqd/111:50:201
200129399142,2cyclictest2198296-21ntpq10:15:221
200129399142,2cyclictest2100074-21grep08:40:241
2001293991412,2cyclictest2095143-21kworker/u8:2+events_unbound09:22:161
2001293991412,2cyclictest2012029-21kworker/u8:0+events_unbound08:02:251
2001293991412,1cyclictest0-21swapper/110:40:151
2001293991411,2cyclictest2339441-21cpuspeed_turbos12:35:141
2001293991411,2cyclictest2215911-21kthreadcore10:35:171
2001293991410,1cyclictest191rcu_preempt08:11:231
200129399140,5cyclictest2008410-21cat07:15:191
200129999139,3cyclictest2095143-21kworker/u8:2+flush-259:009:30:003
200129999139,2cyclictest2277186-21turbostat11:34:593
200129999137,4cyclictest252-21systemd-journal12:04:593
200129999137,4cyclictest252-21systemd-journal11:30:003
200129999131,2cyclictest2225295-21awk10:45:003
2001299991310,2cyclictest2261323-21unixbench_multi11:15:253
200129599139,3cyclictest2241217-21kworker/u8:4+flush-259:011:05:252
200129599139,3cyclictest2225265-21kworker/u8:2+flush-259:011:20:522
200129599139,3cyclictest2158506-21kworker/u8:2+flush-259:009:54:352
2001295991311,1cyclictest0-21swapper/211:40:222
2001295991310,2cyclictest2257315-21irqcore11:15:172
2001295991310,2cyclictest2225265-21kworker/u8:2+events_unbound11:45:202
2001295991310,2cyclictest2059551-21kworker/u8:3+flush-259:008:25:252
2001295991310,2cyclictest2012029-21kworker/u8:0+flush-259:007:35:162
200129399139,3cyclictest2242482-21kthreadcore11:00:171
200129399139,3cyclictest2184008-21cpuspeed_turbos10:05:131
200129399139,3cyclictest2184008-21cpuspeed_turbos10:05:131
200129399137,1cyclictest191rcu_preempt08:20:191
200129399135,1cyclictest0-21swapper/110:10:211
200129399134,2cyclictest0-21swapper/109:35:221
200129399133,6cyclictest32-21ksoftirqd/108:55:121
200129399132,3cyclictest32-21ksoftirqd/107:50:231
200129399132,2cyclictest2004250-21grep07:10:141
200129399131,3cyclictest32-21ksoftirqd/111:45:201
200129399131,3cyclictest32-21ksoftirqd/107:45:201
200129399131,3cyclictest2284014-21cut11:40:181
200129399131,2cyclictest32-21ksoftirqd/111:30:121
200129399131,2cyclictest2281515-21ntpq11:35:211
200129399131,2cyclictest2271237-21ntpq11:25:221
2001293991311,1cyclictest311ktimers/110:45:241
2001293991310,2cyclictest2334240-21cpuspeed_turbos12:30:131
2001293991310,2cyclictest2164390-21kthreadcore09:45:191
2001293991310,2cyclictest2070546-21sed08:15:171
2001293991310,1cyclictest191rcu_preempt11:11:571
2001293991310,1cyclictest191rcu_preempt11:05:081
2001293991310,1cyclictest0-21swapper/107:39:591
200129399130,8cyclictest2204547-21/usr/sbin/munin10:25:221
200129399130,4cyclictest2308761-21taskset12:05:181
200129999129,2cyclictest2326173-21kthreadcore12:20:173
200129999129,2cyclictest2158506-21kworker/u8:2+flush-259:009:40:163
200129999127,3cyclictest2339090-21awk12:35:003
200129999127,3cyclictest2214867-21/usr/sbin/munin10:35:203
200129999126,4cyclictest2137041-21/usr/sbin/munin09:20:163
200129999121,3cyclictest2193962-21latency_hist10:15:003
200129599129,2cyclictest2241217-21kworker/u8:4+flush-259:012:10:152
200129599129,2cyclictest2012029-21kworker/u8:0+events_unbound08:04:422
200129399124,1cyclictest191rcu_preempt08:25:151
200129399123,1cyclictest191rcu_preempt10:50:141
200129399122,3cyclictest32-21ksoftirqd/107:55:221
200129399122,2cyclictest32-21ksoftirqd/107:30:141
200129399121,4cyclictest32-21ksoftirqd/110:00:251
2001293991210,1cyclictest301rcuc/108:32:291
200129399120,2cyclictest2322698-21ntpq12:15:211
200129399120,2cyclictest2063772-21ntpq08:05:201
200129999118,2cyclictest2111459-21cpuspeed_turbos08:55:143
200129999118,2cyclictest0-21swapper/308:20:183
200129999118,1cyclictest0-21swapper/311:00:003
200129999118,1cyclictest0-21swapper/307:45:173
200129999117,3cyclictest2225265-21kworker/u8:2+flush-259:011:25:003
200129999117,3cyclictest2044063-21kworker/u8:1+flush-259:007:55:463
200129999116,3cyclictest2318729-21apt-get12:15:133
200129999116,3cyclictest2044063-21kworker/u8:1+flush-259:008:20:013
200129999110,3cyclictest2173401-21latency_hist09:55:003
200129599118,2cyclictest2163441-21cpuspeed_turbos09:45:142
200129599118,2cyclictest2085355-21kworker/u8:0+events_unbound08:35:202
200129599117,3cyclictest2158506-21kworker/u8:2+flush-259:010:05:532
200129599117,3cyclictest2158506-21kworker/u8:2+flush-259:010:05:532
200129599111,2cyclictest2131679-21latency_hist09:15:002
200129399118,1cyclictest191rcu_preempt11:55:141
200129399111,3cyclictest32-21ksoftirqd/108:45:211
200129999105,4cyclictest2282621-21apt-get11:40:153
200129999105,3cyclictest2241432-21/usr/sbin/munin11:00:163
200129599105,4cyclictest2044279-21/usr/sbin/munin07:50:202
20012999996,2cyclictest2024707-21kthreadcore07:30:183
20012999994,4cyclictest2246846-21uniq11:05:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional