You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 09:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed Jul 22, 2026 00:43:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
203161499514,37cyclictest151ktimers/019:30:440
2031614995040,8cyclictest421-21dbus-daemon20:10:010
2031616994940,6cyclictest2064592-21systemctl21:00:002
2031614994941,5cyclictest2066126-21systemd-run21:05:020
2031614994739,5cyclictest421-21dbus-daemon19:40:010
2031614994739,5cyclictest2119748-21aten_r2power_en22:35:420
2031615994638,5cyclictest1001-21mta-sts-daemon22:00:441
2031614994632,11cyclictest0-21swapper/000:40:010
2031616994537,5cyclictest2040638-21systemd-run19:40:012
203161599446,4cyclictest2102285-21ssh22:05:511
2031616994335,5cyclictest2186041-21systemd-run00:25:012
2031616994333,6cyclictest2146545-21munin-run23:20:012
2031615994312,14cyclictest62750lldpd22:50:481
2031614994332,7cyclictest151ktimers/023:25:430
2031616994234,4cyclictest2154591-21kworker/u8:12-rpciod00:00:022
2031616994232,6cyclictest420-21avahi-daemon00:11:232
2031615994234,5cyclictest2122296-21idleruntime-cro22:40:011
2031616994131,6cyclictest277787-21cups-browsed23:41:292
2031615994134,4cyclictest2195110-21sendmail00:40:011
2031614994134,4cyclictest421-21dbus-daemon22:00:010
2031614994132,4cyclictest0-21swapper/021:40:480
2031614994131,6cyclictest420-21avahi-daemon21:36:560
203161499412,32cyclictest2077808-21sh21:25:450
203161499412,32cyclictest2077808-21sh21:25:450
203161699404,28cyclictest2193165-21ssh00:35:442
2031616994032,5cyclictest2134468-21latency_hist23:00:022
2031616994032,5cyclictest2089873-21sshd21:45:412
2031616994030,6cyclictest420-21avahi-daemon00:31:562
2031615994032,4cyclictest2043750-21latency_hist19:50:031
2031615994010,22cyclictest2071833-21sh21:15:471
2031615994010,22cyclictest2071833-21sh21:15:471
2031614994034,4cyclictest2060096-21cron20:45:010
2031614994032,5cyclictest2113753-21aten_r2power_vo22:25:360
2031614994031,5cyclictest151ktimers/023:35:450
2031614994031,5cyclictest151ktimers/023:35:450
2031616993931,5cyclictest2152626-21munin-run23:30:012
203161699391,7cyclictest171rcu_preempt22:30:492
2031615993932,4cyclictest2117030-21if_wlan022:30:411
2031615993931,5cyclictest2158675-21cron23:40:011
2031615993931,5cyclictest2158675-21cron23:40:011
2031615993931,5cyclictest2105722-21unixbench-2d22:10:571
2031614993925,7cyclictest12650irq/32-eth000:15:480
2031614993918,14cyclictest12650irq/32-eth021:20:500
2031614993918,14cyclictest12650irq/32-eth021:20:500
2031616993831,4cyclictest2073700-21ssh21:20:012
2031616993831,4cyclictest2073700-21ssh21:20:012
2031616993830,5cyclictest2108824-21unixbench_singl22:15:582
2031616993830,5cyclictest2084374-21timerandwakeup21:36:042
2031616993828,8cyclictest2067596-21/usr/sbin/munin21:10:022
2031615993829,5cyclictest1003-21mta-sts-daemon22:55:511
2031614993828,7cyclictest2052585-21cron20:20:010
2031616993727,7cyclictest1-21systemd22:00:012
2031615993731,3cyclictest0-21swapper/123:35:011
2031615993731,3cyclictest0-21swapper/121:45:011
2031615993729,5cyclictest2149671-21latency_hist23:25:021
2031615993728,6cyclictest439-21systemd-logind23:05:011
203161599372,26cyclictest2114359-21ssh22:25:481
2031614993727,7cyclictest2174841-21missed_timers00:05:410
2031614993722,13cyclictest2131369-21ssh22:55:010
2031616993628,5cyclictest2181637-21grep00:15:582
2031616993626,7cyclictest1-21systemd23:10:012
2031616993624,8cyclictest277787-21cups-browsed20:10:472
203161699362,31cyclictest2126462-21ssh22:45:462
2031615993629,4cyclictest0-21swapper/100:10:561
2031615993628,5cyclictest487-21rs:main1
2031615993628,5cyclictest420-21avahi-daemon20:30:521
2031615993625,8cyclictest2180354-21/usr/sbin/munin00:15:361
2031615993611,20cyclictest271ktimers/100:30:471
2031614993630,3cyclictest0-21swapper/019:20:400
2031614993630,3cyclictest0-21swapper/019:20:400
2031614993622,11cyclictest0-21swapper/023:50:580
2031616993529,3cyclictest0-21swapper/223:45:562
2031616993529,3cyclictest0-21swapper/222:40:452
2031616993528,4cyclictest2042627-21fschecks_count19:45:392
2031616993524,8cyclictest2061595-21munin-plugin-st20:50:012
2031616993524,8cyclictest2061595-21munin-plugin-st20:50:012
203161699351,27cyclictest2156921-21grep23:35:512
203161699351,27cyclictest2156921-21grep23:35:512
2031615993527,5cyclictest2058632-21systemd-run20:40:011
2031615993526,5cyclictest2068767-21ntpq21:10:491
2031615993525,7cyclictest2051934-21ntp_kernel_err20:15:441
2031615993523,9cyclictest2174321-21aten_r2power_cu00:05:341
2031614993529,4cyclictest2104178-21latency_hist22:10:020
2031614993528,4cyclictest2168614-21kernelversion23:55:390
2031614993526,5cyclictest0-21swapper/022:55:490
2031614993522,9cyclictest12650irq/32-eth000:25:490
2031614993521,10cyclictest12650irq/32-eth000:30:440
203161499351,8cyclictest2050515-21ntpq20:10:470
2031616993426,4cyclictest1002-21mta-sts-daemon19:27:402
2031616993425,5cyclictest2116378-21latency_hist22:30:022
203161599343,6cyclictest171rcu_preempt20:25:471
2031615993429,2cyclictest0-21swapper/123:10:011
2031615993427,4cyclictest2032129-21grep19:10:401
2031615993426,5cyclictest2096712-21unixbench_singl21:55:561
2031615993426,5cyclictest2041073-21cpuspeed19:40:351
2031614993428,3cyclictest0-21swapper/021:35:010
2031614993427,4cyclictest2155680-21systemd-run23:35:020
2031614993427,4cyclictest1003-21mta-sts-daemon22:15:200
2031614993427,4cyclictest1000-21mta-sts-daemon22:03:280
2031614993426,5cyclictest2162880-21ntp_kernel_pll_23:45:470
2031614993425,6cyclictest2095020-21kthreadd21:55:010
2031614993424,7cyclictest2032412-21needreboot19:10:440
2031614993420,11cyclictest2053427-21needreboot20:20:420
203161499341,27cyclictest12650irq/32-eth022:10:460
2031616993326,4cyclictest0-21swapper/222:55:012
2031616993326,3cyclictest0-21swapper/219:35:012
2031616993325,5cyclictest2136097-21unixbench_multi23:01:002
2031616993325,5cyclictest2079929-21latency_hist21:30:022
2031616993325,5cyclictest2079929-21latency_hist21:30:022
2031615993325,5cyclictest2064452-21unixbench_multi20:55:591
2031615993325,4cyclictest2082077-21kworker/u8:2-xprtiod21:55:021
2031615993325,4cyclictest0-21swapper/123:55:511
203161599331,21cyclictest2170916-21cups-browsed00:00:521
203161499333,23cyclictest2144577-21ssh23:15:450
2031614993325,6cyclictest151ktimers/020:45:480
2031614993325,6cyclictest151ktimers/020:45:480
2031614993325,4cyclictest0-21swapper/021:15:370
2031614993325,4cyclictest0-21swapper/021:15:370
2031614993322,8cyclictest2045902-21needreboot19:55:420
2031614993317,11cyclictest1000-21mta-sts-daemon00:20:470
2031614993316,11cyclictest151ktimers/023:00:500
2031616993224,5cyclictest439-21systemd-logind20:25:012
2031616993224,5cyclictest2105200-21ssh22:10:462
2031616993224,4cyclictest2094993-21cron21:55:012
2031616993215,13cyclictest958-21sendmail-mta19:10:472
203161599326,5cyclictest171rcu_preempt21:05:461
2031615993226,3cyclictest2057118-21kworker/u8:10-xprtiod22:25:011
2031615993225,4cyclictest2146547-21grep23:20:011
2031615993225,3cyclictest0-21swapper/119:35:011
2031615993222,6cyclictest420-21avahi-daemon20:40:451
203161499324,23cyclictest2172167-21ssh00:00:560
203161499323,23cyclictest2067600-21diskmemload22:20:440
2031614993226,3cyclictest0-21swapper/023:05:470
2031614993225,4cyclictest2147259-21fschecks_count23:20:380
2031614993224,4cyclictest0-21swapper/023:10:580
2031614993221,4cyclictest2119280-21systemd22:35:020
2031614993212,16cyclictest665-21snmpd00:12:030
203161699313,23cyclictest2187169-21ssh00:25:472
2031616993125,3cyclictest0-21swapper/222:22:362
2031616993124,4cyclictest1003-21mta-sts-daemon21:13:072
2031616993124,3cyclictest0-21swapper/222:35:462
2031616993122,5cyclictest0-21swapper/221:21:172
2031616993122,5cyclictest0-21swapper/221:21:172
203161699312,21cyclictest2172073-21ssh00:00:522
2031615993125,3cyclictest0-21swapper/119:25:001
2031615993125,3cyclictest0-21swapper/119:25:001
2031615993124,4cyclictest2060743-21kernelversion20:45:401
2031615993124,4cyclictest2060743-21kernelversion20:45:401
2031615993124,4cyclictest2046596-21run-parts20:00:011
2031615993123,5cyclictest2039625-21fschecks_time19:35:371
2031615993120,8cyclictest2108000-21latency22:15:401
2031614993124,4cyclictest1002-21mta-sts-daemon23:42:140
2031616993024,3cyclictest0-21swapper/221:30:392
203161599302,4cyclictest171rcu_preempt23:40:391
2031615993024,2cyclictest0-21swapper/123:10:391
2031615993023,4cyclictest0-21swapper/122:50:011
2031615993020,8cyclictest2048753-21kernelversion20:05:401
2031615993019,8cyclictest2074738-21missed_timers21:20:471
2031615993019,8cyclictest2074738-21missed_timers21:20:471
2031615993013,13cyclictest2164054-21sshd23:47:481
2031615993013,11cyclictest2090389-21/usr/sbin/munin21:45:551
2031616992921,3cyclictest2060122-21kworker/u8:1320:45:012
2031615992921,5cyclictest2050248-21kernelversion20:10:401
2031615992921,4cyclictest2167031-21sh23:52:401
2031615992920,6cyclictest2066120-21systemd-run21:05:021
2031615992920,4cyclictest0-21swapper/100:25:481
2031614992922,3cyclictest0-21swapper/022:45:480
2031614992922,3cyclictest0-21swapper/019:15:550
2031614992921,5cyclictest2031613-21cyclictest22:40:510
2031614992919,5cyclictest12650irq/32-eth021:45:490
2031614992916,8cyclictest12650irq/32-eth021:10:490
2031616992822,3cyclictest1001-21mta-sts-daemon19:55:032
2031616992822,3cyclictest0-21swapper/223:22:132
2031616992822,3cyclictest0-21swapper/222:00:382
2031616992822,3cyclictest0-21swapper/221:40:442
2031616992822,3cyclictest0-21swapper/219:20:012
2031616992821,4cyclictest1000-21mta-sts-daemon22:08:112
2031616992821,3cyclictest0-21swapper/200:05:342
2031616992820,5cyclictest1-21systemd19:25:012
2031616992820,5cyclictest1-21systemd19:25:012
2031616992820,4cyclictest0-21swapper/220:50:402
203161699281,6cyclictest2153506-21latency23:30:402
2031615992822,3cyclictest1002-21mta-sts-daemon22:41:291
2031615992819,6cyclictest2080712-21memory21:30:401
2031614992822,3cyclictest1000-21mta-sts-daemon20:55:540
2031614992821,4cyclictest151ktimers/020:50:450
2031614992811,12cyclictest421-21dbus-daemon20:35:480
2031616992721,3cyclictest0-21swapper/223:50:572
2031616992719,5cyclictest439-21systemd-logind20:20:012
2031616992719,5cyclictest2049621-21systemd-run20:10:012
2031616992718,6cyclictest277787-21cups-browsed20:25:532
2031616992717,7cyclictest2046591-21/usr/sbin/munin20:00:022
2031615992720,4cyclictest1003-21mta-sts-daemon21:25:321
2031615992720,4cyclictest1003-21mta-sts-daemon21:25:321
2031615992717,7cyclictest2184034-21nfs_client00:20:411
2031614992721,3cyclictest0-21swapper/019:45:020
203161699262,7cyclictest2141598-21ntpq23:10:462
2031616992619,4cyclictest2066120-21systemd-run21:05:022
2031616992615,7cyclictest1002-21mta-sts-daemon19:40:452
2031615992620,3cyclictest0-21swapper/120:05:011
2031615992619,4cyclictest2037631-21kworker/u8:15-xprtiod19:30:011
2031614992620,4cyclictest2037609-21munin-plugin-st19:30:000
2031614992618,5cyclictest277787-21cups-browsed20:00:180
203161599257,11cyclictest1001-21mta-sts-daemon20:50:491
203161599252,16cyclictest2151318-21users23:25:561
2031615992518,5cyclictest277746-21cupsd19:53:101
2031615992518,3cyclictest0-21swapper/120:20:561
2031614992519,3cyclictest2042334-21/usr/sbin/munin19:45:480
2031614992518,3cyclictest0-21swapper/019:55:020
2031614992517,5cyclictest277787-21cups-browsed21:06:250
2031614992516,6cyclictest2057083-21cron20:35:010
2031616992418,3cyclictest0-21swapper/220:40:012
2031616992414,7cyclictest2047476-21ntp_kernel_pll_20:00:492
203161599248,12cyclictest421-21dbus-daemon19:15:591
2031616992316,4cyclictest1002-21mta-sts-daemon20:34:522
2031614992310,3cyclictest171rcu_preempt20:25:480
203161499202,15cyclictest2031568-21munin-run19:10:010
203161799183,8cyclictest0-21swapper/321:45:523
2031616991811,4cyclictest2031563-21latency_hist19:10:022
203161799171,8cyclictest0-21swapper/320:40:503
203161799171,7cyclictest0-21swapper/300:15:453
203161599172,3cyclictest171rcu_preempt19:10:021
203161799163,7cyclictest0-21swapper/322:55:503
203161799163,5cyclictest0-21swapper/321:50:453
203161799161,9cyclictest0-21swapper/322:05:483
203161799161,7cyclictest0-21swapper/323:40:583
203161799152,7cyclictest0-21swapper/322:45:463
203161799152,7cyclictest0-21swapper/300:30:443
203161799144,4cyclictest0-21swapper/320:25:503
203161799143,5cyclictest0-21swapper/323:20:443
203161799142,8cyclictest0-21swapper/321:55:463
203161799142,7cyclictest0-21swapper/322:40:013
203161799142,7cyclictest0-21swapper/320:00:403
203161799142,6cyclictest0-21swapper/323:00:443
203161799142,5cyclictest0-21swapper/319:10:453
203161799141,8cyclictest0-21swapper/323:05:433
203161799141,7cyclictest0-21swapper/323:35:513
203161799141,7cyclictest0-21swapper/323:35:513
203161799141,7cyclictest0-21swapper/322:50:463
203161799141,7cyclictest0-21swapper/322:15:413
203161799141,7cyclictest0-21swapper/321:35:443
203161799141,6cyclictest0-21swapper/320:20:483
203161799141,6cyclictest0-21swapper/320:10:463
203161799141,6cyclictest0-21swapper/319:15:463
203161799141,4cyclictest0-21swapper/300:25:463
203161799132,7cyclictest0-21swapper/322:25:433
203161799132,7cyclictest0-21swapper/321:35:003
203161799132,7cyclictest0-21swapper/300:02:513
203161799132,6cyclictest0-21swapper/321:25:403
203161799132,6cyclictest0-21swapper/321:25:403
203161799131,7cyclictest0-21swapper/323:55:493
203161799131,7cyclictest0-21swapper/323:50:443
203161799131,7cyclictest0-21swapper/323:15:523
203161799131,7cyclictest0-21swapper/323:10:423
203161799131,7cyclictest0-21swapper/321:15:563
203161799131,7cyclictest0-21swapper/321:15:563
203161799131,7cyclictest0-21swapper/319:50:023
203161799131,6cyclictest0-21swapper/321:00:463
203161799131,6cyclictest0-21swapper/320:30:433
203161799131,6cyclictest0-21swapper/319:55:443
203161799123,4cyclictest0-21swapper/320:05:483
203161799122,6cyclictest0-21swapper/319:45:013
203161799122,6cyclictest0-21swapper/300:20:413
203161799122,6cyclictest0-21swapper/300:10:343
203161799122,4cyclictest0-21swapper/319:20:423
203161799122,4cyclictest0-21swapper/319:20:423
203161799121,8cyclictest0-21swapper/321:10:393
203161799121,7cyclictest0-21swapper/322:30:363
203161799121,7cyclictest0-21swapper/321:20:463
203161799121,7cyclictest0-21swapper/321:20:463
203161799121,7cyclictest0-21swapper/300:35:413
203161799121,6cyclictest0-21swapper/322:40:443
203161799121,6cyclictest0-21swapper/320:55:483
203161799121,6cyclictest0-21swapper/320:45:383
203161799121,6cyclictest0-21swapper/320:45:383
203161799121,6cyclictest0-21swapper/319:50:423
203161799121,6cyclictest0-21swapper/300:05:353
203161799121,5cyclictest0-21swapper/320:15:443
203161799120,6cyclictest0-21swapper/323:25:443
203161799112,6cyclictest0-21swapper/323:45:353
203161799112,6cyclictest0-21swapper/321:40:483
203161799112,5cyclictest0-21swapper/319:35:433
203161799112,4cyclictest0-21swapper/320:35:413
203161799111,7cyclictest0-21swapper/322:20:383
203161799111,7cyclictest0-21swapper/322:10:343
203161799111,5cyclictest0-21swapper/322:00:343
203161799111,5cyclictest0-21swapper/319:25:443
203161799103,5cyclictest0-21swapper/323:30:333
203161799101,6cyclictest0-21swapper/321:05:533
20316179991,5cyclictest0-21swapper/320:50:333
20316179991,4cyclictest0-21swapper/319:35:023
20316179982,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional