You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 03:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Sat May 16, 2026 00:43:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1829121996657,6cyclictest439-21systemd-logind20:20:010
1829122995517,8cyclictest171rcu_preempt23:10:581
1829121995420,27cyclictest2010301-21ssh00:10:520
1829121995138,5cyclictest0-21swapper/021:26:000
1829121995042,5cyclictest1836625-21systemd-run19:35:010
1829121994925,15cyclictest1946751-21ssh22:50:560
1829123994833,8cyclictest1017-21sendmail-mta19:35:512
1829123994833,8cyclictest1017-21sendmail-mta19:35:512
182912299482,42cyclictest2002437-21ssh00:01:001
1829121994739,3cyclictest151ktimers/020:45:490
1829121994739,3cyclictest151ktimers/020:45:490
1829121994720,19cyclictest12350irq/32-eth000:15:560
1829121994720,19cyclictest12350irq/32-eth000:15:560
1829121994628,15cyclictest1832157-21latency_hist19:20:020
182912399456,32cyclictest1919095-21ssh22:16:012
1829121994535,7cyclictest1921654-21/usr/sbin/munin22:20:010
182912199442,36cyclictest1927051-21ssh22:25:540
182912199442,36cyclictest1927051-21ssh22:25:540
1829121994423,13cyclictest1961524-21latency_hist23:10:030
182912399434,31cyclictest1866403-21ssh21:10:522
1829122994335,5cyclictest2032920-21systemd-run00:40:011
1829122994334,5cyclictest1-21systemd00:00:011
182912299431,6cyclictest171rcu_preempt22:40:581
1829121994319,6cyclictest1865105-21diskmemload23:25:060
182912399424,29cyclictest1882912-21awk21:31:032
1829122994234,5cyclictest2012990-21munin-plugin-st00:15:011
1829121994233,6cyclictest1885948-21aten_r2power_cu21:35:430
1829121994223,16cyclictest1868561-21sh21:13:520
1829121994211,9cyclictest1994504-21ntp_states23:50:560
1829123994134,5cyclictest1829120-21cyclictest23:05:562
1829123994133,5cyclictest421-21dbus-daemon22:50:012
182912199418,12cyclictest1940523-21ssh22:43:060
1829121994130,8cyclictest1905561-21systemd-run22:00:010
1829121994110,17cyclictest2001430-21sh00:00:170
1829123994032,4cyclictest0-21swapper/221:10:012
1829123994030,6cyclictest420-21avahi-daemon22:06:462
1829123994030,6cyclictest0-21swapper/221:51:002
1829122994032,5cyclictest1946875-21ntpq22:51:011
182912199403,31cyclictest1950810-21ssh22:55:570
1829121994032,5cyclictest1922711-21/usr/sbin/munin22:20:480
1829121994030,6cyclictest1970163-21fschecks_count23:20:460
1829121994030,6cyclictest1970163-21fschecks_count23:20:460
1829121994029,4cyclictest1856985-21ntpq20:40:520
1829121994029,4cyclictest1856985-21ntpq20:40:520
1829121994028,10cyclictest1909576-21cron22:05:010
1829123993932,4cyclictest2016975-21ssh00:20:012
1829123993932,4cyclictest2016975-21ssh00:20:012
1829123993931,5cyclictest2018946-21head00:21:062
1829123993931,5cyclictest2001042-21cron00:00:012
1829122993931,5cyclictest1989106-21latency_hist23:45:001
1829122993931,5cyclictest1885329-21munin-run21:35:011
1829122993930,5cyclictest1907441-21unixbench_multi22:01:031
1829122993927,9cyclictest1875200-21unixbench_multi21:21:061
1829121993932,5cyclictest2001049-21sh00:00:010
1829121993932,4cyclictest151ktimers/020:55:510
1829121993931,5cyclictest1966791-21proc_pri23:15:590
1829121993930,3cyclictest0-21swapper/021:16:000
1829121993928,6cyclictest12450irq/33-eth023:00:570
1829123993832,3cyclictest0-21swapper/220:15:012
1829123993829,6cyclictest420-21avahi-daemon00:01:082
1829123993828,6cyclictest1934862-21ntp_kernel_pll_22:35:552
182912299384,26cyclictest1950745-21ssh22:55:551
1829122993830,5cyclictest1937500-21grep22:40:001
1829122993829,6cyclictest4149711-21cups-browsed21:36:301
1829121993813,9cyclictest1898033-21/usr/sbin/munin21:51:070
1829123993729,5cyclictest261-21systemd-journal00:15:022
1829123993728,5cyclictest1985143-21systemd-run23:40:012
1829122993731,3cyclictest0-21swapper/123:20:011
1829122993729,5cyclictest1862080-21cron21:00:001
1829122993727,7cyclictest1914755-21missed_timers22:10:521
1829122993725,9cyclictest421-21dbus-daemon00:10:021
1829121993730,5cyclictest1989135-21ssh23:45:010
1829121993730,4cyclictest1911459-21grep22:06:050
1829121993729,5cyclictest1963025-21threads23:11:030
1829121993727,7cyclictest1838099-21cut19:40:000
1829121993727,7cyclictest1838099-21cut19:40:000
1829121993719,10cyclictest1842916-21aten_r2power_fr19:55:430
1829121993714,7cyclictest1932438-21sh22:32:550
182912199371,25cyclictest1882700-21ssh21:30:560
182912399363,24cyclictest1878789-21ssh21:25:592
1829122993628,5cyclictest2028919-21ssh00:35:011
1829122993628,5cyclictest1957351-21munin-run23:05:011
1829122993627,6cyclictest0-21swapper/122:25:521
1829122993627,6cyclictest0-21swapper/122:25:521
1829122993626,7cyclictest4149707-21cupsd19:54:191
1829121993629,4cyclictest1895069-21mailstats21:46:020
1829121993629,4cyclictest1895069-21mailstats21:46:020
1829121993627,7cyclictest1842608-21sh19:55:010
1829121993626,5cyclictest2030262-21ntpq00:35:560
1829121993624,7cyclictest1854078-21ntpq20:30:580
1829121993623,8cyclictest1860017-21ntpq20:50:550
1829121993623,8cyclictest1851027-21ntpq20:20:540
1829121993622,10cyclictest1839041-21ntpq19:40:540
1829121993622,10cyclictest1839041-21ntpq19:40:540
1829121993620,11cyclictest2021423-21ssh00:25:320
1829121993611,9cyclictest1977984-21cpu23:30:450
1829121993611,9cyclictest1977984-21cpu23:30:450
1829121993611,8cyclictest1914973-21modprobe22:10:580
1829123993525,5cyclictest0-21swapper/223:01:012
1829122993525,7cyclictest2001030-21cups-browsed00:22:101
182912199359,9cyclictest1989622-21ssh23:45:350
182912199359,7cyclictest1846538-21ntpq20:05:560
1829121993526,6cyclictest1890501-21latency21:40:480
1829121993526,6cyclictest1890501-21latency21:40:480
1829121993524,7cyclictest1845082-21ntpq20:01:010
1829121993523,7cyclictest2006444-21ntpq00:05:590
1829121993523,7cyclictest1862949-21ntpq21:00:520
1829121993522,9cyclictest1852531-21ntpq20:25:550
1829121993522,8cyclictest1942859-21ntpq22:45:560
1829121993522,8cyclictest1834533-21ntpq19:25:550
1829121993518,13cyclictest1832350-21/usr/sbin/munin19:21:070
182912399344,25cyclictest1939032-21ssh22:41:012
1829123993426,5cyclictest1963245-21unixbench_singl23:11:042
1829123993426,5cyclictest1953606-21latency_hist23:00:032
1829123993426,5cyclictest1894711-21awk21:45:512
1829123993426,5cyclictest1894711-21awk21:45:512
1829123993425,6cyclictest420-21avahi-daemon00:08:352
1829122993427,4cyclictest1848596-21munin-plugin-st20:15:011
1829122993427,3cyclictest0-21swapper/122:18:291
1829122993426,5cyclictest1943447-21unixbench_singl22:46:071
1829122993426,5cyclictest1833639-21systemd-run19:25:011
1829122993425,6cyclictest1859047-21head20:46:031
1829122993425,6cyclictest1859047-21head20:46:031
1829122993424,7cyclictest421-21dbus-daemon21:54:401
1829122993424,7cyclictest2016969-21latency_hist00:20:011
1829122993424,7cyclictest2016969-21latency_hist00:20:011
1829122993424,6cyclictest1-21systemd21:11:011
1829122993422,9cyclictest1993739-21aten_r2power_en23:50:421
182912299342,23cyclictest1840470-21sendmail-mta19:45:511
1829121993424,8cyclictest2020117-21sshd00:23:240
1829121993424,8cyclictest1875579-21ssh21:21:420
1829121993422,9cyclictest1855727-21sensors_fan20:36:000
1829121993422,8cyclictest1848064-21ntpq20:10:550
1829121993421,8cyclictest1829952-21ntpq19:10:520
1829121993421,7cyclictest1865105-21diskmemload22:35:550
1829123993325,5cyclictest1978434-21needreboot23:30:522
1829123993325,5cyclictest1978434-21needreboot23:30:522
1829123993325,5cyclictest1905517-21cut22:00:002
1829123993325,5cyclictest1875175-21grep21:21:062
1829123993324,6cyclictest1890940-21sendmail_mailqu21:41:002
1829123993324,6cyclictest1890940-21sendmail_mailqu21:41:002
1829123993322,8cyclictest1870867-21switchtime21:16:052
182912299333,23cyclictest1894949-21ssh21:45:591
182912299333,23cyclictest1894949-21ssh21:45:591
1829122993327,4cyclictest1829120-21cyclictest23:30:491
1829122993327,4cyclictest1829120-21cyclictest23:30:491
1829122993327,3cyclictest0-21swapper/120:45:001
1829122993327,3cyclictest0-21swapper/120:45:001
1829122993325,5cyclictest1973959-21aten_r2power_po23:25:451
1829122993325,5cyclictest1971140-21unixbench-2d23:21:051
1829122993325,5cyclictest1971140-21unixbench-2d23:21:051
182912199333,20cyclictest1982410-21ssh23:35:510
1829121993323,4cyclictest2026320-21ntpq00:30:550
1829123993227,3cyclictest1881829-21kworker/u8:1-rpciod21:36:012
1829123993222,7cyclictest1994320-21needreboot23:50:502
1829123993222,6cyclictest261-21systemd-journal22:33:282
1829123993221,8cyclictest1926720-21if_wlan022:25:482
1829123993221,8cyclictest1926720-21if_wlan022:25:482
1829123993215,6cyclictest261-21systemd-journal22:55:012
1829122993226,3cyclictest0-21swapper/100:30:011
1829122993224,5cyclictest421-21dbus-daemon20:35:011
1829122993224,5cyclictest4149711-21cups-browsed19:28:151
1829122993224,5cyclictest1922288-21aten_r2power_cu22:20:421
1829122993224,5cyclictest1865142-21latency_hist21:10:021
1829122993223,5cyclictest0-21swapper/120:20:581
1829122993221,8cyclictest1930624-21latency22:30:491
1829122993221,8cyclictest1902503-21interrupts21:55:491
1829121993221,7cyclictest1840497-21ntpq19:45:530
182912399312,8cyclictest171rcu_preempt20:35:522
1829123993125,3cyclictest0-21swapper/220:00:012
1829123993123,5cyclictest1993101-21grep23:50:002
1829123993120,8cyclictest1914747-21missed_timers22:10:522
182912399311,24cyclictest1906849-21ssh22:00:522
1829122993125,3cyclictest0-21swapper/123:45:491
1829122993125,3cyclictest0-21swapper/122:10:011
1829122993124,4cyclictest1870713-21proc_pri21:16:021
1829122993124,3cyclictest0-21swapper/121:30:011
1829122993123,5cyclictest4149711-21cups-browsed19:11:121
1829122993122,6cyclictest4149711-21cups-browsed23:06:371
182912299312,24cyclictest1982476-21ssh23:35:531
1829121993123,4cyclictest1864524-21ntpq21:05:550
182912399303,21cyclictest4149711-21cups-browsed20:40:542
182912399303,21cyclictest4149711-21cups-browsed20:40:542
1829123993024,3cyclictest0-21swapper/200:30:002
1829123993022,6cyclictest0-21swapper/222:20:592
1829123993022,5cyclictest67868-21munin-node20:05:482
1829123993022,5cyclictest420-21avahi-daemon19:51:002
1829123993022,5cyclictest1-21systemd20:50:012
1829123993022,5cyclictest1-21systemd20:50:012
1829123993022,5cyclictest0-21swapper/219:25:012
1829122993025,2cyclictest0-21swapper/120:10:011
1829122993023,4cyclictest0-21swapper/120:55:001
1829122993022,5cyclictest1890148-21head21:40:431
1829122993022,5cyclictest1890148-21head21:40:431
1829122993022,5cyclictest1843133-21gltestperf19:55:471
1829122993021,6cyclictest4149711-21cups-browsed21:03:271
1829123992923,3cyclictest0-21swapper/223:45:002
1829123992923,3cyclictest0-21swapper/220:01:012
1829123992921,5cyclictest1966194-21fschecks_time23:15:452
1829123992921,4cyclictest0-21swapper/221:00:572
1829123992920,6cyclictest4149711-21cups-browsed23:21:072
1829123992920,6cyclictest4149711-21cups-browsed23:21:072
182912299296,15cyclictest428-21mta-sts-daemon19:40:501
182912299296,15cyclictest428-21mta-sts-daemon19:40:501
1829122992923,3cyclictest0-21swapper/120:20:011
182912399282,20cyclictest2026284-21ssh00:30:542
1829123992820,5cyclictest4149711-21cups-browsed23:25:422
1829123992820,5cyclictest2032872-21cron00:40:002
1829123992721,3cyclictest0-21swapper/221:00:002
1829123992721,3cyclictest0-21swapper/219:15:012
1829122992719,5cyclictest4149711-21cups-browsed20:29:311
1829122992712,12cyclictest261rcuc/119:15:571
1829123992620,3cyclictest0-21swapper/220:35:002
1829123992618,5cyclictest1853032-21unixbench_singl20:26:052
1829123992618,5cyclictest1841033-21sed19:46:042
1829122992612,4cyclictest1838023-21unixbench_singl19:36:021
1829122992612,4cyclictest1838023-21unixbench_singl19:36:021
1829121992619,3cyclictest1829076-21munin-run19:10:010
1829123992518,5cyclictest1850445-21aten_r2power_fr20:20:432
1829123992518,4cyclictest1859633-21fschecks_time20:50:472
1829123992515,8cyclictest1831472-21ntp_kernel_pll_19:15:542
1829123992511,5cyclictest1838453-21aten2.4-expect19:40:422
1829123992511,5cyclictest1838453-21aten2.4-expect19:40:422
1829122992514,8cyclictest1845007-21ntp_kernel_pll_20:00:571
1829123992416,5cyclictest1850101-21munin-run20:20:012
1829122992417,4cyclictest1855623-21proc_pri20:35:571
1829123992313,8cyclictest1835916-21needreboot19:30:502
182912299232,16cyclictest1836023-21ntpq19:30:551
182912399201,14cyclictest1834589-21ntpq19:25:582
182912499193,9cyclictest0-21swapper/300:05:013
1829123991914,2cyclictest0-21swapper/219:10:022
182912499183,8cyclictest0-21swapper/323:30:553
182912499183,8cyclictest0-21swapper/323:30:553
182912499181,9cyclictest0-21swapper/321:25:513
1829122991811,4cyclictest1829070-21latency_hist19:10:021
182912499172,9cyclictest0-21swapper/321:55:533
182912499171,9cyclictest0-21swapper/322:05:553
182912499163,8cyclictest0-21swapper/300:20:013
182912499163,8cyclictest0-21swapper/300:20:013
182912499163,7cyclictest0-21swapper/320:50:003
182912499163,7cyclictest0-21swapper/320:50:003
182912499163,6cyclictest0-21swapper/323:05:523
182912499162,8cyclictest0-21swapper/323:15:513
182912499162,8cyclictest0-21swapper/322:51:013
182912499161,9cyclictest0-21swapper/323:56:023
182912499161,9cyclictest0-21swapper/323:40:553
182912499161,9cyclictest0-21swapper/321:50:543
182912499161,9cyclictest0-21swapper/321:30:573
182912499161,9cyclictest0-21swapper/321:15:573
182912499161,7cyclictest0-21swapper/323:35:533
182912499161,10cyclictest0-21swapper/321:20:523
182912499153,8cyclictest0-21swapper/323:55:023
182912499153,8cyclictest0-21swapper/322:30:513
182912499153,6cyclictest0-21swapper/320:40:003
182912499152,8cyclictest0-21swapper/323:20:433
182912499152,8cyclictest0-21swapper/323:20:433
182912499152,8cyclictest0-21swapper/322:20:553
182912499152,8cyclictest0-21swapper/321:40:513
182912499152,8cyclictest0-21swapper/321:40:513
182912499152,8cyclictest0-21swapper/321:10:473
182912499152,8cyclictest0-21swapper/300:20:503
182912499151,9cyclictest0-21swapper/322:55:563
182912499151,9cyclictest0-21swapper/300:05:543
182912499151,8cyclictest0-21swapper/322:00:523
182912499151,6cyclictest0-21swapper/323:30:023
182912499151,6cyclictest0-21swapper/320:00:543
182912499151,6cyclictest0-21swapper/319:25:513
182912499144,7cyclictest0-21swapper/323:00:543
182912499144,6cyclictest0-21swapper/322:36:003
182912499144,6cyclictest0-21swapper/319:15:013
182912499143,7cyclictest0-21swapper/300:25:563
182912499142,7cyclictest0-21swapper/323:10:423
182912499142,7cyclictest0-21swapper/322:25:503
182912499142,7cyclictest0-21swapper/322:25:503
182912499142,7cyclictest0-21swapper/319:30:563
182912499141,9cyclictest0-21swapper/321:35:453
182912499141,8cyclictest0-21swapper/322:45:533
182912499141,8cyclictest0-21swapper/300:35:553
182912499141,8cyclictest0-21swapper/300:30:503
182912499141,7cyclictest0-21swapper/323:45:523
182912499141,7cyclictest0-21swapper/322:45:023
182912499141,7cyclictest0-21swapper/321:45:513
182912499141,7cyclictest0-21swapper/321:45:513
182912499141,7cyclictest0-21swapper/321:00:533
182912499141,6cyclictest0-21swapper/320:30:513
182912499141,6cyclictest0-21swapper/320:20:523
182912499141,6cyclictest0-21swapper/319:40:543
182912499141,6cyclictest0-21swapper/319:40:543
182912499141,6cyclictest0-21swapper/319:15:523
182912499133,6cyclictest0-21swapper/320:40:533
182912499133,6cyclictest0-21swapper/320:40:533
182912499133,4cyclictest0-21swapper/320:15:583
182912499133,4cyclictest0-21swapper/319:45:563
182912499132,7cyclictest0-21swapper/322:10:563
182912499132,7cyclictest0-21swapper/321:10:013
182912499131,7cyclictest0-21swapper/319:55:013
182912499131,7cyclictest0-21swapper/319:35:513
182912499131,7cyclictest0-21swapper/319:35:513
182912499131,7cyclictest0-21swapper/300:10:423
182912499122,6cyclictest0-21swapper/322:15:453
182912499122,5cyclictest0-21swapper/320:55:533
182912499122,4cyclictest0-21swapper/319:55:503
182912499121,6cyclictest0-21swapper/320:50:523
182912499121,5cyclictest0-21swapper/320:25:483
182912499112,6cyclictest0-21swapper/320:05:533
182912499111,6cyclictest0-21swapper/319:20:543
182912499101,6cyclictest0-21swapper/320:10:453
18291249981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional