You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-17 - 23:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Thu Jul 17, 2025 12:40:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1874125995541,7cyclictest12450irq/32-eth009:01:070
1874126995241,7cyclictest0-21swapper/110:40:011
1874125994833,12cyclictest2063219-21munin-run12:40:010
1874125994833,12cyclictest2063219-21munin-run12:40:010
1874127994739,5cyclictest1944958-21idleruntime-cro10:00:012
1874126994738,4cyclictest0-21swapper/109:23:011
1874126994735,8cyclictest1880137-21systemd-run07:30:011
1874127994638,5cyclictest1056789-21cups-browsed11:39:592
1874127994637,6cyclictest1955379-21latency_hist10:15:032
1874126994636,7cyclictest2032066-21ls12:00:011
1874126994636,7cyclictest2032066-21ls12:00:011
1874127994537,5cyclictest1917225-21grep09:20:012
1874126994537,5cyclictest1901146-21latency_hist08:40:021
1874126994537,5cyclictest1901146-21latency_hist08:40:021
1874127994435,6cyclictest491-21gdbus12:15:022
1874127994435,6cyclictest491-21gdbus12:15:022
1874126994434,6cyclictest2051476-21sh12:25:011
1874127994337,3cyclictest0-21swapper/210:18:002
1874127994332,8cyclictest2001560-21sh11:20:012
1874126994334,6cyclictest1992784-21mailstats11:08:091
1874126994334,6cyclictest1992784-21mailstats11:08:091
1874125994329,6cyclictest1056789-21cups-browsed11:34:190
1874127994235,4cyclictest2024317-21ssh11:50:012
1874127994235,4cyclictest1958695-21kworker/u8:15-rpciod11:25:012
1874127994234,5cyclictest2062295-21unixbench_singl12:38:162
1874127994234,5cyclictest2062295-21unixbench_singl12:38:162
1874127994231,8cyclictest1908631-21sh09:05:012
1874126994231,8cyclictest422-21dbus-daemon09:40:031
1874125994235,4cyclictest1963878-21interrupts10:28:010
1874126994135,3cyclictest0-21swapper/109:13:121
1874126994129,9cyclictest1984391-21cpu10:57:531
1874126994129,9cyclictest1984391-21cpu10:57:531
1874125994135,3cyclictest0-21swapper/011:03:070
1874125994117,20cyclictest2030555-21ssh11:58:030
1874125994117,20cyclictest2030555-21ssh11:58:030
187412699404,30cyclictest2026651-21rm11:53:001
1874126994032,5cyclictest2057107-21aten_r2power_fr12:32:531
1874126994026,11cyclictest2015047-21needreboot11:38:001
1874125994031,6cyclictest1056789-21cups-browsed07:48:120
1874127993934,2cyclictest0-21swapper/209:53:032
1874127993933,3cyclictest0-21swapper/212:25:012
1874127993933,3cyclictest0-21swapper/210:30:012
1874127993933,3cyclictest0-21swapper/209:43:082
1874127993933,3cyclictest0-21swapper/209:33:032
1874127993933,3cyclictest0-21swapper/209:15:022
1874127993931,5cyclictest1897461-21ntp_kernel_err08:28:012
1874127993929,6cyclictest1690079-21ntpd11:13:042
1874126993933,3cyclictest0-21swapper/112:08:091
1874126993931,5cyclictest422-21dbus-daemon12:18:031
1874126993931,5cyclictest1999699-21fschecks_time11:17:581
187412699392,29cyclictest2053793-21ssh12:28:021
1874126993922,10cyclictest0-21swapper/110:13:081
1874125993933,4cyclictest1982671-21munin-run10:55:010
1874127993832,3cyclictest0-21swapper/211:05:032
1874127993831,4cyclictest957-21mta-sts-daemon10:38:392
1874127993828,7cyclictest1979360-21sh10:50:022
1874125993830,5cyclictest422-21dbus-daemon09:15:340
1874125993828,7cyclictest1868815-21kworker/u8:6+xprtiod10:00:010
1874125993827,8cyclictest1951093-21unixbench_multi10:08:140
1874127993731,3cyclictest0-21swapper/211:30:012
1874127993730,3cyclictest0-21swapper/208:45:012
1874127993727,7cyclictest421-21avahi-daemon10:05:242
1874127993725,8cyclictest955-21mta-sts-daemon11:56:592
1874127993725,8cyclictest955-21mta-sts-daemon11:56:592
187412699374,26cyclictest2042214-21sh12:13:061
187412699374,26cyclictest2042214-21sh12:13:061
1874126993729,5cyclictest2034920-21unixbench-2d12:03:121
1874125993730,4cyclictest1938097-21sh09:50:010
1874125993728,7cyclictest2057690-21ntpq12:33:020
1874125993726,9cyclictest1875631-21unin-run07:15:020
1874127993628,4cyclictest1888134-21fschecks_time07:57:572
1874127993627,6cyclictest1910024-21unixbench_multi09:08:152
1874127993627,6cyclictest1910024-21unixbench_multi09:08:152
1874127993621,8cyclictest957-21mta-sts-daemon12:15:182
1874126993628,5cyclictest2007234-21fschecks_time11:27:591
1874126993627,6cyclictest1951113-21unixbench_multi10:08:141
1874126993625,4cyclictest0-21swapper/111:23:011
187412699360,26cyclictest1996408-21sh11:13:081
1874125993630,4cyclictest2053724-21cut12:28:000
1874125993630,4cyclictest2020478-21grep11:45:010
1874125993629,4cyclictest1962166-21ssh10:25:020
1874125993629,4cyclictest1962166-21ssh10:25:020
1874125993628,5cyclictest1056789-21cups-browsed09:28:170
1874125993623,10cyclictest1996867-21unixbench-2d11:13:170
1874127993529,3cyclictest0-21swapper/211:33:032
1874127993529,3cyclictest0-21swapper/207:40:012
1874127993526,6cyclictest421-21avahi-daemon10:57:082
1874127993526,6cyclictest421-21avahi-daemon10:57:082
1874127993525,7cyclictest956-21mta-sts-daemon08:18:162
1874126993527,5cyclictest1988174-21aten_r2power_vo11:02:521
1874126993527,5cyclictest1937327-21head09:48:141
1874126993527,5cyclictest1933807-21tail09:43:151
1874126993525,7cyclictest1941521-21kworker/u8:3-xprtiod09:55:021
1874125993527,6cyclictest1901991-21ntp_kernel_pll_08:43:020
1874125993527,5cyclictest1969766-21sh10:36:250
1874125993527,4cyclictest12450irq/32-eth012:23:080
1874127993428,3cyclictest1975592-21kworker/u8:6-xprtiod10:45:012
1874127993428,3cyclictest0-21swapper/212:10:022
1874127993426,5cyclictest1920674-21systemd-run09:25:022
1874127993425,6cyclictest421-21avahi-daemon12:34:552
1874127993423,8cyclictest1910110-21diskmemload10:53:052
1874127993422,8cyclictest266-21systemd-journal08:38:132
1874127993422,8cyclictest266-21systemd-journal08:38:132
187412699343,25cyclictest1978037-21ssh10:48:041
1874126993427,4cyclictest957-21mta-sts-daemon09:28:091
1874126993426,5cyclictest1910130-21systemd-run09:10:021
1874126993426,5cyclictest1910130-21systemd-run09:10:021
1874126993424,7cyclictest266-21systemd-journal08:55:011
1874125993427,5cyclictest1952227-21sh10:10:290
1874125993426,5cyclictest1056789-21cups-browsed09:35:590
1874125993424,7cyclictest1932533-21/usr/sbin/munin09:43:160
187412599342,28cyclictest2003876-21ssh11:23:030
1874125993415,7cyclictest12550irq/33-eth009:13:100
1874125993414,16cyclictest2026570-21ssh11:52:570
1874127993324,4cyclictest0-21swapper/211:52:512
1874127993323,7cyclictest266-21systemd-journal11:45:012
1874126993326,4cyclictest2020473-21latency_hist11:45:011
1874126993326,4cyclictest1957711-21ssh10:18:121
1874126993325,4cyclictest0-21swapper/108:48:011
1874126993324,6cyclictest1968175-21unixbench_multi10:33:161
187412599333,24cyclictest2034410-21ssh12:03:030
1874125993327,3cyclictest0-21swapper/009:33:030
1874125993326,5cyclictest2009205-21cat11:30:020
1874125993324,6cyclictest421-21avahi-daemon10:15:080
1874125993324,6cyclictest1999860-21latency11:18:000
187412799324,25cyclictest1922913-21ssh09:28:142
187412799324,13cyclictest1910110-21diskmemload12:29:002
1874127993225,4cyclictest2035942-21kworker/u8:4-xprtiod12:05:022
1874127993225,3cyclictest0-21swapper/211:08:042
1874127993225,3cyclictest0-21swapper/211:08:042
1874127993225,3cyclictest0-21swapper/209:47:542
1874127993224,5cyclictest1878772-21latency_hist07:25:022
1874127993222,7cyclictest1960042-21sh10:22:502
1874127993222,7cyclictest1960042-21sh10:22:502
187412799322,24cyclictest1929803-21ssh09:38:052
1874126993225,4cyclictest955-21mta-sts-daemon10:23:031
1874126993225,4cyclictest955-21mta-sts-daemon10:23:031
1874126993223,5cyclictest2063211-21cron12:40:011
1874126993223,5cyclictest2063211-21cron12:40:011
1874126993220,5cyclictest1887834-21/usr/sbin/munin07:58:131
187412699321,26cyclictest1974628-21sshd10:43:001
1874125993226,4cyclictest2043669-21munin-plugin-st12:15:010
1874125993226,4cyclictest2043669-21munin-plugin-st12:15:010
1874125993226,4cyclictest1874124-21cyclictest10:58:020
1874125993226,4cyclictest1874124-21cyclictest10:58:020
1874125993224,6cyclictest151ktimers/007:28:000
1874125993224,5cyclictest1056789-21cups-browsed10:00:240
1874127993125,3cyclictest0-21swapper/210:03:142
1874127993124,5cyclictest0-21swapper/210:35:022
1874127993124,5cyclictest0-21swapper/208:15:012
1874127993124,5cyclictest0-21swapper/208:15:012
1874126993125,3cyclictest0-21swapper/110:28:181
1874126993124,3cyclictest1856891-21kworker/u8:9+xprtiod07:15:021
1874126993124,3cyclictest0-21swapper/110:53:111
1874126993123,5cyclictest1927679-21cron09:35:011
1874126993123,5cyclictest1893602-21grep08:15:011
1874126993123,5cyclictest1893602-21grep08:15:011
1874126993123,5cyclictest0-21swapper/107:20:011
1874126993123,3cyclictest0-21swapper/109:01:051
187412599313,24cyclictest2046062-21ssh12:18:030
1874125993124,5cyclictest0-21swapper/012:08:000
1874125993119,9cyclictest422-21dbus-daemon08:50:020
187412799302,23cyclictest1904966-21sshd08:53:012
1874127993021,7cyclictest1906365-21missed_timers08:58:002
1874127993021,6cyclictest453-21systemd-logind07:55:012
1874126993024,3cyclictest0-21swapper/108:20:011
1874126993024,3cyclictest0-21swapper/107:55:011
1874126993023,4cyclictest1902601-21cron08:45:001
187412699302,23cyclictest2011483-21sh11:33:071
1874126993021,6cyclictest421-21avahi-daemon09:17:241
1874126993021,6cyclictest1899621-21systemd-run08:35:011
1874126993021,6cyclictest1899621-21systemd-run08:35:011
1874126993019,7cyclictest2022906-21sh11:48:081
1874125993024,4cyclictest2024386-21sh11:50:020
1874125993024,4cyclictest1882146-21fschecks_time07:37:530
1874125993024,4cyclictest1881612-21munin-run07:35:010
1874125993023,3cyclictest0-21swapper/010:32:350
187412599302,15cyclictest2015268-21ntp_states11:38:100
1874125993012,15cyclictest12550irq/33-eth011:08:000
1874125993012,15cyclictest12550irq/33-eth011:08:000
1874127992921,5cyclictest1883647-21fschecks_count07:42:542
1874127992919,6cyclictest0-21swapper/207:18:062
1874126992924,2cyclictest0-21swapper/108:05:011
1874126992923,3cyclictest0-21swapper/109:58:001
1874126992922,4cyclictest1886048-21sed07:48:141
1874126992916,10cyclictest1947046-21ssh10:03:041
187412599293,7cyclictest437-21mta-sts-daemon10:42:150
1874125992923,4cyclictest1899596-21cron08:35:010
1874125992923,4cyclictest1899596-21cron08:35:010
1874127992820,5cyclictest1879986-21unixbench-2d07:28:142
187412799281,21cyclictest1880997-21sshd07:33:002
1874126992822,3cyclictest0-21swapper/107:40:011
1874126992820,5cyclictest1881503-21unixbench_multi07:33:121
187412599283,6cyclictest1919312-21ntpq09:23:030
187412599283,21cyclictest1940213-21ssh09:53:060
1874125992823,3cyclictest957-21mta-sts-daemon07:25:010
1874125992822,3cyclictest0-21swapper/010:48:040
1874125992819,6cyclictest955-21mta-sts-daemon08:27:560
1874125992811,13cyclictest1889088-21latency_hist08:00:010
1874127992721,3cyclictest0-21swapper/207:13:012
1874127992720,4cyclictest956-21mta-sts-daemon08:07:072
1874127992719,5cyclictest1903989-21unixbench_multi08:48:132
1874126992718,6cyclictest1906126-21fschecks_time08:57:571
1874125992723,2cyclictest0-21swapper/008:40:010
1874125992723,2cyclictest0-21swapper/008:40:010
1874125992720,5cyclictest1904473-21aten_r2power_po08:52:530
1874125992718,7cyclictest1881834-21kworker/u8:12-xprtiod07:43:130
1874127992620,3cyclictest0-21swapper/208:25:012
1874126992618,5cyclictest1896561-21unixbench_singl08:23:121
187412599262,17cyclictest1892957-21sshd08:13:000
187412599262,17cyclictest1892957-21sshd08:13:000
1874125992621,2cyclictest0-21swapper/008:23:000
1874125992621,2cyclictest0-21swapper/008:20:010
1874125992620,3cyclictest0-21swapper/009:10:020
1874125992620,3cyclictest0-21swapper/009:10:020
1874125992617,6cyclictest12450irq/32-eth008:58:070
1874127992519,3cyclictest0-21swapper/208:03:052
1874126992519,3cyclictest0-21swapper/107:23:011
187412699251,17cyclictest1056785-21cupsd08:08:051
1874125992520,3cyclictest1183579-21kworker/u8:3-xprtiod07:10:020
1874125992519,3cyclictest0-21swapper/007:18:160
1874125992518,3cyclictest1891527-21ntpq08:08:040
187412599251,5cyclictest171rcu_preempt07:53:090
1874127992416,5cyclictest1056789-21cups-browsed07:45:362
187412699243,15cyclictest437-21mta-sts-daemon07:43:031
1874125992417,4cyclictest1889653-21fschecks_time08:02:560
187412799232,16cyclictest1899058-21ntpq08:33:052
187412799232,16cyclictest1899058-21ntpq08:33:052
1874126992318,2cyclictest0-21swapper/107:10:021
187412899222,10cyclictest0-21swapper/309:33:033
187412899221,13cyclictest0-21swapper/312:08:063
1874126992214,5cyclictest461-21thd08:28:051
187412899201,13cyclictest0-21swapper/312:28:093
187412899192,10cyclictest0-21swapper/311:43:013
187412899191,8cyclictest0-21swapper/308:08:043
187412899191,11cyclictest0-21swapper/312:18:553
187412899191,10cyclictest0-21swapper/308:13:003
187412899191,10cyclictest0-21swapper/308:13:003
187412899181,10cyclictest0-21swapper/309:28:143
187412899172,8cyclictest0-21swapper/309:01:073
187412899171,9cyclictest0-21swapper/311:03:043
187412899171,8cyclictest0-21swapper/309:13:103
187412899171,7cyclictest0-21swapper/312:03:033
187412899171,7cyclictest0-21swapper/310:38:053
187412899171,6cyclictest0-21swapper/309:18:003
187412899171,6cyclictest0-21swapper/307:38:003
187412899171,10cyclictest0-21swapper/310:33:063
187412799177,7cyclictest422-21dbus-daemon07:10:012
187412899163,7cyclictest0-21swapper/311:38:113
187412899162,8cyclictest0-21swapper/312:13:013
187412899162,8cyclictest0-21swapper/312:13:013
187412899162,6cyclictest0-21swapper/308:38:003
187412899162,6cyclictest0-21swapper/308:38:003
187412899162,6cyclictest0-21swapper/307:18:083
187412899161,9cyclictest0-21swapper/310:23:083
187412899161,9cyclictest0-21swapper/310:23:083
187412899161,9cyclictest0-21swapper/309:48:043
187412899161,8cyclictest0-21swapper/312:23:033
187412899161,8cyclictest0-21swapper/311:58:003
187412899161,8cyclictest0-21swapper/311:58:003
187412899161,8cyclictest0-21swapper/311:53:033
187412899161,8cyclictest0-21swapper/310:18:003
187412899161,6cyclictest0-21swapper/309:53:033
187412899152,6cyclictest0-21swapper/307:53:043
187412899151,8cyclictest0-21swapper/312:38:103
187412899151,8cyclictest0-21swapper/312:38:103
187412899151,7cyclictest0-21swapper/310:08:043
187412899151,7cyclictest0-21swapper/307:48:053
187412899151,10cyclictest0-21swapper/311:08:013
187412899151,10cyclictest0-21swapper/311:08:013
187412899143,6cyclictest0-21swapper/311:23:043
187412899142,7cyclictest0-21swapper/312:33:083
187412899142,6cyclictest0-21swapper/311:13:093
187412899141,8cyclictest0-21swapper/311:18:073
187412899141,7cyclictest0-21swapper/311:33:023
187412899141,7cyclictest0-21swapper/309:43:083
187412899141,7cyclictest0-21swapper/309:40:023
187412899141,7cyclictest0-21swapper/308:03:073
187412899141,6cyclictest0-21swapper/307:42:593
187412899141,6cyclictest0-21swapper/307:27:583
187412899141,5cyclictest0-21swapper/308:53:013
187412899141,5cyclictest0-21swapper/307:33:003
187412899132,6cyclictest0-21swapper/310:15:013
187412899131,7cyclictest0-21swapper/309:58:003
187412899131,6cyclictest0-21swapper/307:57:593
187412899131,5cyclictest0-21swapper/309:08:023
187412899131,5cyclictest0-21swapper/309:08:023
187412899122,6cyclictest0-21swapper/310:42:533
187412899122,6cyclictest0-21swapper/308:18:053
187412899122,5cyclictest0-21swapper/309:22:563
187412899121,7cyclictest0-21swapper/311:48:113
187412899121,7cyclictest0-21swapper/310:53:053
187412899121,7cyclictest0-21swapper/310:03:033
187412899121,7cyclictest0-21swapper/308:23:003
187412899121,6cyclictest0-21swapper/310:45:373
187412899121,6cyclictest0-21swapper/307:23:013
187412899121,4cyclictest0-21swapper/308:28:033
187412899112,7cyclictest0-21swapper/310:58:003
187412899112,7cyclictest0-21swapper/310:58:003
187412899111,7cyclictest0-21swapper/310:27:583
187412899111,6cyclictest0-21swapper/311:28:053
187412899111,6cyclictest0-21swapper/308:58:053
187412899111,6cyclictest0-21swapper/308:48:023
187412899111,6cyclictest0-21swapper/307:13:123
187412899111,5cyclictest0-21swapper/308:42:553
187412899101,5cyclictest0-21swapper/308:33:003
187412899101,5cyclictest0-21swapper/308:33:003
18741289971,3cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional