You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 23:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Sun Nov 30, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19392999910891,1084cyclictest2206527-21sleep10:20:003
1939299993340,330cyclictest2262794-21turbostat3
23114122200,1sleep02311509-21latency_hist0
21963242201,8sleep02196551-21users0
23505422180,8sleep014-21ksoftirqd/00
1939299991815,1cyclictest0-21swapper/33
1939299991714,1cyclictest0-21swapper/33
193929999167,6cyclictest2174372-21turbostat3
1939299991614,1cyclictest0-21swapper/309:15:013
1939299991613,1cyclictest0-21swapper/307:40:013
193929999161,12cyclictest2335746-21turbostat3
193929999161,11cyclictest2080971-21taskset09:00:123
193929999155,8cyclictest2049036-21turbostat08:39:543
1939299991513,1cyclictest0-21swapper/33
1939299991513,1cyclictest0-21swapper/33
1939299991513,1cyclictest0-21swapper/33
1939299991513,0cyclictest0-21swapper/33
1939299991512,1cyclictest0-21swapper/33
1939299991512,1cyclictest0-21swapper/33
1939299991512,1cyclictest0-21swapper/33
1939299991512,1cyclictest0-21swapper/33
1939299991512,1cyclictest0-21swapper/33
193929999151,11cyclictest1992268-21taskset3
193929999151,10cyclictest2220453-21taskset3
190898150,2rtkit-daemon171rcu_preempt0
24313812144,3sleep12093479-21diskmemload12:36:271
20369242141,4sleep12037915-21irqstats1
193929999141,9cyclictest19545452sleep33
1939299991412,1cyclictest0-21swapper/33
1939299991412,1cyclictest0-21swapper/33
1939299991412,1cyclictest0-21swapper/33
1939299991411,1cyclictest0-21swapper/33
1939299991411,1cyclictest0-21swapper/33
193929999141,10cyclictest2378493-21taskset3
193929999141,10cyclictest2367041-21taskset3
193929999141,10cyclictest2244937-21chrt10:40:433
193929999141,10cyclictest2231037-21taskset3
193929999141,10cyclictest2188554-21taskset3
193929999141,10cyclictest2181813-21taskset10:03:323
193929999141,10cyclictest2150415-21taskset3
193929999141,10cyclictest2148627-21taskset3
193929999141,10cyclictest2120196-21taskset09:25:173
193929999141,10cyclictest2117259-21taskset3
193929999141,10cyclictest2064335-21taskset3
193929999141,10cyclictest2016017-21taskset3
193929999141,10cyclictest1965488-21taskset07:25:213
193929999135,6cyclictest21907562sleep33
193929999134,7cyclictest2035891-21taskset3
193929999131,9cyclictest2401204-21taskset3
193929999131,9cyclictest2325988-21taskset3
193929999131,9cyclictest2206793-21taskset10:20:063
193929999131,9cyclictest2133850-21turbostat3
193929999131,9cyclictest2029667-21turbostat3
193929999131,9cyclictest1986706-21taskset3
1939299991312,0cyclictest0-21swapper/33
1939299991312,0cyclictest0-21swapper/308:42:473
1939299991311,1cyclictest0-21swapper/33
1939299991311,1cyclictest0-21swapper/33
193929999131,10cyclictest2318383-21taskset3
193929999131,10cyclictest2310641-21chrt3
193929999131,10cyclictest2080722-21taskset3
1939299991310,1cyclictest0-21swapper/33
1939299991310,1cyclictest0-21swapper/33
1939299991310,1cyclictest0-21swapper/33
193929999130,11cyclictest2408965-21taskset3
193929999130,10cyclictest2335993-21taskset11:40:023
193929999130,10cyclictest2087177-21taskset3
190898130,9rtkit-daemon2318992-21ssh0
190898130,7rtkit-daemon2155800-21python30
22229392121,2sleep00-21swapper/010:30:150
193929999121,9cyclictest2398871-21chrt3
193929999121,8cyclictest2222890-21taskset3
193929999121,8cyclictest20429992sleep308:30:133
1939299991211,1cyclictest0-21swapper/308:11:003
193929999120,9cyclictest2004803-21taskset3
193929999120,10cyclictest2295539-21taskset11:15:013
19332212122,5sleep20-21swapper/207:05:132
190898120,4rtkit-daemon421-21avahi-daemon0
21967632111,7sleep22196771-21ssh2
20621342110,1sleep10-21swapper/11
20562342110,8sleep22056540-21irqcore08:40:182
20444552110,3sleep02045649-21kthreadcore0
190898110,7rtkit-daemon697-21runrttasks11:42:250
190898110,7rtkit-daemon436-21systemd-logind0
190898110,4rtkit-daemon2205169-21ssh1
190898110,4rtkit-daemon1945674-21munin-node0
23170682101,6sleep22311750-21/usr/sbin/munin2
23027982100,7sleep22302802-21ssh11:18:502
22874992101,1sleep20-21swapper/22
22874992101,1sleep20-21swapper/22
21719382101,5sleep02172063-21python30
20781482100,1sleep23309230-21kworker/2:2+mm_percpu_wq08:55:202
19394642101,5sleep01939497-21apt-get0
190898100,6rtkit-daemon671-21ntpd12:30:110
190898100,6rtkit-daemon473-21gmain0
190898100,5rtkit-daemon2414941-21ssh0
190898100,5rtkit-daemon2374770-21ssh0
190898100,5rtkit-daemon2134354-21df_inode09:35:120
190898100,3rtkit-daemon473-21gmain0
190898100,1rtkit-daemon0-21swapper/008:02:200
2409497290,6sleep22410501-21irqstats2
2269404290,6sleep02269412-21ssh0
2262758291,4sleep12262778-21munin-run10:54:591
2230995291,4sleep22231032-21apt-get2
2214973290,2sleep00-21swapper/00
2212092291,5sleep22212156-21ntp_states2
2109281291,6sleep12109283-21ssh09:19:181
2086130291,1sleep0171rcu_preempt0
1966715291,4sleep11966748-21apt-get07:30:121
1952942291,4sleep11953019-21ntp_states07:15:221
19089890,6rtkit-daemon473-21gmain0
19089890,6rtkit-daemon319-21systemd-udevd11:11:380
19089890,6rtkit-daemon319-21systemd-udevd0
19089890,5rtkit-daemon667-21snmpd08:46:590
19089890,5rtkit-daemon436-21systemd-logind09:33:370
19089890,5rtkit-daemon2384899-21ssh0
19089890,4rtkit-daemon2163582-21memory0
19089890,4rtkit-daemon2004535-21cron0
19089890,3rtkit-daemon81364-21packagekitd0
19089890,1rtkit-daemon0-21swapper/110:07:381
19089890,1rtkit-daemon0-21swapper/00
2384592281,4sleep02384642-21munin-run12:10:000
2383957281,4sleep22383965-21ssh2
2366830280,5sleep02366838-21ssh0
2350560281,5sleep22350568-21ssh2
2240511280,6sleep12241931-21cut10:40:171
2214549285,1sleep1321rcuc/110:24:551
2204056281,4sleep02204117-21ntp_states0
2136500280,5sleep12138560-21tr1
1984548281,4sleep21984625-21ntp_states2
1972948281,3sleep21972981-21apt-get2
19089880,5rtkit-daemon81364-21packagekitd0
19089880,5rtkit-daemon667-21snmpd08:42:460
19089880,5rtkit-daemon436-21systemd-logind1
19089880,5rtkit-daemon1928-21sendmail-mta1
19089880,5rtkit-daemon1928-21sendmail-mta07:33:550
19089880,4rtkit-daemon81364-21packagekitd0
19089880,4rtkit-daemon667-21snmpd11:50:331
19089880,4rtkit-daemon438-21wpa_supplicant07:22:300
19089880,4rtkit-daemon2109918-21ssh2
19089880,4rtkit-daemon2099642-21kworker/u16:2+rpciod2
19089880,4rtkit-daemon1945674-21munin-node1
19089880,4rtkit-daemon1945674-21munin-node0
19089880,4rtkit-daemon1945674-21munin-node0
19089880,4rtkit-daemon1945674-21munin-node0
19089880,4rtkit-daemon0-21swapper/00
19089880,3rtkit-daemon438-21wpa_supplicant1
19089880,3rtkit-daemon2335025-21ssh0
19089880,1rtkit-daemon0-21swapper/11
19089880,1rtkit-daemon0-21swapper/00
19089880,1rtkit-daemon0-21swapper/00
19089880,1rtkit-daemon0-21swapper/00
2432410270,4sleep22432417-21sh2
2423553271,4sleep22423554-21sh12:31:542
2405447270,1sleep00-21swapper/00
2399118271,4sleep22399126-21ssh12:16:152
2390080271,3sleep12390202-21/usr/sbin/munin1
2377162270,5sleep12377552-21irqcore12:05:161
2358080270,4sleep22358087-21sh2
2303313270,5sleep12303536-21latency_hist11:20:001
2255330270,4sleep22255383-21tune2fs2
2248585271,3sleep22249892-21cat2
2228976270,1sleep20-21swapper/22
2166644271,4sleep22166685-21ssh2
2165056270,1sleep20-21swapper/209:52:102
2150370271,2sleep22150405-21apt-get09:45:122
2109921270,3sleep12109958-21apt-get09:20:101
2093853270,5sleep12093952-21cpuspeed_turbos1
2068602270,4sleep22068756-21/usr/sbin/munin08:50:142
2031269270,4sleep22032429-21cat2
2024114270,5sleep02025094-21kernelversion0
1998932270,5sleep11999071-21irqcore07:55:171
1985724271,3sleep21985757-21apt-get2
1940918270,4sleep11942083-21tr1
19089870,4rtkit-daemon81364-21packagekitd09:15:170
19089870,4rtkit-daemon1945674-21munin-node2
19089870,4rtkit-daemon1945674-21munin-node07:36:480
19089870,3rtkit-daemon671-21ntpd0
19089870,3rtkit-daemon421-21avahi-daemon11:15:180
19089870,3rtkit-daemon2101548-21munin-run0
19089870,3rtkit-daemon2074446-21kworker/0:2+kacpid0
19089870,3rtkit-daemon1979828-21hddtemp_smartct0
19089870,3rtkit-daemon1945674-21munin-node1
19089870,3rtkit-daemon1945674-21munin-node0
19089870,3rtkit-daemon1928-21sendmail-mta0
19089870,1rtkit-daemon0-21swapper/00
2419090262,2sleep10-21swapper/11
2414477260,3sleep12414696-21grep1
2401468260,3sleep12402235-21irqrtprio1
2374878260,1sleep10-21swapper/11
2368073260,3sleep22368397-21latency_hist12:00:012
2342470260,1sleep20-21swapper/211:41:572
2334711260,1sleep20-21swapper/211:37:442
2312822260,4sleep12313970-21tr1
2309138260,3sleep12309396-21unixbench_multi1
2285964260,1sleep20-21swapper/22
2271980261,3sleep10-21swapper/111:00:171
2263091260,3sleep12263124-21apt-get1
2239028261,2sleep00-21swapper/00
2236895260,3sleep02236903-21ssh0
2220344261,2sleep20-21swapper/22
2212705260,3sleep02212712-21sh10:20:330
2202178260,1sleep20-21swapper/22
2188770261,3sleep22188778-21ssh10:05:572
2187836260,4sleep00-21swapper/00
2156676260,1sleep10-21swapper/11
2148170260,1sleep10-21swapper/11
2143221260,4sleep22144214-21kthreadcore2
2140909260,4sleep22140916-21sh2
2127532262,2sleep12128749-21kthreadcore1
2056033260,3sleep12056233-21hddtemp_smartct1
2043184260,4sleep12043000-21/usr/sbin/munin1
2028807260,3sleep12028892-21mailstats08:15:241
2016266261,2sleep10-21swapper/11
1998153260,1sleep10-21swapper/11
1991766260,3sleep21991777-21apt-get2
1971844260,4sleep21971884-21proc_pri07:30:212
1949300260,3sleep21950706-21kthreadcore2
1939044260,1sleep10-21swapper/11
1933204260,3sleep30-21swapper/33
19089860,3rtkit-daemon0-21swapper/109:28:101
2402236250,3sleep22403390-21cut2
2399811250,1sleep10-21swapper/11
2386426250,3sleep22387535-21cut2
2374611250,2sleep20-21swapper/22
2365632250,2sleep10-21swapper/11
2338317250,2sleep10-21swapper/11
2330782251,2sleep10-21swapper/111:35:201
2319805251,2sleep20-21swapper/211:30:042
2303682250,2sleep22303717-21apt-get2
2289455250,3sleep12291443-21cut1
2289455250,3sleep12291443-21cut1
2272859251,2sleep20-21swapper/22
2270321251,2sleep20-21swapper/22
2247095251,2sleep12247132-21apt-get10:45:151
2239227251,2sleep20-21swapper/210:40:152
2182271250,1sleep20-21swapper/22
2115855251,2sleep00-21swapper/00
2102818250,3sleep22103695-21grep2
2087918250,3sleep20-21swapper/22
2081187250,2sleep10-21swapper/11
2080934251,1sleep20-21swapper/209:00:132
2069909251,2sleep12071939-21grep1
2063133250,3sleep22064333-21tr2
2055411250,1sleep10-21swapper/11
2017540250,3sleep12017568-21cut1
2014077250,3sleep22016015-21latency2
2010627250,1sleep20-21swapper/22
2004806250,1sleep10-21swapper/108:00:091
1998549250,2sleep20-21swapper/207:55:132
1991489250,1sleep10-21swapper/107:46:261
1966315250,1sleep20-21swapper/207:28:132
1966217250,1sleep10-21swapper/107:26:561
1942084250,2sleep20-21swapper/22
2357428240,1sleep00-21swapper/011:50:210
2326067240,2sleep10-21swapper/11
2286178240,2sleep12286186-21ssh1
2164759240,2sleep12164767-21ssh1
2131853240,1sleep20-21swapper/22
2049537240,1sleep20-21swapper/22
2049019240,2sleep20-21swapper/22
2037916240,2sleep20-21swapper/22
2029923240,2sleep10-21swapper/11
1979728240,2sleep11979890-21if_err_eno11
1960026240,3sleep20-21swapper/22
1954188240,1sleep10-21swapper/11
2166599230,1sleep10-21swapper/11
2093127230,1sleep10-21swapper/11
2074746230,1sleep10-21swapper/11
2029377230,1sleep20-21swapper/22
2023077230,1sleep20-21swapper/22
1972939230,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional