You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 13:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Fri Dec 05, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19657799913240,1321cyclictest23910592sleep33
19657799911720,1168cyclictest20881992sleep33
21976392360,33sleep02068656-21kworker/u16:1+efi_rts_wq0
1965779992118,1cyclictest0-21swapper/33
20110022190,2sleep12011125-21cut1
1965779991917,1cyclictest0-21swapper/33
1965779991916,1cyclictest0-21swapper/33
196577999176,8cyclictest23509592sleep33
1965779991714,1cyclictest0-21swapper/33
1965779991714,1cyclictest0-21swapper/33
196577999166,6cyclictest22358612sleep33
196577999165,8cyclictest2180164-21turbostat3
196577999162,11cyclictest2113385-21taskset3
1965779991614,1cyclictest0-21swapper/33
1965779991613,1cyclictest0-21swapper/33
196577999161,12cyclictest2277271-21turbostat3
190898160,5rtkit-daemon2382434-21ssh0
1965779991513,1cyclictest0-21swapper/33
1965779991513,1cyclictest0-21swapper/33
1965779991513,1cyclictest0-21swapper/33
1965779991513,1cyclictest0-21swapper/33
1965779991513,1cyclictest0-21swapper/33
1965779991513,1cyclictest0-21swapper/33
1965779991513,1cyclictest0-21swapper/33
1965779991513,0cyclictest0-21swapper/33
1965779991512,1cyclictest0-21swapper/33
1965779991512,1cyclictest0-21swapper/33
1965779991512,1cyclictest0-21swapper/33
1965779991512,1cyclictest0-21swapper/33
1965779991512,1cyclictest0-21swapper/33
196577999151,11cyclictest2455784-21chrt3
196577999151,11cyclictest2210029-21taskset3
196577999151,11cyclictest2172056-21taskset3
196577999151,11cyclictest2069100-21taskset3
196577999151,11cyclictest2069100-21taskset3
20879572141,8sleep12087972-21munin-run1
20879572141,8sleep12087972-21munin-run1
20124812140,11sleep02012744-21irqcore0
196577999141,9cyclictest2221271-21taskset3
196577999141,9cyclictest2006181-21taskset3
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,1cyclictest0-21swapper/33
1965779991412,0cyclictest0-21swapper/33
1965779991411,1cyclictest0-21swapper/33
1965779991411,1cyclictest0-21swapper/33
1965779991411,1cyclictest0-21swapper/33
196577999141,10cyclictest2438025-21taskset3
196577999141,10cyclictest2350652-21turbostat3
196577999141,10cyclictest2292642-21taskset3
196577999141,10cyclictest2285366-21turbostat3
196577999141,10cyclictest1992583-21taskset3
196577999141,10cyclictest1980423-21turbostat3
196577999134,7cyclictest21279942sleep33
196577999131,9cyclictest2339953-21taskset3
196577999131,9cyclictest2125856-21taskset3
196577999131,9cyclictest2081916-21taskset3
196577999131,9cyclictest2081916-21taskset3
196577999131,9cyclictest2018396-21taskset3
196577999131,9cyclictest1986334-21chrt3
196577999131,9cyclictest1965929-21taskset3
1965779991311,1cyclictest0-21swapper/33
196577999131,10cyclictest2365568-21taskset3
196577999130,10cyclictest2440226-21taskset3
196577999130,10cyclictest2406184-21taskset3
196577999130,10cyclictest2049203-21taskset3
196577999130,10cyclictest2036023-21chrt3
196577999130,10cyclictest1965813-21taskset3
190898130,4rtkit-daemon2365560-21ssh1
22505202120,1sleep02250541-21/usr/sbin/munin0
1965779991210,1cyclictest0-21swapper/33
190898120,8rtkit-daemon2109197-21cut0
190898120,7rtkit-daemon2196313-21awk2
23643992110,9sleep02364485-21swap0
22931452110,8sleep02293155-21ssh0
190898110,6rtkit-daemon2093393-21ntp_states0
190898110,4rtkit-daemon2389218-21ssh0
24138322100,3sleep12413840-21ssh1
22937582101,4sleep12293790-21apt-get1
21726032100,7sleep02172631-21fschecks_count0
21073432101,5sleep12107498-21diskstats1
190898100,6rtkit-daemon697-21runrttasks1
190898100,6rtkit-daemon2154844-21ssh0
190898100,5rtkit-daemon2398669-21ssh0
190898100,5rtkit-daemon2170872-21ssh0
190898100,5rtkit-daemon2034003-21kthreadcore2
190898100,5rtkit-daemon2017348-21/usr/sbin/munin2
190898100,4rtkit-daemon2252823-21ssh1
190898100,3rtkit-daemon1972151-21munin-node1
2233942291,5sleep12234175-21missed_timers1
2221867291,5sleep02222829-21cut0
2215938290,6sleep22212807-21/usr/sbin/munin2
2209740291,5sleep02209812-21/usr/sbin/munin0
2171285291,6sleep12171293-21ssh1
2161578290,3sleep12161708-21grep1
2075596291,4sleep02075628-21apt-get0
2062800291,4sleep12062833-21apt-get1
2006487292,5sleep02007456-21cut0
1959740291,4sleep21959853-21diskstats2
19089890,6rtkit-daemon421-21avahi-daemon2
19089890,5rtkit-daemon667-21snmpd2
19089890,5rtkit-daemon2455337-21ssh0
19089890,5rtkit-daemon2449885-21cut2
19089890,5rtkit-daemon2219279-21runrttasks1
19089890,5rtkit-daemon2119984-21apt-get0
19089890,5rtkit-daemon0-21swapper/00
19089890,4rtkit-daemon2422148-21ssh0
19089890,4rtkit-daemon2321366-21grep1
19089890,4rtkit-daemon2316653-21ssh1
19089890,4rtkit-daemon2299766-21ssh0
19089890,4rtkit-daemon2291500-21ssh1
19089890,4rtkit-daemon2139841-21smartctl0
19089890,4rtkit-daemon2078626-21cut2
19089890,2rtkit-daemon0-21swapper/11
19089890,1rtkit-daemon0-21swapper/11
2432071280,4sleep12432110-21apt-get1
2423492281,4sleep12423498-21ssh1
2326424281,3sleep12326457-21apt-get1
2279505280,6sleep12281468-21kthreadcore1
2277626281,4sleep02277783-21df_inode0
2244319281,4sleep22244327-21ssh2
2188569281,3sleep02188604-21apt-get0
2187668280,2sleep12187676-21ssh1
2178646281,4sleep12178649-21ssh1
2095984281,4sleep12097887-21kthreadcore1
2088212281,3sleep12088245-21apt-get1
2082899280,5sleep02083842-21kthreadcore0
2082899280,5sleep02083842-21kthreadcore0
2030937281,4sleep02031093-21diskstats0
1993246280,5sleep21993393-21df2
1972028280,6sleep21972615-21cat2
1969848281,4sleep01970949-21memory0
1965932281,3sleep11965965-21apt-get1
1964698280,4sleep11964775-21ntp_states1
19089880,4rtkit-daemon667-21snmpd2
19089880,4rtkit-daemon667-21snmpd1
19089880,4rtkit-daemon667-21snmpd0
19089880,4rtkit-daemon473-21gmain2
19089880,4rtkit-daemon438-21wpa_supplicant2
19089880,4rtkit-daemon421-21avahi-daemon2
19089880,4rtkit-daemon2308510-21ssh2
19089880,4rtkit-daemon2251198-21ssh2
19089880,4rtkit-daemon2037718-21grep2
19089880,4rtkit-daemon1972151-21munin-node1
19089880,4rtkit-daemon1972151-21munin-node0
19089880,4rtkit-daemon1223-21cron0
19089880,3rtkit-daemon81364-21packagekitd0
19089880,3rtkit-daemon438-21wpa_supplicant2
19089880,1rtkit-daemon0-21swapper/22
19089880,1rtkit-daemon0-21swapper/11
19089880,1rtkit-daemon0-21swapper/00
2446469270,4sleep12446477-21ssh1
2438583270,1sleep00-21swapper/00
2430087271,3sleep02430095-21ssh0
2414704270,1sleep20-21swapper/22
2383301271,2sleep10-21swapper/11
2382426270,1sleep20-21swapper/22
2330483270,5sleep02331505-21cat0
2319771270,5sleep02321292-21kthreadcore0
2318693270,5sleep22318820-21irqcore2
2285896270,4sleep22286073-21cut2
2229003270,3sleep20-21swapper/22
2194637270,1sleep10-21swapper/11
2180884270,4sleep02181015-21sed0
2170244272,2sleep22170252-21ssh2
2148601271,3sleep22148739-21ps2
2141189271,3sleep02141466-21sed0
2133211270,4sleep22133294-21ntp_states2
2113318270,4sleep22113384-21sed2
2094107270,5sleep22094108-21sshd2
2026255270,5sleep12028264-21kthreadcore1
2005509271,3sleep02005625-21ls0
1987019271,3sleep01987127-21diskstats0
1959649270,3sleep00-21swapper/00
19089870,4rtkit-daemon667-21snmpd2
19089870,4rtkit-daemon667-21snmpd0
19089870,4rtkit-daemon473-21gmain2
19089870,4rtkit-daemon473-21gmain1
19089870,4rtkit-daemon421-21avahi-daemon2
19089870,4rtkit-daemon1972151-21munin-node2
19089870,4rtkit-daemon1972151-21munin-node2
19089870,4rtkit-daemon1972151-21munin-node2
19089870,3rtkit-daemon671-21ntpd2
19089870,3rtkit-daemon667-21snmpd0
19089870,3rtkit-daemon2445995-21ssh2
19089870,3rtkit-daemon2261843-21ssh1
19089870,3rtkit-daemon2128180-21cpuspeed_turbos0
19089870,3rtkit-daemon1972151-21munin-node2
19089870,3rtkit-daemon1972151-21munin-node1
19089870,1rtkit-daemon0-21swapper/22
19089870,1rtkit-daemon0-21swapper/22
19089870,1rtkit-daemon0-21swapper/22
2440331260,3sleep02440498-21gltestperf0
2399783261,2sleep12399817-21apt-get1
2397688260,1sleep20-21swapper/22
2390286260,1sleep20-21swapper/22
2366547260,1sleep20-21swapper/22
2358021260,1sleep00-21swapper/00
2350167260,1sleep20-21swapper/22
2342826260,1sleep00-21swapper/00
2333751260,1sleep20-21swapper/22
2316226260,3sleep02316234-21ssh0
2281472260,3sleep22282684-21memory2
2275995260,4sleep22276002-21sh2
2259751260,4sleep00-21swapper/00
2243289260,4sleep12243297-21ssh1
2235521260,3sleep02235529-21ssh0
2222831260,3sleep12224857-21cut1
2206219261,2sleep22196371-21kworker/2:0-events2
2162233261,2sleep20-21swapper/22
2154181260,1sleep10-21swapper/11
2140826261,2sleep10-21swapper/11
2133467260,3sleep12133676-21grep1
2126707260,1sleep10-21swapper/11
2118697261,3sleep12118766-21ntpq1
2100743260,2sleep12100775-21apt-get1
2063103261,2sleep00-21swapper/00
2058942260,3sleep02061037-21grep0
2052094260,4sleep00-21swapper/00
2050522261,3sleep10-21swapper/11
2036192260,3sleep12036301-21find1
2020393260,3sleep02022418-21tr0
2004590260,1sleep10-21swapper/11
1993210260,3sleep11993243-21apt-get1
1980778260,3sleep01980874-21entropy0
1966885260,3sleep21967807-21cat2
19089860,3rtkit-daemon671-21ntpd0
2391770250,3sleep10-21swapper/11
2373675251,2sleep2431ktimers/22
2358711250,2sleep20-21swapper/22
2336196250,3sleep22338127-21tr2
2334942250,2sleep00-21swapper/00
2275326250,2sleep02275334-21ssh0
2268845250,2sleep20-21swapper/22
2262915250,2sleep00-21swapper/00
2259203250,3sleep22259211-21ssh2
2242084250,3sleep00-21swapper/00
2142419250,2sleep20-21swapper/22
2140508250,1sleep10-21swapper/11
2140306250,1sleep20-21swapper/22
2120142250,3sleep22120214-21tune2fs2
2119346250,1sleep20-21swapper/22
2102197251,2sleep20-21swapper/22
2100226250,1sleep20-21swapper/22
2087194250,1sleep20-21swapper/22
2087194250,1sleep20-21swapper/22
2075631250,3sleep12075785-21df1
2043976250,2sleep10-21swapper/11
2043214250,1sleep00-21swapper/00
2022419250,2sleep12023332-21missed_timers1
1998936250,1sleep00-21swapper/00
1975826251,2sleep10-21swapper/11
19089850,1rtkit-daemon0-21swapper/11
2407028240,2sleep22407036-21ssh2
2381024240,2sleep12381032-21ssh1
2373854240,2sleep10-21swapper/11
2316182240,2sleep20-21swapper/22
2294826240,2sleep20-21swapper/22
2218905240,3sleep00-21swapper/00
2012745240,2sleep10-21swapper/11
1992756240,3sleep10-21swapper/11
2074272230,1sleep10-21swapper/11
2074272230,1sleep10-21swapper/11
2069257230,1sleep00-21swapper/00
2069257230,1sleep00-21swapper/00
2061038230,1sleep10-21swapper/11
2043583230,1sleep00-21swapper/00
2037240230,1sleep10-21swapper/11
2024589230,1sleep00-21swapper/00
1984412230,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional