You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 16:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Tue Sep 01, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
107129211954,60sleep30-21swapper/33
10727729363,12sleep10-21swapper/11
10704128555,17sleep00-21swapper/00
10731428464,10sleep20-21swapper/22
2753992220,4sleep10-21swapper/11
10747199212,14cyclictest570339-21chrt3
10747199211,15cyclictest0-21swapper/33
4247652201,9sleep2424499-21sendmail-msp2
10747199201,2cyclictest0-21swapper/33
10747199201,15cyclictest569901-21taskset3
10747199201,14cyclictest0-21swapper/33
5719712190,7sleep0151ktimers/00
28550421913,2sleep30-21swapper/33
10747199191,15cyclictest0-21swapper/33
10747199191,15cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
10747199191,14cyclictest0-21swapper/33
4472962181,5sleep0235617-21diskmemload0
4472962181,5sleep0235617-21diskmemload0
2948322181,2sleep10-21swapper/11
2420822181,3sleep00-21swapper/00
107471991811,4cyclictest525435-21chrt3
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,14cyclictest0-21swapper/33
10747199181,13cyclictest0-21swapper/33
10747199181,13cyclictest0-21swapper/33
10747199181,13cyclictest0-21swapper/33
10747199181,13cyclictest0-21swapper/33
10747199181,12cyclictest0-21swapper/33
10747199180,14cyclictest0-21swapper/33
10747199180,14cyclictest0-21swapper/33
5158472171,5sleep0235617-21diskmemload0
4990262171,3sleep20-21swapper/22
4749322170,4sleep10-21swapper/11
46697421710,2sleep2235617-21diskmemload2
3444722171,4sleep2344486-21ssh2
3369462171,2sleep00-21swapper/00
3131872171,2sleep00-21swapper/00
10747199172,10cyclictest505174-21taskset3
107471991715,1cyclictest0-21swapper/33
107471991711,3cyclictest0-21swapper/33
10747199171,13cyclictest0-21swapper/33
10747199171,13cyclictest0-21swapper/33
10747199171,13cyclictest0-21swapper/33
10747199170,14cyclictest0-21swapper/33
5164072161,4sleep1235617-21diskmemload1
4927852161,4sleep0235617-21diskmemload0
4312572160,3sleep00-21swapper/00
4135472160,1sleep00-21swapper/00
2852422160,1sleep20-21swapper/22
10747199166,7cyclictest66950irq/132-enp1s0-TxRx-33
10747199161,3cyclictest0-21swapper/33
10747199161,13cyclictest0-21swapper/33
10747199160,14cyclictest0-21swapper/33
10747199160,14cyclictest0-21swapper/33
10747199160,13cyclictest0-21swapper/33
10747199160,13cyclictest0-21swapper/33
10747199160,13cyclictest0-21swapper/33
10747199160,13cyclictest0-21swapper/33
10747199160,13cyclictest0-21swapper/33
10747199160,13cyclictest0-21swapper/33
4762882150,2sleep00-21swapper/00
4645692151,4sleep1235617-21diskmemload1
3546912151,2sleep20-21swapper/22
2676072151,3sleep0235617-21diskmemload0
2145432150,1sleep10-21swapper/11
107471991512,2cyclictest0-21swapper/33
107471991512,1cyclictest0-21swapper/33
10747199151,12cyclictest0-21swapper/33
107471991510,3cyclictest0-21swapper/33
10747199150,12cyclictest0-21swapper/33
10747199150,12cyclictest0-21swapper/33
10747199150,12cyclictest0-21swapper/33
10747199150,12cyclictest0-21swapper/33
5602772140,2sleep20-21swapper/22
5019922141,7sleep1502056-21ls1
4943672140,2sleep0221rcuc/00
4420962141,7sleep2442111-21cpu2
4420962141,7sleep2442111-21cpu2
4409662141,8sleep0235617-21diskmemload0
4240462141,8sleep1424058-21ssh1
4054842141,3sleep0235617-21diskmemload0
3549052141,4sleep0354916-21ssh0
3206052140,3sleep10-21swapper/11
2667842141,4sleep1266798-21ssh1
2256292141,9sleep04192556-21in:imuxsock0
10747199148,3cyclictest1932602sleep33
107471991411,2cyclictest0-21swapper/33
10747199140,2cyclictest0-21swapper/33
10747199140,12cyclictest0-21swapper/33
5945312131,7sleep1235617-21diskmemload1
5774432131,6sleep10-21swapper/11
4863202131,8sleep1487899-21cut1
4651562131,2sleep0465167-21ssh0
3793852131,8sleep1379396-21sh1
3733172130,7sleep2373310-21apt-get2
3218712131,6sleep2321497-21sendmail-msp2
3125522131,8sleep2312566-21ssh2
2874192131,7sleep2287144-21sendmail-msp2
1362552131,7sleep1136464-21hddtemp_smartct1
10747199133,7cyclictest0-21swapper/33
107471991311,1cyclictest0-21swapper/33
10747199130,2cyclictest0-21swapper/33
10747199130,2cyclictest0-21swapper/33
5874132120,2sleep00-21swapper/00
5845702121,6sleep1235617-21diskmemload1
5352542121,7sleep0535257-21ssh0
5349672121,6sleep1534978-21ssh1
5035612124,4sleep2505169-21kthreadcore2
4764112121,5sleep1476415-21apt-get1
4744872120,8sleep2474499-21sh2
4391592122,3sleep2439173-21ssh2
4218752120,8sleep2421887-21sh2
4130142120,1sleep10-21swapper/11
3819272121,7sleep0382103-21cat0
3561142121,5sleep2355808-21sendmail-msp2
3284632121,7sleep1328474-21ssh1
2850112121,5sleep00-21swapper/00
2513402121,7sleep0251351-21ssh0
2090062123,5sleep139542-21systemd-journal1
5854002111,4sleep0235617-21diskmemload0
5528182110,5sleep20-21swapper/22
5422502110,2sleep1542261-21ssh1
5073662111,3sleep00-21swapper/00
4999502111,4sleep10-21swapper/11
4759772111,6sleep01-21systemd0
4562842110,1sleep10-21swapper/11
4512232112,5sleep0451551-21rm0
4151242111,6sleep2415139-21ssh2
3965702111,4sleep2235617-21diskmemload2
3867912110,1sleep10-21swapper/11
3716162111,4sleep0235617-21diskmemload0
3617882110,7sleep0235617-21diskmemload0
3561012111,3sleep10-21swapper/11
3408392111,6sleep1342527-21grep1
3208112112,4sleep2235617-21diskmemload2
3109742111,3sleep00-21swapper/00
2603912111,4sleep1235617-21diskmemload1
2598752110,2sleep00-21swapper/00
2506592111,2sleep1107462-21cyclictest1
2360272111,4sleep1236022-21apt-get1
2136812111,6sleep2213765-21head2
1728042111,5sleep1172823-21apt-get1
1201732111,2sleep20-21swapper/22
5785052101,4sleep239542-21systemd-journal2
5693292101,4sleep0235617-21diskmemload0
5686172101,4sleep2235617-21diskmemload2
5587192101,4sleep1235617-21diskmemload1
5519692101,4sleep0235617-21diskmemload0
5516762101,4sleep1235617-21diskmemload1
5415902101,4sleep00-21swapper/00
5275562101,3sleep2527247-21sendmail-msp2
5256852101,4sleep0235617-21diskmemload0
5244572102,4sleep1235617-21diskmemload1
5166542101,3sleep20-21swapper/22
4849232101,3sleep20-21swapper/22
4389442101,2sleep10-21swapper/11
4264152101,5sleep1424499-21sendmail-msp1
4224992100,1sleep00-21swapper/00
4035882101,2sleep10-21swapper/11
3991362101,2sleep20-21swapper/22
3973422101,4sleep0235617-21diskmemload0
3877402101,2sleep20-21swapper/22
3784292101,3sleep00-21swapper/00
3712972101,3sleep2235617-21diskmemload2
3708862101,3sleep10-21swapper/11
3526342101,5sleep1352648-21ssh1
3443542101,4sleep0235617-21diskmemload0
3297832102,4sleep0329794-21sh0
3053182101,5sleep1306100-21cut1
3028762100,4sleep1321rcuc/11
3014972101,4sleep2301508-21ssh2
2952092101,4sleep0235617-21diskmemload0
2843242101,2sleep10-21swapper/11
2761992101,4sleep2235617-21diskmemload2
2703112102,4sleep0270329-21apt-get0
2690242101,4sleep2235617-21diskmemload2
2553132101,5sleep2256901-21kthreadcore2
2444772101,4sleep2244488-21ssh2
2372432102,4sleep2238840-21grep2
2306622102,4sleep0230658-21apt-get0
2253372100,6sleep2225116-21sendmail-msp2
2201722101,4sleep2220375-21diskstats2
2148742101,3sleep1214893-21apt-get1
2085512101,5sleep2204095-21sendmail-msp2
1993362101,4sleep1199545-21hddtemp_smartct1
1885962101,4sleep0188850-21df0
1882132100,1sleep00-21swapper/00
1874782101,4sleep2183050-21sendmail-msp2
1832682100,2sleep10-21swapper/11
1413222101,4sleep1141535-21df_inode1
1152042102,4sleep0115328-21perl0
1126572101,5sleep2112681-21latency_hist2
584282290,1sleep20-21swapper/22
565516291,4sleep1565881-21tail1
560646291,2sleep00-21swapper/00
543051290,1sleep20-21swapper/22
523231290,2sleep20-21swapper/22
482703291,3sleep2235617-21diskmemload2
447751291,4sleep1447763-21rm1
447751291,4sleep1447763-21rm1
390592291,3sleep10-21swapper/11
337656290,3sleep134-21ksoftirqd/11
335852290,5sleep2335865-21ssh2
296120291,3sleep0235617-21diskmemload0
204892291,5sleep0205676-21kthreadcore0
181206291,4sleep1181948-21missed_timers1
179525291,4sleep0181204-21tr0
178039291,3sleep2178058-21apt-get2
162839293,3sleep1163528-21kthreadcore1
147945291,4sleep2149627-21grep2
147149291,4sleep1147943-21cut1
131741291,4sleep0133393-21grep0
129353290,2sleep2171rcu_preempt2
125208291,4sleep0125233-21latency_hist0
124577291,5sleep1124824-21/usr/sbin/munin1
107631291,3sleep1107650-21apt-get1
594393281,3sleep2594407-21ssh2
454919280,2sleep20-21swapper/22
235601281,4sleep1235608-21unix_chkpwd1
200899281,3sleep0202632-21kthreadcore0
199044281,2sleep20-21swapper/22
167569280,3sleep2167588-21apt-get2
162277281,2sleep00-21swapper/00
159285281,3sleep10-21swapper/11
157037281,3sleep00-21swapper/00
156747281,4sleep2157003-21latency_hist2
156487280,1sleep00-21swapper/00
141536281,3sleep2141659-21perl2
135658281,4sleep2135811-21wc2
130173280,1sleep00-21swapper/00
230926271,2sleep20-21swapper/22
225341271,2sleep1225360-21apt-get1
225046270,1sleep10-21swapper/11
220086270,2sleep0220105-21apt-get0
198445270,1sleep00-21swapper/00
193798270,2sleep20-21swapper/22
193179270,1sleep10-21swapper/11
162444270,4sleep20-21swapper/22
151795270,2sleep20-21swapper/22
141801271,3sleep00-21swapper/00
140675270,1sleep00-21swapper/00
135963271,1sleep2135982-21apt-get2
115485271,2sleep20-21swapper/22
112465270,1sleep00-21swapper/00
219649260,1sleep00-21swapper/00
215437260,2sleep20-21swapper/22
173372261,2sleep20-21swapper/22
172207260,1sleep10-21swapper/11
169396261,2sleep00-21swapper/00
151178260,4sleep00-21swapper/00
134891260,2sleep1135035-21wc1
129676261,2sleep10-21swapper/11
114917260,1sleep10-21swapper/11
209930250,1sleep00-21swapper/00
194093250,1sleep10-21swapper/11
192929250,1sleep20-21swapper/22
177706250,1sleep00-21swapper/00
152085240,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional