You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 20:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri Mar 27, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16781162156114,38sleep30-21swapper/33
167789428357,17sleep00-21swapper/00
167785028063,9sleep10-21swapper/11
167779627950,11sleep20-21swapper/22
1678243992517,4cyclictest2143539-21taskset3
167824399221,18cyclictest0-21swapper/33
20121562211,16sleep22012178-21latency_hist2
1678243992117,2cyclictest0-21swapper/33
167824399211,16cyclictest0-21swapper/33
167824399211,15cyclictest66950irq/132-enp1s0-TxRx-33
167824399211,15cyclictest0-21swapper/33
167824399209,8cyclictest0-21swapper/33
167824399201,8cyclictest0-21swapper/33
1678243992016,3cyclictest0-21swapper/33
167824399201,15cyclictest66950irq/132-enp1s0-TxRx-33
167824399201,15cyclictest0-21swapper/33
1678243992010,7cyclictest1843345-21chrt3
167824399192,12cyclictest1893590-21taskset3
1678243991914,3cyclictest0-21swapper/33
1678243991912,5cyclictest0-21swapper/33
167824399191,15cyclictest0-21swapper/33
167824399191,15cyclictest0-21swapper/33
167824399191,15cyclictest0-21swapper/33
167824399191,15cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399191,14cyclictest0-21swapper/33
167824399190,7cyclictest0-21swapper/33
20438542181,3sleep00-21swapper/00
167824399187,8cyclictest1683521-21turbostat3
167824399181,14cyclictest1904871-21taskset3
167824399181,14cyclictest0-21swapper/33
167824399181,14cyclictest0-21swapper/33
167824399181,14cyclictest0-21swapper/33
167824399181,14cyclictest0-21swapper/33
167824399181,14cyclictest0-21swapper/33
167824399181,14cyclictest0-21swapper/33
167824399181,13cyclictest0-21swapper/33
167824399181,13cyclictest0-21swapper/33
167824399181,13cyclictest0-21swapper/33
167824399181,13cyclictest0-21swapper/33
167824399181,13cyclictest0-21swapper/33
167824399180,15cyclictest0-21swapper/33
21043482171,5sleep12104404-21ntp_states1
20167072171,5sleep12016763-21ntpq1
18207012172,7sleep21822396-21cut2
167824399172,13cyclictest0-21swapper/33
1678243991714,1cyclictest0-21swapper/33
167824399171,13cyclictest0-21swapper/33
167824399171,13cyclictest0-21swapper/33
167824399171,13cyclictest0-21swapper/33
167824399171,13cyclictest0-21swapper/33
167824399170,14cyclictest0-21swapper/33
167824399170,14cyclictest0-21swapper/33
167824399170,14cyclictest0-21swapper/33
167824399170,13cyclictest0-21swapper/33
21765762161,4sleep11809691-21diskmemload1
20795572160,1sleep10-21swapper/11
20555192161,2sleep00-21swapper/00
20445232161,2sleep12044532-21sh1
20429832161,3sleep22043043-21ntpq2
19662852160,1sleep00-21swapper/00
167824399167,6cyclictest1750537-21turbostat3
167824399161,3cyclictest0-21swapper/33
167824399161,3cyclictest0-21swapper/33
167824399161,13cyclictest0-21swapper/33
167824399161,13cyclictest0-21swapper/33
167824399161,12cyclictest0-21swapper/33
167824399161,12cyclictest0-21swapper/33
167824399161,12cyclictest0-21swapper/33
167824399161,11cyclictest0-21swapper/33
1678243991610,4cyclictest0-21swapper/33
167824399160,13cyclictest0-21swapper/33
21771992151,3sleep20-21swapper/22
21668502150,2sleep10-21swapper/11
20343852152,7sleep22034498-21/usr/sbin/munin2
19856502151,9sleep11809691-21diskmemload1
19641892151,6sleep11964245-21gcc1
19333052150,2sleep00-21swapper/00
19117892151,4sleep01911801-21ssh0
19048922151,2sleep10-21swapper/11
18935312155,2sleep2171rcu_preempt2
1678243991512,2cyclictest0-21swapper/33
167824399150,13cyclictest0-21swapper/33
167824399150,12cyclictest0-21swapper/33
21505902141,2sleep10-21swapper/11
21356652145,5sleep22135986-21grep2
21184212141,9sleep12120102-21tr1
20165592140,2sleep00-21swapper/00
19999722140,2sleep10-21swapper/11
19935182140,3sleep11809691-21diskmemload1
19324422140,3sleep20-21swapper/22
18166412147,2sleep00-21swapper/00
17764342140,1sleep10-21swapper/11
167824399140,12cyclictest0-21swapper/33
21672742131,6sleep20-21swapper/22
21429372131,8sleep12142950-21ssh1
21319602131,2sleep10-21swapper/11
21319522131,8sleep02131959-21ssh0
21078342131,6sleep21809691-21diskmemload2
20956512132,6sleep22095789-21run-parts2
20888302130,7sleep00-21swapper/00
20796962131,8sleep02079710-21ssh0
20255292131,8sleep12025595-21ntpq1
19839782131,8sleep01983990-21ssh0
19665732131,7sleep21809691-21diskmemload2
18775822131,7sleep21809691-21diskmemload2
18680632131,3sleep01868077-21ssh0
18440712135,2sleep00-21swapper/00
18330272131,5sleep20-21swapper/22
20561912121,7sleep22056199-21ssh2
20541542121,5sleep11809691-21diskmemload1
19386632120,8sleep21938673-21ssh2
18774662121,7sleep11877469-21ssh1
18163502121,6sleep11809691-21diskmemload1
17565042121,7sleep21756704-21/usr/sbin/munin2
17565042121,7sleep21756704-21/usr/sbin/munin2
17276052121,6sleep01727659-21ntpq0
17180862121,5sleep21718105-21apt-get2
17122082120,7sleep11712143-31find1
16899112121,6sleep21689968-21ntp_states2
16824632121,5sleep21682520-21ntpq2
167824399120,10cyclictest0-21swapper/33
21592842111,4sleep00-21swapper/00
21345222111,7sleep22134537-21ssh2
21154802111,3sleep20-21swapper/22
21042932111,5sleep02104346-21ntpq0
20875702111,4sleep11809691-21diskmemload1
20728422110,6sleep01809691-21diskmemload0
20367872111,7sleep12036800-21ssh1
20195932111,6sleep22019604-21ssh2
20000612110,2sleep00-21swapper/00
19995522111,5sleep21809691-21diskmemload2
19933762111,5sleep20-21swapper/22
19845982111,4sleep20-21swapper/22
19737982110,3sleep20-21swapper/22
19566512111,6sleep11956664-21ssh1
19485492111,4sleep11809691-21diskmemload1
19198612111,6sleep01919921-21pluginstate0
19109292111,5sleep11911019-21ntpq1
18973622111,4sleep11809691-21diskmemload1
18892712112,4sleep01889290-21apt-get0
18690012110,2sleep10-21swapper/11
18610812111,3sleep10-21swapper/11
18430312111,3sleep20-21swapper/22
18261762111,5sleep00-21swapper/00
18253822111,6sleep11825395-21sh1
17833282111,4sleep21783347-21apt-get2
17529812111,6sleep21754694-21cat2
17331832111,5sleep11733232-21ntpq1
17239032111,5sleep21724019-21irqstats2
16828172111,6sleep01682944-21sed0
21695892101,3sleep00-21swapper/00
21583842101,3sleep11809691-21diskmemload1
21527982101,4sleep21809691-21diskmemload2
21488782101,3sleep20-21swapper/22
21479872101,5sleep02148043-21ntpq0
21396372101,4sleep02139783-21timerwakeupswit0
21244322101,3sleep21809691-21diskmemload2
21217072101,4sleep02121762-21ntpq0
21141892100,3sleep00-21swapper/00
20971922101,3sleep00-21swapper/00
20867822101,5sleep22086841-21ntpq2
20650112101,4sleep21809691-21diskmemload2
20625552101,3sleep00-21swapper/00
20537632101,3sleep20-21swapper/22
20378372100,2sleep00-21swapper/00
20249372101,4sleep22025401-21strings2
20092852101,3sleep11809691-21diskmemload1
19928982100,2sleep01992912-21ssh0
19744772100,6sleep01974493-21sshd0
19728632101,4sleep11972923-21ssh1
19510412101,3sleep00-21swapper/00
19421332100,7sleep21942213-21sort2
19330372101,4sleep11809691-21diskmemload1
19299872102,4sleep01930001-21ssh0
19246142101,4sleep11924639-21apt-get1
19232702101,4sleep11809691-21diskmemload1
19110202101,4sleep21809691-21diskmemload2
19031292101,4sleep21809691-21diskmemload2
19023092101,5sleep01902373-21ld0
18864652101,4sleep01809691-21diskmemload0
18846952101,5sleep21884751-21ntpq2
18846232101,5sleep11884694-21ssh1
18793292101,2sleep00-21swapper/00
18711482101,3sleep20-21swapper/22
18596712101,3sleep20-21swapper/22
18536702101,4sleep21809691-21diskmemload2
18531962101,3sleep10-21swapper/11
18154502100,1sleep20-21swapper/22
17999222101,5sleep11801568-21kthreadcore1
17993112101,4sleep01799330-21apt-get0
17981212101,5sleep01798196-21awk0
17401052101,4sleep21740315-21hddtemp_smartct2
17291122101,4sleep21729373-21df_inode2
17115722100,6sleep21707201-21/usr/sbin/munin2
17112682101,5sleep01711314-21ntpq0
17009262101,4sleep21701170-21users2
2177065291,2sleep00-21swapper/00
2114408291,4sleep12114419-21ssh1
2091263290,1sleep10-21swapper/11
2073683291,2sleep20-21swapper/22
2070223291,3sleep11809691-21diskmemload1
2061663291,2sleep10-21swapper/11
2021173291,2sleep00-21swapper/00
2009272291,4sleep02009284-21ssh0
1949508291,2sleep00-21swapper/00
1920076291,4sleep21920094-21ssh2
1859007291,3sleep01809691-21diskmemload0
1852072291,2sleep00-21swapper/00
1843823290,1sleep10-21swapper/11
1833668290,1sleep10-21swapper/11
1833312291,4sleep01833314-21ssh0
1779300291,4sleep11781045-21cut1
1765656291,4sleep11765796-21ntp_192.168.1141
1757386291,4sleep01759129-21cut0
1757386291,4sleep01759129-21cut0
1751389291,4sleep11752980-21cut1
1712709291,4sleep01712728-21apt-get0
1956809281,3sleep21956820-21ssh2
1804579280,1sleep10-21swapper/11
1777190280,1sleep20-21swapper/22
1772407281,3sleep01772426-21apt-get0
1759130281,3sleep11760058-21sed1
1759130281,3sleep11760058-21sed1
1729060281,3sleep01728564-21apt-get0
1713487280,1sleep1171rcu_preempt1
1706809280,2sleep20-21swapper/22
1691167281,3sleep01691419-21df0
1690789280,1sleep10-21swapper/11
1804636271,2sleep01804888-21df0
1798845270,1sleep10-21swapper/11
1793961270,2sleep20-21swapper/22
1793688270,1sleep10-21swapper/11
1787679272,2sleep01787808-21cat0
1783004270,1sleep20-21swapper/22
1782748270,1sleep00-21swapper/00
1771881270,1sleep20-21swapper/22
1766456270,1sleep00-21swapper/00
1761735270,2sleep20-21swapper/22
1755777270,1sleep00-21swapper/00
1750186270,1sleep00-21swapper/00
1745426270,1sleep20-21swapper/22
1744576270,1sleep10-21swapper/11
1739229270,1sleep10-21swapper/11
1728479270,1sleep10-21swapper/11
1723014270,2sleep10-21swapper/11
1706691270,1sleep00-21swapper/00
1701847270,1sleep10-21swapper/11
1701567270,1sleep10-21swapper/11
1701216270,1sleep00-21swapper/00
1696018270,1sleep20-21swapper/22
1789201261,2sleep20-21swapper/22
1787206261,2sleep11787344-21netstat1
1750285260,4sleep10-21swapper/11
1740416261,2sleep00-21swapper/00
1739306260,1sleep00-21swapper/00
1734469260,1sleep20-21swapper/22
1717467261,1sleep20-21swapper/22
1690738260,4sleep00-21swapper/00
1801570250,1sleep20-21swapper/22
1793425250,2sleep00-21swapper/00
1767371250,2sleep10-21swapper/11
1682945250,1sleep10-21swapper/11
1804936240,1sleep20-21swapper/22
1766873240,1sleep00-21swapper/00
1718361240,1sleep00-21swapper/00
1690863240,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional