You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 12:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Sep 03, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1607379212656,64sleep30-21swapper/33
160741628454,11sleep10-21swapper/11
160729328362,11sleep20-21swapper/22
160759527956,7sleep00-21swapper/00
19288722292,10sleep11928539-21sendmail-msp1
19288722292,10sleep11928539-21sendmail-msp1
1607713992722,2cyclictest0-21swapper/33
160771399241,17cyclictest2024287-21taskset3
160771399225,13cyclictest0-21swapper/33
160771399221,16cyclictest0-21swapper/33
17356832211,10sleep01735910-21/usr/sbin/munin0
160771399219,8cyclictest66950irq/132-enp1s0-TxRx-33
160771399211,6cyclictest1656462-21taskset3
160771399211,16cyclictest0-21swapper/33
19835562200,4sleep10-21swapper/11
19457502201,8sleep11734969-21diskmemload1
160771399209,6cyclictest1878139-21chrt3
1607713992011,5cyclictest66950irq/132-enp1s0-TxRx-33
160771399201,14cyclictest0-21swapper/33
1607713992010,6cyclictest66950irq/132-enp1s0-TxRx-33
20420712191,7sleep02042085-21ssh0
17621082191,13sleep01762165-21fschecks_count0
160771399191,15cyclictest0-21swapper/33
160771399191,15cyclictest0-21swapper/33
160771399191,15cyclictest0-21swapper/33
160771399191,14cyclictest0-21swapper/33
160771399191,14cyclictest0-21swapper/33
160771399191,14cyclictest0-21swapper/33
160771399190,16cyclictest0-21swapper/33
19248222181,3sleep00-21swapper/00
160771399184,2cyclictest0-21swapper/33
160771399181,5cyclictest0-21swapper/33
160771399181,4cyclictest0-21swapper/33
1607713991814,2cyclictest0-21swapper/33
1607713991814,2cyclictest0-21swapper/33
1607713991812,4cyclictest66950irq/132-enp1s0-TxRx-33
1607713991811,4cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,14cyclictest0-21swapper/33
160771399181,13cyclictest0-21swapper/33
160771399181,13cyclictest0-21swapper/33
160771399181,13cyclictest0-21swapper/33
160771399181,13cyclictest0-21swapper/33
20520352171,2sleep10-21swapper/11
18677962171,6sleep21867930-21kthreadcore2
18046552171,2sleep00-21swapper/00
1607713991714,1cyclictest0-21swapper/33
1607713991714,1cyclictest0-21swapper/33
1607713991713,2cyclictest0-21swapper/33
1607713991713,2cyclictest0-21swapper/33
1607713991711,3cyclictest0-21swapper/33
160771399171,13cyclictest0-21swapper/33
160771399171,13cyclictest0-21swapper/33
160771399171,13cyclictest0-21swapper/33
160771399171,13cyclictest0-21swapper/33
160771399171,12cyclictest0-21swapper/33
160771399171,12cyclictest0-21swapper/33
160771399170,15cyclictest0-21swapper/33
160771399170,14cyclictest0-21swapper/33
20902412161,11sleep12090323-21kernelversion1
20772312161,2sleep20-21swapper/22
20418862161,10sleep21734969-21diskmemload2
19977832161,3sleep10-21swapper/11
19810672161,4sleep01734969-21diskmemload0
19639192160,6sleep10-21swapper/11
19633852161,4sleep01734969-21diskmemload0
19172782160,4sleep20-21swapper/22
17497962161,3sleep00-21swapper/00
1607713991612,2cyclictest0-21swapper/33
160771399161,12cyclictest0-21swapper/33
160771399161,11cyclictest0-21swapper/33
160771399161,11cyclictest0-21swapper/33
160771399160,13cyclictest0-21swapper/33
160771399160,13cyclictest0-21swapper/33
160771399160,13cyclictest0-21swapper/33
160771399160,12cyclictest0-21swapper/33
160771399160,12cyclictest0-21swapper/33
19722242151,8sleep21734969-21diskmemload2
19517462151,2sleep20-21swapper/22
18908682151,2sleep10-21swapper/11
18906732150,5sleep21734969-21diskmemload2
17930752151,4sleep01793090-21ssh0
17243232150,2sleep20-21swapper/22
17243232150,2sleep20-21swapper/22
160771399150,13cyclictest0-21swapper/33
160771399150,13cyclictest0-21swapper/33
160771399150,12cyclictest0-21swapper/33
20882712141,9sleep12088283-21ssh1
20632472142,5sleep12063277-21apt-get1
20542232141,3sleep02054366-21ssh0
20042442141,7sleep12004257-21sshd-session1
19640092141,9sleep01734969-21diskmemload0
18999552141,2sleep10-21swapper/11
18644102145,3sleep21734969-21diskmemload2
18110042141,9sleep01734969-21diskmemload0
17290402141,4sleep21724586-21sendmail-msp2
17290402141,4sleep21724586-21sendmail-msp2
16991672141,2sleep10-21swapper/11
16780452141,9sleep21678169-21perl2
160771399146,5cyclictest1657101-21taskset3
160771399140,2cyclictest0-21swapper/33
160771399140,12cyclictest0-21swapper/33
160771399140,12cyclictest0-21swapper/33
20957172131,8sleep22095731-21ssh2
20885162130,2sleep00-21swapper/00
20723342131,7sleep02071686-21sendmail-msp0
20605982134,4sleep1331ktimers/11
20313972131,3sleep22031408-21ssh2
19196242131,4sleep01-21systemd0
18984922131,7sleep21734969-21diskmemload2
18936072131,4sleep01893642-21ssh0
18766812131,7sleep21878138-21cat2
18728012131,6sleep00-21swapper/00
18653982131,7sleep11734969-21diskmemload1
18298432131,4sleep21829858-21ssh2
18038872130,1sleep20-21swapper/22
17579432130,3sleep00-21swapper/00
160771399138,3cyclictest0-21swapper/33
160771399130,2cyclictest0-21swapper/33
160771399130,11cyclictest0-21swapper/33
20420412120,1chrt2034357-40find1
20313812121,7sleep12031396-21ssh1
20232052121,7sleep12023218-21sh1
20219242121,2sleep00-21swapper/00
20141532121,7sleep22014168-21ssh2
19858082121,7sleep01985897-21ssh0
19820092121,7sleep21982232-21hddtemp_smartct2
19734882121,7sleep11973605-21grep1
19728232121,7sleep21972830-21ssh2
19515632121,7sleep01951574-21ssh0
18896412121,3sleep01889654-21ssh0
18835042121,2sleep00-21swapper/00
18807162121,7sleep11880789-21ssh1
18494902120,8sleep21849502-21sh2
18347842121,5sleep11835987-21memory1
18316862121,7sleep11831934-21date1
18316142121,7sleep01831630-21ssh0
18213462121,7sleep11821361-21ssh1
18198662121,7sleep01819877-21ssh0
18031452121,6sleep11734969-21diskmemload1
17774252121,7sleep11777438-21ssh1
17343422121,5sleep11734570-21users1
16908762121,5sleep21692075-21memory2
16413582122,5sleep01641376-21apt-get0
16360892121,6sleep01636108-21apt-get0
160771399121,8cyclictest1607900-21taskset3
20978662111,5sleep00-21swapper/00
20868582110,3sleep20-21swapper/22
20711632111,4sleep21734969-21diskmemload2
20631242111,2sleep00-21swapper/00
20522662110,1sleep20-21swapper/22
20255862111,3sleep00-21swapper/00
20226362111,6sleep22022651-21ssh2
20058792111,3sleep20-21swapper/22
19984032111,3sleep00-21swapper/00
19611902111,4sleep21734969-21diskmemload2
19295002111,5sleep21930163-21tr2
19295002111,5sleep21930163-21tr2
19253792110,3sleep11925393-21ssh1
19072932111,6sleep11907306-21sh1
18668062111,3sleep00-21swapper/00
18546082110,1sleep10-21swapper/11
18495952111,5sleep01849863-21df0
18415582111,6sleep01843107-21kthreadcore0
18407252111,3sleep10-21swapper/11
18221712110,2sleep20-21swapper/22
17787892111,4sleep01734969-21diskmemload0
17708562111,4sleep21771077-21hddtemp_smartct2
17530632110,6sleep21752597-21sendmail-msp2
17402282111,4sleep21734969-21diskmemload2
17288922111,6sleep11724813-21/usr/sbin/munin1
17288922111,6sleep11724813-21/usr/sbin/munin1
16876092111,5sleep072-21kcompactd00
16256652111,5sleep01625919-21df0
20985072101,4sleep12098531-21apt-get1
20774302101,3sleep10-21swapper/11
20590502100,2sleep20-21swapper/22
20509102101,2sleep00-21swapper/00
20127682101,3sleep00-21swapper/00
19557572101,5sleep11956217-21kthreadcore1
19465002101,3sleep11946502-21apt-get1
19428272101,2sleep20-21swapper/22
19281322101,2sleep20-21swapper/22
19111432101,4sleep11734969-21diskmemload1
18731862101,3sleep10-21swapper/11
18491322100,5sleep20-21swapper/22
18058392101,4sleep21734969-21diskmemload2
18057722101,4sleep11805783-21ssh1
17931382101,3sleep10-21swapper/11
17881142101,3sleep21787749-21sendmail-msp2
17858532101,3sleep01734969-21diskmemload0
17662532101,5sleep11766505-21users1
17528312101,3sleep10-21swapper/11
17518252100,2sleep20-21swapper/22
17481492101,2sleep10-21swapper/11
17156972101,5sleep21717381-21kthreadcore2
16826752100,4sleep00-21swapper/00
16621612100,6sleep01662399-21expr0
16544862101,4sleep01655577-21memory0
16253032101,6sleep1106713-21snmpd1
16166932101,5sleep21618388-21kthreadcore2
16139682101,4sleep01615016-21df_inode0
2105274291,3sleep01734969-21diskmemload0
2105147290,2sleep20-21swapper/22
2013093291,3sleep12013104-21ssh1
2005905291,4sleep02005919-21ssh0
1996956290,2sleep20-21swapper/22
1942741290,1sleep00-21swapper/00
1934144291,2sleep00-21swapper/00
1934144291,2sleep00-21swapper/00
1907961291,2sleep00-21swapper/00
1837090291,2sleep00-21swapper/00
1836958291,3sleep21734969-21diskmemload2
1784346291,3sleep11734969-21diskmemload1
1779303291,4sleep21779318-21ssh2
1767363291,4sleep21767378-21ssh2
1743628291,4sleep11743643-21ssh1
1709193291,4sleep21709257-21/usr/sbin/munin2
1708747290,2sleep20-21swapper/22
1703953291,3sleep139542-21systemd-journal1
1698610291,4sleep01698629-21apt-get0
1693435290,3sleep01693454-21apt-get0
1688251291,3sleep01688263-21apt-get0
1687606291,3sleep24192556-21in:imuxsock2
1672595291,2sleep00-21swapper/00
1667392291,3sleep21667411-21apt-get2
1661846291,4sleep11661869-21latency_hist1
1656952293,2sleep20-21swapper/22
1656143291,4sleep11656285-21vmstat1
1641985291,5sleep11643449-21kthreadcore1
1640657290,1sleep10-21swapper/11
1630876291,4sleep21630944-21wc2
1609866291,4sleep01611631-21kthreadcore0
1907733281,2sleep21734969-21diskmemload2
1734185283,2sleep01734326-21timerandwakeup0
1730054280,2sleep20-21swapper/22
1718974280,2sleep00-21swapper/00
1709171281,2sleep10-21swapper/11
1703866281,3sleep0205-21jbd2/sda2-80
1677120280,1sleep20-21swapper/22
1676612281,3sleep10-21swapper/11
1662131281,3sleep11661912-21sendmail-msp1
1661815280,4sleep01661818-21sshd-auth0
1637511282,2sleep21639215-21cut2
1635801280,1sleep00-21swapper/00
1624279281,3sleep01624421-21ntp_kernel_err0
1607899280,1sleep20-21swapper/22
1728828271,3sleep00-21swapper/00
1728828271,3sleep00-21swapper/00
1723194270,2sleep11723540-21strings1
1723194270,2sleep11723540-21strings1
1714437270,2sleep10-21swapper/11
1699811271,2sleep21701454-21kthreadcore2
1698175270,1sleep20-21swapper/22
1693998271,2sleep10-21swapper/11
1689241271,2sleep10-21swapper/11
1685217271,2sleep10-21swapper/11
1668807270,3sleep01670510-21kthreadcore0
1651784270,2sleep20-21swapper/22
1651177270,1sleep20-21swapper/22
1646622270,2sleep10-21swapper/11
1643452271,3sleep21641097-21sendmail-msp2
1630854270,2sleep10-21swapper/11
1630421270,1sleep20-21swapper/22
1624990271,2sleep20-21swapper/22
1619787270,1sleep10-21swapper/11
1721439261,2sleep00-21swapper/00
1721439261,2sleep00-21swapper/00
1709462260,3sleep00-21swapper/00
1682607260,1sleep10-21swapper/11
1662137261,2sleep20-21swapper/22
1646933261,2sleep00-21swapper/00
1612213260,3sleep10-21swapper/11
1670513250,2sleep10-21swapper/11
1628319250,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional