You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 13:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri May 08, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1634793211857,57sleep30-21swapper/33
163508028554,12sleep20-21swapper/22
163006728064,10sleep10-21swapper/11
163483127454,14sleep00-21swapper/00
1635191992311,7cyclictest1765580-21turbostat3
1635191992211,7cyclictest18495172sleep33
163519199221,16cyclictest0-21swapper/33
163519199217,12cyclictest0-21swapper/33
1635191992111,7cyclictest2019124-21taskset3
163519199211,16cyclictest66950irq/132-enp1s0-TxRx-33
163519199211,16cyclictest0-21swapper/33
163519199211,15cyclictest66950irq/132-enp1s0-TxRx-33
163519199210,15cyclictest0-21swapper/33
163519199203,12cyclictest2060392-21taskset3
163519199202,13cyclictest1834199-21chrt3
1635191992017,1cyclictest0-21swapper/33
1635191992017,1cyclictest0-21swapper/33
1635191992011,6cyclictest0-21swapper/33
163519199201,15cyclictest0-21swapper/33
163519199201,15cyclictest0-21swapper/33
163519199201,15cyclictest0-21swapper/33
163519199201,14cyclictest1875769-21taskset3
1635191992010,8cyclictest0-21swapper/33
163519199200,18cyclictest0-21swapper/33
163519199200,16cyclictest1808728-21taskset3
18761602191,6sleep11876197-21munin-run1
18652752191,3sleep11865289-21ssh1
163519199191,15cyclictest0-21swapper/33
163519199191,14cyclictest0-21swapper/33
163519199191,14cyclictest0-21swapper/33
163519199191,14cyclictest0-21swapper/33
163519199191,14cyclictest0-21swapper/33
163519199191,14cyclictest0-21swapper/33
163519199191,14cyclictest0-21swapper/33
1635191991910,3cyclictest0-21swapper/33
163519199189,6cyclictest1653146-21turbostat3
163519199181,5cyclictest0-21swapper/33
1635191991812,3cyclictest0-21swapper/33
163519199181,14cyclictest0-21swapper/33
163519199181,14cyclictest0-21swapper/33
163519199181,14cyclictest0-21swapper/33
163519199181,14cyclictest0-21swapper/33
163519199181,13cyclictest0-21swapper/33
163519199180,15cyclictest0-21swapper/33
163519199180,14cyclictest0-21swapper/33
20792402171,3sleep01765565-21diskmemload0
18916932171,3sleep20-21swapper/22
17908192170,1sleep10-21swapper/11
163519199179,5cyclictest1898959-21taskset3
163519199179,5cyclictest1674455-21taskset3
163519199171,5cyclictest0-21swapper/33
1635191991714,1cyclictest0-21swapper/33
1635191991712,3cyclictest0-21swapper/33
163519199171,13cyclictest1822962-21taskset3
163519199171,13cyclictest0-21swapper/33
163519199171,13cyclictest0-21swapper/33
163519199171,13cyclictest0-21swapper/33
163519199171,13cyclictest0-21swapper/33
163519199170,14cyclictest0-21swapper/33
18504182161,2sleep00-21swapper/00
18155062161,4sleep11765565-21diskmemload1
17659392161,5sleep11766204-21df1
163519199162,9cyclictest1706678-21taskset3
163519199161,13cyclictest0-21swapper/33
163519199160,13cyclictest0-21swapper/33
163519199160,13cyclictest0-21swapper/33
163519199160,13cyclictest0-21swapper/33
163519199160,13cyclictest0-21swapper/33
20854182150,1sleep10-21swapper/11
20689472150,2sleep10-21swapper/11
20089372151,9sleep01765565-21diskmemload0
19999822151,5sleep12000036-21ssh1
19949942151,8sleep01765565-21diskmemload0
19611092150,1sleep20-21swapper/22
19366412151,10sleep11936699-21kernelversion1
19253162151,2sleep10-21swapper/11
19071442150,2sleep10-21swapper/11
18637572151,9sleep01863822-21ntpq0
17784562151,9sleep11778499-21ntpq1
17071902154,3sleep21707629-21cut2
16775332151,8sleep21678627-21memory2
16468372151,4sleep01646885-21ntpq0
163519199155,6cyclictest1669139-21turbostat3
163519199151,2cyclictest0-21swapper/33
1635191991512,1cyclictest0-21swapper/33
163519199150,13cyclictest0-21swapper/33
163519199150,13cyclictest0-21swapper/33
163519199150,13cyclictest0-21swapper/33
163519199150,12cyclictest0-21swapper/33
20364832141,3sleep11765565-21diskmemload1
20265212141,3sleep22026535-21ssh2
19740472141,7sleep11974095-21ntpq1
19077202141,2sleep20-21swapper/22
18489722141,2sleep10-21swapper/11
18383432141,9sleep21838373-21ssh2
18146812141,9sleep21814693-21ssh2
16520922141,8sleep01652140-21ntpq0
1635191991411,2cyclictest0-21swapper/33
163519199140,13cyclictest0-21swapper/33
163519199140,12cyclictest0-21swapper/33
163519199140,11cyclictest0-21swapper/33
20935922131,7sleep01765565-21diskmemload0
20935922131,7sleep01765565-21diskmemload0
20110512131,8sleep12011063-21ssh1
19930882131,3sleep11993099-21ssh1
19105242133,5sleep01910548-21apt-get0
18928722131,8sleep11892882-21ssh1
18845772131,2sleep10-21swapper/11
18595082131,5sleep21765565-21diskmemload2
18340442130,4sleep01834055-21ssh0
18322892131,8sleep11765565-21diskmemload1
17428012131,8sleep21742937-21/usr/sbin/munin2
17270592131,9sleep00-21swapper/00
16895392131,8sleep01689658-21find0
16482562131,5sleep21648372-21diskstats2
20878792120,2sleep00-21swapper/00
20610842121,8sleep12061095-21sh1
20530682121,2sleep20-21swapper/22
20029332121,3sleep20-21swapper/22
19697902121,7sleep21969801-21ssh2
19615022121,7sleep21961523-21ssh2
19596632121,7sleep11959674-21sh1
19264432120,5sleep00-21swapper/00
19150902121,7sleep21915145-21wc2
18979822121,5sleep20-21swapper/22
18000302122,5sleep11799520-21apt-get1
17969652121,7sleep21796976-21ssh2
17834162121,7sleep01783946-21ssh0
17659072121,5sleep00-21swapper/00
17485392121,5sleep21748590-21ntpq2
16681032121,7sleep01668155-21ld0
20960422111,5sleep10-21swapper/11
20960422111,5sleep10-21swapper/11
20758422111,5sleep12075904-21ntpq1
20715282110,5sleep20-21swapper/22
20433192111,3sleep10-21swapper/11
20183652111,4sleep01765565-21diskmemload0
19925772110,7sleep01992583-21diskmemload0
19859582110,2sleep21985969-21ssh2
19832392111,4sleep01765565-21diskmemload0
19670172111,4sleep01765565-21diskmemload0
19425812111,5sleep01765565-21diskmemload0
19360222111,3sleep20-21swapper/22
19156882111,5sleep10-21swapper/11
19011352110,3sleep10-21swapper/11
18874052114,3sleep01889163-21kthreadcore0
18774002111,2sleep20-21swapper/22
18570982111,5sleep21765565-21diskmemload2
18213802111,5sleep11821436-21ntpq1
18212332111,6sleep01821377-21ntpq0
17701422111,5sleep21770205-21ntp_states2
17282222112,4sleep01728240-21apt-get0
16574852111,6sleep01657538-21ld0
21202132101,4sleep21765565-21diskmemload2
21189032101,3sleep11765565-21diskmemload1
21139402101,5sleep02114119-21expr0
21096432101,5sleep22109696-21awk2
21026132100,2sleep20-21swapper/22
21011852101,4sleep01765565-21diskmemload0
20943572101,4sleep21765565-21diskmemload2
20943572101,4sleep21765565-21diskmemload2
20853862101,4sleep21765565-21diskmemload2
20695342101,4sleep21765565-21diskmemload2
20531902101,2sleep00-21swapper/00
20441032101,3sleep00-21swapper/00
20434702101,3sleep20-21swapper/22
20352412101,4sleep01765565-21diskmemload0
20289872101,3sleep20-21swapper/22
20257952101,5sleep12025806-21ssh1
20248732101,5sleep02024932-21sed0
20198022101,6sleep22019810-21ssh2
20180402101,3sleep10-21swapper/11
20078082101,4sleep22007868-21ntpq2
19931402101,4sleep21765565-21diskmemload2
19767832101,3sleep00-21swapper/00
19690572100,3sleep10-21swapper/11
19498992101,4sleep11765565-21diskmemload1
19487292101,6sleep01765565-21diskmemload0
19281122101,5sleep11928195-21irqstats1
19275542101,3sleep01765565-21diskmemload0
19024622101,4sleep01765565-21diskmemload0
18937542101,4sleep00-21swapper/00
18793072101,6sleep01880807-21latency0
18727172101,4sleep21765565-21diskmemload2
18579932101,4sleep01765565-21diskmemload0
18479692101,3sleep20-21swapper/22
18417052101,3sleep10-21swapper/11
18320162101,4sleep21765565-21diskmemload2
18229912101,5sleep21823005-21ssh2
18128222101,5sleep01812921-21ntpq0
18072482101,3sleep20-21swapper/22
17913112100,2sleep10-21swapper/11
17816802100,2sleep00-21swapper/00
17645472101,4sleep01764597-21ntpq0
17265862101,4sleep11726674-21ntpq1
17178182101,5sleep21717995-21kthreadcore2
16945442100,2sleep11694689-21strings1
16616492101,5sleep01662633-21memory0
2111321291,2sleep10-21swapper/11
2105136291,3sleep01765565-21diskmemload0
2102650291,2sleep10-21swapper/11
2067388290,1sleep00-21swapper/00
2059756291,2sleep00-21swapper/00
2056206291,5sleep22057873-21kthreadcore2
2053770291,2sleep10-21swapper/11
1983533291,4sleep11765565-21diskmemload1
1958146291,4sleep01765565-21diskmemload0
1949562290,1sleep20-21swapper/22
1933976291,3sleep20-21swapper/22
1924300291,4sleep21924312-21rm2
1874186291,3sleep01765565-21diskmemload0
1856264291,4sleep11856274-21ssh1
1830165290,1sleep00-21swapper/00
1799574291,4sleep01800029-21ssh0
1799316291,3sleep01765565-21diskmemload0
1788999291,3sleep21765565-21diskmemload2
1780110291,2sleep20-21swapper/22
1759222291,3sleep01759272-21ntpq0
1758228291,4sleep21759074-21missed_timers2
1738526291,2sleep10-21swapper/11
1732644291,4sleep21732896-21grep2
1721437291,3sleep01721490-21ntp_states0
1709365291,4sleep01710540-21memory0
1706393291,4sleep21706466-21cat2
1696568291,5sleep21697586-21kthreadcore2
1690698291,3sleep21690717-21apt-get2
1680555291,4sleep11680721-21grep1
1675279294,2sleep00-21swapper/00
1673546291,4sleep11669406-21/usr/sbin/munin1
1664334291,4sleep21664437-21perl2
1658463290,4sleep21658513-21munin-run2
1657539291,4sleep11657588-21open_inodes1
1760563281,3sleep11760779-21diskstats1
1754882281,3sleep11754895-21latency_hist1
1729588281,3sleep11731356-21kthreadcore1
1723208281,4sleep21711422-21kworker/2:0+events2
1710832280,1sleep10-21swapper/11
1706385281,2sleep10-21swapper/11
1669428280,5sleep01669614-21cstates0
1659295281,3sleep21659913-21cut2
1658719280,1sleep10-21swapper/11
1640311280,2sleep00-21swapper/00
1765538270,1sleep20-21swapper/22
1751653271,2sleep21751919-21kthreadcore2
1749329270,1sleep10-21swapper/11
1743646271,3sleep13599524-21sshd1
1743640270,1sleep00-21swapper/00
1734610270,3sleep01736339-21cut0
1716799270,1sleep10-21swapper/11
1706218270,1sleep00-21swapper/00
1700911270,1sleep10-21swapper/11
1695741270,4sleep01695756-21latency_hist0
1689443270,3sleep20-21swapper/22
1685396270,2sleep10-21swapper/11
1648106271,3sleep10-21swapper/11
1640176270,1sleep10-21swapper/11
1755238260,3sleep10-21swapper/11
1749220260,4sleep00-21swapper/00
1722249260,1sleep10-21swapper/11
1712309260,3sleep21712517-21if_enp1s02
1680722261,2sleep20-21swapper/22
1680038260,3sleep00-21swapper/00
1679446260,4sleep10-21swapper/11
1674444260,1sleep20-21swapper/22
1668156260,2sleep11668205-21ntpq1
1647594260,1sleep10-21swapper/11
1640501260,2sleep20-21swapper/22
1635386260,1sleep20-21swapper/22
1753834250,1sleep00-21swapper/00
1737493250,1sleep20-21swapper/22
1712007250,1sleep00-21swapper/00
1690864240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional