You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 07:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Mon Dec 01, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14944139912130,1209cyclictest17528762sleep33
14944139910891,1079cyclictest1940709-21turbostat3
1494413995850,580cyclictest18168852chrt3
1494413993310,328cyclictest1687799-21turbostat3
17929822210,5sleep2431ktimers/22
1494413991912,4cyclictest1609884-21sleep3
1494413991815,1cyclictest0-21swapper/33
149441399181,13cyclictest1889920-21taskset3
149441399175,9cyclictest1553491-21taskset3
1494413991714,1cyclictest0-21swapper/33
149441399171,12cyclictest1857690-21taskset3
149441399161,6cyclictest0-21swapper/33
1494413991613,1cyclictest0-21swapper/33
149441399161,11cyclictest1655617-21turbostat3
149441399161,10cyclictest14945732sleep33
149441399151,9cyclictest15407282sleep33
1494413991513,1cyclictest0-21swapper/33
1494413991513,1cyclictest0-21swapper/33
1494413991512,1cyclictest0-21swapper/33
1494413991512,1cyclictest0-21swapper/33
1494413991512,1cyclictest0-21swapper/33
1494413991512,1cyclictest0-21swapper/33
1494413991512,1cyclictest0-21swapper/33
1494413991512,1cyclictest0-21swapper/33
149441399151,11cyclictest1775009-21taskset3
19339472141,10sleep11935053-21grep1
17511592140,3sleep01751167-21ssh0
149441399145,6cyclictest1973017-21turbostat3
1494413991412,1cyclictest0-21swapper/33
1494413991412,1cyclictest0-21swapper/33
1494413991412,1cyclictest0-21swapper/33
1494413991412,1cyclictest0-21swapper/33
1494413991411,1cyclictest0-21swapper/33
149441399141,10cyclictest1746358-21taskset3
149441399141,10cyclictest1731833-21taskset3
149441399141,10cyclictest1727964-21taskset3
149441399141,10cyclictest1686863-21taskset3
149441399141,10cyclictest1641087-21chrt3
149441399141,10cyclictest1622708-21taskset3
149441399141,10cyclictest1590784-21taskset3
149441399141,10cyclictest1584699-21turbostat3
19491242130,6sleep11949157-21apt-get1
149441399134,7cyclictest1743384-21taskset3
149441399132,8cyclictest1831794-21chrt3
149441399131,9cyclictest1849615-21taskset3
149441399131,9cyclictest1669897-21taskset3
149441399131,9cyclictest1641770-21taskset3
1494413991312,0cyclictest0-21swapper/33
1494413991311,1cyclictest0-21swapper/33
1494413991311,1cyclictest0-21swapper/33
1494413991311,1cyclictest0-21swapper/33
1494413991311,1cyclictest0-21swapper/33
149441399131,10cyclictest1888922-21taskset3
149441399131,10cyclictest1872765-21chrt3
149441399131,10cyclictest1849059-21taskset3
149441399131,10cyclictest1622431-21turbostat3
1494413991310,1cyclictest0-21swapper/33
1494413991310,1cyclictest0-21swapper/33
1494413991310,1cyclictest0-21swapper/33
149441399130,9cyclictest15467362sleep33
149441399130,10cyclictest1920013-21taskset3
149441399130,10cyclictest1908196-21taskset3
149441399130,10cyclictest1879311-21taskset3
149441399130,10cyclictest1597512-21taskset3
149441399130,10cyclictest1572378-21taskset3
149441399130,10cyclictest1509250-21chrt3
149441399121,9cyclictest1719156-21taskset3
149441399121,9cyclictest1671973-21taskset3
149441399121,9cyclictest1521839-21taskset3
149441399121,8cyclictest1704257-21taskset3
149441399120,9cyclictest1799161-21chrt3
19414852111,3sleep01942480-21irqstats0
19207742111,7sleep11920782-21ssh1
190898110,7rtkit-daemon1623628-21cut2
190898110,6rtkit-daemon1931290-21ssh0
190898110,3rtkit-daemon1500786-21munin-node2
18990102111,7sleep21899981-21tr2
16704882110,4sleep01670496-21ssh0
149441399111,8cyclictest1494435-21taskset3
19537002101,6sleep01954229-21/usr/sbin/munin0
19312262100,1sleep20-21swapper/22
190898100,6rtkit-daemon438-21wpa_supplicant2
190898100,6rtkit-daemon438-21wpa_supplicant1
190898100,5rtkit-daemon299-21systemd-journal1
190898100,5rtkit-daemon1898319-21perl0
190898100,5rtkit-daemon1514458-21ntp_states2
190898100,5rtkit-daemon1493513-21awk2
190898100,4rtkit-daemon1784517-21ssh0
190898100,3rtkit-daemon1928-21sendmail-mta0
190898100,2rtkit-daemon1928448-21ssh2
18869302101,6sleep01887100-21awk0
18832042100,7sleep11884225-21grep1
16868652100,6sleep01686871-21ssh0
16429182101,2sleep10-21swapper/11
14993692100,1sleep20-21swapper/22
1963212290,6sleep11963220-21ssh1
1938223291,5sleep21938349-21python32
1929854290,1sleep10-21swapper/11
19089890,5rtkit-daemon436-21systemd-logind2
19089890,5rtkit-daemon1929253-21ssh0
19089890,5rtkit-daemon1911719-21unixbench-2d0
19089890,5rtkit-daemon1804676-21kthreadcore2
19089890,5rtkit-daemon1647575-21diskmemload0
19089890,5rtkit-daemon1544514-21grep2
19089890,5rtkit-daemon0-21swapper/00
19089890,4rtkit-daemon1952479-21cut2
19089890,4rtkit-daemon1939709-21ssh0
19089890,4rtkit-daemon1856398-21ssh0
19089890,4rtkit-daemon1775606-21ssh0
19089890,4rtkit-daemon1763673-21kthreadcore1
19089890,4rtkit-daemon1744784-21ssh1
19089890,4rtkit-daemon1686512-21ssh2
19089890,3rtkit-daemon667-21snmpd1
19089890,3rtkit-daemon1928-21sendmail-mta2
19089890,1rtkit-daemon0-21swapper/11
19089890,1rtkit-daemon0-21swapper/11
19089890,1rtkit-daemon0-21swapper/11
1895327290,6sleep11895449-21sort1
1874771290,7sleep01875762-21grep0
1835805291,5sleep01837867-21kthreadcore0
1656149291,6sleep01656355-21hddtemp_smartct0
1641520291,4sleep01641553-21apt-get0
1635279291,4sleep01635434-21diskstats0
1559805291,4sleep11559838-21apt-get1
1974842281,4sleep01976382-21grep0
19089880,5rtkit-daemon1928-21sendmail-mta2
19089880,5rtkit-daemon1500786-21munin-node2
19089880,4rtkit-daemon436-21systemd-logind2
19089880,4rtkit-daemon1928-21sendmail-mta0
19089880,4rtkit-daemon1809239-21ssh2
19089880,4rtkit-daemon1799056-21ssh0
19089880,4rtkit-daemon1646805-21find2
19089880,4rtkit-daemon1600763-21grep2
19089880,4rtkit-daemon1556711-21cut2
19089880,4rtkit-daemon1500786-21munin-node2
19089880,4rtkit-daemon1500786-21munin-node2
19089880,4rtkit-daemon1500786-21munin-node2
19089880,4rtkit-daemon1500786-21munin-node2
19089880,3rtkit-daemon1824007-21ssh0
19089880,3rtkit-daemon1824007-21ssh0
19089880,3rtkit-daemon1575453-21ps2
19089880,1rtkit-daemon0-21swapper/22
19089880,1rtkit-daemon0-21swapper/22
1906477280,5sleep11907077-21sed1
1865654281,4sleep11865660-21ssh1
1842002280,5sleep01842456-21sed0
1760329281,4sleep11760337-21ssh1
1743523280,2sleep11743531-21ssh1
1736812280,4sleep00-21swapper/00
1648120281,4sleep01648327-21hddtemp_smartct0
1634726281,5sleep10-21swapper/11
1564957280,5sleep21565009-21/usr/sbin/munin2
1547700280,5sleep11548595-21cut1
1540754281,4sleep01540910-21diskstats0
1499773281,5sleep01499823-21proc_pri0
1971948270,1sleep00-21swapper/00
1964102270,1sleep00-21swapper/00
1912431270,1sleep20-21swapper/22
19089870,4rtkit-daemon81364-21packagekitd2
19089870,4rtkit-daemon674-21lldpd2
19089870,4rtkit-daemon674-21lldpd2
19089870,4rtkit-daemon671-21ntpd2
19089870,4rtkit-daemon667-21snmpd2
19089870,4rtkit-daemon667-21snmpd1
19089870,4rtkit-daemon1500786-21munin-node2
19089870,3rtkit-daemon671-21ntpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon436-21systemd-logind2
19089870,3rtkit-daemon1947230-21ssh2
19089870,3rtkit-daemon1727070-21ssh1
19089870,3rtkit-daemon1677933-21ssh2
19089870,3rtkit-daemon1500786-21munin-node2
19089870,1rtkit-daemon0-21swapper/22
1873026270,4sleep01873034-21ssh0
1865387270,4sleep11865443-21ls1
1857659271,2sleep21857683-21apt-get2
1856428270,5sleep21856436-21ssh2
1841554271,2sleep11841061-21apt-get1
1778624270,5sleep11780531-21tr1
1767269270,1sleep00-21swapper/00
1758329271,3sleep21758451-21switchtime2
1745361270,4sleep21746357-21cut2
1728107271,4sleep21728115-21ssh2
1715003271,3sleep01717078-21cut0
1618351270,4sleep01620432-21tr0
1615317270,4sleep11615421-21awk1
1604026270,4sleep01604226-21hddtemp_smartct0
1602671270,4sleep11602730-21ntp_states1
1567635273,2sleep10-21swapper/11
1566714271,2sleep00-21swapper/00
1559525271,4sleep11559773-21date1
1973556261,2sleep10-21swapper/11
1964488260,3sleep21964495-21sh2
1889884261,2sleep20-21swapper/22
1888259260,1sleep20-21swapper/22
1855875261,2sleep10-21swapper/11
1849212260,1sleep20-21swapper/22
1839389260,3sleep11839397-21ssh1
1833344261,2sleep20-21swapper/22
1832271260,1sleep20-21swapper/22
1823598260,1sleep20-21swapper/22
1823598260,1sleep20-21swapper/22
1809240261,2sleep10-21swapper/11
1801577261,2sleep00-21swapper/00
1800800260,1sleep10-21swapper/11
1786655260,4sleep11788138-21grep1
1769080260,3sleep11769113-21apt-get1
1742699260,1sleep20-21swapper/22
1736292260,3sleep21736300-21ssh2
1735656260,1sleep00-21swapper/00
1720646261,2sleep01720670-21apt-get0
1710174260,1sleep00-21swapper/00
1702409260,1sleep00-21swapper/00
1697676260,4sleep11699161-21cut1
1694160260,1sleep00-21swapper/00
1677853260,1sleep00-21swapper/00
1662840261,2sleep10-21swapper/11
1653929260,1sleep10-21swapper/11
1634272261,3sleep01634410-21timerandwakeup0
1596502261,2sleep00-21swapper/00
1586010261,3sleep01587139-21kthreadcore0
1580105260,3sleep01581442-21grep0
1578835260,3sleep11578941-21grep1
1578214260,1sleep00-21swapper/00
1565500260,1sleep00-21swapper/00
1558844260,3sleep01559103-21tail0
1544315261,3sleep11545702-21memory1
1533328260,1sleep00-21swapper/00
1527140260,3sleep11527249-21find1
1505933260,3sleep01507871-21latency0
1967960251,2sleep10-21swapper/11
1903533250,2sleep11903785-21ssh1
1879963250,3sleep20-21swapper/22
1808942250,3sleep01809148-21ssh0
1801231250,2sleep10-21swapper/11
1770541251,2sleep20-21swapper/22
1768261250,1sleep20-21swapper/22
1760632250,3sleep00-21swapper/00
1713036250,2sleep20-21swapper/22
1674692250,3sleep1321rcuc/11
1669821250,3sleep10-21swapper/11
1641207250,1sleep10-21swapper/11
1625194251,2sleep00-21swapper/00
1610832251,2sleep00-21swapper/00
1603824251,1sleep10-21swapper/11
1603260250,1sleep00-21swapper/00
1578257250,1sleep10-21swapper/11
1552962250,1sleep00-21swapper/00
1534444250,3sleep11534477-21apt-get1
1523352250,2sleep00-21swapper/00
1515283250,1sleep10-21swapper/11
1508853250,1sleep10-21swapper/11
1497218251,2sleep11499364-21kthreadcore1
1491524250,2sleep11493179-21latency1
1947692240,2sleep11947700-21ssh1
1930942240,2sleep11930950-21ssh1
1831537240,2sleep11831545-21ssh1
1783047240,2sleep21783055-21ssh2
1711620240,1sleep20-21swapper/22
1628506240,3sleep10-21swapper/11
1622061240,3sleep10-21swapper/11
1596969240,3sleep10-21swapper/11
1587141240,2sleep10-21swapper/11
1528122240,1sleep10-21swapper/11
1520258240,2sleep01520629-21needreboot0
1515546240,1sleep10-21swapper/11
1511348241,1sleep00-21swapper/00
1488318240,1sleep00-21swapper/00
1914185230,1sleep00-21swapper/00
1680038230,1sleep10-21swapper/11
1534645230,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional