You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 03:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri May 22, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137946021250,2sleep10-21swapper/11
1221296212059,56sleep30-21swapper/33
122567129058,12sleep20-21swapper/22
122584828564,11sleep10-21swapper/11
122585828057,17sleep00-21swapper/00
16469222740,2sleep10-21swapper/11
122599499251,20cyclictest0-21swapper/33
14575702232,7sleep11457627-21ntpq1
1225994992211,7cyclictest1655743-21turbostat3
122599499221,15cyclictest12698842sleep33
122599499220,20cyclictest0-21swapper/33
1225994992117,2cyclictest0-21swapper/33
122599499201,15cyclictest66950irq/132-enp1s0-TxRx-33
122599499201,15cyclictest0-21swapper/33
122599499200,17cyclictest0-21swapper/33
15397912190,2sleep00-21swapper/00
1225994991916,1cyclictest0-21swapper/33
122599499191,15cyclictest0-21swapper/33
122599499191,14cyclictest66950irq/132-enp1s0-TxRx-33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499191,14cyclictest0-21swapper/33
122599499190,16cyclictest0-21swapper/33
15856412181,2sleep10-21swapper/11
15296252181,3sleep00-21swapper/00
122599499187,1cyclictest0-21swapper/33
122599499181,7cyclictest1661433-21taskset3
1225994991814,2cyclictest0-21swapper/33
122599499181,14cyclictest1383001-21chrt3
1225994991811,4cyclictest13720952sleep33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,14cyclictest0-21swapper/33
122599499181,13cyclictest0-21swapper/33
122599499181,13cyclictest0-21swapper/33
122599499180,2cyclictest0-21swapper/33
122599499180,15cyclictest0-21swapper/33
122599499180,14cyclictest0-21swapper/33
122599499180,13cyclictest0-21swapper/33
16365722171,3sleep00-21swapper/00
16100222170,3sleep00-21swapper/00
15968872176,7sleep21598472-21cat2
14891442171,5sleep01489364-21hddtemp_smartct0
13073222171,5sleep11307423-21ntpq1
122599499172,11cyclictest1548100-21taskset3
122599499171,13cyclictest1637268-21taskset3
122599499171,13cyclictest0-21swapper/33
122599499171,13cyclictest0-21swapper/33
122599499170,14cyclictest66950irq/132-enp1s0-TxRx-33
122599499170,14cyclictest0-21swapper/33
122599499170,14cyclictest0-21swapper/33
122599499170,14cyclictest0-21swapper/33
16691832161,9sleep01669226-21ntpq0
16096902161,2sleep20-21swapper/22
15757022161,2sleep10-21swapper/11
15318002161,7sleep21356374-21diskmemload2
15035922161,11sleep01503603-21ssh0
13992702161,3sleep20-21swapper/22
122599499167,5cyclictest1242939-21taskset3
1225994991614,1cyclictest0-21swapper/33
122599499161,3cyclictest0-21swapper/33
122599499161,12cyclictest1330305-21taskset3
122599499160,14cyclictest0-21swapper/33
122599499160,14cyclictest0-21swapper/33
122599499160,13cyclictest0-21swapper/33
16604702151,8sleep21660514-21ntpq2
15647792151,4sleep11356374-21diskmemload1
14401672150,1sleep00-21swapper/00
13998282151,2sleep10-21swapper/11
13353392150,3sleep01335358-21apt-get0
122599499150,13cyclictest0-21swapper/33
122599499150,12cyclictest0-21swapper/33
122599499150,12cyclictest0-21swapper/33
122599499150,12cyclictest0-21swapper/33
16709712141,4sleep21356374-21diskmemload2
16303932141,7sleep21630473-21irqstats2
15490622140,2sleep20-21swapper/22
15483152141,10sleep00-21swapper/00
15364382140,3sleep10-21swapper/11
15055032140,1sleep20-21swapper/22
14663662140,9sleep11466424-21ntpq1
14524652141,3sleep01452472-21ssh0
14418852141,3sleep11356374-21diskmemload1
14339122141,3sleep11433926-21ssh1
14147212141,3sleep21356374-21diskmemload2
14015912142,7sleep21403196-21grep2
13813582140,2sleep20-21swapper/22
13637722141,9sleep11363782-21ssh1
122599499148,3cyclictest12761512sleep33
122599499148,3cyclictest12761512sleep33
122599499141,1cyclictest0-21swapper/33
1225994991411,2cyclictest0-21swapper/33
122599499140,2cyclictest0-21swapper/33
122599499140,2cyclictest0-21swapper/33
122599499140,11cyclictest0-21swapper/33
17251282131,2sleep01602831-21kworker/u16:0+rpciod0
16544962131,7sleep00-21swapper/00
16256472131,4sleep11625679-21tr1
15654372131,7sleep00-21swapper/00
15233792131,6sleep00-21swapper/00
15067772131,8sleep11506826-21ssh1
14956572131,8sleep11356374-21diskmemload1
14773282131,5sleep00-21swapper/00
14528902131,2sleep20-21swapper/22
14251482131,3sleep11356374-21diskmemload1
13732802131,8sleep01373290-21ssh0
13728492131,8sleep21372860-21ssh2
13638782131,7sleep21356374-21diskmemload2
122599499138,3cyclictest1287379-21chrt3
122599499130,2cyclictest0-21swapper/33
16976592121,7sleep11697673-21ssh1
16865912121,6sleep01686640-21ntpq0
16557292121,2sleep10-21swapper/11
15995382121,6sleep01599552-21ssh0
15768272121,7sleep01576840-21rm0
14839712121,5sleep21484029-21ntpq2
14353772120,3sleep01435391-21ssh0
14175662121,6sleep00-21swapper/00
14047972121,6sleep11404897-21ntpq1
13725832121,3sleep10-21swapper/11
13287482121,5sleep21328794-21ntpq2
17250572111,5sleep21356374-21diskmemload2
17238762114,3sleep11723883-21ssh1
17000652111,5sleep01700278-21diskstats0
17000652111,5sleep01700278-21diskstats0
16978242110,4sleep272-21kcompactd02
16885412111,3sleep20-21swapper/22
16847012111,6sleep11686366-21ls1
16778712111,5sleep21677928-21ntpq2
15769732111,6sleep21577183-21cstates2
15762102111,5sleep21356374-21diskmemload2
15734562111,7sleep01356374-21diskmemload0
15663802111,3sleep20-21swapper/22
15582792111,3sleep20-21swapper/22
15461782111,4sleep11546243-21ntp_states1
15407872111,3sleep20-21swapper/22
15285642111,4sleep10-21swapper/11
15131812110,3sleep01513195-21ssh0
14971462111,7sleep21497159-21sh2
14797612111,3sleep10-21swapper/11
14708612111,4sleep21356374-21diskmemload2
14615252111,5sleep01356374-21diskmemload0
14600212111,5sleep21356374-21diskmemload2
14337752111,1sleep20-21swapper/22
14252142111,2sleep21425228-21ssh2
14052932111,6sleep01405490-21cat0
13465432111,7sleep01346599-21kthreadcore0
13457102111,5sleep01345738-21munin-run0
13340312111,5sleep11334088-21ntpq1
13127132111,5sleep21312751-21ntpq2
12495232111,5sleep21249778-21df2
17145992101,4sleep01356374-21diskmemload0
17087562101,5sleep21708864-21tr2
17070552100,6sleep11707068-21ssh1
17070552100,6sleep11707068-21ssh1
16954202101,5sleep01695479-21ntpq0
16616682101,4sleep11356374-21diskmemload1
16428242101,5sleep01356374-21diskmemload0
16385132101,4sleep21356374-21diskmemload2
16351712101,5sleep11635182-21ssh1
16277472101,2sleep20-21swapper/22
16193212100,2sleep00-21swapper/00
16164822101,5sleep11616532-21ntpq1
16101332100,1sleep10-21swapper/11
15915822101,2sleep00-21swapper/00
15843972101,4sleep11356374-21diskmemload1
15223672101,3sleep10-21swapper/11
15206742101,4sleep21356374-21diskmemload2
15026892101,3sleep10-21swapper/11
14866562100,6sleep01356374-21diskmemload0
14703812100,1sleep0171rcu_preempt0
14696942101,4sleep21356374-21diskmemload2
14517142100,1sleep10-21swapper/11
14367602101,5sleep21438174-21ssh2
14182672101,5sleep00-21swapper/00
14161902101,2sleep10-21swapper/11
13972172100,1sleep00-21swapper/00
13885322101,4sleep21356374-21diskmemload2
13884712101,4sleep11356374-21diskmemload1
13811312100,2sleep00-21swapper/00
13487552101,4sleep21349878-21memory2
13303342101,5sleep01330556-21cat0
13223102101,4sleep11323243-21memory1
12911492100,2sleep10-21swapper/11
12837442101,5sleep01285568-21kthreadcore0
12751952101,4sleep01275244-21ntpq0
12751952101,4sleep01275244-21ntpq0
12535442101,5sleep11253568-21ntpq1
12442652101,4sleep01244481-21diskstats0
12302572101,4sleep21230298-21ntpq2
12301622101,4sleep11230250-21ntpq1
1714264291,3sleep11356374-21diskmemload1
1678799291,3sleep00-21swapper/00
1677745290,3sleep11677870-21ntpq1
1663889291,4sleep01663906-21ssh0
1654566291,3sleep21356374-21diskmemload2
1625286291,4sleep01625334-21ntpq0
1616541291,3sleep21616594-21ntpq2
1601455291,2sleep10-21swapper/11
1590695291,2sleep20-21swapper/22
1558170291,4sleep11558183-21ssh1
1555812291,2sleep00-21swapper/00
1512888291,2sleep20-21swapper/22
1471283291,4sleep11471394-21perl1
1387374291,4sleep01387442-21ntpq0
1357067291,4sleep01357198-21perl0
1330638291,3sleep21332166-21grep2
1319054291,4sleep11319218-21latency_hist1
1308983291,4sleep11309193-21hddtemp_smartct1
1296466291,4sleep01296505-21ntpq0
1291202291,4sleep21291260-21ntpq2
1280459291,4sleep11280507-21ntpq1
1266315291,4sleep21266632-21cat2
1264533291,3sleep21264583-21ntpq2
1257127291,4sleep11258957-21tr1
1240135291,5sleep01241882-21kthreadcore0
1699956281,3sleep21699985-21apt-get2
1699956281,3sleep21699985-21apt-get2
1669946281,2sleep11356374-21diskmemload1
1352979281,3sleep01353392-21kthreadcore0
1340185281,3sleep21340189-21sshd2
1324922280,1sleep00-21swapper/00
1320569281,3sleep01322309-21kthreadcore0
1314021281,3sleep01314040-21apt-get0
1303320280,3sleep20-21swapper/22
1271414281,3sleep20-21swapper/22
1271414281,3sleep20-21swapper/22
1258958281,3sleep21259090-21uname2
1242819281,2sleep20-21swapper/22
1231257281,3sleep21231621-21pmu-power2
1356167270,4sleep10-21swapper/11
1335911271,2sleep10-21swapper/11
1313430270,1sleep00-21swapper/00
1306600271,3sleep01307319-21missed_timers0
1302652270,1sleep10-21swapper/11
1297747270,2sleep00-21swapper/00
1297253270,1sleep10-21swapper/11
1296564272,2sleep21225988-21cyclictest2
1281162270,1sleep00-21swapper/00
1277619271,3sleep21279326-21cut2
1276139270,1sleep10-21swapper/11
1276139270,1sleep10-21swapper/11
1270797270,1sleep10-21swapper/11
1270539270,2sleep00-21swapper/00
1260467271,3sleep10-21swapper/11
1249035270,1sleep10-21swapper/11
1238865270,1sleep10-21swapper/11
1234083271,2sleep00-21swapper/00
1356011260,4sleep20-21swapper/22
1347054261,2sleep10-21swapper/11
1328905261,2sleep10-21swapper/11
1323951260,3sleep20-21swapper/22
1291059260,2sleep00-21swapper/00
1286728260,1sleep10-21swapper/11
1281788260,2sleep20-21swapper/22
1265469260,4sleep00-21swapper/00
1260187261,2sleep00-21swapper/00
1248189260,2sleep20-21swapper/22
1235038261,2sleep10-21swapper/11
1342002250,2sleep11342736-21kthreadcore1
1340425250,1sleep20-21swapper/22
1316077250,1sleep20-21swapper/22
1301856250,1sleep20-21swapper/22
1254282250,1sleep00-21swapper/00
1226451240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional