You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 10:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Jul 01, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3724015212554,65sleep30-21swapper/33
372418928353,11sleep20-21swapper/22
372410828352,12sleep10-21swapper/11
372415128053,7sleep00-21swapper/00
372435399310,21cyclictest0-21swapper/33
372435399260,23cyclictest0-21swapper/33
3724353992510,12cyclictest0-21swapper/33
41231102241,2sleep20-21swapper/22
372435399233,17cyclictest0-21swapper/33
372435399231,16cyclictest0-21swapper/33
372435399221,18cyclictest0-21swapper/33
372435399221,17cyclictest0-21swapper/33
39245242211,7sleep23924532-21ssh2
38578202211,3sleep20-21swapper/22
372435399212,13cyclictest4130924-21taskset3
372435399211,5cyclictest0-21swapper/33
372435399211,15cyclictest0-21swapper/33
372435399210,9cyclictest0-21swapper/33
372435399210,10cyclictest0-21swapper/33
40936472201,8sleep14093867-21hddtemp_smartct1
372435399201,8cyclictest4033383-21taskset3
372435399201,3cyclictest37422482sleep33
3724353992012,5cyclictest0-21swapper/33
372435399201,15cyclictest0-21swapper/33
372435399200,2cyclictest37942542sleep33
372435399200,1cyclictest0-21swapper/33
41066552191,2sleep10-21swapper/11
40731292191,3sleep20-21swapper/22
39658482191,5sleep10-21swapper/11
39605842192,5sleep23960608-21apt-get2
38740062193,5sleep10-21swapper/11
372435399191,7cyclictest4117680-21turbostat3
372435399191,7cyclictest4011-21turbostat3
372435399191,7cyclictest3868717-21turbostat3
372435399191,7cyclictest3831015-21turbostat3
372435399191,7cyclictest3789111-21turbostat3
3724353991916,1cyclictest0-21swapper/33
372435399191,5cyclictest3885258-21turbostat3
3724353991912,5cyclictest0-21swapper/33
372435399191,15cyclictest0-21swapper/33
372435399191,14cyclictest0-21swapper/33
372435399191,14cyclictest0-21swapper/33
372435399191,14cyclictest0-21swapper/33
372435399191,13cyclictest0-21swapper/33
372435399190,2cyclictest0-21swapper/33
16492191,7sleep13851964-21diskmemload1
40660082182,4sleep03851964-21diskmemload0
40020212182,5sleep24002046-21apt-get2
39936482181,7sleep13851964-21diskmemload1
38580502181,5sleep13858060-21ssh1
372435399183,6cyclictest0-21swapper/33
372435399183,5cyclictest0-21swapper/33
372435399182,7cyclictest0-21swapper/33
372435399182,13cyclictest66950irq/132-enp1s0-TxRx-33
3724353991812,3cyclictest0-21swapper/33
372435399181,14cyclictest0-21swapper/33
372435399181,14cyclictest0-21swapper/33
372435399181,14cyclictest0-21swapper/33
372435399181,13cyclictest0-21swapper/33
372435399181,13cyclictest0-21swapper/33
372435399181,13cyclictest0-21swapper/33
372435399180,3cyclictest38152272sleep33
372435399180,2cyclictest38046532sleep33
372435399180,2cyclictest37835322sleep33
372435399180,15cyclictest0-21swapper/33
41540762171,3sleep10-21swapper/11
40079992171,3sleep10-21swapper/11
39415622171,2sleep10-21swapper/11
38885482171,3sleep20-21swapper/22
38417572171,2sleep00-21swapper/00
38214012171,4sleep03822791-21cut0
372435399172,8cyclictest4093351-21taskset3
3724353991714,1cyclictest0-21swapper/33
372435399171,3cyclictest3851613-21chrt3
372435399171,13cyclictest0-21swapper/33
372435399171,13cyclictest0-21swapper/33
372435399171,11cyclictest3767990-21taskset3
372435399170,14cyclictest0-21swapper/33
372435399170,14cyclictest0-21swapper/33
372435399170,13cyclictest0-21swapper/33
372435399170,13cyclictest0-21swapper/33
372435399170,12cyclictest0-21swapper/33
41150432161,2sleep00-21swapper/00
408243821612,1sleep04082449-21sh0
39840272161,2sleep00-21swapper/00
38203232160,2sleep10-21swapper/11
372435399160,13cyclictest0-21swapper/33
31002161,2sleep20-21swapper/22
41635862151,4sleep23851964-21diskmemload2
40919872151,9sleep23851964-21diskmemload2
39846282150,1sleep20-21swapper/22
39076912151,2sleep00-21swapper/00
38998722152,2sleep0171rcu_preempt0
37687592151,9sleep03768969-21hddtemp_smartct0
372435399150,12cyclictest0-21swapper/33
41416702141,6sleep24134188-40sort2
40982902141,8sleep03851964-21diskmemload0
40337142141,8sleep13851964-21diskmemload1
39990692141,8sleep03851964-21diskmemload0
39684142141,6sleep03968504-21munin-run0
39523232141,6sleep10-21swapper/11
39523032141,5sleep03952322-21munin-node0
39439592141,4sleep03944217-21ssh0
38523522141,8sleep03852383-21ssh0
38517502140,1sleep10-21swapper/11
37431752141,8sleep23744809-21cut2
372435399149,3cyclictest0-21swapper/33
372435399147,4cyclictest3732265-21chrt3
372435399140,12cyclictest0-21swapper/33
372435399140,10cyclictest0-21swapper/33
41890042131,8sleep14189015-21sh1
41869832130,6sleep20-21swapper/22
41810882131,2sleep10-21swapper/11
41638362131,7sleep04163848-21ssh0
41312502131,7sleep03851964-21diskmemload0
41242262131,8sleep14124237-21ssh1
41153932131,8sleep24115404-21ssh2
41092162131,3sleep24109219-21ssh2
40926732131,8sleep14092684-21ssh1
40068682131,7sleep03851964-21diskmemload0
39330892131,8sleep23851964-21diskmemload2
38308942130,1sleep00-21swapper/00
37776822131,8sleep03777733-21/usr/sbin/munin0
37629542131,8sleep23763001-21true2
372435399130,11cyclictest0-21swapper/33
372435399130,10cyclictest38204552sleep33
41611952121,6sleep13851964-21diskmemload1
41486182121,5sleep24148884-21df2
41379112121,7sleep14134298-40find1
40745252121,6sleep13851964-21diskmemload1
40653542121,7sleep14065364-21ssh1
40495362121,6sleep03851964-21diskmemload0
40403952121,2sleep10-21swapper/11
40151672120,2sleep04015179-21ssh0
40143692121,7sleep24010251-21/usr/sbin/munin2
40140722121,7sleep14014361-21ssh1
39854992121,5sleep23985635-21diskstats2
39405062121,7sleep0106713-21snmpd0
39024692121,5sleep13902734-21df1
38770212121,7sleep03877032-21ssh0
38685102121,7sleep03868521-21ssh0
37515412121,5sleep13751594-21ntp_states1
16382121,7sleep03851964-21diskmemload0
41866352111,1sleep00-21swapper/00
41792322111,2sleep20-21swapper/22
41535362111,3sleep00-21swapper/00
41480132111,3sleep20-21swapper/22
41316272110,6sleep14131638-21ssh1
41104872111,6sleep14110530-21kthreadcore1
41062562111,6sleep04106273-21ssh0
41004172111,3sleep20-21swapper/22
40570322111,3sleep10-21swapper/11
40399462110,6sleep23851964-21diskmemload2
40236852111,2sleep04023696-21ssh0
39991372111,4sleep20-21swapper/22
39931062111,4sleep13851964-21diskmemload1
39757902111,4sleep00-21swapper/00
39579782110,2sleep20-21swapper/22
39519032111,5sleep23951958-21ssh2
39420032111,3sleep20-21swapper/22
39343272111,6sleep03934337-21ssh0
39163272111,6sleep23916338-21sh2
39097172111,3sleep20-21swapper/22
39008742111,3sleep20-21swapper/22
38821222111,3sleep10-21swapper/11
38469862111,5sleep23847005-21apt-get2
38417792111,3sleep10-21swapper/11
38348892111,6sleep23835246-21needreboot2
37890442110,2sleep2106713-21snmpd2
37844472111,4sleep03784656-21hddtemp_smartct0
37583092111,5sleep13758518-21hddtemp_smartct1
37298732111,6sleep23731334-21cut2
41755052101,4sleep04177212-21tr0
41723942101,4sleep13851964-21diskmemload1
41458032100,6sleep14145813-21ssh1
41455732101,4sleep03851964-21diskmemload0
41365732101,5sleep04136586-21sh0
41318742101,2sleep20-21swapper/22
40819852101,3sleep20-21swapper/22
40818802101,2sleep10-21swapper/11
40747442100,6sleep04074752-21ssh0
40598992101,2sleep20-21swapper/22
40582082101,3sleep23851964-21diskmemload2
40574772101,3sleep00-21swapper/00
40485972100,1sleep10-21swapper/11
40390832101,5sleep04039239-21grep0
40326122100,1sleep00-21swapper/00
40319812101,3sleep20-21swapper/22
40253122101,1sleep10-21swapper/11
40233132101,2sleep20-21swapper/22
39920962100,6sleep03992109-21sh0
39773192101,3sleep10-21swapper/11
39757562100,1sleep10-21swapper/11
39739162101,3sleep23851964-21diskmemload2
39516542100,2sleep13951657-21ssh1
39331282101,3sleep10-21swapper/11
39250632101,4sleep13851964-21diskmemload1
39191322101,2sleep00-21swapper/00
39169162104,2sleep00-21swapper/00
39150262101,5sleep13915102-21ssh1
38988392101,4sleep13851964-21diskmemload1
38920252101,1sleep00-21swapper/00
38911662101,2sleep10-21swapper/11
38847052101,3sleep20-21swapper/22
38826412101,3sleep03851964-21diskmemload0
38741102101,3sleep20-21swapper/22
38659102100,4sleep20-21swapper/22
38610402101,5sleep13861177-21grep1
38155412101,3sleep03815265-21apt-get0
38046162100,1sleep10-21swapper/11
37789212101,4sleep23778940-21apt-get2
37632292101,4sleep13763248-21apt-get1
37524842101,4sleep03752517-21munin-run0
37472232101,4sleep13747252-21munin-run1
37323542101,2sleep00-21swapper/00
37245302101,4sleep23724529-21apt-get2
4170451291,2sleep00-21swapper/00
4170362291,4sleep24170373-21ssh2
4123692291,3sleep03851964-21diskmemload0
4090182291,3sleep03851964-21diskmemload0
4048499291,2sleep20-21swapper/22
3831388291,4sleep13831559-21diskstats1
3789376291,3sleep10-21swapper/11
3768481291,3sleep13768483-21apt-get1
3763526290,1sleep00-21swapper/00
3753364291,4sleep03754908-21grep0
3742250291,4sleep03742272-21apt0
3737594291,4sleep03738666-21cut0
3724794291,4sleep03724891-21grep0
3837926281,3sleep03838603-21kthreadcore0
3836503281,2sleep23836522-21apt-get2
3830835280,1sleep10-21swapper/11
3830119281,3sleep23830242-21mailstats2
3810305280,2sleep00-21swapper/00
3805647281,3sleep13805847-21cut1
3804517281,3sleep03804520-21sshd-auth0
3784657280,1sleep10-21swapper/11
3783734280,1sleep10-21swapper/11
3778914281,3sleep03778688-21sendmail-msp0
3778315280,1sleep20-21swapper/22
3775156281,3sleep13776857-21grep1
3757501280,1sleep10-21swapper/11
3753055281,3sleep23753267-21smartctl2
3845819271,3sleep23845909-21cut2
3841176270,1sleep10-21swapper/11
3831843271,2sleep00-21swapper/00
3825484270,1sleep20-21swapper/22
3809657270,2sleep20-21swapper/22
3805091270,2sleep00-21swapper/00
3799271270,1sleep00-21swapper/00
3794894271,2sleep13795108-21smartctl1
3794595270,2sleep20-21swapper/22
3794327271,2sleep00-21swapper/00
3763251271,2sleep20-21swapper/22
3748762271,3sleep23750435-21kthreadcore2
3737005271,2sleep13724348-21cyclictest1
3735849271,3sleep13731782-21/usr/sbin/munin1
3820809260,1sleep10-21swapper/11
3819832261,2sleep20-21swapper/22
3814904260,1sleep10-21swapper/11
3799864260,2sleep23799883-21apt-get2
3790230261,2sleep20-21swapper/22
3768486260,3sleep23768742-21sh2
3741929260,1sleep20-21swapper/22
3729199260,4sleep10-21swapper/11
3851631250,3sleep00-21swapper/00
3810568250,2sleep20-21swapper/22
3757790250,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional