You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 02:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Mon Aug 17, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2879592212453,66sleep30-21swapper/33
287951728962,13sleep20-21swapper/22
287953028759,7sleep00-21swapper/00
287973927959,12sleep10-21swapper/11
287991399290,18cyclictest0-21swapper/33
287991399231,17cyclictest3206119-21taskset3
32445022224,5sleep23245955-21cut2
287991399229,10cyclictest30513832sleep33
2879913992215,3cyclictest3109248-21taskset3
32841892211,6sleep03284207-21sh0
33496242200,1sleep00-21swapper/00
31869432202,9sleep23187015-21/usr/sbin/munin2
30863992201,2sleep20-21swapper/22
287991399201,15cyclictest3104006-21turbostat3
287991399201,15cyclictest0-21swapper/33
287991399201,15cyclictest0-21swapper/33
31548922190,7sleep014-21ksoftirqd/00
287991399192,11cyclictest3354466-21taskset3
2879913991915,3cyclictest66950irq/132-enp1s0-TxRx-33
2879913991914,3cyclictest0-21swapper/33
2879913991912,5cyclictest0-21swapper/33
287991399191,15cyclictest0-21swapper/33
287991399191,14cyclictest0-21swapper/33
287991399191,14cyclictest0-21swapper/33
287991399191,14cyclictest0-21swapper/33
287991399191,14cyclictest0-21swapper/33
287991399191,13cyclictest0-21swapper/33
287991399190,13cyclictest0-21swapper/33
33198732180,11sleep10-21swapper/11
32172462181,11sleep13217312-21munin-run1
31451452181,12sleep23008070-21diskmemload2
31388872181,12sleep03138924-21munin-run0
31267312182,11sleep0106713-21snmpd0
30160912181,8sleep13008070-21diskmemload1
287991399181,7cyclictest3164982-21turbostat3
2879913991812,4cyclictest0-21swapper/33
2879913991811,4cyclictest2955006-21taskset3
287991399181,14cyclictest0-21swapper/33
287991399181,14cyclictest0-21swapper/33
287991399181,14cyclictest0-21swapper/33
287991399181,14cyclictest0-21swapper/33
287991399181,14cyclictest0-21swapper/33
287991399181,14cyclictest0-21swapper/33
287991399181,14cyclictest0-21swapper/33
287991399181,13cyclictest0-21swapper/33
287991399181,13cyclictest0-21swapper/33
287991399181,13cyclictest0-21swapper/33
287991399181,13cyclictest0-21swapper/33
287991399181,13cyclictest0-21swapper/33
287991399180,16cyclictest0-21swapper/33
287991399180,15cyclictest0-21swapper/33
287991399180,14cyclictest0-21swapper/33
287991399180,13cyclictest0-21swapper/33
2879913991714,1cyclictest0-21swapper/33
287991399171,3cyclictest0-21swapper/33
2879913991711,4cyclictest66950irq/132-enp1s0-TxRx-33
2879913991711,3cyclictest2908076-21turbostat3
287991399171,13cyclictest0-21swapper/33
287991399171,13cyclictest0-21swapper/33
287991399171,13cyclictest0-21swapper/33
287991399171,13cyclictest0-21swapper/33
287991399171,12cyclictest0-21swapper/33
287991399171,12cyclictest0-21swapper/33
287991399170,14cyclictest0-21swapper/33
287991399170,13cyclictest0-21swapper/33
33161232161,4sleep03316344-21hddtemp_smartct0
32977012161,7sleep13299194-21ssh1
32519132160,1sleep10-21swapper/11
32432882161,9sleep23008070-21diskmemload2
31201132161,4sleep13120128-21ssh1
31049972161,10sleep13105574-21cut1
287991399161,3cyclictest0-21swapper/33
2879913991613,1cyclictest0-21swapper/33
2879913991611,3cyclictest0-21swapper/33
287991399160,14cyclictest0-21swapper/33
287991399160,14cyclictest0-21swapper/33
287991399160,13cyclictest0-21swapper/33
287991399160,13cyclictest0-21swapper/33
287991399160,13cyclictest0-21swapper/33
33671192151,9sleep13008070-21diskmemload1
33119062151,4sleep23304338-21sendmail-msp2
32322922151,2sleep20-21swapper/22
31464052151,3sleep03008070-21diskmemload0
31130492151,4sleep03113060-21ssh0
30754242151,2sleep00-21swapper/00
30154782151,4sleep23008070-21diskmemload2
29142002151,5sleep22913358-21sendmail-msp2
287991399159,3cyclictest0-21swapper/33
287991399158,4cyclictest29156722sleep33
2879913991513,1cyclictest0-21swapper/33
2879913991513,1cyclictest0-21swapper/33
2879913991512,1cyclictest0-21swapper/33
287991399151,12cyclictest0-21swapper/33
287991399150,12cyclictest0-21swapper/33
287991399150,12cyclictest0-21swapper/33
287991399150,12cyclictest0-21swapper/33
33594512141,7sleep03359476-21ssh0
33410672141,7sleep20-21swapper/22
32753232141,9sleep13275338-21ssh1
32654332141,7sleep20-21swapper/22
32394562141,9sleep03239632-21grep0
32243282141,8sleep13008070-21diskmemload1
32067012140,9sleep03008070-21diskmemload0
31821602141,2sleep20-21swapper/22
31354042141,8sleep13008070-21diskmemload1
30768642141,8sleep23008070-21diskmemload2
30517502140,3sleep10-21swapper/11
30429132146,4sleep1331ktimers/11
30429132146,4sleep1331ktimers/11
2879913991411,2cyclictest0-21swapper/33
287991399141,10cyclictest0-21swapper/33
287991399140,2cyclictest0-21swapper/33
33387772131,2sleep10-21swapper/11
33236672131,8sleep23323681-21ssh2
32765232131,7sleep23008070-21diskmemload2
32306612131,2sleep00-21swapper/00
32001712131,7sleep23008070-21diskmemload2
31728202131,8sleep03172831-21ssh0
31660672131,8sleep13167492-21cat1
31545632131,8sleep23154578-21ssh2
30787202131,8sleep03078802-21ssh0
30655882131,2sleep00-21swapper/00
30561702132,7sleep03008070-21diskmemload0
30414592134,2sleep03041468-21sh0
30414592134,2sleep03041468-21sh0
29503812130,1sleep10-21swapper/11
33733772121,2sleep10-21swapper/11
33724912121,6sleep03372648-21/usr/sbin/munin0
33148672121,6sleep03008070-21diskmemload0
32694712120,3sleep13269485-21ssh1
32519972121,7sleep03252011-21sh0
32060482121,7sleep13206062-21ssh1
31970452121,7sleep03197057-21ssh0
31714092121,3sleep23171420-21ssh2
31467222121,7sleep13146736-21ssh1
31347252121,7sleep23134803-21ntp_states2
31296852121,6sleep23008070-21diskmemload2
30868782122,5sleep23086875-21apt-get2
30676242122,2sleep2171rcu_preempt2
30313412122,4sleep23008070-21diskmemload2
30222062121,7sleep23022219-21rm2
30155072121,2sleep00-21swapper/00
30020932121,7sleep03002296-21sed0
29719602121,2sleep10-21swapper/11
28850772121,7sleep21-21systemd2
33681752111,5sleep23368189-21ssh2
33562422111,3sleep00-21swapper/00
33119942110,6sleep13304558-40find1
32942872111,3sleep10-21swapper/11
32925802111,6sleep03292595-21ssh0
32350322111,6sleep10-21swapper/11
32236212111,2sleep23223635-21ssh2
32233342111,5sleep014-21ksoftirqd/00
31920562111,5sleep23192160-21irqstats2
31803602111,5sleep03008070-21diskmemload0
31639432111,6sleep23163955-21ssh2
31571802111,5sleep13157286-21irqstats1
31531682111,6sleep13153179-21sh1
31183342111,3sleep20-21swapper/22
31036982111,5sleep23008070-21diskmemload2
31026042111,3sleep10-21swapper/11
30347622111,5sleep23034975-21diskstats2
30347622111,5sleep23034975-21diskstats2
30312272111,6sleep13031238-21ssh1
30234132111,6sleep03008070-21diskmemload0
30232782111,6sleep13023288-21ssh1
29544332111,6sleep22954529-21proc_pri2
28814952111,6sleep12883170-21kthreadcore1
33490232101,2sleep10-21swapper/11
33470812101,4sleep23008070-21diskmemload2
33396492103,4sleep00-21swapper/00
33313252101,4sleep03008070-21diskmemload0
33293362101,4sleep13008070-21diskmemload1
33033172102,3sleep03008070-21diskmemload0
33012062101,3sleep20-21swapper/22
32915212101,4sleep23287300-21/usr/sbin/munin2
32838952101,2sleep10-21swapper/11
32760362101,2sleep00-21swapper/00
32675072101,3sleep03008070-21diskmemload0
32593662101,4sleep03008070-21diskmemload0
32593662101,4sleep03008070-21diskmemload0
32575922101,4sleep23008070-21diskmemload2
32575922101,4sleep23008070-21diskmemload2
32158242101,5sleep23215835-21ssh2
32089672101,3sleep00-21swapper/00
31827882101,4sleep13008070-21diskmemload1
31757722101,5sleep13176302-21kthreadcore1
31623252101,2sleep00-21swapper/00
31217692101,3sleep10-21swapper/11
31122832101,2sleep00-21swapper/00
31104192100,5sleep23008070-21diskmemload2
30945342101,4sleep13008070-21diskmemload1
30924292101,3sleep00-21swapper/00
30863442101,5sleep13086358-21ssh1
30756332100,2sleep10-21swapper/11
30607242101,3sleep10-21swapper/11
30584552101,3sleep10-21swapper/11
30550292101,4sleep23056017-21memory2
30499572101,3sleep00-21swapper/00
30497782101,3sleep23008070-21diskmemload2
30265242101,6sleep03027599-21kthreadcore0
29927922100,2sleep00-21swapper/00
29819882101,4sleep02981708-21apt-get0
29810662101,4sleep02981310-21users0
29757752101,4sleep02975992-21users0
29668442101,5sleep02968413-21kthreadcore0
29556372100,2sleep00-21swapper/00
29296442101,5sleep12929854-21hddtemp_smartct1
29228742101,5sleep12922938-21grep1
3364576292,3sleep23364579-21ssh2
3358734290,1sleep10-21swapper/11
3352819292,3sleep20-21swapper/22
3324873291,3sleep23325093-21hddtemp_smartct2
3285851291,4sleep23285863-21ssh2
3256877291,3sleep13008070-21diskmemload1
3256877291,3sleep13008070-21diskmemload1
3226089290,1sleep10-21swapper/11
3197629291,5sleep13197640-21ssh1
3183110291,3sleep03008070-21diskmemload0
3102179291,3sleep03008070-21diskmemload0
2998357291,4sleep22997545-21sendmail-msp2
2987757291,4sleep12987828-21irqstats1
2975482291,4sleep22975605-21/usr/sbin/munin2
2962229291,4sleep12963903-21tr1
2955664291,2sleep10-21swapper/11
2946104290,5sleep02947729-21kthreadcore0
2945474291,4sleep22945682-21if_enp1s02
2919994291,4sleep22920980-21kthreadcore2
2914097291,4sleep12914197-21sed1
2903407290,1sleep10-21swapper/11
2884936290,1sleep00-21swapper/00
2997759282,2sleep10-21swapper/11
2992522280,2sleep20-21swapper/22
2991870281,3sleep22987257-21/usr/sbin/munin2
2983847280,4sleep12985579-21kthreadcore1
2978135281,4sleep22979672-21kthreadcore2
2977239281,3sleep12978134-21grep1
2960962280,3sleep02960958-21apt-get0
2955634281,3sleep22955415-21sendmail-msp2
2940069280,3sleep12940192-21diskstats1
2926156281,3sleep02926214-21kthreadcore0
2924114280,3sleep22924133-21apt-get2
2897880281,2sleep10-21swapper/11
2892584281,2sleep22892603-21apt-get2
2887899281,3sleep12887934-21kthreadcore1
2887899281,3sleep12887934-21kthreadcore1
2885551283,2sleep00-21swapper/00
2885551283,2sleep00-21swapper/00
2997223270,1sleep10-21swapper/11
2970866270,1sleep20-21swapper/22
2968417271,2sleep12970020-21latency1
2965547270,1sleep20-21swapper/22
2944668270,1sleep20-21swapper/22
2944538270,1sleep00-21swapper/00
2938547271,2sleep20-21swapper/22
2935839271,3sleep00-21swapper/00
2934607271,2sleep10-21swapper/11
2934027270,1sleep00-21swapper/00
2929349270,2sleep20-21swapper/22
2928381271,2sleep10-21swapper/11
2923590270,1sleep00-21swapper/00
2917269271,3sleep02917634-21grep0
2913137270,1sleep20-21swapper/22
2908334270,2sleep00-21swapper/00
2903115270,2sleep20-21swapper/22
2902649270,1sleep20-21swapper/22
2892876271,3sleep10-21swapper/11
2892739270,3sleep02892875-21diskstats0
2888992270,4sleep20-21swapper/22
2888992270,4sleep20-21swapper/22
3007860260,4sleep10-21swapper/11
3003696260,3sleep00-21swapper/00
2950369260,1sleep00-21swapper/00
2913156260,1sleep10-21swapper/11
2905106261,2sleep00-21swapper/00
3003079250,1sleep20-21swapper/22
2901564250,1sleep00-21swapper/00
2987547240,1sleep00-21swapper/00
2982291240,1sleep20-21swapper/22
2945427240,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional