You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 12:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu May 28, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2960951212953,71sleep30-21swapper/33
296085328654,12sleep20-21swapper/22
296072227961,12sleep10-21swapper/11
296082127859,14sleep00-21swapper/00
296120999291,3cyclictest0-21swapper/33
296120999271,3cyclictest0-21swapper/33
296120999271,3cyclictest0-21swapper/33
296120999261,14cyclictest0-21swapper/33
31102382250,2sleep00-21swapper/00
296120999250,14cyclictest0-21swapper/33
32382672240,3sleep20-21swapper/22
33099232231,10sleep23092016-21diskmemload2
296120999231,15cyclictest0-21swapper/33
296120999231,12cyclictest0-21swapper/33
296120999230,20cyclictest511irq_work/33
296120999222,17cyclictest3256021-21taskset3
296120999220,2cyclictest0-21swapper/33
31947522211,3sleep00-21swapper/00
296120999215,8cyclictest0-21swapper/33
2961209992112,7cyclictest0-21swapper/33
296120999211,15cyclictest0-21swapper/33
296120999211,15cyclictest0-21swapper/33
296120999210,13cyclictest66950irq/132-enp1s0-TxRx-33
33873632201,3sleep20-21swapper/22
31359092202,8sleep13135955-21systemd-run1
31228582201,5sleep03123009-21ntpq0
296120999204,13cyclictest0-21swapper/33
296120999203,14cyclictest0-21swapper/33
2961209992017,2cyclictest0-21swapper/33
2961209992014,4cyclictest0-21swapper/33
296120999201,16cyclictest0-21swapper/33
296120999201,16cyclictest0-21swapper/33
296120999201,15cyclictest66950irq/132-enp1s0-TxRx-33
296120999201,15cyclictest0-21swapper/33
296120999201,15cyclictest0-21swapper/33
296120999200,12cyclictest0-21swapper/33
34480372191,14sleep03671801-21sshd0
34317302191,5sleep00-21swapper/00
33785072191,8sleep23378517-21ssh2
33287962191,14sleep13329203-21date1
33187702190,9sleep014-21ksoftirqd/00
31680682191,6sleep03092016-21diskmemload0
2961209991914,3cyclictest0-21swapper/33
296120999191,3cyclictest0-21swapper/33
296120999191,3cyclictest0-21swapper/33
296120999191,3cyclictest0-21swapper/33
2961209991912,5cyclictest0-21swapper/33
2961209991912,4cyclictest0-21swapper/33
296120999191,16cyclictest0-21swapper/33
296120999191,15cyclictest0-21swapper/33
296120999191,14cyclictest0-21swapper/33
296120999191,14cyclictest0-21swapper/33
296120999191,14cyclictest0-21swapper/33
296120999191,14cyclictest0-21swapper/33
296120999191,14cyclictest0-21swapper/33
2961209991910,6cyclictest3399750-21taskset3
296120999190,15cyclictest34099552sleep33
296120999190,10cyclictest0-21swapper/33
31964162181,7sleep13196427-21ssh1
30062412182,11sleep03006260-21apt-get0
296120999189,7cyclictest0-21swapper/33
296120999184,11cyclictest3059931-21turbostat3
296120999181,7cyclictest2995354-21turbostat3
296120999181,5cyclictest0-21swapper/33
296120999181,3cyclictest0-21swapper/33
296120999181,3cyclictest0-21swapper/33
296120999181,3cyclictest0-21swapper/33
296120999181,3cyclictest0-21swapper/33
296120999181,13cyclictest66950irq/132-enp1s0-TxRx-33
296120999181,13cyclictest0-21swapper/33
296120999180,2cyclictest0-21swapper/33
34588532170,2sleep10-21swapper/11
34239392171,4sleep03423953-21ssh0
33353932171,1sleep10-21swapper/11
31711952171,4sleep03092016-21diskmemload0
31339982171,2sleep20-21swapper/22
31133552171,12sleep13114138-21sort1
30749202171,10sleep23074966-21ntpq2
296120999175,3cyclictest0-21swapper/33
296120999171,6cyclictest3005998-21turbostat3
296120999171,6cyclictest2979370-21turbostat3
296120999171,4cyclictest0-21swapper/33
2961209991712,3cyclictest0-21swapper/33
296120999170,1cyclictest0-21swapper/33
34508292161,4sleep23092016-21diskmemload2
33443062161,5sleep13092016-21diskmemload1
32906362161,3sleep00-21swapper/00
32844482161,3sleep10-21swapper/11
31253102161,3sleep10-21swapper/11
31177682161,3sleep20-21swapper/22
296120999163,5cyclictest66950irq/132-enp1s0-TxRx-33
296120999163,2cyclictest0-21swapper/33
296120999162,9cyclictest3028139-21taskset3
296120999161,12cyclictest3092041-21sleep3
296120999160,13cyclictest0-21swapper/33
34590702151,5sleep23459085-21ssh2
34223772153,4sleep1171rcu_preempt1
34140972151,8sleep03092016-21diskmemload0
34058112151,9sleep03092016-21diskmemload0
33374692151,4sleep23092016-21diskmemload2
32989322151,2sleep10-21swapper/11
32748462150,1sleep10-21swapper/11
32593572150,6sleep03259465-21ps0
32288512156,5sleep23228863-21sh2
32288512156,5sleep23228863-21sh2
31866242150,1sleep00-21swapper/00
31861472151,3sleep23092016-21diskmemload2
31755622151,10sleep13175618-21ssh1
31607332151,4sleep23092016-21diskmemload2
29671152157,4sleep02968306-21cut0
296120999150,13cyclictest0-21swapper/33
296120999150,13cyclictest0-21swapper/33
296120999150,12cyclictest0-21swapper/33
296120999150,11cyclictest0-21swapper/33
34408962147,4sleep12961203-21cyclictest1
34335942141,3sleep13092016-21diskmemload1
33973292140,2sleep20-21swapper/22
33806492142,8sleep03092016-21diskmemload0
33717692140,9sleep03092016-21diskmemload0
32045232141,3sleep10-21swapper/11
31868992141,8sleep13186909-21ssh1
31628252141,9sleep23163032-21tail2
31590522147,4sleep13159065-21sh1
31258822141,7sleep23092016-21diskmemload2
29784482142,8sleep12978514-21/usr/sbin/munin1
34183272132,6sleep23419919-21cut2
33577762131,8sleep23359517-21kthreadcore2
33540672131,3sleep13092016-21diskmemload1
32902102131,4sleep23290213-21ssh2
32321692131,4sleep13232290-21cat1
32321692131,4sleep13232290-21cat1
32123112131,8sleep03092016-21diskmemload0
30588622132,6sleep03058910-21ntpq0
29890952131,9sleep12989150-21awk1
29783962130,4sleep00-21swapper/00
34605552121,5sleep00-21swapper/00
34495522121,7sleep13449562-21ssh1
34395542121,7sleep03439687-21cat0
33896952121,3sleep13389706-21ssh1
33610042120,4sleep00-21swapper/00
33261082120,4sleep00-21swapper/00
33168372122,3sleep2171rcu_preempt2
33081522121,7sleep13308165-21rm1
32745472121,7sleep23274558-21ssh2
32740182122,4sleep0151ktimers/00
32472142121,7sleep13247227-21rm1
32113612121,7sleep13211370-21sh1
31888192121,5sleep23188843-21apt-get2
31701932121,6sleep13092016-21diskmemload1
31526952120,1sleep10-21swapper/11
31421162121,3sleep23142130-21ssh2
31092832121,6sleep10-21swapper/11
34350222112,4sleep23435048-21apt-get2
34343702111,2sleep20-21swapper/22
33983222111,4sleep10-21swapper/11
33882862111,3sleep00-21swapper/00
33718502110,4sleep20-21swapper/22
33469302110,2sleep20-21swapper/22
33442402111,4sleep03092016-21diskmemload0
33344162111,4sleep03092016-21diskmemload0
33082872111,6sleep03308301-21ssh0
32818482111,5sleep03092016-21diskmemload0
32402132111,3sleep00-21swapper/00
31771152111,3sleep20-21swapper/22
31450052111,4sleep23145033-21apt-get2
31363092111,5sleep13136331-21apt-get1
31330142111,6sleep03133028-21ssh0
30977112111,3sleep10-21swapper/11
30959962110,7sleep03092661-21/usr/sbin/munin0
30932732111,2sleep20-21swapper/22
30909822111,5sleep23091023-21ntpq2
30856882111,5sleep23085746-21ntpq2
30660002111,5sleep13066101-21irqstats1
30589122110,7sleep13058967-21ntpq1
30482472111,4sleep23048295-21ntpq2
30375182112,4sleep03037566-21ntpq0
29966062111,6sleep02998317-21grep0
29872932111,6sleep02984951-21/usr/sbin/munin0
29782012111,4sleep22978342-21ntpq2
34145222101,3sleep10-21swapper/11
34091452101,5sleep23409195-21kthreadcore2
34053302101,2sleep20-21swapper/22
33992192101,4sleep03092016-21diskmemload0
33782362101,3sleep10-21swapper/11
33701072101,2sleep10-21swapper/11
33612092101,3sleep10-21swapper/11
33490452101,6sleep03350680-21kthreadcore0
33380692101,2sleep20-21swapper/22
33268042100,1sleep20-21swapper/22
33183222100,3sleep10-21swapper/11
33015592101,3sleep20-21swapper/22
32933692101,4sleep13092016-21diskmemload1
32770522101,4sleep23277125-21irqstats2
32663672101,4sleep13092016-21diskmemload1
32645712101,3sleep20-21swapper/22
32562362101,4sleep03092016-21diskmemload0
32554302101,3sleep10-21swapper/11
32500152101,2sleep20-21swapper/22
32468972100,2sleep00-21swapper/00
32402652101,3sleep10-21swapper/11
32279062101,4sleep03228006-21ntpq0
32279062101,4sleep03228006-21ntpq0
32225652101,3sleep10-21swapper/11
32205892100,1sleep00-21swapper/00
32192582101,5sleep23092016-21diskmemload2
32146022100,3sleep20-21swapper/22
32048912101,3sleep20-21swapper/22
32017582100,6sleep03201825-21collect20
31613202101,2sleep00-21swapper/00
31501202101,3sleep00-21swapper/00
31431482101,4sleep03092016-21diskmemload0
31065012101,4sleep03092016-21diskmemload0
30815622101,4sleep03081581-21apt-get0
30738642104,3sleep00-21swapper/00
30695892101,5sleep23069638-21ntpq2
30540052101,4sleep13054197-21users1
30437052101,3sleep272-21kcompactd02
30321812101,4sleep23032227-21ntpq2
30320672101,4sleep13032180-21ntpq1
30232812101,4sleep23023352-21irqstats2
30009162101,4sleep23000935-21apt-get2
29801802100,6sleep22980973-21cut2
29689422100,4sleep12968961-21apt-get1
29656272101,4sleep12965669-21ntpq1
3404048291,4sleep13404113-21modprobe1
3248044290,3sleep20-21swapper/22
3082165291,5sleep13083732-21kthreadcore1
3080332291,4sleep23080389-21/usr/sbin/munin2
3059218291,4sleep23059312-21sed2
3049486291,3sleep23049505-21apt-get2
3033477291,4sleep13033454-21/usr/sbin/munin1
3024598291,4sleep03026393-21grep0
3022254291,4sleep03022366-21awk0
3005292292,3sleep03005447-21cat0
2966081291,4sleep02966298-21users0
2965677290,4sleep22965726-21ntpq2
3299173281,3sleep03299185-21ssh0
3105608282,3sleep2431ktimers/22
3060872281,3sleep03062534-21cut0
3051654281,4sleep00-21swapper/00
3042942281,3sleep03042996-21ld0
2999966280,1sleep10-21swapper/11
2994212281,3sleep02994325-21ntpq0
2985458281,3sleep22985613-21kthreadcore2
2984557280,1sleep10-21swapper/11
3076196270,2sleep00-21swapper/00
3049179270,1sleep00-21swapper/00
3044162270,2sleep10-21swapper/11
3026394271,3sleep13026792-21needreboot1
3017164270,1sleep10-21swapper/11
3016615271,3sleep10-21swapper/11
3016414270,1sleep20-21swapper/22
3010297271,3sleep20-21swapper/22
3006263271,2sleep13006418-21cstates1
3000905270,2sleep10-21swapper/11
2990277270,2sleep10-21swapper/11
2968964271,3sleep22969151-21cstates2
3091134261,2sleep10-21swapper/11
3076768260,3sleep1321rcuc/11
3074782260,2sleep10-21swapper/11
3065822261,2sleep00-21swapper/00
3042171261,2sleep10-21swapper/11
3022061260,1sleep20-21swapper/22
3012113261,2sleep00-21swapper/00
3000443260,1sleep20-21swapper/22
2991679261,2sleep20-21swapper/22
3091943250,2sleep00-21swapper/00
3062535250,2sleep10-21swapper/11
3033236250,1sleep20-21swapper/22
3033148250,1sleep00-21swapper/00
3060488240,1sleep20-21swapper/22
2979391240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional