You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 16:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Sep 02, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2513480211758,55sleep30-21swapper/33
251376228559,10sleep10-21swapper/11
251348928050,12sleep20-21swapper/22
251334528052,17sleep00-21swapper/00
27969732330,5sleep2431ktimers/22
28625382300,3sleep1321rcuc/11
251396699251,9cyclictest2820558-21taskset3
29402512231,18sleep12940263-21ssh1
2513966992219,1cyclictest0-21swapper/33
251396699221,17cyclictest2589183-21taskset3
25271462211,9sleep22526382-21sendmail-msp2
251396699215,13cyclictest0-21swapper/33
251396699211,17cyclictest0-21swapper/33
251396699211,16cyclictest66950irq/132-enp1s0-TxRx-33
251396699202,14cyclictest2934084-21turbostat3
251396699201,15cyclictest0-21swapper/33
26735792191,3sleep20-21swapper/22
251396699192,12cyclictest2822831-21taskset3
251396699192,12cyclictest2822831-21taskset3
251396699191,15cyclictest2642072-21taskset3
251396699191,15cyclictest2642072-21taskset3
251396699191,15cyclictest0-21swapper/33
251396699191,14cyclictest0-21swapper/33
251396699191,14cyclictest0-21swapper/33
251396699191,14cyclictest0-21swapper/33
251396699190,16cyclictest0-21swapper/33
251396699190,15cyclictest66950irq/132-enp1s0-TxRx-33
28851452181,12sleep22886882-21grep2
28142722181,2sleep10-21swapper/11
25908142181,13sleep22589578-21sendmail-msp2
2513966991815,1cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,14cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,13cyclictest0-21swapper/33
251396699181,12cyclictest0-21swapper/33
2513966991810,4cyclictest0-21swapper/33
251396699180,15cyclictest66950irq/132-enp1s0-TxRx-33
251396699180,15cyclictest2707765-21chrt3
251396699180,15cyclictest0-21swapper/33
29430872171,9sleep00-21swapper/00
28035372171,3sleep10-21swapper/11
251396699179,5cyclictest2552753-21turbostat3
2513966991714,2cyclictest0-21swapper/33
251396699171,13cyclictest0-21swapper/33
251396699171,13cyclictest0-21swapper/33
251396699170,14cyclictest0-21swapper/33
251396699170,13cyclictest0-21swapper/33
251396699170,13cyclictest0-21swapper/33
29665752161,2sleep22642093-21diskmemload2
29644662161,10sleep12964599-21find1
29583682160,7sleep22958380-21ssh2
28551932160,3sleep10-21swapper/11
28386382161,9sleep02642093-21diskmemload0
28036072161,2sleep20-21swapper/22
28020452161,3sleep039542-21systemd-journal0
27942062161,4sleep12642093-21diskmemload1
27708272161,3sleep00-21swapper/00
251396699161,3cyclictest0-21swapper/33
251396699161,3cyclictest0-21swapper/33
2513966991612,2cyclictest0-21swapper/33
2513966991612,2cyclictest0-21swapper/33
251396699161,12cyclictest0-21swapper/33
251396699160,13cyclictest0-21swapper/33
251396699160,13cyclictest0-21swapper/33
251396699160,13cyclictest0-21swapper/33
29087062150,2sleep00-21swapper/00
27615962151,10sleep02761611-21ssh0
27341182151,4sleep02642093-21diskmemload0
26568502150,3sleep00-21swapper/00
2513966991512,1cyclictest0-21swapper/33
2513966991511,2cyclictest0-21swapper/33
251396699150,3cyclictest0-21swapper/33
251396699150,13cyclictest0-21swapper/33
251396699150,13cyclictest0-21swapper/33
29859082140,6sleep20-21swapper/22
29829542141,7sleep22642093-21diskmemload2
28628802141,8sleep02862890-21ssh0
27628192141,7sleep22763086-21df2
26840242141,4sleep12684037-21ssh1
26563162141,9sleep10-21swapper/11
25684902140,2sleep20-21swapper/22
251396699141,2cyclictest0-21swapper/33
2513966991412,1cyclictest0-21swapper/33
251396699141,10cyclictest2542432-21chrt3
251396699140,11cyclictest0-21swapper/33
29952032131,4sleep20-21swapper/22
29832512131,8sleep12983261-21ssh1
29728562131,9sleep00-21swapper/00
29687372131,7sleep22968482-21sendmail-msp2
29681332131,8sleep02968148-21ssh0
29419882131,3sleep00-21swapper/00
29308022130,6sleep10-21swapper/11
29174042131,6sleep02917416-21ssh0
29061172131,5sleep10-21swapper/11
28746902130,8sleep02874724-21grep0
27841722131,8sleep02784242-21/usr/sbin/munin0
27714262130,2sleep00-21swapper/00
27351892131,7sleep22642093-21diskmemload2
27239952131,6sleep12724076-21ntp_states1
27094362131,8sleep12709450-21ssh1
27089252131,6sleep00-21swapper/00
26499572131,7sleep22642093-21diskmemload2
25740962131,7sleep02574098-21apt-get0
25266102131,8sleep12526629-21apt-get1
251396699137,3cyclictest25682662sleep33
2513966991310,2cyclictest0-21swapper/33
28732702121,7sleep02873281-21ssh0
28655642121,5sleep22865329-21sendmail-msp2
28231282121,7sleep12823348-21/usr/sbin/munin1
28231282121,7sleep12823348-21/usr/sbin/munin1
28228042121,5sleep22642093-21diskmemload2
28228042121,5sleep22642093-21diskmemload2
28206522120,3sleep22820666-21ssh2
27934512121,7sleep02642093-21diskmemload0
27777282121,6sleep12642093-21diskmemload1
27162132121,5sleep20-21swapper/22
27106502120,1sleep20-21swapper/22
26834652120,8sleep02683476-21ssh0
26769122120,1sleep20-21swapper/22
26199702121,7sleep22620109-21/usr/sbin/munin2
26003442121,5sleep22600356-21apt-get2
25793002120,2sleep20-21swapper/22
2513966991210,1cyclictest0-21swapper/33
29432782111,4sleep22943411-21diskstats2
29137272111,5sleep22642093-21diskmemload2
28976502111,4sleep22642093-21diskmemload2
28803262111,2sleep10-21swapper/11
28635952111,6sleep22863607-21sh2
28559722111,3sleep00-21swapper/00
28548992113,3sleep2431ktimers/22
28481662110,7sleep22848431-21date2
28454152111,3sleep10-21swapper/11
28370192111,4sleep20-21swapper/22
28125842111,4sleep22642093-21diskmemload2
28114242110,2sleep10-21swapper/11
27870262111,6sleep22787037-21ssh2
27869062111,3sleep10-21swapper/11
27696442111,5sleep12642093-21diskmemload1
27611162110,2sleep10-21swapper/11
27415422111,5sleep02741787-21ssh0
27335262110,1sleep10-21swapper/11
27260482111,4sleep00-21swapper/00
26993122110,7sleep22699323-21ssh2
26860882111,6sleep12686142-21kthreadcore1
26854722111,4sleep02685500-21apt-get0
26751752111,3sleep00-21swapper/00
26598712111,5sleep22659330-21sendmail-msp2
26492622111,3sleep10-21swapper/11
26305572111,5sleep22630627-21ntp_states2
26108402111,6sleep02610841-21http0
25956372111,6sleep12595900-21kthreadcore1
25795872111,6sleep12579708-21fschecks_time1
25734452110,2sleep00-21swapper/00
25729142111,6sleep22573008-21processes2
25693412113,4sleep12569908-21tr1
25424092111,5sleep12542420-21sort1
25319832111,5sleep12532199-21df_inode1
25257422111,5sleep12525927-21users1
29995672101,4sleep02642093-21diskmemload0
29925802101,2sleep10-21swapper/11
29914792101,3sleep00-21swapper/00
29815452100,1sleep00-21swapper/00
29749562101,3sleep10-21swapper/11
29546562101,4sleep12955634-21memory1
29530662101,6sleep02953805-21kthreadcore0
29350922101,5sleep22934126-21sendmail-msp2
29329802101,4sleep02932991-21ssh0
29323082101,3sleep22642093-21diskmemload2
29250622100,6sleep12925076-21ssh1
29051382101,3sleep22642093-21diskmemload2
28972882101,3sleep00-21swapper/00
28918482101,4sleep12642093-21diskmemload1
28891922101,2sleep10-21swapper/11
28723472101,4sleep12642093-21diskmemload1
28377052101,3sleep10-21swapper/11
28190962101,4sleep02642093-21diskmemload0
28057432101,5sleep02806007-21df0
27775952101,4sleep22642093-21diskmemload2
27608332101,4sleep22642093-21diskmemload2
27522332103,3sleep22642093-21diskmemload2
27510362101,3sleep02642093-21diskmemload0
27498902101,4sleep12750055-21systemctl1
27370412100,1sleep20-21swapper/22
27370292101,4sleep12642093-21diskmemload1
27276312101,5sleep22727644-21ssh2
27154322101,4sleep12715495-21ntp_states1
26940982101,4sleep02694363-21df0
26655702101,3sleep10-21swapper/11
26496802101,2sleep00-21swapper/00
26420762101,4sleep22642106-21munin-run2
26420762101,4sleep22642106-21munin-run2
26371362101,4sleep02637393-21df0
26371362101,4sleep02637393-21df0
26362062101,5sleep02636449-21head0
26218982101,5sleep22623228-21kthreadcore2
26055742101,3sleep22605593-21apt-get2
25845642101,5sleep22584682-21tr2
25741012100,1sleep10-21swapper/11
25477392101,4sleep12547751-21apt-get1
25307832100,2sleep0171rcu_preempt0
2999149291,3sleep12642093-21diskmemload1
2943712291,4sleep12944906-21cut1
2904449291,4sleep039542-21systemd-journal0
2874404291,2sleep20-21swapper/22
2845349291,2sleep00-21swapper/00
2828162290,1sleep00-21swapper/00
2828162290,1sleep00-21swapper/00
2716295291,3sleep00-21swapper/00
2698896291,3sleep12642093-21diskmemload1
2686691291,5sleep22688243-21kthreadcore2
2673513291,2sleep10-21swapper/11
2665809291,3sleep02642093-21diskmemload0
2657123291,3sleep22642093-21diskmemload2
2641879290,5sleep10-21swapper/11
2641879290,5sleep10-21swapper/11
2635888291,4sleep12635945-21ntp_states1
2631813291,3sleep22631589-21sendmail-msp2
2631516290,1sleep00-21swapper/00
2621324291,3sleep12621343-21apt-get1
2600336290,1sleep10-21swapper/11
2590359291,4sleep12590813-21cat1
2565451291,4sleep02567182-21kthreadcore0
2557995291,4sleep02558007-21latency_hist0
2553029291,2sleep10-21swapper/11
2552365291,3sleep22547473-21sendmail-msp2
2542329290,1sleep00-21swapper/00
2515144291,4sleep02516757-21grep0
2917357280,2sleep20-21swapper/22
2911380280,4sleep10-21swapper/11
2882982281,2sleep00-21swapper/00
2620294281,3sleep12620464-21cat1
2612921281,4sleep22610591-21sendmail-msp2
2606930281,3sleep02608606-21kthreadcore0
2595101281,1sleep00-21swapper/00
2589804281,3sleep02589823-21apt-get0
2589446280,2sleep172-21kcompactd01
2553008281,3sleep02553027-21apt-get0
2548029283,2sleep00-21swapper/00
2542939281,3sleep22543200-21kthreadcore2
2537052280,1sleep10-21swapper/11
2531935281,3sleep02531982-21grep0
2518776280,1sleep20-21swapper/22
2630432270,4sleep10-21swapper/11
2617475271,3sleep00-21swapper/00
2615806270,1sleep10-21swapper/11
2599851270,1sleep20-21swapper/22
2580615271,2sleep00-21swapper/00
2568444270,1sleep10-21swapper/11
2560136271,3sleep12561869-21grep1
2553937271,3sleep20-21swapper/22
2541372270,3sleep2421rcuc/22
2537722271,3sleep00-21swapper/00
2536507271,3sleep2106713-21snmpd2
2518829270,1sleep10-21swapper/11
2621313260,4sleep00-21swapper/00
2610339260,4sleep10-21swapper/11
2602579261,2sleep00-21swapper/00
2578814260,1sleep20-21swapper/22
2558557260,1sleep20-21swapper/22
2519252260,2sleep00-21swapper/00
2587624250,1sleep00-21swapper/00
2520712250,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional