You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 04:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri Mar 27, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2613882212852,71sleep30-21swapper/33
261367028567,12sleep20-21swapper/22
261359628464,11sleep10-21swapper/11
261385428161,14sleep00-21swapper/00
261402599230,18cyclictest29253172sleep33
261402599221,17cyclictest2663805-21taskset3
261402599221,17cyclictest2663805-21taskset3
29547242211,9sleep12954947-21hddtemp_smartct1
2614025992117,2cyclictest0-21swapper/33
2614025992116,3cyclictest0-21swapper/33
2614025992112,5cyclictest0-21swapper/33
2614025992111,7cyclictest2845829-21chrt3
2614025992111,6cyclictest66950irq/132-enp1s0-TxRx-33
2614025992016,2cyclictest0-21swapper/33
261402599201,15cyclictest0-21swapper/33
261402599200,1cyclictest0-21swapper/33
302393921913,2sleep23014860-21kworker/2:0+events_freezable2
261402599191,6cyclictest3015747-21taskset3
2614025991916,1cyclictest0-21swapper/33
261402599191,15cyclictest2977081-21taskset3
261402599191,15cyclictest0-21swapper/33
261402599191,15cyclictest0-21swapper/33
261402599191,15cyclictest0-21swapper/33
2614025991911,4cyclictest28260892sleep33
261402599191,14cyclictest0-21swapper/33
261402599191,14cyclictest0-21swapper/33
2614025991910,6cyclictest2838434-21taskset3
2614025991910,6cyclictest2812035-21taskset3
261402599190,15cyclictest66950irq/132-enp1s0-TxRx-33
261402599190,15cyclictest3005063-21taskset3
28393682181,3sleep00-21swapper/00
27582212181,6sleep12758287-21ntpq1
261402599188,6cyclictest2696157-21turbostat3
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,14cyclictest0-21swapper/33
261402599181,13cyclictest0-21swapper/33
261402599181,13cyclictest0-21swapper/33
261402599181,12cyclictest2744635-21sleep3
261402599180,15cyclictest0-21swapper/33
261402599180,15cyclictest0-21swapper/33
261402599180,15cyclictest0-21swapper/33
261402599180,14cyclictest0-21swapper/33
30314482171,3sleep00-21swapper/00
29763942171,4sleep12744611-21diskmemload1
29246262171,3sleep00-21swapper/00
28350632171,3sleep20-21swapper/22
28126552171,2sleep00-21swapper/00
261402599179,6cyclictest66950irq/132-enp1s0-TxRx-33
261402599171,14cyclictest0-21swapper/33
2614025991711,3cyclictest0-21swapper/33
261402599171,13cyclictest0-21swapper/33
261402599171,12cyclictest0-21swapper/33
261402599170,13cyclictest0-21swapper/33
261402599170,13cyclictest0-21swapper/33
261402599170,13cyclictest0-21swapper/33
29774992161,2sleep00-21swapper/00
28474012168,4sleep22847415-21ssh2
28154142160,1sleep20-21swapper/22
261402599169,2cyclictest0-21swapper/33
261402599161,12cyclictest0-21swapper/33
261402599161,12cyclictest0-21swapper/33
261402599161,12cyclictest0-21swapper/33
261402599161,12cyclictest0-21swapper/33
261402599160,13cyclictest0-21swapper/33
261402599160,13cyclictest0-21swapper/33
30828172151,8sleep20-21swapper/22
261402599151,2cyclictest0-21swapper/33
2614025991512,2cyclictest0-21swapper/33
2614025991511,2cyclictest0-21swapper/33
261402599151,12cyclictest0-21swapper/33
2614025991510,2cyclictest0-21swapper/33
261402599150,13cyclictest0-21swapper/33
261402599150,13cyclictest0-21swapper/33
261402599150,12cyclictest0-21swapper/33
261402599150,11cyclictest0-21swapper/33
30691042141,8sleep23069251-21/usr/sbin/munin2
30214312140,9sleep12744611-21diskmemload1
29699652141,3sleep00-21swapper/00
28672162142,3sleep00-21swapper/00
28459002142,2sleep1171rcu_preempt1
27976982141,2sleep20-21swapper/22
261402599140,2cyclictest0-21swapper/33
30769782131,8sleep13077046-21grep1
30325652131,1sleep20-21swapper/22
29984982131,8sleep02998613-21perl0
29806342131,7sleep22744611-21diskmemload2
29799592131,8sleep22979973-21ssh2
29545892131,8sleep02954723-21diskstats0
29349782131,7sleep22934992-21ssh2
28810642131,8sleep22881078-21ssh2
28089712130,1sleep20-21swapper/22
27944242131,7sleep12744611-21diskmemload1
27684102131,8sleep22768422-21rm2
27435912130,8sleep12743632-21ntpq1
27186072132,7sleep22718809-21cat2
26446222131,8sleep22646398-21cut2
2614025991311,1cyclictest0-21swapper/33
261402599130,12cyclictest0-21swapper/33
30832592121,6sleep02744611-21diskmemload0
30602242121,7sleep23060338-21ssh2
30568552120,1sleep20-21swapper/22
30131192121,5sleep00-21swapper/00
30085802121,7sleep13010242-21cut1
29654232121,7sleep12967121-21ls1
29584882122,5sleep22958578-21ntpq2
29102712123,1sleep0171rcu_preempt0
28559382121,4sleep20-21swapper/22
28247252121,7sleep02744611-21diskmemload0
28191872121,4sleep12818640-21apt-get1
28039992121,6sleep20-21swapper/22
27854042124,3sleep12744611-21diskmemload1
27771672121,7sleep02777178-21ssh0
27546822125,3sleep02756246-21kthreadcore0
27497462121,5sleep12744611-21diskmemload1
27383172121,7sleep02738365-21ntpq0
27264202121,5sleep22727484-21memory2
27223392121,6sleep02722379-21ntpq0
27168532121,7sleep12716980-21systemctl1
30815732111,5sleep13081632-21ntpq1
30697922111,6sleep03071409-21ssh0
30680692111,3sleep00-21swapper/00
30584712111,4sleep12744611-21diskmemload1
30563742111,5sleep00-21swapper/00
30507422111,5sleep03050823-21munin-run0
30497412111,3sleep23049754-21ssh2
30479932111,4sleep12744611-21diskmemload1
30341302111,6sleep13034262-21tr1
30056112111,3sleep20-21swapper/22
30047872110,7sleep13004798-21ssh1
29947872111,4sleep12744611-21diskmemload1
29946532111,3sleep20-21swapper/22
29812952111,6sleep12982722-21cut1
29358632111,3sleep00-21swapper/00
29188232111,5sleep22918734-21apt-get2
29174582111,6sleep22917471-21rm2
29102312111,6sleep12910242-21ssh1
28995782111,3sleep10-21swapper/11
28904052111,4sleep12744611-21diskmemload1
28841932110,2sleep20-21swapper/22
28809962111,6sleep02881007-21ssh0
28583792111,6sleep12858782-21cut1
28549012111,3sleep10-21swapper/11
28392782111,3sleep20-21swapper/22
28093752111,4sleep12744611-21diskmemload1
27934982111,5sleep02793552-21ntpq0
27757842111,4sleep22775848-21ntpq2
27711582111,4sleep12744611-21diskmemload1
27685352111,5sleep02744611-21diskmemload0
27395552111,5sleep22739813-21df_inode2
26854642111,5sleep12685512-21munin-run1
26590722111,5sleep12659094-21apt-get1
26590722111,5sleep12659094-21apt-get1
26537672112,4sleep12653786-21apt-get1
26316982110,1sleep10-21swapper/11
26259062111,6sleep22626173-21head2
30657952101,2sleep10-21swapper/11
30386842101,5sleep02744611-21diskmemload0
30303842101,2sleep10-21swapper/11
30157602101,4sleep03015795-21ssh0
30138132101,4sleep22744611-21diskmemload2
29883892101,3sleep00-21swapper/00
29685222101,2sleep20-21swapper/22
29521392101,4sleep02744611-21diskmemload0
29518362101,3sleep12744611-21diskmemload1
29455722101,4sleep20-21swapper/22
29434992101,5sleep02943513-21ssh0
29194082101,4sleep12919673-21df1
29077222101,2sleep20-21swapper/22
29058022101,3sleep00-21swapper/00
28928972101,4sleep02893124-21df_inode0
28758692101,4sleep12875981-21ps1
28665152101,3sleep20-21swapper/22
28632252101,5sleep20-21swapper/22
28617902101,6sleep02861959-21needreboot0
28557132101,2sleep00-21swapper/00
28406422101,2sleep00-21swapper/00
28333092101,5sleep02833318-21sh0
28310462101,2sleep10-21swapper/11
28302622101,3sleep00-21swapper/00
28298342101,4sleep22744611-21diskmemload2
28193172100,3sleep20-21swapper/22
28181322101,4sleep02744611-21diskmemload0
28150482101,2sleep10-21swapper/11
28139402101,4sleep12744611-21diskmemload1
28110062101,2sleep20-21swapper/22
28096632101,2sleep00-21swapper/00
27861252101,2sleep20-21swapper/22
27855502100,5sleep02744611-21diskmemload0
27784492100,1sleep10-21swapper/11
27532532101,3sleep20-21swapper/22
27329512101,4sleep02732992-21ntpq0
27241262101,5sleep02724697-21kthreadcore0
27126322101,3sleep22712651-21apt-get2
27077562101,4sleep02707875-21irqstats0
26803992101,4sleep02680418-21apt-get0
26701342104,3sleep1321rcuc/11
26537922101,4sleep22654024-21cstates2
3033611291,4sleep23033871-21expr2
2993685291,3sleep00-21swapper/00
2941462291,4sleep12941688-21grep1
2936897291,2sleep20-21swapper/22
2933840291,2sleep10-21swapper/11
2908532291,3sleep12744611-21diskmemload1
2896840291,2sleep20-21swapper/22
2883985291,3sleep00-21swapper/00
2836317291,3sleep12744611-21diskmemload1
2833677291,3sleep12744611-21diskmemload1
2805404291,3sleep12744611-21diskmemload1
2804067291,3sleep02744611-21diskmemload0
2759846291,2sleep20-21swapper/22
2745249291,3sleep02745277-21apt-get0
2706733290,2sleep00-21swapper/00
2702516291,4sleep22702933-21grep2
2701963290,2sleep10-21swapper/11
2690785291,4sleep0555-21dbus-daemon0
2664897291,4sleep22664968-21irqstats2
2659958293,3sleep22661659-21kthreadcore2
2659958293,3sleep22661659-21kthreadcore2
2652470291,3sleep22652521-21ntpq2
2641541291,4sleep12641593-21gcc1
2639165291,4sleep22640958-21cut2
2636968290,2sleep10-21swapper/11
2632275291,4sleep22632456-21diskstats2
2622725291,4sleep12624454-21grep1
2870825282,3sleep12870896-21/usr/sbin/munin1
2728645281,3sleep12728807-21latency_hist1
2707326281,3sleep12707454-21tr1
2696391280,3sleep20-21swapper/22
2672780281,3sleep02673679-21sed0
2669747280,2sleep20-21swapper/22
2642542281,3sleep01-21systemd0
2632165281,3sleep02632184-21apt-get0
2622117281,3sleep02622724-21kthreadcore0
2739104270,1sleep10-21swapper/11
2739013270,1sleep20-21swapper/22
2729806270,3sleep1321rcuc/11
2723142270,1sleep10-21swapper/11
2717031271,2sleep00-21swapper/00
2708995271,2sleep22710782-21kthreadcore2
2701145270,1sleep10-21swapper/11
2696020270,1sleep20-21swapper/22
2676966271,3sleep20-21swapper/22
2653457270,1sleep00-21swapper/00
2643098271,3sleep10-21swapper/11
2626839270,2sleep00-21swapper/00
2619083270,1sleep20-21swapper/22
2614219271,2sleep10-21swapper/11
2614197271,2sleep00-21swapper/00
2733963262,2sleep22614018-21cyclictest2
2701205260,4sleep00-21swapper/00
2695971260,4sleep10-21swapper/11
2691089260,3sleep00-21swapper/00
2690683260,1sleep10-21swapper/11
2675651261,2sleep10-21swapper/11
2666223261,2sleep00-21swapper/00
2664057260,1sleep00-21swapper/00
2664057260,1sleep00-21swapper/00
2658794260,1sleep00-21swapper/00
2647064260,2sleep00-21swapper/00
2631729260,1sleep20-21swapper/22
2743472250,2sleep00-21swapper/00
2685745250,1sleep20-21swapper/22
2682659250,2sleep20-21swapper/22
2679098250,1sleep00-21swapper/00
2648745250,1sleep10-21swapper/11
2664138240,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional