You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 23:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Tue Mar 31, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306121021340,4sleep0221rcuc/00
297999921290,2sleep00-21swapper/00
2780812212664,58sleep30-21swapper/33
278075528656,12sleep10-21swapper/11
278068028252,7sleep00-21swapper/00
278077128062,12sleep20-21swapper/22
31822722380,6sleep10-21swapper/11
30473882380,2sleep20-21swapper/22
278111499248,13cyclictest0-21swapper/33
2781114992422,1cyclictest0-21swapper/33
278111499237,13cyclictest0-21swapper/33
2781114992320,1cyclictest0-21swapper/33
2781114992320,1cyclictest0-21swapper/33
278111499229,11cyclictest0-21swapper/33
2781114992220,1cyclictest0-21swapper/33
2781114992220,1cyclictest0-21swapper/33
2781114992219,1cyclictest0-21swapper/33
2781114992219,1cyclictest0-21swapper/33
2781114992218,2cyclictest0-21swapper/33
2781114992218,2cyclictest0-21swapper/33
278111499216,12cyclictest0-21swapper/33
2781114992119,1cyclictest0-21swapper/33
2781114992119,1cyclictest0-21swapper/33
2781114992117,2cyclictest0-21swapper/33
2781114992116,2cyclictest0-21swapper/33
278111499211,3cyclictest0-21swapper/33
278111499211,2cyclictest0-21swapper/33
278111499211,16cyclictest2943744-21taskset3
31576032201,6sleep23157799-21cstates2
29005122201,11sleep12900588-21timerandwakeup1
278111499205,12cyclictest0-21swapper/33
2781114992018,1cyclictest0-21swapper/33
2781114992018,1cyclictest0-21swapper/33
2781114992017,1cyclictest0-21swapper/33
2781114992017,1cyclictest0-21swapper/33
2781114992017,1cyclictest0-21swapper/33
2781114992017,1cyclictest0-21swapper/33
2781114992016,2cyclictest0-21swapper/33
2781114992015,2cyclictest0-21swapper/33
2781114992015,2cyclictest0-21swapper/33
278111499201,16cyclictest2895544-21taskset3
278111499201,16cyclictest0-21swapper/33
278111499201,15cyclictest2911571-21taskset3
278111499201,15cyclictest2799537-21taskset3
278111499201,15cyclictest0-21swapper/33
31916232191,9sleep23191634-21ssh2
29999232191,3sleep03000035-21ssh0
278111499194,12cyclictest0-21swapper/33
2781114991917,1cyclictest0-21swapper/33
2781114991917,1cyclictest0-21swapper/33
2781114991917,1cyclictest0-21swapper/33
2781114991916,1cyclictest0-21swapper/33
2781114991915,2cyclictest0-21swapper/33
278111499191,15cyclictest0-21swapper/33
278111499191,14cyclictest66950irq/132-enp1s0-TxRx-33
278111499190,12cyclictest0-21swapper/33
278111499181,5cyclictest0-21swapper/33
2781114991815,1cyclictest0-21swapper/33
2781114991814,2cyclictest0-21swapper/33
278111499181,14cyclictest0-21swapper/33
278111499181,14cyclictest0-21swapper/33
32602642178,2sleep2171rcu_preempt2
30423732172,8sleep244-21ksoftirqd/22
278111499172,13cyclictest0-21swapper/33
2781114991715,1cyclictest0-21swapper/33
2781114991715,1cyclictest0-21swapper/33
2781114991714,2cyclictest0-21swapper/33
2781114991713,2cyclictest0-21swapper/33
2781114991713,2cyclictest0-21swapper/33
278111499171,13cyclictest0-21swapper/33
278111499171,13cyclictest0-21swapper/33
278111499170,14cyclictest0-21swapper/33
32234352162,7sleep13223499-21ntpq1
30930102161,4sleep22911893-21diskmemload2
30513632161,10sleep12911893-21diskmemload1
30511062161,11sleep03051119-21rm0
28150472161,10sleep22815060-31mv2
2781114991614,1cyclictest0-21swapper/33
2781114991613,1cyclictest0-21swapper/33
2781114991612,2cyclictest0-21swapper/33
278111499161,12cyclictest0-21swapper/33
278111499160,14cyclictest0-21swapper/33
278111499160,14cyclictest0-21swapper/33
278111499160,13cyclictest0-21swapper/33
278111499160,13cyclictest0-21swapper/33
32521872151,9sleep22911893-21diskmemload2
32322532151,9sleep13232316-21ntp_states1
32187912152,3sleep10-21swapper/11
31670122151,10sleep03168509-21cut0
30759692150,1sleep10-21swapper/11
30692772151,3sleep20-21swapper/22
30162832151,3sleep02911893-21diskmemload0
30139172150,1sleep20-21swapper/22
29522352150,2sleep10-21swapper/11
2781114991513,1cyclictest0-21swapper/33
2781114991512,1cyclictest0-21swapper/33
32637472141,7sleep13263961-21diskstats1
31899262142,5sleep13189937-21pool-udisksd1
31118752141,6sleep00-21swapper/00
31067982141,3sleep23108510-21cut2
30831932141,7sleep13083251-21ntpq1
30744282141,8sleep23074486-21ntpq2
30570412141,7sleep23057136-21/usr/sbin/munin2
30253992143,5sleep2431ktimers/22
29192552141,9sleep02919269-21ssh0
28160852141,5sleep12816200-21fschecks_time1
28160852141,5sleep12816200-21fschecks_time1
32808592131,8sleep03280953-21ls0
32510602131,3sleep12911893-21diskmemload1
32059272131,6sleep23205995-21sed2
32002642131,6sleep00-21swapper/00
31932482131,7sleep23194698-21cat2
31744472131,5sleep20-21swapper/22
31554022131,7sleep20-21swapper/22
31057852131,3sleep13106562-21ssh1
30406152131,7sleep02911893-21diskmemload0
30310112132,5sleep244-21ksoftirqd/22
30006482131,7sleep23000778-21irqstats2
29620082130,9sleep02962020-21rm0
29335102135,4sleep12933790-21cat1
29262172131,6sleep12911893-21diskmemload1
28141022131,5sleep12814148-21ntpq1
32534962121,7sleep03253508-21ssh0
32370012122,4sleep13237025-21apt-get1
32353892124,4sleep23198960-21kworker/u16:1+nfsiod2
32074072122,4sleep12911893-21diskmemload1
31901192121,7sleep03190133-21ssh0
31444182121,7sleep23140163-21/usr/sbin/munin2
31283192121,7sleep23128333-21ssh2
31141982121,7sleep23114318-21fschecks_time2
31036922121,6sleep12911893-21diskmemload1
30880852121,2sleep13088307-21smartctl1
30745552121,7sleep03074613-21/usr/sbin/munin0
30227412121,7sleep12911893-21diskmemload1
29704312121,5sleep20-21swapper/22
29704312121,5sleep20-21swapper/22
29631972121,3sleep22963211-21ssh2
29190672121,3sleep10-21swapper/11
29118832121,6sleep22912023-21grep2
28894492121,5sleep22889538-21ntpq2
28656772121,3sleep12867449-21kthreadcore1
28528992120,1sleep20-21swapper/22
28250562121,5sleep02825189-21ntpq0
27926702121,2sleep02786063-21kworker/u16:3+events_unbound0
27863622121,7sleep12786655-21gltestperf1
32793342111,6sleep12911893-21diskmemload1
32679492111,5sleep23268016-21ntp_states2
32437152111,3sleep00-21swapper/00
32278022111,7sleep03227816-21ssh0
32233732110,1sleep00-21swapper/00
31970852111,6sleep13197150-21sh1
31792802111,5sleep03179333-21ntpq0
31748082111,3sleep20-21swapper/22
31617072111,6sleep12911893-21diskmemload1
31566652111,6sleep13156681-21sh1
31469202111,4sleep00-21swapper/00
31212982111,4sleep12911893-21diskmemload1
31145002111,6sleep03115281-21cut0
30850852111,5sleep00-21swapper/00
30685532111,4sleep12911893-21diskmemload1
30407842111,3sleep10-21swapper/11
30089012111,4sleep12911893-21diskmemload1
29979402111,4sleep22911893-21diskmemload2
29975562113,3sleep00-21swapper/00
29968582111,4sleep12911893-21diskmemload1
29895442110,1sleep00-21swapper/00
29893532110,7sleep12911893-21diskmemload1
29833722111,6sleep22984997-21kthreadcore2
29801212111,3sleep20-21swapper/22
29437722111,6sleep12943783-21ssh1
29168642111,5sleep22917128-21unixbench_singl2
28855282111,5sleep02885766-21df_inode0
28635802111,3sleep072-21kcompactd00
28521402111,5sleep12852186-21ntpq1
28158262111,5sleep02816084-21df_inode0
28158262111,5sleep02816084-21df_inode0
28150452110,1sleep00-21swapper/00
28087812111,5sleep22808835-21ntpq2
32642562104,3sleep03265273-21kthreadcore0
32550562101,2sleep10-21swapper/11
32370352101,4sleep23237299-21df2
32191472101,4sleep22911893-21diskmemload2
31552762101,3sleep00-21swapper/00
31402002101,4sleep12911893-21diskmemload1
31371712100,2sleep10-21swapper/11
31367472101,2sleep00-21swapper/00
31363252101,3sleep20-21swapper/22
31293152101,3sleep12911893-21diskmemload1
31244062101,6sleep03126125-21kthreadcore0
31029912101,4sleep22911893-21diskmemload2
31015392101,2sleep00-21swapper/00
30876812101,3sleep00-21swapper/00
30848492101,3sleep20-21swapper/22
30313682101,4sleep12911893-21diskmemload1
29788822101,3sleep10-21swapper/11
29693972100,4sleep00-21swapper/00
29693972100,4sleep00-21swapper/00
29644182101,4sleep12911893-21diskmemload1
29548212100,4sleep00-21swapper/00
29472332101,2sleep20-21swapper/22
29427402101,4sleep22942790-21ntpq2
29344482101,4sleep22934700-21users2
29317652101,5sleep02933508-21cut0
29267912101,4sleep22911893-21diskmemload2
29216322101,4sleep02922574-21cut0
29033542101,5sleep22905175-21kthreadcore2
28949622101,4sleep02895015-21ntp_states0
28857672101,5sleep12885907-21head1
28801772101,4sleep02880432-21df0
28750582101,5sleep12875113-21grep1
28739032101,4sleep12874147-21users1
28590812101,4sleep12859205-21perl1
28534392101,4sleep12853691-21df1
28469252101,5sleep02847050-21wc0
28374312101,5sleep12837442-21/usr/sbin/munin1
28279652101,5sleep22829703-21kthreadcore2
28267622101,4sleep12826765-21apt-get1
28046692101,4sleep12804688-21apt-get1
3278634291,4sleep23278647-21ssh2
3260943290,1sleep00-21swapper/00
3216138290,1sleep00-21swapper/00
3210750291,4sleep23210885-21perl2
3207806291,2sleep00-21swapper/00
3170620291,3sleep172-21kcompactd01
3157511291,3sleep02911893-21diskmemload0
3066801291,4sleep00-21swapper/00
3056972291,4sleep13057038-21ntpq1
3032737291,2sleep00-21swapper/00
3021835290,4sleep03021893-21ntpq0
3005681291,3sleep12911893-21diskmemload1
2973063291,3sleep12911893-21diskmemload1
2973063291,3sleep12911893-21diskmemload1
2896459291,4sleep22896664-21hddtemp_smartct2
2890808291,3sleep22890827-21apt-get2
2874518291,4sleep01-21systemd0
2868184291,4sleep02868239-21ntpq0
2853692291,4sleep22853723-21fschecks_count2
2836171291,3sleep02836218-21ntpq0
2826207291,4sleep12826258-21systemd-run1
2809750291,4sleep02809894-21cut0
2799980291,4sleep22801654-21cut2
2799322291,3sleep12799341-21apt-get1
2794566291,4sleep12795455-21tr1
2782835292,3sleep12783368-21kthreadcore1
2938868281,3sleep00-21swapper/00
2911699280,2sleep172-21kcompactd01
2901484280,3sleep02901503-21apt-get0
2846775281,3sleep12846813-21ntpq1
2842207280,5sleep2802-21snmpd2
2831659281,3sleep072-21kcompactd00
2792777280,3sleep22792830-21ntp_states2
2786080280,1sleep00-21swapper/00
2905551271,2sleep10-21swapper/11
2884866270,1sleep10-21swapper/11
2879690270,1sleep00-21swapper/00
2870025271,3sleep20-21swapper/22
2868297270,3sleep22868385-21ps2
2853417270,2sleep00-21swapper/00
2843422271,2sleep20-21swapper/22
2837148270,1sleep20-21swapper/22
2836457270,4sleep10-21swapper/11
2820879271,3sleep22820927-21munin-run2
2820879271,3sleep22820927-21munin-run2
2804122270,1sleep00-21swapper/00
2911189261,2sleep00-21swapper/00
2895713260,1sleep10-21swapper/11
2884061260,3sleep20-21swapper/22
2878993260,2sleep20-21swapper/22
2862976261,2sleep20-21swapper/22
2822435261,2sleep20-21swapper/22
2795456261,2sleep20-21swapper/22
2896184250,1sleep00-21swapper/00
2851998250,1sleep00-21swapper/00
2842260250,3sleep00-21swapper/00
2794411251,1sleep00-21swapper/00
2780980250,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional