You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 02:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu May 14, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
357070721250,2sleep1171rcu_preempt1
3240844212253,64sleep30-21swapper/33
324083728668,12sleep10-21swapper/11
324102228453,12sleep20-21swapper/22
324088427959,14sleep00-21swapper/00
324129999241,18cyclictest3418380-21chrt3
324129999221,17cyclictest3429348-21taskset3
324129999213,5cyclictest0-21swapper/33
324129999211,16cyclictest3378800-21taskset3
324129999211,16cyclictest0-21swapper/33
35643932200,8sleep134-21ksoftirqd/11
34569932202,6sleep13457017-21apt-get1
33858512201,4sleep23371482-21diskmemload2
324129999204,13cyclictest0-21swapper/33
324129999203,14cyclictest66950irq/132-enp1s0-TxRx-33
324129999201,15cyclictest0-21swapper/33
34958292191,7sleep13495841-21rm1
3241299991916,1cyclictest0-21swapper/33
324129999191,2cyclictest0-21swapper/33
324129999191,15cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
324129999191,14cyclictest0-21swapper/33
3241299991910,6cyclictest0-21swapper/33
324129999190,16cyclictest66950irq/132-enp1s0-TxRx-33
324129999190,16cyclictest3505267-21taskset3
33077872181,5sleep03307805-21apt-get0
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,14cyclictest0-21swapper/33
324129999181,13cyclictest0-21swapper/33
324129999181,13cyclictest0-21swapper/33
324129999181,13cyclictest0-21swapper/33
324129999181,13cyclictest0-21swapper/33
324129999180,15cyclictest0-21swapper/33
324129999180,14cyclictest0-21swapper/33
34039712171,3sleep20-21swapper/22
3241299991715,1cyclictest0-21swapper/33
3241299991713,2cyclictest0-21swapper/33
3241299991712,2cyclictest0-21swapper/33
324129999171,13cyclictest0-21swapper/33
324129999171,13cyclictest0-21swapper/33
324129999171,13cyclictest0-21swapper/33
324129999171,12cyclictest0-21swapper/33
324129999170,14cyclictest0-21swapper/33
324129999170,14cyclictest0-21swapper/33
37023022162,8sleep03702524-21diskstats0
36731992161,4sleep23673207-21ssh2
35398792161,4sleep23371482-21diskmemload2
35130992161,2sleep20-21swapper/22
34553712161,2sleep20-21swapper/22
33952822161,3sleep03371482-21diskmemload0
32595672161,4sleep13259586-21apt-get1
3241299991613,1cyclictest0-21swapper/33
324129999160,13cyclictest0-21swapper/33
324129999160,13cyclictest0-21swapper/33
324129999160,13cyclictest0-21swapper/33
324129999160,13cyclictest0-21swapper/33
36333872151,3sleep13371482-21diskmemload1
36139182150,4sleep10-21swapper/11
36139182150,4sleep10-21swapper/11
35814102150,2sleep10-21swapper/11
35637232151,2sleep00-21swapper/00
35152182151,4sleep13371482-21diskmemload1
34811022151,2sleep00-21swapper/00
34386962151,3sleep23371482-21diskmemload2
3241299991511,3cyclictest0-21swapper/33
324129999151,12cyclictest0-21swapper/33
324129999150,13cyclictest0-21swapper/33
324129999150,12cyclictest0-21swapper/33
324129999150,12cyclictest0-21swapper/33
324129999150,12cyclictest0-21swapper/33
37079362141,8sleep23371482-21diskmemload2
36831312145,5sleep1331ktimers/11
35372272141,7sleep13537278-21ntpq1
35042882142,8sleep13371482-21diskmemload1
34793892141,4sleep23371482-21diskmemload2
34116662141,2sleep20-21swapper/22
34049812140,2sleep10-21swapper/11
32921662141,9sleep00-21swapper/00
3241299991411,2cyclictest0-21swapper/33
324129999140,13cyclictest0-21swapper/33
324129999140,12cyclictest0-21swapper/33
324129999140,12cyclictest0-21swapper/33
324129999140,12cyclictest0-21swapper/33
324129999140,11cyclictest0-21swapper/33
324129999140,11cyclictest0-21swapper/33
36994972130,2sleep10-21swapper/11
36920872131,7sleep23371482-21diskmemload2
36575352131,3sleep23657546-21ssh2
36227912131,8sleep23371482-21diskmemload2
36216882130,1sleep00-21swapper/00
36078582131,8sleep23607872-21ssh2
35577952131,4sleep03557806-21sh0
35423242131,8sleep20-21swapper/22
34655112132,5sleep03465540-21apt-get0
34482302131,3sleep13448242-21ssh1
34471232131,7sleep23371482-21diskmemload2
34229122131,2sleep23422968-21ssh2
34187352131,7sleep03418835-21sed0
33243282131,8sleep13324768-21cut1
33225172131,2sleep20-21swapper/22
324129999138,3cyclictest3291527-21chrt3
37269912121,3sleep23727003-21ssh2
37157022121,6sleep23371482-21diskmemload2
36853552121,6sleep03685620-21df0
36329112121,7sleep03632921-21ssh0
36065752121,7sleep13606585-21ssh1
35896422120,1sleep10-21swapper/11
35422402121,6sleep13542323-21irqstats1
35298642121,3sleep23529875-21ssh2
34874932121,7sleep23487505-21ssh2
34736172121,5sleep13473667-21munin-run1
34612152121,6sleep23461271-21ntpq2
34313842121,4sleep10-21swapper/11
34196472121,7sleep13419657-21ssh1
33623122121,7sleep03364068-21tr0
33574502121,6sleep03359250-21cut0
33464212121,7sleep03348167-21tr0
33078082121,5sleep13308066-21df_inode1
32636582121,6sleep23263708-21ntpq2
32593082121,7sleep23259452-21cat2
37241912111,3sleep00-21swapper/00
37075902111,5sleep13371482-21diskmemload1
37000982110,7sleep23700112-21ssh2
36938262111,5sleep03694094-21df_inode0
36836332111,4sleep00-21swapper/00
36740902111,6sleep03371482-21diskmemload0
36738782111,6sleep13673886-21ssh1
36583982110,2sleep10-21swapper/11
36488462111,1sleep20-21swapper/22
36088132110,7sleep23608825-21ssh2
36088132110,7sleep23608825-21ssh2
35985012111,5sleep20-21swapper/22
35752972111,5sleep23575325-21apt-get2
35719222110,5sleep20-21swapper/22
35296172111,3sleep00-21swapper/00
35212812111,2sleep20-21swapper/22
35203602111,5sleep03520423-21ntpq0
35152062110,2sleep00-21swapper/00
34980162111,3sleep20-21swapper/22
34890092111,4sleep03371482-21diskmemload0
34885922111,3sleep10-21swapper/11
34814952111,3sleep10-21swapper/11
34541372111,4sleep03371482-21diskmemload0
34372792111,4sleep03371482-21diskmemload0
34309192110,4sleep10-21swapper/11
34264652111,6sleep03427025-21head0
34129012111,3sleep00-21swapper/00
33791692110,2sleep10-21swapper/11
33772122111,4sleep03371482-21diskmemload0
33302542111,6sleep03331962-21kthreadcore0
32968792111,5sleep03297134-21df0
32529952111,5sleep03253056-21ntpq0
37250082101,3sleep10-21swapper/11
37154862101,3sleep10-21swapper/11
36881702101,4sleep13689354-21memory1
36826322101,3sleep23371482-21diskmemload2
36660162101,3sleep00-21swapper/00
36651802101,4sleep23371482-21diskmemload2
36485502101,4sleep13371482-21diskmemload1
36483092100,6sleep03648322-21sh0
36387982101,3sleep10-21swapper/11
36345732102,4sleep20-21swapper/22
36173532100,1sleep10-21swapper/11
36155832101,3sleep03371482-21diskmemload0
36155832101,3sleep03371482-21diskmemload0
36065102101,4sleep03371482-21diskmemload0
36000242101,4sleep03371482-21diskmemload0
35896812101,4sleep03371482-21diskmemload0
35882102101,4sleep23371482-21diskmemload2
35725822100,1sleep00-21swapper/00
35652272101,4sleep20-21swapper/22
35551932101,3sleep13371482-21diskmemload1
35501432101,4sleep23550170-21apt-get2
35420182101,4sleep03371482-21diskmemload0
35322992101,3sleep13371482-21diskmemload1
35012682101,5sleep23501494-21kthreadcore2
34983812101,4sleep03371482-21diskmemload0
34700632101,4sleep20-21swapper/22
34639832101,4sleep03371482-21diskmemload0
34055762100,2sleep10-21swapper/11
33976332101,5sleep03397761-21irqstats0
33941732101,3sleep20-21swapper/22
33929522101,4sleep13393020-21ntp_states1
33759972101,4sleep23376047-21ntpq2
33704932101,4sleep23370542-21ntpq2
33548612101,4sleep23355105-21users2
33510082101,5sleep03352397-21cut0
32901692101,4sleep23290258-21ntpq2
32808772101,4sleep23280896-21apt-get2
32760852101,5sleep23276417-21kthreadcore2
3665746290,1sleep10-21swapper/11
3656935291,4sleep03656947-21ssh0
3634276291,3sleep03634298-21apt-get0
3629737291,2sleep20-21swapper/22
3596270291,3sleep13371482-21diskmemload1
3581733291,2sleep00-21swapper/00
3533010290,4sleep03533036-21apt-get0
3522047290,1sleep10-21swapper/11
3499330291,4sleep03499355-21apt-get0
3444101291,4sleep13444155-21ntp_states1
3440586291,4sleep03442205-21grep0
3414079291,2sleep20-21swapper/22
3387391290,1sleep00-21swapper/00
3380837291,4sleep10-21swapper/11
3364069291,4sleep13364991-21grep1
3352399291,4sleep13354244-21cut1
3344831290,2sleep10-21swapper/11
3335025291,4sleep03335517-21kthreadcore0
3328517290,1sleep00-21swapper/00
3323817290,5sleep23323796-21/usr/sbin/munin2
3286486291,4sleep13286607-21tune2fs1
3272564291,4sleep13274112-21/usr/sbin/munin1
3265446291,4sleep03266050-21cut0
3249338291,4sleep23249454-21irqstats2
3715963281,2sleep00-21swapper/00
3308067281,3sleep20-21swapper/22
3303409280,4sleep2421rcuc/22
3295628280,4sleep10-21swapper/11
3275862281,3sleep13276084-21cat1
3258408280,1sleep00-21swapper/00
3253268281,3sleep13253527-21/usr/sbin/munin1
3246263281,3sleep03246264-21sshd0
3371158270,1sleep00-21swapper/00
3366950272,2sleep134-21ksoftirqd/11
3365924270,1sleep20-21swapper/22
3349046270,2sleep20-21swapper/22
3339891270,2sleep00-21swapper/00
3335521271,2sleep10-21swapper/11
3334000270,1sleep10-21swapper/11
3333006270,2sleep20-21swapper/22
3321532271,2sleep00-21swapper/00
3313792271,3sleep23315473-21kthreadcore2
3313116270,2sleep10-21swapper/11
3307089270,1sleep10-21swapper/11
3290309270,4sleep00-21swapper/00
3285658270,1sleep10-21swapper/11
3270246271,3sleep03270416-21cstates0
3269829270,1sleep10-21swapper/11
3259142270,1sleep10-21swapper/11
3359917260,3sleep13360030-21ps1
3344722262,2sleep2766-21ntpd2
3338600261,2sleep20-21swapper/22
3306379261,2sleep00-21swapper/00
3300950260,2sleep10-21swapper/11
3295685261,2sleep20-21swapper/22
3285251261,2sleep03285405-21timerwakeupswit0
3275555260,4sleep00-21swapper/00
3264528260,1sleep00-21swapper/00
3246570260,1sleep10-21swapper/11
3246566260,1sleep20-21swapper/22
3356190250,1sleep20-21swapper/22
3322688250,1sleep10-21swapper/11
3316985250,1sleep00-21swapper/00
3267785250,2sleep20-21swapper/22
3345419240,1sleep10-21swapper/11
3297156240,1sleep20-21swapper/22
3270222240,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional