You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 00:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Mon May 18, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
341260721260,2sleep190-21kswapd01
3377819211760,53sleep30-21swapper/33
387201021090,2sleep13872012-21sed1
337777928654,11sleep10-21swapper/11
337781328556,11sleep20-21swapper/22
337774827752,6sleep00-21swapper/00
337812699271,23cyclictest0-21swapper/33
36905742262,9sleep13690554-21zstd1
337812699252,18cyclictest3493100-21taskset3
36793482243,17sleep00-21swapper/00
35257532240,2sleep00-21swapper/00
35218972240,2sleep20-21swapper/22
337812699241,6cyclictest0-21swapper/33
337812699228,4cyclictest0-21swapper/33
337812699221,18cyclictest0-21swapper/33
337812699220,12cyclictest0-21swapper/33
337812699220,12cyclictest0-21swapper/33
38436762211,5sleep13843690-21ssh1
38106502211,7sleep03508834-21diskmemload0
337812699218,4cyclictest0-21swapper/33
337812699215,12cyclictest3744114-21taskset3
337812699214,13cyclictest0-21swapper/33
337812699212,16cyclictest0-21swapper/33
337812699211,5cyclictest0-21swapper/33
337812699211,16cyclictest3785143-21taskset3
337812699211,15cyclictest0-21swapper/33
337812699210,12cyclictest0-21swapper/33
337812699210,12cyclictest0-21swapper/33
337812699207,3cyclictest0-21swapper/33
337812699203,8cyclictest0-21swapper/33
337812699203,13cyclictest0-21swapper/33
337812699203,13cyclictest0-21swapper/33
337812699202,2cyclictest0-21swapper/33
337812699201,15cyclictest0-21swapper/33
337812699201,15cyclictest0-21swapper/33
337812699200,17cyclictest0-21swapper/33
337812699200,12cyclictest0-21swapper/33
38112902191,3sleep23811305-21ssh2
337812699197,3cyclictest0-21swapper/33
337812699197,3cyclictest0-21swapper/33
337812699196,2cyclictest0-21swapper/33
337812699191,7cyclictest3654040-21turbostat3
337812699191,7cyclictest3428741-21turbostat3
337812699191,6cyclictest3466144-21turbostat3
337812699191,4cyclictest0-21swapper/33
3378126991913,4cyclictest0-21swapper/33
337812699191,15cyclictest0-21swapper/33
337812699191,14cyclictest0-21swapper/33
337812699191,14cyclictest0-21swapper/33
337812699190,9cyclictest0-21swapper/33
337812699190,17cyclictest0-21swapper/33
337812699190,16cyclictest0-21swapper/33
36023842181,6sleep03508834-21diskmemload0
337812699189,6cyclictest3455416-21chrt3
337812699183,11cyclictest0-21swapper/33
337812699181,6cyclictest3444738-21turbostat3
337812699181,5cyclictest66950irq/132-enp1s0-TxRx-33
337812699181,4cyclictest0-21swapper/33
337812699181,14cyclictest3635080-21chrt3
337812699181,14cyclictest0-21swapper/33
337812699181,14cyclictest0-21swapper/33
337812699181,14cyclictest0-21swapper/33
337812699181,13cyclictest0-21swapper/33
337812699181,13cyclictest0-21swapper/33
337812699181,13cyclictest0-21swapper/33
337812699180,14cyclictest0-21swapper/33
337812699180,10cyclictest0-21swapper/33
37868022171,3sleep00-21swapper/00
37016432171,4sleep03701667-21apt-get0
36905852173,8sleep23690555-21zstd2
337812699171,6cyclictest3423396-21turbostat3
337812699171,4cyclictest0-21swapper/33
3378126991714,2cyclictest0-21swapper/33
337812699171,13cyclictest0-21swapper/33
337812699170,8cyclictest0-21swapper/33
337812699170,15cyclictest0-21swapper/33
38872712161,3sleep20-21swapper/22
38869682161,2sleep00-21swapper/00
38508132161,10sleep23850871-21ntpq2
38181152161,3sleep20-21swapper/22
37358592161,3sleep20-21swapper/22
37172242161,6sleep23717235-21ssh2
35903792160,1sleep10-21swapper/11
337812699161,11cyclictest0-21swapper/33
337812699160,13cyclictest0-21swapper/33
37501772151,3sleep13508834-21diskmemload1
37257972151,2sleep10-21swapper/11
37257972151,2sleep10-21swapper/11
37011322152,2sleep2171rcu_preempt2
36999072151,4sleep13508834-21diskmemload1
36434972151,5sleep10-21swapper/11
34687292156,2sleep1171rcu_preempt1
337812699158,4cyclictest3383148-21chrt3
337812699153,8cyclictest0-21swapper/33
337812699150,13cyclictest0-21swapper/33
337812699150,12cyclictest0-21swapper/33
337812699150,12cyclictest0-21swapper/33
38206462141,2sleep10-21swapper/11
37914942141,6sleep03791542-21ntpq0
37773412141,7sleep13508834-21diskmemload1
37626872141,5sleep10-21swapper/11
37158002141,10sleep03715814-21ssh0
35321362141,9sleep23532148-21ssh2
35305712141,2sleep13530639-21ntpq1
35174072141,4sleep00-21swapper/00
34626812146,4sleep13464482-21cut1
337812699140,12cyclictest0-21swapper/33
38347102130,4sleep00-21swapper/00
37347242130,2sleep10-21swapper/11
37342602130,5sleep23734274-21ssh2
36586232131,8sleep23658689-21ntpq2
36352622131,6sleep20-21swapper/22
36197412131,8sleep13508834-21diskmemload1
36112922130,2sleep10-21swapper/11
35748182130,4sleep10-21swapper/11
35495672131,6sleep10-21swapper/11
35339652131,8sleep03533976-21ssh0
35247392131,3sleep13524750-21ssh1
35037692131,8sleep13503782-21wc1
34705762131,4sleep23470704-21readlink2
34445792130,1sleep20-21swapper/22
33786352131,8sleep23378845-21idleruntime2
38016592121,3sleep23801670-21ssh2
37325702127,2sleep00-21swapper/00
37254522120,4sleep00-21swapper/00
37254522120,4sleep00-21swapper/00
36267892121,7sleep03626799-21ssh0
36267102121,3sleep13626721-21ssh1
35515472121,6sleep03508834-21diskmemload0
35429182120,1sleep20-21swapper/22
35151172121,7sleep13515127-21ssh1
35025042121,6sleep23502550-21ntpq2
34493792121,7sleep23449514-21/usr/sbin/munin2
34490992121,6sleep13449148-21ntpq1
34173372121,7sleep13412959-21/usr/sbin/munin1
38888252111,7sleep13888918-21wc1
38799332111,3sleep20-21swapper/22
38719312111,6sleep03871954-21latency_hist0
38620242110,4sleep20-21swapper/22
38529042110,3sleep00-21swapper/00
38442102112,3sleep23508834-21diskmemload2
38293422111,5sleep13829404-21munin-run1
38035502111,3sleep13803563-21sh1
37743902111,5sleep03774502-21ntpq0
37657572111,6sleep23765816-21/usr/sbin/munin2
37608302111,6sleep13760840-21ssh1
37359312111,4sleep03508834-21diskmemload0
36585652111,5sleep13658621-21ntpq1
36534372111,4sleep23508834-21diskmemload2
36457712110,5sleep172-21kcompactd01
36424882111,6sleep23642499-21sh2
36354452110,1sleep10-21swapper/11
36093332111,4sleep20-21swapper/22
36001392111,6sleep23508834-21diskmemload2
35845602111,2sleep20-21swapper/22
35748292110,4sleep20-21swapper/22
35666842111,2sleep00-21swapper/00
35606452111,6sleep13560871-21cstates1
35572282115,3sleep20-21swapper/22
35483712111,4sleep23508834-21diskmemload2
35427572111,5sleep13508834-21diskmemload1
35077222111,5sleep03507802-21ntpq0
35024692111,4sleep13502502-21ntpq1
34402182111,6sleep03441465-21cut0
34398122111,5sleep13439934-21diskstats1
34328832110,1sleep00-21swapper/00
34232992110,1sleep10-21swapper/11
33951912111,5sleep13395241-21ntpq1
38774802101,4sleep13508834-21diskmemload1
38531132101,5sleep13508834-21diskmemload1
38445322101,2sleep00-21swapper/00
38302922100,6sleep23830382-21kernelversion2
38258342101,3sleep00-21swapper/00
38254592101,4sleep23825520-21ntpq2
38181852100,2sleep03508834-21diskmemload0
37933862101,4sleep23508834-21diskmemload2
37930902101,3sleep10-21swapper/11
37859292101,3sleep20-21swapper/22
37774972101,3sleep20-21swapper/22
37605672101,3sleep20-21swapper/22
37599402101,2sleep00-21swapper/00
37359622101,4sleep13736234-21df_inode1
37244512101,3sleep20-21swapper/22
37244512101,3sleep20-21swapper/22
37181132101,4sleep13508834-21diskmemload1
37065552100,2sleep10-21swapper/11
37012772101,3sleep23508834-21diskmemload2
36930442101,4sleep03693272-21df_inode0
36610422101,3sleep00-21swapper/00
36516212101,3sleep03508834-21diskmemload0
36424182101,3sleep00-21swapper/00
36367362101,2sleep00-21swapper/00
36257212101,4sleep23508834-21diskmemload2
36159652101,3sleep20-21swapper/22
36109892101,4sleep03508834-21diskmemload0
35934962101,3sleep23508834-21diskmemload2
35912202101,5sleep03508834-21diskmemload0
35771812101,2sleep00-21swapper/00
35669012101,5sleep23508834-21diskmemload2
35569672101,3sleep10-21swapper/11
35413392101,3sleep00-21swapper/00
35168032101,3sleep20-21swapper/22
35078602101,4sleep23507905-21ntpq2
34865302101,4sleep23486587-21ntp_states2
34819742101,3sleep00-21swapper/00
34760982101,5sleep13476293-21/usr/sbin/munin1
34610642101,5sleep03461177-21expr0
34557092101,4sleep13455721-21apt-get1
34503522101,4sleep23450608-21df2
34383462101,4sleep03438397-21ntpq0
34125692100,1sleep20-21swapper/22
3874046291,4sleep00-21swapper/00
3869650291,2sleep20-21swapper/22
3860014291,3sleep03508834-21diskmemload0
3855226291,4sleep13855429-21df_inode1
3833598291,3sleep13508834-21diskmemload1
3809274291,2sleep1171rcu_preempt1
3801543291,2sleep00-21swapper/00
3786008291,2sleep10-21swapper/11
3752352291,3sleep03508834-21diskmemload0
3750437290,1sleep20-21swapper/22
3616388291,3sleep00-21swapper/00
3594725291,3sleep13508834-21diskmemload1
3580891290,6sleep03581754-21sed0
3577642291,2sleep10-21swapper/11
3557776291,3sleep00-21swapper/00
3487478290,1sleep00-21swapper/00
3482438291,5sleep13482670-21cstates1
3476792290,5sleep23476964-21sed2
3438399291,4sleep13438445-21ntpq1
3433079291,4sleep23433133-21ld2
3418339291,4sleep03418375-21grep0
3412482290,4sleep03412495-31find0
3390818291,4sleep13390848-21munin-run1
3379056291,5sleep03380738-21kthreadcore0
3766530281,3sleep03766544-21ssh0
3498142281,4sleep13498196-21idleruntime-cro1
3491796280,4sleep00-21swapper/00
3477645281,3sleep13478884-21cut1
3445562281,3sleep03446804-21cut0
3422687281,4sleep23422899-21sed2
3407021280,2sleep00-21swapper/00
3397013281,3sleep03398078-21kthreadcore0
3385708280,2sleep20-21swapper/22
3498050270,1sleep20-21swapper/22
3492611270,1sleep10-21swapper/11
3481911270,1sleep20-21swapper/22
3472594271,3sleep03474302-21kthreadcore0
3467053271,3sleep00-21swapper/00
3456849271,3sleep20-21swapper/22
3455387270,1sleep00-21swapper/00
3450331270,2sleep10-21swapper/11
3428983270,2sleep10-21swapper/11
3428543270,1sleep20-21swapper/22
3417809270,2sleep20-21swapper/22
3402034270,3sleep13401539-21apt-get1
3398080270,3sleep1321rcuc/11
3395136271,2sleep00-21swapper/00
3390673270,1sleep00-21swapper/00
3494780261,2sleep00-21swapper/00
3461328261,2sleep20-21swapper/22
3460591260,1sleep00-21swapper/00
3436434261,2sleep20-21swapper/22
3428600260,4sleep10-21swapper/11
3428418260,1sleep00-21swapper/00
3402064260,2sleep20-21swapper/22
3395243260,2sleep20-21swapper/22
3383181260,4sleep10-21swapper/11
3490535250,1sleep20-21swapper/22
3399876250,1sleep20-21swapper/22
3498424240,1sleep00-21swapper/00
3413284240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional