You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 07:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Mon Jul 20, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3067725212860,63sleep30-21swapper/33
335247621260,6sleep13352478-21latency_hist1
306341728559,11sleep20-21swapper/22
306770328251,9sleep10-21swapper/11
306774227956,17sleep00-21swapper/00
34216812251,3sleep13421692-21ssh1
306800299251,20cyclictest3511374-21chrt3
306800299241,19cyclictest3471100-21chrt3
34912572221,8sleep13491485-21munin-node1
306800299225,13cyclictest0-21swapper/33
306800299225,13cyclictest0-21swapper/33
32917042211,7sleep13291737-21ssh1
32490732213,3sleep23250620-21kthreadcore2
32145032211,4sleep03216064-21grep0
306800299213,15cyclictest66950irq/132-enp1s0-TxRx-33
3068002992118,1cyclictest0-21swapper/33
306800299211,16cyclictest0-21swapper/33
3068002992017,1cyclictest0-21swapper/33
3068002992017,1cyclictest0-21swapper/33
306800299201,16cyclictest0-21swapper/33
306800299201,15cyclictest0-21swapper/33
306800299201,15cyclictest0-21swapper/33
306800299200,18cyclictest0-21swapper/33
35530452191,5sleep00-21swapper/00
306800299191,7cyclictest3132449-21taskset3
3068002991915,2cyclictest0-21swapper/33
306800299191,15cyclictest0-21swapper/33
306800299191,14cyclictest0-21swapper/33
306800299191,14cyclictest0-21swapper/33
306800299191,14cyclictest0-21swapper/33
306800299191,14cyclictest0-21swapper/33
306800299191,13cyclictest0-21swapper/33
306800299190,16cyclictest66950irq/132-enp1s0-TxRx-33
34475692181,2sleep00-21swapper/00
33091292181,11sleep23309164-21ssh2
32572052181,5sleep23257339-21perl2
306800299181,5cyclictest0-21swapper/33
3068002991814,2cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,14cyclictest0-21swapper/33
306800299181,13cyclictest0-21swapper/33
306800299181,13cyclictest0-21swapper/33
306800299181,13cyclictest0-21swapper/33
306800299181,13cyclictest0-21swapper/33
306800299181,13cyclictest0-21swapper/33
306800299181,13cyclictest0-21swapper/33
306800299181,12cyclictest0-21swapper/33
306800299181,12cyclictest0-21swapper/33
306800299180,15cyclictest0-21swapper/33
306800299180,14cyclictest0-21swapper/33
306800299180,14cyclictest0-21swapper/33
35628852171,3sleep03195650-21diskmemload0
34817382171,4sleep23481752-21ssh2
34017302171,12sleep23401747-21ssh2
33672662171,3sleep00-21swapper/00
32814412171,3sleep00-21swapper/00
30804012171,12sleep04192558-21rs:main0
306800299176,9cyclictest0-21swapper/33
306800299176,8cyclictest3090897-21turbostat3
3068002991711,4cyclictest0-21swapper/33
306800299171,13cyclictest0-21swapper/33
306800299171,13cyclictest0-21swapper/33
306800299171,12cyclictest0-21swapper/33
306800299170,14cyclictest0-21swapper/33
306800299170,14cyclictest0-21swapper/33
306800299170,14cyclictest0-21swapper/33
306800299170,14cyclictest0-21swapper/33
306800299170,13cyclictest0-21swapper/33
34829542161,3sleep20-21swapper/22
33418172161,4sleep03341829-21rm0
33239222161,2sleep00-21swapper/00
31802202161,4sleep13180278-21tr1
306800299161,4cyclictest0-21swapper/33
306800299161,12cyclictest0-21swapper/33
3068002991610,4cyclictest0-21swapper/33
306800299160,13cyclictest31221072chrt3
306800299160,13cyclictest0-21swapper/33
35276472150,1sleep20-21swapper/22
34565282152,9sleep13456619-21munin-run1
34565282152,9sleep13456619-21munin-run1
34398862151,5sleep23440107-21/usr/sbin/munin2
34380392151,8sleep23195650-21diskmemload2
34180712152,7sleep23418266-21users2
32889132151,10sleep23288926-21ssh2
32815232151,2sleep20-21swapper/22
32564332150,2sleep00-21swapper/00
32466042150,1sleep10-21swapper/11
32054882156,2sleep1171rcu_preempt1
32025112151,2sleep00-21swapper/00
306800299151,11cyclictest31586462sleep33
306800299151,10cyclictest0-21swapper/33
306800299150,2cyclictest0-21swapper/33
306800299150,12cyclictest0-21swapper/33
306800299150,12cyclictest0-21swapper/33
34618762140,3sleep23195650-21diskmemload2
34568572140,2sleep10-21swapper/11
34096582141,2sleep10-21swapper/11
34034532140,1sleep00-21swapper/00
33859902141,3sleep03386001-21ssh0
33830922141,4sleep23383243-21xargs2
33430032141,8sleep13195650-21diskmemload1
32461472141,8sleep23195650-21diskmemload2
32271782140,2sleep00-21swapper/00
32100182141,4sleep23210021-21ssh2
3068002991411,1cyclictest0-21swapper/33
306800299140,11cyclictest0-21swapper/33
306800299140,11cyclictest0-21swapper/33
33829632131,7sleep13383091-21sendmail_mailqu1
33576982131,8sleep23357712-21ssh2
32881262131,7sleep13195650-21diskmemload1
32476562131,8sleep03247669-21ssh0
32369942131,5sleep20-21swapper/22
32217202131,6sleep13221786-21munin-run1
31608962131,7sleep23162617-21tr2
306800299138,3cyclictest31697242sleep33
3068002991311,1cyclictest0-21swapper/33
306800299130,2cyclictest0-21swapper/33
35521862121,3sleep13552197-21ssh1
35283322121,7sleep03528497-21cat0
35196362121,7sleep13519666-21latency_hist1
34904242124,4sleep03448776-21kworker/u16:2+xprtiod0
33706962121,7sleep13371925-21cat1
33509412120,8sleep03350955-21ssh0
33156702121,3sleep13315681-21ssh1
32998512121,5sleep00-21swapper/00
32915662121,7sleep23291577-21ssh2
32792972121,6sleep13195650-21diskmemload1
32646632121,5sleep03195650-21diskmemload0
32631042121,7sleep13263118-21ssh1
32191712121,7sleep23219186-21ssh2
31755132121,8sleep20-21swapper/22
31173742122,4sleep23117386-21apt-get2
35629672111,3sleep20-21swapper/22
35551352111,6sleep13555345-21cut1
35441792111,6sleep23195650-21diskmemload2
35427412111,5sleep13195650-21diskmemload1
35343672111,4sleep20-21swapper/22
35165812111,4sleep23195650-21diskmemload2
35108962110,7sleep23510977-2192
35025002110,2sleep10-21swapper/11
34899862111,2sleep13490000-21ssh1
34711712110,7sleep03195650-21diskmemload0
34705722110,1sleep10-21swapper/11
34553182111,4sleep23195650-21diskmemload2
34553182111,4sleep23195650-21diskmemload2
34256772111,6sleep13422222-21/usr/sbin/munin1
34111492111,4sleep23195650-21diskmemload2
34097472110,7sleep03409761-21ssh0
33957412111,4sleep23395807-21munin-run2
33955772112,4sleep13195650-21diskmemload1
33778802111,7sleep03377892-21ssh0
33755062111,2sleep2171rcu_preempt2
33674272111,3sleep10-21swapper/11
33613842112,4sleep23361408-21apt-get2
33575192111,2sleep00-21swapper/00
33316962111,5sleep23195650-21diskmemload2
33313102110,2sleep00-21swapper/00
33073342111,3sleep20-21swapper/22
33070562110,2sleep03195650-21diskmemload0
32716112115,2sleep23195650-21diskmemload2
32716112115,2sleep23195650-21diskmemload2
32336582110,1sleep00-21swapper/00
32288332111,4sleep10-21swapper/11
31959272112,1sleep2171rcu_preempt2
31907062112,4sleep23190708-21apt-get2
31596912111,5sleep03159806-21irqstats0
31281422111,5sleep23128355-21hddtemp_smartct2
31060472111,5sleep03106232-21users0
31060472111,5sleep03106232-21users0
35515262101,2sleep20-21swapper/22
35432392100,2sleep00-21swapper/00
35367012101,2sleep00-21swapper/00
35260442101,3sleep10-21swapper/11
35171182100,2sleep00-21swapper/00
35079192100,1sleep00-21swapper/00
34992082100,2sleep23499222-21ssh2
34814072101,4sleep03195650-21diskmemload0
34808672100,1sleep10-21swapper/11
34568462101,3sleep00-21swapper/00
34551402101,5sleep03455151-21ssh0
34551402101,5sleep03455151-21ssh0
34446022100,1sleep10-21swapper/11
34381632101,5sleep03438177-21ssh0
34222192101,4sleep23422243-21apt-get2
34194112101,4sleep03195650-21diskmemload0
34037642100,1sleep10-21swapper/11
33932172101,4sleep03195650-21diskmemload0
33402992101,3sleep20-21swapper/22
33347122101,3sleep13195650-21diskmemload1
33238402101,3sleep10-21swapper/11
33160142101,4sleep03195650-21diskmemload0
33059372100,2sleep10-21swapper/11
32888332100,1sleep00-21swapper/00
32359572101,3sleep10-21swapper/11
32302692100,1sleep20-21swapper/22
32043362100,1sleep00-21swapper/00
32024272101,6sleep13195650-21diskmemload1
31488062101,5sleep23148819-21grep2
31383132101,3sleep00-21swapper/00
30951822101,5sleep03095245-21/usr/sbin/munin0
30806882101,4sleep23080467-21sendmail-msp2
30794512101,4sleep23079502-21ntp_states2
3535687291,4sleep13535701-21ssh1
3494133291,3sleep03491508-40find0
3467809291,4sleep23469551-21tr2
3430478290,4sleep1321rcuc/11
3422193291,2sleep00-21swapper/00
3358659291,3sleep13358666-21ssh1
3351823290,1sleep20-21swapper/22
3320246291,2sleep20-21swapper/22
3271878290,1sleep00-21swapper/00
3271878290,1sleep00-21swapper/00
3270459290,1sleep10-21swapper/11
3270459290,1sleep10-21swapper/11
3254408290,3sleep10-21swapper/11
3185460291,2sleep10-21swapper/11
3171062291,4sleep13172722-21grep1
3168473290,1sleep20-21swapper/22
3154022291,2sleep10-21swapper/11
3137284293,3sleep0221rcuc/00
3133077291,3sleep13133096-21apt-get1
3128434291,4sleep03129947-21tr0
3122328291,4sleep03122509-21latency_hist0
3107462291,4sleep13108338-21cut1
3092991291,4sleep13094730-21cut1
3068747291,5sleep23069135-21kthreadcore2
3181860281,4sleep23183573-21kthreadcore2
3180198280,3sleep03180217-21apt-get0
3174531280,1sleep00-21swapper/00
3164784281,3sleep13164997-21hddtemp_smartct1
3164484280,5sleep00-21swapper/00
3148784281,2sleep13148803-21apt-get1
3133645281,3sleep20-21swapper/22
3132745280,2sleep10-21swapper/11
3127368280,1sleep00-21swapper/00
3122619281,2sleep20-21swapper/22
3112147280,3sleep13112149-21apt-get1
3106413280,1sleep20-21swapper/22
3106413280,1sleep20-21swapper/22
3106355281,3sleep13106356-21fwupdmgr1
3106355281,3sleep13106356-21fwupdmgr1
3096409280,2sleep20-21swapper/22
3096409280,2sleep20-21swapper/22
3085347280,2sleep10-21swapper/11
3072839280,4sleep00-21swapper/00
3194851271,2sleep10-21swapper/11
3190060270,1sleep20-21swapper/22
3185162271,2sleep00-21swapper/00
3179837270,2sleep10-21swapper/11
3179672270,1sleep00-21swapper/00
3159807270,3sleep13160894-21kthreadcore1
3158670270,1sleep00-21swapper/00
3148417270,1sleep10-21swapper/11
3148192270,1sleep00-21swapper/00
3142466271,3sleep23138310-21/usr/sbin/munin2
3122512271,2sleep10-21swapper/11
3122075270,4sleep10-21swapper/11
3117086270,4sleep00-21swapper/00
3117014270,1sleep20-21swapper/22
3106894271,2sleep00-21swapper/00
3101275270,1sleep00-21swapper/00
3101275270,1sleep00-21swapper/00
3095394270,3sleep20-21swapper/22
3090564270,1sleep00-21swapper/00
3195246260,1sleep00-21swapper/00
3143571260,1sleep20-21swapper/22
3143031260,4sleep10-21swapper/11
3111688260,1sleep20-21swapper/22
3100660261,2sleep10-21swapper/11
3100660261,2sleep10-21swapper/11
3090055261,2sleep10-21swapper/11
3080697260,1sleep00-21swapper/00
3072975260,4sleep10-21swapper/11
3173909250,1sleep20-21swapper/22
3158020250,1sleep20-21swapper/22
3152003250,1sleep00-21swapper/00
3090625250,1sleep20-21swapper/22
3075456240,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional