You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 18:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Apr 16, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
350975022480,2sleep0471rcuog/20
3265640212265,53sleep30-21swapper/33
326560928452,12sleep10-21swapper/11
326566127958,15sleep00-21swapper/00
326562827658,12sleep20-21swapper/22
35617912230,2sleep20-21swapper/22
36508602221,6sleep23651182-21grep2
34634312221,17sleep23396333-21diskmemload2
35602682211,4sleep00-21swapper/00
326580199215,12cyclictest0-21swapper/33
3265801992116,2cyclictest0-21swapper/33
35086992201,10sleep13396333-21diskmemload1
326580199205,12cyclictest3289216-21taskset3
326580199201,6cyclictest3455008-21taskset3
326580199201,3cyclictest0-21swapper/33
3265801992011,5cyclictest0-21swapper/33
326580199201,15cyclictest0-21swapper/33
326580199201,15cyclictest0-21swapper/33
326580199201,14cyclictest0-21swapper/33
326580199200,1cyclictest0-21swapper/33
326580199199,6cyclictest35519532sleep33
326580199191,15cyclictest0-21swapper/33
326580199191,15cyclictest0-21swapper/33
326580199191,14cyclictest3310864-21taskset3
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199191,14cyclictest0-21swapper/33
326580199190,17cyclictest0-21swapper/33
326580199190,16cyclictest0-21swapper/33
35012052181,3sleep20-21swapper/22
326580199181,14cyclictest3641741-21taskset3
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,14cyclictest0-21swapper/33
326580199181,13cyclictest0-21swapper/33
326580199181,13cyclictest0-21swapper/33
326580199180,2cyclictest0-21swapper/33
326580199180,14cyclictest0-21swapper/33
36109742171,10sleep03611040-21ntpq0
34985102171,4sleep00-21swapper/00
326580199179,5cyclictest3527401-21turbostat3
3265801991711,4cyclictest0-21swapper/33
326580199171,14cyclictest0-21swapper/33
326580199171,13cyclictest0-21swapper/33
326580199171,13cyclictest0-21swapper/33
326580199171,13cyclictest0-21swapper/33
326580199171,12cyclictest3372580-21taskset3
326580199170,14cyclictest66950irq/132-enp1s0-TxRx-33
326580199170,13cyclictest0-21swapper/33
326580199170,13cyclictest0-21swapper/33
326580199170,13cyclictest0-21swapper/33
36496422161,3sleep20-21swapper/22
35792482160,1sleep00-21swapper/00
35534192160,2sleep20-21swapper/22
35357132161,2sleep00-21swapper/00
34667652161,7sleep23467010-21expr2
3265801991612,2cyclictest0-21swapper/33
326580199161,11cyclictest33428882sleep33
326580199160,13cyclictest0-21swapper/33
37549912151,2sleep00-21swapper/00
37262702151,2sleep20-21swapper/22
36840532150,1sleep20-21swapper/22
36119892151,4sleep13396333-21diskmemload1
34743002151,3sleep13396333-21diskmemload1
34471902151,4sleep00-21swapper/00
326580199151,12cyclictest0-21swapper/33
326580199151,11cyclictest0-21swapper/33
326580199150,13cyclictest0-21swapper/33
37629812141,6sleep23762984-21ssh2
37504132141,7sleep03750480-21ntpq0
37133642141,5sleep10-21swapper/11
36543142141,9sleep13654460-21/usr/sbin/munin1
35953422140,5sleep13595353-21ssh1
35796222141,9sleep23579634-21ssh2
34918422141,5sleep23491856-21ssh2
34754752143,6sleep23475496-21apt-get2
346521521410,1sleep00-21swapper/00
34565822141,2sleep13396333-21diskmemload1
34485052141,3sleep13396333-21diskmemload1
3265801991412,1cyclictest0-21swapper/33
3265801991412,1cyclictest0-21swapper/33
3265801991411,1cyclictest0-21swapper/33
326580199141,11cyclictest0-21swapper/33
326580199140,2cyclictest0-21swapper/33
326580199140,12cyclictest0-21swapper/33
326580199140,12cyclictest0-21swapper/33
326580199140,11cyclictest0-21swapper/33
37353212131,8sleep13735334-21ssh1
36578782131,5sleep00-21swapper/00
36383502131,8sleep03638364-21ssh0
36327762131,3sleep13632790-21ssh1
35802062131,7sleep03396333-21diskmemload0
35323522131,7sleep13396333-21diskmemload1
35162502131,7sleep20-21swapper/22
34929312131,6sleep13492966-21ssh1
34732372131,4sleep03473249-21ssh0
34306772131,8sleep23430691-21ssh2
34133422131,5sleep00-21swapper/00
33219252130,2sleep10-21swapper/11
32999892130,3chrt3278450-21kworker/u16:2+flush-8:00
37417322121,5sleep00-21swapper/00
37344882121,7sleep23396333-21diskmemload2
36984122121,7sleep23694064-21/usr/sbin/munin2
36662632121,3sleep03666277-21ssh0
36488202121,2sleep10-21swapper/11
36296092121,5sleep00-21swapper/00
36210882121,3sleep00-21swapper/00
35527132121,6sleep13396333-21diskmemload1
35164262121,6sleep03396333-21diskmemload0
35156762121,5sleep10-21swapper/11
34905292121,7sleep13490540-21ssh1
34650792121,5sleep13396333-21diskmemload1
34486662121,5sleep20-21swapper/22
34396032121,7sleep23439616-21rm2
33651262123,5sleep23366205-21kthreadcore2
33272542121,5sleep03327506-21df0
33055482120,1sleep00-21swapper/00
32899592121,7sleep13290026-21tune2fs1
326580199127,3cyclictest3316098-21chrt3
37446052111,3sleep20-21swapper/22
37166622111,4sleep23396333-21diskmemload2
37120662111,6sleep03713363-21grep0
36908232110,7sleep03690837-21ssh0
36732162111,4sleep23396333-21diskmemload2
36660322111,4sleep13396333-21diskmemload1
36390712111,3sleep20-21swapper/22
36154242111,5sleep13615451-21apt-get1
36059552111,6sleep03605966-21ssh0
35948032111,3sleep20-21swapper/22
35693002110,7sleep03569311-21ssh0
35520072114,4sleep0151ktimers/00
35379032111,5sleep13539549-21cut1
35332412111,1sleep20-21swapper/22
35252922111,5sleep23396333-21diskmemload2
34893282111,4sleep00-21swapper/00
34816412111,5sleep13396333-21diskmemload1
34546202111,4sleep00-21swapper/00
34328202111,6sleep03434424-21cut0
34273512111,6sleep13427363-21/usr/sbin/munin1
34096412111,5sleep13409701-21ntp_states1
33917622111,5sleep13391833-21irqstats1
33792952111,5sleep13379344-21ntpq1
33693292110,3sleep00-21swapper/00
33543442111,5sleep03354449-21irqstats0
33325702111,5sleep03332588-21apt-get0
33118302111,2sleep03311249-21/usr/sbin/munin0
32999672111,5sleep13299912-31find1
32938022111,5sleep03293829-21ntpq0
32828152111,5sleep13282862-21ntpq1
32664882111,5sleep23266559-21irqstats2
37618112101,4sleep13396333-21diskmemload1
37532962101,3sleep10-21swapper/11
37505372101,2sleep20-21swapper/22
37430452101,4sleep13396333-21diskmemload1
37359652101,4sleep03396333-21diskmemload0
37105022101,4sleep03396333-21diskmemload0
37072462100,3sleep20-21swapper/22
37027742101,3sleep13396333-21diskmemload1
36903452101,4sleep23396333-21diskmemload2
36830962100,7sleep13683109-21sh1
36826812100,1sleep00-21swapper/00
36758432101,4sleep13396333-21diskmemload1
36303182101,2sleep10-21swapper/11
36222102101,2sleep20-21swapper/22
36125982101,3sleep20-21swapper/22
36034942100,3sleep20-21swapper/22
35864072101,6sleep13396333-21diskmemload1
35860732101,4sleep23396333-21diskmemload2
35431652100,1sleep20-21swapper/22
35364032100,4sleep00-21swapper/00
35250362101,2sleep10-21swapper/11
35196432103,4sleep03520732-21kthreadcore0
35077142101,3sleep20-21swapper/22
34543922101,3sleep20-21swapper/22
34370332101,4sleep13396333-21diskmemload1
34228292101,5sleep03422853-21apt-get0
34208312101,3sleep13396333-21diskmemload1
34208312101,3sleep13396333-21diskmemload1
34198602101,3sleep03396333-21diskmemload0
34198602101,3sleep03396333-21diskmemload0
34190202101,2sleep20-21swapper/22
34190202101,2sleep20-21swapper/22
34097022101,5sleep23409828-21gzip2
34044302101,4sleep13396333-21diskmemload1
34032252101,5sleep23403240-21ssh2
33846862101,4sleep03384740-21ntpq0
33525182101,4sleep03352607-21ntpq0
33023012101,5sleep23304094-21cut2
32783112101,6sleep0802-21snmpd0
3726107291,3sleep13396333-21diskmemload1
3725282291,3sleep03396333-21diskmemload0
3695251291,4sleep03696842-21cut0
3694370291,4sleep13396333-21diskmemload1
3691623290,1sleep10-21swapper/11
3669150291,5sleep03670837-21sh0
3665741291,2sleep20-21swapper/22
3647143291,4sleep03647157-21ssh0
3629337290,1sleep20-21swapper/22
3592644291,5sleep03593094-21ssh0
3576899291,4sleep13576913-21ssh1
3569736291,4sleep23569748-21rm2
3567768291,3sleep13396333-21diskmemload1
3553957291,3sleep13554062-21expr1
3483021291,4sleep03483033-21sh0
3400890291,3sleep00-21swapper/00
3385918291,3sleep03385936-21apt-get0
3370879291,4sleep23372579-21cut2
3358888290,3sleep13358873-21fwupd1
3353454291,2sleep23265795-21cyclictest2
3343247290,4sleep23343266-21apt-get2
3331440291,4sleep23331557-21sendmail_mailst2
3325837290,4sleep03325950-21ntpq0
3310299291,4sleep23310543-21sed2
3305920291,3sleep03305939-21apt-get0
3289054290,1sleep00-21swapper/00
3288529291,4sleep23288645-21rt-features2
3282725290,4sleep00-21swapper/00
3602790281,3sleep13396333-21diskmemload1
3390910280,2sleep10-21swapper/11
3377063281,3sleep20-21swapper/22
3370444281,3sleep10-21swapper/11
3357973281,3sleep23358027-21ntpq2
3338180280,4sleep10-21swapper/11
3321947281,3sleep20-21swapper/22
3270923280,1sleep10-21swapper/11
3396263270,1sleep00-21swapper/00
3391834271,3sleep20-21swapper/22
3390787270,1sleep20-21swapper/22
3374643270,1sleep00-21swapper/00
3369561270,1sleep10-21swapper/11
3363656270,3sleep23363900-21users2
3353617271,3sleep10-21swapper/11
3346422270,4sleep00-21swapper/00
3329838270,4sleep1321rcuc/11
3321440270,1sleep20-21swapper/22
3321321270,1sleep00-21swapper/00
3318031271,3sleep10-21swapper/11
3316081270,1sleep10-21swapper/11
3315981270,1sleep20-21swapper/22
3310763271,3sleep13310764-21pool-udisksd1
3295020270,1sleep20-21swapper/22
3288722270,3sleep13284358-21/usr/sbin/munin1
3283467270,1sleep20-21swapper/22
3278094270,1sleep10-21swapper/11
3380841261,2sleep10-21swapper/11
3375236260,2sleep00-21swapper/00
3359831261,2sleep03361441-21kthreadcore0
3342859260,1sleep20-21swapper/22
3339528261,2sleep00-21swapper/00
3333955261,2sleep13335698-21kthreadcore1
3332850261,2sleep23332971-21fschecks_time2
3304657260,3sleep13304699-21ntpq1
3290029261,2sleep20-21swapper/22
3270838260,1sleep00-21swapper/00
3380977250,1sleep20-21swapper/22
3347215250,1sleep10-21swapper/11
3278376250,3sleep20-21swapper/22
3359023240,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional