You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 03:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Tue Apr 28, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3319566213657,73sleep30-21swapper/33
362573521270,2sleep00-21swapper/00
331963328156,11sleep20-21swapper/22
331942228052,14sleep10-21swapper/11
331932427652,18sleep00-21swapper/00
3319818992826,1cyclictest0-21swapper/33
37548362261,8sleep03449886-21diskmemload0
34570862250,2sleep10-21swapper/11
331981899251,21cyclictest0-21swapper/33
331981899221,18cyclictest0-21swapper/33
331981899211,16cyclictest0-21swapper/33
331981899211,15cyclictest0-21swapper/33
3319818992010,6cyclictest3535150-21turbostat3
331981899198,7cyclictest3704816-21taskset3
331981899198,7cyclictest3704816-21taskset3
3319818991917,1cyclictest0-21swapper/33
3319818991915,2cyclictest0-21swapper/33
331981899191,15cyclictest3685826-21taskset3
331981899191,15cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899191,14cyclictest0-21swapper/33
331981899190,16cyclictest66950irq/132-enp1s0-TxRx-33
38045352181,3sleep00-21swapper/00
34813052181,2sleep30-21swapper/33
331981899187,7cyclictest3406694-21chrt3
3319818991811,4cyclictest66950irq/132-enp1s0-TxRx-33
331981899181,14cyclictest3737191-21chrt3
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,14cyclictest0-21swapper/33
331981899181,13cyclictest0-21swapper/33
331981899181,13cyclictest0-21swapper/33
331981899181,13cyclictest0-21swapper/33
3319818991810,5cyclictest3337921-21taskset3
331981899180,14cyclictest3390808-21taskset3
331981899180,14cyclictest0-21swapper/33
37212922170,1sleep20-21swapper/22
36876122171,3sleep10-21swapper/11
36731742171,3sleep00-21swapper/00
35532902171,6sleep23554859-21grep2
331981899172,10cyclictest3577952-21taskset3
3319818991714,1cyclictest0-21swapper/33
331981899171,13cyclictest0-21swapper/33
331981899171,13cyclictest0-21swapper/33
331981899171,13cyclictest0-21swapper/33
331981899171,12cyclictest0-21swapper/33
3319818991710,4cyclictest3348283-21chrt3
37363462161,3sleep13449886-21diskmemload1
37129822160,1sleep20-21swapper/22
36609822161,3sleep20-21swapper/22
35868962160,4sleep1321rcuc/11
34828952161,2sleep10-21swapper/11
34237562161,6sleep13424223-21cut1
331981899162,10cyclictest3412555-21taskset3
331981899161,3cyclictest0-21swapper/33
3319818991613,1cyclictest0-21swapper/33
3319818991613,1cyclictest0-21swapper/33
331981899161,13cyclictest0-21swapper/33
331981899161,13cyclictest0-21swapper/33
331981899160,13cyclictest0-21swapper/33
331981899160,13cyclictest0-21swapper/33
331981899160,13cyclictest0-21swapper/33
37717892150,3sleep00-21swapper/00
37643912151,2sleep20-21swapper/22
37538862151,4sleep23753899-21ssh2
36874352151,2sleep20-21swapper/22
36462092151,3sleep13449886-21diskmemload1
36121452152,7sleep23612410-21df2
35530532151,9sleep13553289-21grep1
35508262151,4sleep03550841-21ssh0
34730012151,2sleep10-21swapper/11
331981899151,2cyclictest0-21swapper/33
3319818991512,2cyclictest0-21swapper/33
331981899151,11cyclictest33855132sleep33
331981899151,10cyclictest3337686-21turbostat3
331981899150,12cyclictest0-21swapper/33
331981899150,12cyclictest0-21swapper/33
331981899150,11cyclictest0-21swapper/33
37822002141,4sleep23449886-21diskmemload2
36931802142,6sleep03693244-21ntpq0
35679952141,3sleep03568006-21ssh0
35146402141,9sleep03514900-21/usr/sbin/munin0
35146402141,9sleep03514900-21/usr/sbin/munin0
34740492141,8sleep23474058-21ssh2
34665292141,3sleep13466531-21ssh1
33645702141,8sleep03364828-21df_inode0
33645702141,8sleep03364828-21df_inode0
3319818991411,2cyclictest0-21swapper/33
331981899140,12cyclictest0-21swapper/33
331981899140,12cyclictest0-21swapper/33
331981899140,10cyclictest34387172sleep33
331981899140,10cyclictest0-21swapper/33
37981562131,7sleep13449886-21diskmemload1
37886042131,7sleep03449886-21diskmemload0
37464912131,3sleep13449886-21diskmemload1
37186512131,6sleep03718707-21ntpq0
37044432131,8sleep03704455-21ssh0
37044432131,8sleep03704455-21ssh0
36426412131,8sleep13642652-21sh1
36279152131,8sleep23627929-21ssh2
35949332131,8sleep03594943-21ssh0
35937782130,2sleep00-21swapper/00
35396452131,8sleep13539708-21ssh1
35329762131,8sleep23532989-21ssh2
35326702131,3sleep13449886-21diskmemload1
33367032131,5sleep13336757-21ntpq1
33231242131,8sleep13323937-21sed1
37890942121,7sleep23789108-21ssh2
37483862122,5sleep2555-21dbus-daemon2
37130912121,6sleep03449886-21diskmemload0
37016002121,5sleep23701664-21ntpq2
37016002121,5sleep23701664-21ntpq2
36634342121,7sleep03663447-21/usr/sbin/munin0
36597642121,5sleep00-21swapper/00
36546722122,5sleep13654698-21apt-get1
36542172120,1sleep00-21swapper/00
36000152121,7sleep13449886-21diskmemload1
35243812121,3sleep23524395-21ssh2
35142082121,3sleep23514270-21ssh2
35142082121,3sleep23514270-21ssh2
34812202120,5sleep20-21swapper/22
34288012121,8sleep03428932-21tail0
38048822111,3sleep10-21swapper/11
37990012110,7sleep23449886-21diskmemload2
37708752111,7sleep23770889-21ssh2
37614002111,5sleep13761452-21ntpq1
37534242111,5sleep13753640-21users1
37394942111,6sleep03449886-21diskmemload0
37289192110,7sleep03449886-21diskmemload0
37275142110,2sleep10-21swapper/11
37147472111,5sleep13714881-21perl1
37123232111,5sleep10-21swapper/11
36935212111,6sleep13693809-21sed1
36788412111,6sleep23678855-21ssh2
36633482111,3sleep10-21swapper/11
36162712111,5sleep13616326-21ntpq1
36161052111,6sleep03616269-21cut0
36101992111,6sleep13610210-21ssh1
36101992111,6sleep13610210-21ssh1
35991352111,5sleep23599191-21ntpq2
35930412111,6sleep23593052-21ssh2
35779202111,4sleep23449886-21diskmemload2
35443142111,7sleep13544473-21grep1
35397092111,5sleep23539763-21ntpq2
35060612111,4sleep03449886-21diskmemload0
34969572111,5sleep13497058-21ntpq1
34911342110,7sleep13491145-21sh1
34508492111,6sleep23450979-21/usr/sbin/munin2
34436002111,5sleep03443657-21ntp_states0
34377482111,5sleep13437889-21ntpq1
33902552111,6sleep23390492-21/usr/sbin/munin2
33648292110,7sleep13364940-21grep1
33648292110,7sleep13364940-21grep1
33529762110,7sleep03353233-21grep0
37745382101,4sleep13774757-21hddtemp_smartct1
37744002101,4sleep03774535-21diskstats0
37662982101,6sleep13767063-21kthreadcore1
37316552101,3sleep23731690-21ssh2
37033662101,4sleep10-21swapper/11
37033662101,4sleep10-21swapper/11
36685052101,3sleep20-21swapper/22
36366152101,4sleep13449886-21diskmemload1
36258052100,6sleep13625816-21diskmemload1
36066792100,6sleep23607555-21/usr/sbin/munin2
36066792100,6sleep23607555-21/usr/sbin/munin2
36041712101,5sleep03604214-21kthreadcore0
36041712101,5sleep03604214-21kthreadcore0
35855462101,4sleep13585556-21ssh1
35771132100,2sleep20-21swapper/22
35752962101,4sleep13449886-21diskmemload1
35670902101,3sleep20-21swapper/22
35662632101,4sleep13449886-21diskmemload1
35444772102,3sleep23449886-21diskmemload2
35327492101,4sleep03449886-21diskmemload0
35245492101,2sleep00-21swapper/00
35141522101,5sleep13514206-21ntpq1
35141522101,5sleep13514206-21ntpq1
35081712101,3sleep20-21swapper/22
35077722101,4sleep13507786-21ssh1
34979162101,2sleep00-21swapper/00
34934962101,6sleep23494514-21kthreadcore2
34775342101,5sleep00-21swapper/00
34637642101,5sleep23463980-21grep2
34637102101,4sleep00-21swapper/00
34068942101,4sleep03406923-21munin-run0
34059112101,4sleep13405958-21ntpq1
33757462101,5sleep13375816-21kthreadcore1
33526312101,3sleep23352679-21ntpq2
33435422101,4sleep03343645-21perl0
33432612101,4sleep20-21swapper/22
33353982101,4sleep23336511-21memory2
3799445291,3sleep23449886-21diskmemload2
3790071291,3sleep13449886-21diskmemload1
3764174292,2sleep03319812-21cyclictest0
3744448291,3sleep03449886-21diskmemload0
3729786290,2sleep20-21swapper/22
3694216291,2sleep23449886-21diskmemload2
3687190290,1sleep00-21swapper/00
3677046291,4sleep10-21swapper/11
3643725290,1sleep20-21swapper/22
3642670291,3sleep03449886-21diskmemload0
3633464291,4sleep23633532-21gcc2
3584114290,4sleep03584125-21ssh0
3559012291,3sleep03449886-21diskmemload0
3541054291,4sleep03541067-21ssh0
3491095291,2sleep20-21swapper/22
3489662291,2sleep00-21swapper/00
3451141291,4sleep03452152-21cut0
3439438291,4sleep13438917-21apt-get1
3438000291,5sleep03438065-21ls0
3416590291,4sleep23416637-21ntpq2
3412169291,3sleep03412188-21apt-get0
3407440291,4sleep13407578-21grep1
3392038290,5sleep03393264-21kthreadcore0
3376169291,5sleep23377891-21kthreadcore2
3359366290,4sleep23359531-21diskstats2
3342024291,4sleep03342065-21ntpq0
3332640291,2sleep00-21swapper/00
3327858291,4sleep03328836-21cut0
3797758281,2sleep00-21swapper/00
3646223281,3sleep23646236-21ssh2
3634170281,3sleep03634177-21ssh0
3575894281,2sleep00-21swapper/00
3524222281,4sleep13449886-21diskmemload1
3473839281,2sleep00-21swapper/00
3438909281,4sleep23439121-21latency_hist2
3432505280,4sleep10-21swapper/11
3422933281,3sleep1802-21snmpd1
3417884281,2sleep20-21swapper/22
3401565281,3sleep11-21systemd1
3396522280,2sleep00-21swapper/00
3387656280,4sleep13389483-21kthreadcore1
3385406280,1sleep20-21swapper/22
3380174280,1sleep00-21swapper/00
3328837281,3sleep10-21swapper/11
3321367281,3sleep03323122-21kthreadcore0
3319975281,3sleep23319992-21apt2
3444831271,3sleep23444976-21cstates2
3443497270,4sleep20-21swapper/22
3428530270,3sleep23428581-21grep2
3424224271,3sleep23425949-21kthreadcore2
3415836270,3sleep03412533-21/usr/sbin/munin0
3401380270,1sleep20-21swapper/22
3396100270,2sleep10-21swapper/11
3385432273,2sleep03325195-21munin-node0
3385244270,4sleep10-21swapper/11
3369862270,2sleep20-21swapper/22
3368443270,3sleep20-21swapper/22
3368443270,3sleep20-21swapper/22
3364284270,1sleep10-21swapper/11
3359243270,2sleep00-21swapper/00
3353528270,1sleep10-21swapper/11
3445313261,2sleep10-21swapper/11
3418160261,2sleep00-21swapper/00
3412522260,4sleep10-21swapper/11
3406197261,2sleep20-21swapper/22
3391471261,2sleep20-21swapper/22
3386438261,2sleep00-21swapper/00
3374928260,1sleep00-21swapper/00
3374612260,1sleep10-21swapper/11
3354450261,2sleep10-21swapper/11
3348110260,1sleep10-21swapper/11
3332359260,2sleep20-21swapper/22
3427094250,1sleep00-21swapper/00
3411724250,1sleep20-21swapper/22
3342112250,2sleep20-21swapper/22
3342073250,1sleep10-21swapper/11
3445099240,1sleep00-21swapper/00
3354228240,1sleep00-21swapper/00
3353697240,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional