You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 08:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu May 21, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3018543211756,56sleep30-21swapper/33
301850028670,10sleep10-21swapper/11
301356328556,7sleep00-21swapper/00
301840028353,11sleep20-21swapper/22
31817072241,19sleep03148826-21diskmemload0
3018787992112,6cyclictest0-21swapper/33
301878799211,16cyclictest3289889-21taskset3
301878799211,16cyclictest0-21swapper/33
30601642202,14sleep03061978-21tr0
3018787992017,1cyclictest0-21swapper/33
301878799201,16cyclictest3498699-21taskset3
301878799201,15cyclictest0-21swapper/33
301878799201,15cyclictest0-21swapper/33
301878799200,7cyclictest33652882sleep33
301878799200,17cyclictest0-21swapper/33
3018787991915,2cyclictest0-21swapper/33
301878799191,15cyclictest0-21swapper/33
301878799191,15cyclictest0-21swapper/33
301878799191,14cyclictest0-21swapper/33
301878799191,14cyclictest0-21swapper/33
301878799191,14cyclictest0-21swapper/33
301878799191,14cyclictest0-21swapper/33
301878799191,14cyclictest0-21swapper/33
301878799191,14cyclictest0-21swapper/33
301878799190,16cyclictest0-21swapper/33
32545572181,13sleep23254648-21date2
31104752181,13sleep23110520-21/usr/sbin/munin2
301878799187,8cyclictest3342192-21turbostat3
301878799187,8cyclictest3342192-21turbostat3
301878799181,14cyclictest66950irq/132-enp1s0-TxRx-33
301878799181,14cyclictest3173528-21taskset3
301878799181,14cyclictest31432952sleep33
301878799181,14cyclictest0-21swapper/33
301878799181,14cyclictest0-21swapper/33
301878799181,14cyclictest0-21swapper/33
301878799181,14cyclictest0-21swapper/33
301878799181,14cyclictest0-21swapper/33
301878799181,14cyclictest0-21swapper/33
301878799181,14cyclictest0-21swapper/33
301878799181,13cyclictest0-21swapper/33
301878799181,13cyclictest0-21swapper/33
301878799180,16cyclictest0-21swapper/33
301878799180,15cyclictest66950irq/132-enp1s0-TxRx-33
301878799180,15cyclictest0-21swapper/33
301878799180,15cyclictest0-21swapper/33
35164342170,6sleep20-21swapper/22
34305332171,4sleep03148826-21diskmemload0
31428952171,4sleep03143111-21users0
3018787991714,1cyclictest0-21swapper/33
3018787991712,2cyclictest0-21swapper/33
301878799171,14cyclictest0-21swapper/33
301878799171,14cyclictest0-21swapper/33
301878799171,13cyclictest0-21swapper/33
301878799171,12cyclictest0-21swapper/33
301878799170,14cyclictest0-21swapper/33
301878799170,14cyclictest0-21swapper/33
301878799170,13cyclictest0-21swapper/33
301878799170,13cyclictest0-21swapper/33
35007982161,2sleep00-21swapper/00
34222132160,5sleep23148826-21diskmemload2
34014362161,4sleep03148826-21diskmemload0
33929792160,2sleep00-21swapper/00
32636492161,2sleep10-21swapper/11
32440512161,2sleep00-21swapper/00
32327502161,4sleep03232811-21ntpq0
3018787991612,3cyclictest0-21swapper/33
301878799161,12cyclictest3127524-21sleep3
301878799160,13cyclictest0-21swapper/33
301878799160,13cyclictest0-21swapper/33
301878799160,13cyclictest0-21swapper/33
301878799160,13cyclictest0-21swapper/33
35099312151,3sleep13148826-21diskmemload1
34912982150,2sleep00-21swapper/00
33425782151,8sleep23342602-21apt-get2
33425782151,8sleep23342602-21apt-get2
33162092151,3sleep03148826-21diskmemload0
32968082150,1sleep20-21swapper/22
32562032151,7sleep23257857-21kthreadcore2
32367272151,3sleep20-21swapper/22
30838732151,9sleep23083919-21ntpq2
301878799151,12cyclictest0-21swapper/33
301878799151,12cyclictest0-21swapper/33
301878799151,10cyclictest3074189-21turbostat3
301878799150,13cyclictest0-21swapper/33
301878799150,13cyclictest0-21swapper/33
301878799150,12cyclictest0-21swapper/33
301878799150,12cyclictest0-21swapper/33
301878799150,12cyclictest0-21swapper/33
35019452143,7sleep03502185-21cat0
34015032141,2sleep20-21swapper/22
33949222141,3sleep13148826-21diskmemload1
33822052141,8sleep13382263-21ntpq1
32347772140,2sleep1321rcuc/11
32261642141,9sleep03148826-21diskmemload0
32105972141,6sleep03210656-21munin-run0
31889022141,7sleep03188959-21ntpq0
31659092141,2sleep20-21swapper/22
30953052140,1sleep20-21swapper/22
301878799149,3cyclictest0-21swapper/33
301878799140,2cyclictest0-21swapper/33
301878799140,12cyclictest0-21swapper/33
301878799140,12cyclictest0-21swapper/33
301878799140,11cyclictest0-21swapper/33
301878799140,11cyclictest0-21swapper/33
35002522131,8sleep13148826-21diskmemload1
34732132131,3sleep23473223-21ssh2
34386942130,1sleep10-21swapper/11
33652502130,1sleep00-21swapper/00
33651712131,7sleep13148826-21diskmemload1
33388382131,3sleep23338849-21ssh2
33310532130,3sleep13148826-21diskmemload1
33224252138,2sleep13322438-21sh1
32993332131,8sleep03300521-21grep0
31566262130,2sleep20-21swapper/22
30750122131,8sleep13075547-21cut1
301878799130,10cyclictest0-21swapper/33
34548282122,4sleep23148826-21diskmemload2
34393162121,6sleep20-21swapper/22
34203242121,7sleep03420332-21ssh0
34126772121,7sleep03412690-21rm0
34052802121,2sleep20-21swapper/22
33492862121,2sleep13349297-21ssh1
33492862121,2sleep13349297-21ssh1
33488732121,3sleep00-21swapper/00
33488732121,3sleep00-21swapper/00
33215762121,1sleep20-21swapper/22
33120432123,2sleep0171rcu_preempt0
33048182121,3sleep23304832-21ssh2
32691442121,7sleep03148826-21diskmemload0
32520452121,1sleep00-21swapper/00
32109542121,3sleep10-21swapper/11
32067342121,6sleep23206771-21ssh2
31625282121,5sleep03162592-21ntpq0
31571072121,6sleep03157118-21ssh0
31052122121,7sleep03105271-21ntpq0
30957762122,5sleep13095795-21apt-get1
30572442121,6sleep13057290-21ntpq1
30359722122,5sleep23036026-21ntpq2
30359242121,6sleep13035970-21ntpq1
35010702111,6sleep23501083-21ssh2
34824112110,3sleep23482425-21ssh2
34723762111,2sleep10-21swapper/11
34708322111,6sleep03470929-21grep0
34606552111,5sleep23461759-21memory2
34553582111,3sleep00-21swapper/00
34531592111,5sleep13453211-21ntpq1
34480112110,6sleep03148826-21diskmemload0
34457832111,3sleep20-21swapper/22
34368572111,2sleep00-21swapper/00
33837212110,2sleep00-21swapper/00
33746072111,4sleep00-21swapper/00
33359282112,5sleep03337674-21cut0
33136162110,7sleep13313626-21ssh1
32795942111,5sleep1331ktimers/11
32789752111,5sleep00-21swapper/00
32725022111,5sleep23272528-21apt-get2
32614182111,3sleep10-21swapper/11
32249992111,4sleep13148826-21diskmemload1
32097462110,2sleep10-21swapper/11
31993302110,1sleep20-21swapper/22
31861462111,7sleep10-21swapper/11
31799362111,5sleep23180039-21ntpq2
31721052111,3sleep00-21swapper/00
31210222111,6sleep13117109-21/usr/sbin/munin1
30900712110,1sleep10-21swapper/11
30551852111,6sleep23057004-21grep2
30420342110,1sleep00-21swapper/00
35173912101,5sleep03517403-21ssh0
35092202101,3sleep20-21swapper/22
34914602101,4sleep13148826-21diskmemload1
34820982101,4sleep13148826-21diskmemload1
34802442101,2sleep00-21swapper/00
34641442101,4sleep13148826-21diskmemload1
34619112102,4sleep03461954-21ntpq0
34454632101,4sleep13148826-21diskmemload1
34286752101,3sleep10-21swapper/11
34185112102,4sleep23417902-21kworker/u16:2+nfsiod2
34134942104,3sleep10-21swapper/11
34100402101,4sleep13148826-21diskmemload1
33922882101,4sleep23148826-21diskmemload2
33841582101,4sleep23148826-21diskmemload2
33740722101,3sleep20-21swapper/22
33740012101,4sleep13148826-21diskmemload1
33679242101,3sleep20-21swapper/22
33591662101,4sleep23148826-21diskmemload2
33564462101,4sleep13148826-21diskmemload1
33391322101,4sleep13148826-21diskmemload1
33313142101,4sleep23148826-21diskmemload2
33136272101,3sleep23148826-21diskmemload2
33038822101,5sleep13148826-21diskmemload1
32894632101,4sleep13289474-21ssh1
32622842101,3sleep00-21swapper/00
32438732101,3sleep23148826-21diskmemload2
32438622101,5sleep10-21swapper/11
32177872101,4sleep23148826-21diskmemload2
31990762101,3sleep13148826-21diskmemload1
31830002101,3sleep10-21swapper/11
31749392101,2sleep10-21swapper/11
31667412101,3sleep20-21swapper/22
31639462101,4sleep13148826-21diskmemload1
31389272101,5sleep13139089-21grep1
31277572101,4sleep03127776-21apt-get0
31050672101,4sleep23105155-21ntpq2
30954902101,5sleep13095519-21munin-run1
30851152101,3sleep20-21swapper/22
30838262101,5sleep13083871-21ntpq1
30678402101,5sleep03067881-21ntpq0
30265162101,4sleep13026772-21df1
3514870291,5sleep13148826-21diskmemload1
3484415291,4sleep23484644-21sort2
3400152291,3sleep13400221-21ntp_states1
3359388291,4sleep03359400-21ssh0
3331397291,4sleep03331408-21ssh0
3296587291,3sleep13148826-21diskmemload1
3295452290,1sleep00-21swapper/00
3287439291,3sleep03148826-21diskmemload0
3286534290,1sleep20-21swapper/22
3268280291,3sleep23148826-21diskmemload2
3246497291,3sleep13148826-21diskmemload1
3227316291,2sleep20-21swapper/22
3216232291,2sleep00-21swapper/00
3198647291,3sleep00-21swapper/00
3191309290,1sleep20-21swapper/22
3155491291,3sleep10-21swapper/11
3126532291,4sleep03126586-21ntpq0
3117108291,3sleep03117127-21apt-get0
3111756290,3sleep10-21swapper/11
3073131291,3sleep13073161-21ntpq1
3068856290,5sleep23069021-21latency_hist2
3062664290,1sleep10-21swapper/11
3058944290,1sleep20-21swapper/22
3053188291,3sleep03053207-21apt-get0
3047858291,3sleep23048078-21expr2
3046628291,3sleep23046684-21ntp_states2
3043098291,4sleep13044263-21cut1
3035782291,2sleep00-21swapper/00
3137144282,3sleep23018778-21cyclictest2
3133635281,3sleep13134075-21grep1
3132826281,3sleep23132940-21latency_hist2
3125517281,3sleep13126358-21missed_timers1
3115978280,1sleep00-21swapper/00
3112927281,3sleep20-21swapper/22
3106040280,1sleep10-21swapper/11
3079211280,1sleep20-21swapper/22
3024080280,2sleep00-21swapper/00
3024080280,2sleep00-21swapper/00
3019659281,3sleep23019771-21grep2
3019659281,3sleep23019771-21grep2
3148659270,1sleep20-21swapper/22
3143186270,1sleep20-21swapper/22
3131830270,4sleep10-21swapper/11
3117632271,2sleep23117882-21kthreadcore2
3111422270,1sleep10-21swapper/11
3100808270,1sleep00-21swapper/00
3096392271,3sleep20-21swapper/22
3047833270,2sleep10-21swapper/11
3047819270,1sleep00-21swapper/00
3047556270,1sleep00-21swapper/00
3040155271,3sleep10-21swapper/11
3031466270,1sleep20-21swapper/22
3026491270,2sleep00-21swapper/00
3147785260,2sleep00-21swapper/00
3144953261,2sleep10-21swapper/11
3127155260,1sleep20-21swapper/22
3094332260,2sleep00-21swapper/00
3089818260,1sleep00-21swapper/00
3083675261,2sleep00-21swapper/00
3073168260,2sleep23073217-21ntpq2
3069110260,1sleep00-21swapper/00
3064866261,2sleep10-21swapper/11
3041129260,2sleep20-21swapper/22
3023933260,1sleep10-21swapper/11
3023933260,1sleep10-21swapper/11
3133414250,1sleep00-21swapper/00
3109093250,1sleep00-21swapper/00
3078621250,2sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional