You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri Apr 17, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2358635211760,51sleep30-21swapper/33
235870827963,10sleep10-21swapper/11
235844727557,12sleep20-21swapper/22
235858427353,14sleep00-21swapper/00
235884499293,21cyclictest2626911-21taskset3
28574902240,2sleep30-21swapper/33
2358844992218,2cyclictest0-21swapper/33
235884499221,17cyclictest2786105-21taskset3
2358844992118,1cyclictest0-21swapper/33
2358844992118,1cyclictest0-21swapper/33
2358844992118,1cyclictest0-21swapper/33
2358844992118,1cyclictest0-21swapper/33
2358844992117,2cyclictest0-21swapper/33
2358844992117,1cyclictest0-21swapper/33
235884499211,17cyclictest0-21swapper/33
235884499211,16cyclictest2741874-21taskset3
235884499211,16cyclictest2534330-21taskset3
235884499210,19cyclictest0-21swapper/33
26272892201,3sleep10-21swapper/11
235884499206,12cyclictest0-21swapper/33
235884499206,12cyclictest0-21swapper/33
2358844992017,2cyclictest0-21swapper/33
2358844992017,1cyclictest0-21swapper/33
2358844992017,1cyclictest0-21swapper/33
2358844992015,2cyclictest0-21swapper/33
2358844992015,2cyclictest0-21swapper/33
235884499201,16cyclictest0-21swapper/33
235884499201,15cyclictest0-21swapper/33
235884499199,7cyclictest2824329-21chrt3
235884499199,7cyclictest2576946-21chrt3
235884499199,6cyclictest66950irq/132-enp1s0-TxRx-33
235884499191,6cyclictest2523330-21taskset3
235884499191,6cyclictest2359007-21taskset3
2358844991916,1cyclictest0-21swapper/33
2358844991916,1cyclictest0-21swapper/33
2358844991914,2cyclictest0-21swapper/33
2358844991912,4cyclictest26640802sleep33
235884499191,15cyclictest0-21swapper/33
235884499191,14cyclictest0-21swapper/33
235884499191,14cyclictest0-21swapper/33
235884499191,14cyclictest0-21swapper/33
235884499191,14cyclictest0-21swapper/33
235884499191,14cyclictest0-21swapper/33
27433852181,5sleep22743387-21ssh2
26189432181,4sleep02490250-21diskmemload0
235884499181,5cyclictest66950irq/132-enp1s0-TxRx-33
2358844991815,1cyclictest0-21swapper/33
2358844991815,1cyclictest0-21swapper/33
2358844991815,1cyclictest0-21swapper/33
2358844991815,1cyclictest0-21swapper/33
2358844991815,1cyclictest0-21swapper/33
235884499181,4cyclictest0-21swapper/33
2358844991814,2cyclictest0-21swapper/33
2358844991814,2cyclictest0-21swapper/33
2358844991814,2cyclictest0-21swapper/33
2358844991813,2cyclictest0-21swapper/33
28133742171,4sleep00-21swapper/00
27770422171,4sleep12490250-21diskmemload1
27258882171,3sleep20-21swapper/22
26627232170,1sleep00-21swapper/00
26460042170,5sleep20-21swapper/22
26112492171,3sleep20-21swapper/22
25521882171,4sleep22552408-21hddtemp_smartct2
23812122171,7sleep22381287-21sshd2
2358844991714,1cyclictest0-21swapper/33
2358844991714,1cyclictest0-21swapper/33
235884499171,13cyclictest2425386-21taskset3
235884499170,13cyclictest0-21swapper/33
235884499170,13cyclictest0-21swapper/33
235884499170,13cyclictest0-21swapper/33
28028742161,4sleep12490250-21diskmemload1
27510192161,2sleep20-21swapper/22
26573302161,8sleep12657555-21hddtemp_smartct1
25761602161,2sleep00-21swapper/00
2358844991613,1cyclictest0-21swapper/33
2358844991612,2cyclictest0-21swapper/33
2358844991611,2cyclictest0-21swapper/33
235884499160,14cyclictest0-21swapper/33
27634212151,9sleep22765037-21grep2
26649732150,3sleep20-21swapper/22
26534592151,4sleep014-21ksoftirqd/00
26276042151,2sleep20-21swapper/22
24956112154,7sleep22490250-21diskmemload2
2358844991512,2cyclictest0-21swapper/33
2358844991512,1cyclictest0-21swapper/33
2358844991512,1cyclictest0-21swapper/33
235884499151,12cyclictest0-21swapper/33
235884499150,13cyclictest0-21swapper/33
235884499150,12cyclictest0-21swapper/33
28578562141,8sleep02857870-21ssh0
28567132140,3sleep10-21swapper/11
28369272141,6sleep12837162-21users1
27382082141,7sleep12735827-21/usr/sbin/munin1
26796042141,2sleep20-21swapper/22
26730192140,1sleep20-21swapper/22
26310432141,7sleep22631169-21irqstats2
26291682141,7sleep00-21swapper/00
25850102141,9sleep02585024-21ssh0
25768242140,1sleep10-21swapper/11
25412542141,2sleep20-21swapper/22
24427422141,7sleep02442774-21kthreadcore0
2358844991412,1cyclictest0-21swapper/33
235884499140,2cyclictest0-21swapper/33
235884499140,2cyclictest0-21swapper/33
28493372131,8sleep02849570-21latency_hist0
28462442131,6sleep22490250-21diskmemload2
28292232131,8sleep12829234-21ssh1
28209422131,8sleep12490250-21diskmemload1
28061112131,6sleep10-21swapper/11
27848222131,8sleep12490250-21diskmemload1
27773532131,5sleep20-21swapper/22
27513422131,8sleep02751356-21ssh0
27344152131,8sleep22734426-21sh2
27051532131,5sleep12705211-21ntpq1
26994672131,7sleep12490250-21diskmemload1
25787272130,2sleep10-21swapper/11
25592072131,8sleep02559221-21ssh0
25507252131,8sleep02550736-21ssh0
25311092131,8sleep02531122-21ssh0
235884499130,1cyclictest0-21swapper/33
28285912121,8sleep22828603-21sh2
28143642120,1sleep20-21swapper/22
27885822121,6sleep02490250-21diskmemload0
27791572121,5sleep00-21swapper/00
27700932121,7sleep02770105-21sh0
27610492120,7sleep02490250-21diskmemload0
27418362120,8sleep02741850-21ssh0
27237922120,7sleep02490250-21diskmemload0
26554182120,2sleep20-21swapper/22
26368682122,6sleep02636884-21diskmemload0
26107922121,6sleep12490250-21diskmemload1
25789752121,7sleep20-21swapper/22
25669832121,7sleep02566996-21ssh0
25422432121,6sleep02490250-21diskmemload0
25168172122,5sleep22516838-21apt-get2
24025902121,6sleep12402636-21ntpq1
23818372120,1sleep00-21swapper/00
28377742111,2sleep22837788-21ssh2
28015812113,3sleep22801637-21ntpq2
27942612111,3sleep20-21swapper/22
27881662111,2sleep20-21swapper/22
27703142111,4sleep12490250-21diskmemload1
27590562114,3sleep22759057-21sh2
27488272111,5sleep12748883-21ntpq1
27226462111,5sleep12722683-21ntpq1
27075832111,4sleep02490250-21diskmemload0
27006242111,6sleep22700642-21latency_hist2
26825482111,3sleep00-21swapper/00
26661062111,5sleep12666324-21hddtemp_smartct1
26537192111,7sleep12490250-21diskmemload1
25999212111,6sleep22600056-21wc2
25871812111,5sleep22587401-21hddtemp_smartct2
25868132111,5sleep02490250-21diskmemload0
25667092110,3sleep22566723-21ssh2
25319062111,2sleep20-21swapper/22
25309732110,1sleep10-21swapper/11
25135312110,6sleep20-21swapper/22
25135312110,6sleep20-21swapper/22
25048702111,4sleep12490250-21diskmemload1
24856052111,5sleep12485721-21irqstats1
24691532112,4sleep02469172-21apt-get0
24679432111,5sleep02468000-21ntp_states0
24640962111,5sleep12464309-21hddtemp_smartct1
24500762111,4sleep22451390-21memory2
24368372111,5sleep22437095-21df_inode2
24352472111,5sleep22435296-21ntpq2
24352472111,5sleep22435296-21ntpq2
24299052111,5sleep12429954-21ntpq1
24192772111,7sleep22419332-21kworker/u16:32
23969582111,5sleep12397009-21ntpq1
23716692111,5sleep22371927-21df_inode2
23716692111,5sleep22371927-21df_inode2
28321592101,3sleep00-21swapper/00
28259392101,5sleep02827496-21latency0
28040412101,1sleep00-21swapper/00
27938402101,4sleep12490250-21diskmemload1
27868252101,4sleep02490250-21diskmemload0
27609642101,2sleep10-21swapper/11
27139872101,4sleep22714052-21ntp_states2
27069532101,2sleep20-21swapper/22
26993972101,3sleep00-21swapper/00
26896432101,2sleep00-21swapper/00
26896162101,3sleep20-21swapper/22
26889552100,2sleep10-21swapper/11
26721782100,5sleep02490250-21diskmemload0
26474372101,4sleep12490250-21diskmemload1
26360702101,3sleep12490250-21diskmemload1
26208952101,4sleep22490250-21diskmemload2
26196152101,4sleep12490250-21diskmemload1
26125902101,4sleep02490250-21diskmemload0
25956332101,4sleep02490250-21diskmemload0
25955732101,3sleep10-21swapper/11
25917942101,2sleep10-21swapper/11
25763322102,4sleep22576347-21ssh2
25652552101,5sleep12565272-21sensors1
25568482101,4sleep12490250-21diskmemload1
25497382101,3sleep172-21kcompactd01
25401742101,4sleep12490250-21diskmemload1
25247752101,2sleep10-21swapper/11
25231392101,5sleep02513963-21kworker/u16:2+nfsiod0
25162032101,4sleep12490250-21diskmemload1
25162032101,4sleep12490250-21diskmemload1
25073542100,2sleep22507368-21ssh2
24959342101,4sleep02490250-21diskmemload0
24930492100,6sleep12494613-21cat1
24798542101,4sleep02480115-21df_inode0
24785272100,6sleep22478560-21ntpq2
24255652100,1sleep00-21swapper/00
24146302101,3sleep00-21swapper/00
24097882101,5sleep12409792-21find1
23876252101,4sleep22387797-21expr2
23630612101,4sleep22363112-21ntpq2
2849719291,3sleep22849744-21apt-get2
2820751291,2sleep20-21swapper/22
2820079290,1sleep00-21swapper/00
2732180291,3sleep12490250-21diskmemload1
2729037291,4sleep02730774-21cut0
2715444291,3sleep02490250-21diskmemload0
2709782291,4sleep12490250-21diskmemload1
2643585291,2sleep00-21swapper/00
2548552290,1sleep20-21swapper/22
2509350290,6sleep02510982-21cut0
2509350290,6sleep02510982-21cut0
2490180290,1sleep00-21swapper/00
2463806291,3sleep22463825-21apt-get2
2458422291,4sleep02457903-21apt-get0
2446281291,4sleep12446331-21ntpq1
2429956291,4sleep22430015-21ld2
2397124291,4sleep02397241-21sendmail_mailqu0
2397011291,4sleep22397066-21ntpq2
2375694291,4sleep12375731-21ntpq1
2375694291,4sleep12375731-21ntpq1
2847338281,3sleep12490250-21diskmemload1
2674518281,3sleep10-21swapper/11
2499025280,3sleep02499235-21ssh0
2490104280,1sleep20-21swapper/22
2469175281,3sleep12469430-21/usr/sbin/munin1
2441888280,1sleep00-21swapper/00
2398559281,3sleep22398747-21cstates2
2388111281,3sleep12388208-21ps1
2366832280,5sleep12367067-21kthreadcore1
2359298281,3sleep02359422-21grep0
2463287270,3sleep10-21swapper/11
2459007271,3sleep20-21swapper/22
2457830270,2sleep00-21swapper/00
2447494270,2sleep00-21swapper/00
2425849271,3sleep02425859-21find0
2425311270,1sleep20-21swapper/22
2415169270,2sleep10-21swapper/11
2409160270,1sleep10-21swapper/11
2404153270,1sleep00-21swapper/00
2392510270,1sleep00-21swapper/00
2383413271,3sleep00-21swapper/00
2484274260,2sleep10-21swapper/11
2484118260,3sleep20-21swapper/22
2479313260,1sleep10-21swapper/11
2473544261,2sleep20-21swapper/22
2452370260,1sleep10-21swapper/11
2447127260,1sleep20-21swapper/22
2436816261,1sleep10-21swapper/11
2436043260,1sleep10-21swapper/11
2436043260,1sleep10-21swapper/11
2420227260,1sleep00-21swapper/00
2410354261,2sleep20-21swapper/22
2404136260,2sleep20-21swapper/22
2398832260,2sleep02399044-21smartctl0
2387071260,1sleep10-21swapper/11
2386184260,2sleep20-21swapper/22
2367070261,2sleep20-21swapper/22
2363939260,1sleep10-21swapper/11
2478860250,2sleep00-21swapper/00
2453135251,1sleep20-21swapper/22
2420674250,2sleep10-21swapper/11
2381711250,3sleep10-21swapper/11
2375603250,2sleep02375693-21ldconfig0
2375603250,2sleep02375693-21ldconfig0
2371155250,2sleep00-21swapper/00
2453088240,1sleep10-21swapper/11
2430965240,1sleep00-21swapper/00
2430965240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional