You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 21:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Mar 15, 2026 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2335134212654,67sleep30-21swapper/33
233459028361,12sleep10-21swapper/11
233035227963,9sleep20-21swapper/22
233536127756,15sleep00-21swapper/00
25093652630,2sleep0171rcu_preempt0
26647302320,2sleep20-21swapper/22
23743572231,16sleep12374445-21ntpq1
233557499221,17cyclictest0-21swapper/33
233557499220,20cyclictest0-21swapper/33
233557499203,13cyclictest0-21swapper/33
233557499202,14cyclictest2694914-21turbostat3
2335574992017,1cyclictest0-21swapper/33
2335574992015,3cyclictest0-21swapper/33
233557499201,13cyclictest0-21swapper/33
27574682191,12sleep12757500-21ssh1
26395352191,12sleep12466020-21diskmemload1
23762482191,7sleep12376565-21grep1
233557499191,7cyclictest2701639-21taskset3
233557499191,14cyclictest66950irq/132-enp1s0-TxRx-33
233557499191,14cyclictest0-21swapper/33
233557499191,14cyclictest0-21swapper/33
233557499191,14cyclictest0-21swapper/33
233557499191,14cyclictest0-21swapper/33
233557499191,14cyclictest0-21swapper/33
233557499191,14cyclictest0-21swapper/33
233557499191,14cyclictest0-21swapper/33
233557499184,1cyclictest0-21swapper/33
233557499184,1cyclictest0-21swapper/33
233557499181,5cyclictest66950irq/132-enp1s0-TxRx-33
233557499181,5cyclictest0-21swapper/33
2335574991815,1cyclictest0-21swapper/33
2335574991815,1cyclictest0-21swapper/33
2335574991814,2cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,14cyclictest0-21swapper/33
233557499181,13cyclictest0-21swapper/33
233557499181,13cyclictest0-21swapper/33
233557499180,15cyclictest66950irq/132-enp1s0-TxRx-33
233557499180,14cyclictest66950irq/132-enp1s0-TxRx-33
233557499180,14cyclictest0-21swapper/33
27401672171,11sleep02740294-21irqstats0
26658482171,2sleep00-21swapper/00
26326972170,8sleep12466020-21diskmemload1
25991132171,12sleep12599375-21cat1
25552092171,5sleep12555321-21irqstats1
25249192171,2sleep20-21swapper/22
25020192171,3sleep10-21swapper/11
24804282171,3sleep00-21swapper/00
233557499171,13cyclictest0-21swapper/33
233557499171,13cyclictest0-21swapper/33
233557499171,13cyclictest0-21swapper/33
233557499171,13cyclictest0-21swapper/33
233557499171,12cyclictest66950irq/132-enp1s0-TxRx-33
233557499171,12cyclictest2466046-21turbostat3
233557499171,12cyclictest0-21swapper/33
233557499170,14cyclictest0-21swapper/33
233557499170,14cyclictest0-21swapper/33
233557499170,13cyclictest0-21swapper/33
233557499170,13cyclictest0-21swapper/33
233557499170,12cyclictest0-21swapper/33
27180192160,1sleep20-21swapper/22
26759842161,4sleep02466020-21diskmemload0
26749192161,9sleep20-21swapper/22
24615462161,11sleep22463060-21cut2
233557499161,13cyclictest0-21swapper/33
233557499161,12cyclictest0-21swapper/33
233557499160,12cyclictest0-21swapper/33
28262692151,2sleep20-21swapper/22
28084722151,10sleep02808487-21ssh0
26141032150,3sleep12466020-21diskmemload1
25613752150,2sleep00-21swapper/00
25535702151,2sleep20-21swapper/22
24995322151,3sleep20-21swapper/22
24802862151,9sleep12466020-21diskmemload1
233557499158,4cyclictest2428637-21chrt3
233557499158,4cyclictest2428637-21chrt3
233557499152,8cyclictest2455660-21taskset3
233557499151,2cyclictest0-21swapper/33
233557499151,2cyclictest0-21swapper/33
233557499151,11cyclictest23910772sleep33
233557499150,13cyclictest0-21swapper/33
233557499150,13cyclictest0-21swapper/33
233557499150,12cyclictest0-21swapper/33
27996122141,2sleep20-21swapper/22
27980132140,9sleep02798027-21ssh0
27302622141,2sleep00-21swapper/00
27281802141,3sleep12466020-21diskmemload1
27115422141,8sleep22711556-21ssh2
26952512141,2sleep00-21swapper/00
26921692147,4sleep12692180-21sh1
26314152140,3sleep00-21swapper/00
25884022142,8sleep12466020-21diskmemload1
25689002141,9sleep02568910-21ssh0
24340372140,1sleep10-21swapper/11
24236912141,8sleep02423943-21df0
24236912141,8sleep02423943-21df0
24117662141,4sleep22411812-21ntpq2
233557499148,3cyclictest2444554-21chrt3
233557499147,4cyclictest2449916-21chrt3
2335574991411,2cyclictest0-21swapper/33
2335574991411,2cyclictest0-21swapper/33
233557499140,12cyclictest0-21swapper/33
28155282131,3sleep02815542-21ssh0
27976702131,7sleep12466020-21diskmemload1
27912232131,3sleep02791237-21ssh0
27905862131,8sleep12466020-21diskmemload1
27753102131,8sleep02775533-21sort0
27653592131,3sleep20-21swapper/22
27621272131,6sleep02762382-21users0
27127012131,7sleep12466020-21diskmemload1
27025762131,8sleep22702587-21ssh2
26159322131,2sleep20-21swapper/22
25325392131,8sleep12532596-21ntpq1
25186052131,8sleep02518619-21ssh0
25070612131,6sleep22466020-21diskmemload2
24744162131,5sleep20-21swapper/22
24674512131,3sleep12469072-21kthreadcore1
24614682131,8sleep12461545-21/usr/sbin/munin1
24348542131,8sleep22435076-21grep2
233557499138,3cyclictest2364425-21chrt3
233557499138,3cyclictest2364425-21chrt3
28270492121,7sleep11-21systemd1
28182672121,7sleep22818345-21latency_hist2
27202242121,7sleep02720239-21ssh0
26935672121,7sleep02693578-21ssh0
26839762121,3sleep12683987-21ssh1
26839762121,3sleep12683987-21ssh1
26681592121,7sleep12466020-21diskmemload1
26072732121,6sleep02607540-21df0
25982152121,3sleep22598426-21ssh2
25803272121,3sleep22580338-21ssh2
25553222121,7sleep22556396-21kthreadcore2
25509032121,7sleep02466020-21diskmemload0
25439642121,6sleep12466020-21diskmemload1
25435722121,7sleep22543583-21ssh2
25277102121,7sleep12527721-21ssh1
25183082121,5sleep12466020-21diskmemload1
23744462121,7sleep22374502-21awk2
23697832121,5sleep12369718-31find1
23529982121,6sleep22353045-21ntpq2
2335574991210,1cyclictest0-21swapper/33
28333922111,3sleep00-21swapper/00
28245102111,6sleep02824524-21ssh0
28067222111,5sleep12466020-21diskmemload1
27799692111,3sleep20-21swapper/22
27738242111,3sleep20-21swapper/22
27663562110,2sleep00-21swapper/00
27480402111,5sleep22748114-21munin-run2
27314992111,6sleep02731632-21cat0
26551012111,3sleep20-21swapper/22
26464262111,5sleep02646479-21ntpq0
26223712110,7sleep02622386-21ssh0
26042412111,6sleep02604252-21ssh0
25939332111,5sleep12593978-21ntpq1
25890882111,5sleep20-21swapper/22
25882962110,3sleep00-21swapper/00
25782592110,1sleep00-21swapper/00
25684552111,4sleep22466020-21diskmemload2
25684042111,4sleep12466020-21diskmemload1
25349092111,7sleep02534920-21ssh0
24882402111,6sleep12488389-21ssh1
24882402111,6sleep12488389-21ssh1
24821392111,3sleep20-21swapper/22
24663912111,5sleep02466426-21ssh0
24490232111,5sleep12449061-21ntpq1
24399452111,6sleep22439973-21sort2
24291402111,5sleep02429283-21diskstats0
24117072111,5sleep12411764-21ntpq1
23646742111,6sleep01-21systemd0
23646742111,6sleep01-21systemd0
23583262110,2sleep10-21swapper/11
23412532111,5sleep12341285-21munin-node1
28350842101,4sleep12466020-21diskmemload1
28171542100,1sleep10-21swapper/11
28076462101,2sleep20-21swapper/22
27836382101,4sleep22466020-21diskmemload2
27800062101,2sleep10-21swapper/11
27567842101,3sleep20-21swapper/22
27540682101,6sleep02754082-21ssh0
27539812101,3sleep10-21swapper/11
27381732101,3sleep20-21swapper/22
27363462101,2sleep10-21swapper/11
27270832100,2sleep20-21swapper/22
27099042101,4sleep12466020-21diskmemload1
27095972101,2sleep00-21swapper/00
27023612101,4sleep12466020-21diskmemload1
26829802101,4sleep22466020-21diskmemload2
26829802101,4sleep22466020-21diskmemload2
26662142101,4sleep12466020-21diskmemload1
26563662101,4sleep02466020-21diskmemload0
26419012101,3sleep10-21swapper/11
26417142100,4sleep20-21swapper/22
26246442100,1sleep20-21swapper/22
26216142101,4sleep12466020-21diskmemload1
25974292101,3sleep00-21swapper/00
25521382101,3sleep12466020-21diskmemload1
25438012101,4sleep02466020-21diskmemload0
25181232101,3sleep20-21swapper/22
25007802101,2sleep00-21swapper/00
25003892101,2sleep10-21swapper/11
24909192101,3sleep20-21swapper/22
24909192101,3sleep20-21swapper/22
24893072100,1sleep00-21swapper/00
24893072100,1sleep00-21swapper/00
24556382101,4sleep22455657-21apt-get2
24553582101,5sleep02455516-21wc0
24490632101,4sleep22449110-21ntpq2
24262922101,5sleep12427573-21memory1
24262922101,5sleep12427573-21memory1
24130382101,4sleep22413057-21apt-get2
24064002100,6sleep10-21swapper/11
23973352101,5sleep02397548-21hddtemp_smartct0
23922192101,4sleep22392289-21irqstats2
23918312101,4sleep02392008-21diskstats0
23869212101,5sleep02387567-21cut0
23851002101,5sleep02385147-21ntpq0
23765662101,5sleep22378243-21grep2
23698182101,4sleep22369823-40gst-plugin-scan2
2832410291,2sleep20-21swapper/22
2772602291,3sleep12466020-21diskmemload1
2690746291,4sleep22690810-21ntpq2
2655549291,4sleep12655785-21grep1
2642210291,2sleep20-21swapper/22
2612381292,1sleep2171rcu_preempt2
2596430291,2sleep20-21swapper/22
2576791291,4sleep12577044-21grep1
2534167290,1sleep20-21swapper/22
2527805291,2sleep00-21swapper/00
2439670291,3sleep02439689-21apt-get0
2364962291,4sleep02365216-21df_inode0
2348645291,3sleep02348389-21apt-get0
2341286291,4sleep22342405-21cut2
2744583280,2sleep10-21swapper/11
2677006281,2sleep02676873-21apt-get0
2677006281,2sleep02676873-21apt-get0
2634311280,1sleep00-21swapper/00
2418092280,1sleep00-21swapper/00
2402395281,3sleep22402626-21expr2
2397626281,3sleep22399142-21sed2
2391105280,1sleep10-21swapper/11
2348084280,2sleep00-21swapper/00
2340096281,3sleep22340148-21open_files2
2460012271,3sleep02460147-21timerwakeupswit0
2455149270,1sleep20-21swapper/22
2439361270,1sleep10-21swapper/11
2433745270,1sleep20-21swapper/22
2427574270,1sleep20-21swapper/22
2427574270,1sleep20-21swapper/22
2423409271,2sleep10-21swapper/11
2423325270,1sleep20-21swapper/22
2417888270,1sleep10-21swapper/11
2406267270,4sleep00-21swapper/00
2396489270,1sleep10-21swapper/11
2385423271,2sleep22381042-21/usr/sbin/munin2
2382654271,2sleep10-21swapper/11
2375704270,1sleep00-21swapper/00
2364433270,1sleep10-21swapper/11
2364433270,1sleep10-21swapper/11
2363722271,2sleep20-21swapper/22
2363722271,2sleep20-21swapper/22
2359263270,1sleep00-21swapper/00
2358980270,1sleep20-21swapper/22
2353672270,1sleep10-21swapper/11
2336447271,3sleep02336552-21kthreadcore0
2465774260,4sleep00-21swapper/00
2450921260,2sleep12452578-21grep1
2449825260,1sleep00-21swapper/00
2421633260,3sleep0221rcuc/00
2408012261,2sleep00-21swapper/00
2397045260,2sleep10-21swapper/11
2391273260,1sleep20-21swapper/22
2374798261,2sleep00-21swapper/00
2336553261,2sleep10-21swapper/11
2443733250,1sleep10-21swapper/11
2455517240,1sleep10-21swapper/11
2434066240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional