You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 21:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Jun 28, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32121921580,4sleep1321217-21cron1
4079118212452,67sleep30-21swapper/33
378922850,2sleep00-21swapper/00
407909828050,12sleep20-21swapper/22
407905728062,12sleep10-21swapper/11
407880627958,14sleep00-21swapper/00
2947252221,7sleep012970-21diskmemload0
407927399210,18cyclictest66950irq/132-enp1s0-TxRx-33
407927399200,16cyclictest253943-21taskset3
2226862201,4sleep10-21swapper/11
407927399198,8cyclictest163939-21turbostat3
407927399192,13cyclictest0-21swapper/33
407927399192,12cyclictest21723-21taskset3
4079273991912,4cyclictest0-21swapper/33
407927399191,15cyclictest0-21swapper/33
407927399191,14cyclictest66950irq/132-enp1s0-TxRx-33
407927399191,14cyclictest0-21swapper/33
407927399190,15cyclictest0-21swapper/33
407927399185,2cyclictest0-21swapper/33
407927399181,4cyclictest0-21swapper/33
4079273991814,2cyclictest0-21swapper/33
4079273991814,2cyclictest0-21swapper/33
407927399181,14cyclictest66950irq/132-enp1s0-TxRx-33
407927399181,14cyclictest0-21swapper/33
407927399181,14cyclictest0-21swapper/33
407927399181,14cyclictest0-21swapper/33
407927399181,14cyclictest0-21swapper/33
407927399181,14cyclictest0-21swapper/33
407927399181,14cyclictest0-21swapper/33
407927399181,14cyclictest0-21swapper/33
407927399181,13cyclictest0-21swapper/33
407927399181,13cyclictest0-21swapper/33
407927399181,13cyclictest0-21swapper/33
407927399181,13cyclictest0-21swapper/33
407927399181,12cyclictest4079483-21chrt3
392482181,13sleep139586-21cut1
2212052181,3sleep10-21swapper/11
717922171,2sleep10-21swapper/11
547882171,3sleep10-21swapper/11
407927399172,10cyclictest89328-21taskset3
407927399171,4cyclictest193092chrt3
407927399171,3cyclictest0-21swapper/33
4079273991713,2cyclictest0-21swapper/33
4079273991712,3cyclictest0-21swapper/33
4079273991711,4cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399171,13cyclictest0-21swapper/33
407927399170,14cyclictest0-21swapper/33
407927399170,14cyclictest0-21swapper/33
3372152171,4sleep20-21swapper/22
3364502170,1sleep30-21swapper/33
1874882170,7sleep10-21swapper/11
1633142171,3sleep00-21swapper/00
942422162,4sleep212970-21diskmemload2
407927399162,8cyclictest723012sleep33
407927399161,12cyclictest0-21swapper/33
407927399161,12cyclictest0-21swapper/33
407927399161,12cyclictest0-21swapper/33
407927399161,11cyclictest0-21swapper/33
407927399160,13cyclictest0-21swapper/33
407927399160,13cyclictest0-21swapper/33
3285112161,2sleep20-21swapper/22
2954522160,3sleep20-21swapper/22
2275972161,4sleep012970-21diskmemload0
2109992161,3sleep10-21swapper/11
1613912161,2sleep20-21swapper/22
1033332161,10sleep012970-21diskmemload0
407927399159,3cyclictest76052sleep33
407927399151,3cyclictest0-21swapper/33
4079273991512,2cyclictest0-21swapper/33
407927399150,13cyclictest0-21swapper/33
407927399150,12cyclictest0-21swapper/33
407927399150,12cyclictest0-21swapper/33
3537662151,2sleep00-21swapper/00
2538552151,9sleep012970-21diskmemload0
2447982151,3sleep10-21swapper/11
1785952151,3sleep112970-21diskmemload1
1686692151,6sleep00-21swapper/00
1294572151,3sleep00-21swapper/00
1243542151,5sleep1125984-21ssh1
407927399148,4cyclictest4097193-21chrt3
4079273991412,1cyclictest0-21swapper/33
407927399141,10cyclictest0-21swapper/33
4079273991410,2cyclictest0-21swapper/33
407927399140,2cyclictest0-21swapper/33
407927399140,12cyclictest0-21swapper/33
407927399140,11cyclictest0-21swapper/33
3605992141,9sleep2360613-21ssh2
3542582141,9sleep2354272-21ssh2
2551432141,3sleep1255154-21ssh1
2548302141,3sleep212970-21diskmemload2
2101922141,4sleep212970-21diskmemload2
1948042141,3sleep0194816-21ssh0
1535112141,2sleep00-21swapper/00
948562131,8sleep194870-21ssh1
604072131,7sleep112970-21diskmemload1
407927399138,3cyclictest0-21swapper/33
407927399130,10cyclictest0-21swapper/33
381302131,8sleep138139-21ssh1
3540112131,8sleep1354021-21ssh1
3369622131,6sleep00-21swapper/00
3113922131,8sleep1311404-21ssh1
304882131,8sleep230528-21/usr/sbin/munin2
2359992131,8sleep1236009-21ssh1
2118362131,8sleep0211847-21sh0
2049812131,2sleep20-21swapper/22
2033612131,8sleep1203372-21ssh1
1618962131,3sleep1161908-21ssh1
1117672131,2sleep00-21swapper/00
1019632131,8sleep2102127-21/usr/sbin/munin2
41854272121,5sleep14185623-21users1
40794612121,5sleep24079480-21apt-get2
395882121,7sleep241124-21cut2
3456742121,7sleep1345684-21ssh1
3285312121,5sleep012970-21diskmemload0
3269002121,6sleep112970-21diskmemload1
3135902121,7sleep0313695-21cut0
3121112121,5sleep212970-21diskmemload2
2864752120,3sleep0286487-21ssh0
284052122,5sleep212970-21diskmemload2
2791422120,9sleep00-21swapper/00
2638612121,6sleep0263971-21grep0
1799142121,3sleep2179924-21ssh2
1198432120,5sleep10-21swapper/11
1103072121,5sleep1110525-21sed1
864882111,5sleep086525-21ssh0
798832110,6sleep10-21swapper/11
713382111,4sleep212970-21diskmemload2
627412111,3sleep20-21swapper/22
438772111,4sleep012970-21diskmemload0
41554062111,6sleep24155623-21kthreadcore2
41339602111,3sleep10-21swapper/11
41030132111,6sleep24104307-21kthreadcore2
40929812111,6sleep04094618-21grep0
3608302111,5sleep112970-21diskmemload1
3462282111,4sleep012970-21diskmemload0
3457252111,5sleep212970-21diskmemload2
268232110,2sleep00-21swapper/00
2603902111,4sleep212970-21diskmemload2
2277232110,7sleep212970-21diskmemload2
2219252111,3sleep20-21swapper/22
2033062111,5sleep012970-21diskmemload0
198032111,3sleep00-21swapper/00
1941222110,2sleep20-21swapper/22
1793992111,6sleep012970-21diskmemload0
1462542110,1sleep20-21swapper/22
1460112111,4sleep112970-21diskmemload1
1310772111,5sleep2131102-21ssh2
1308752111,5sleep1130978-21ssh1
1114202110,5sleep20-21swapper/22
908802101,6sleep092492-21kthreadcore0
885742100,4sleep20-21swapper/22
863552101,3sleep112970-21diskmemload1
802102101,5sleep080220-21ssh0
77422101,5sleep27963-21latency_hist2
773492101,5sleep277363-21ssh2
549702101,4sleep012970-21diskmemload0
520782101,3sleep20-21swapper/22
41915082101,3sleep04191527-21apt-get0
41912362101,5sleep04191424-21ls0
41496702101,4sleep04149924-21df0
41287492101,4sleep14129008-21df_inode1
41284502101,4sleep04128507-21munin-run0
41276362100,6sleep24127777-21switchtime2
41173242101,4sleep24117568-21users2
41129902101,3sleep04112686-21apt-get0
3369232101,4sleep112970-21diskmemload1
3185662101,3sleep212970-21diskmemload2
3038822101,3sleep112970-21diskmemload1
3034042101,3sleep20-21swapper/22
3021282100,1sleep00-21swapper/00
2874832101,2sleep20-21swapper/22
2844442100,1sleep10-21swapper/11
2777462101,5sleep1277751-21sh1
272212101,4sleep112970-21diskmemload1
2610332100,1sleep10-21swapper/11
2562012101,4sleep0256278-21irqstats0
2445712100,1sleep20-21swapper/22
2443622101,3sleep012970-21diskmemload0
2373292101,3sleep20-21swapper/22
2149252101,5sleep0215584-21kthreadcore0
1943982101,4sleep112970-21diskmemload1
192452101,4sleep112970-21diskmemload1
1865212101,3sleep20-21swapper/22
184572101,4sleep212970-21diskmemload2
1718622101,3sleep112970-21diskmemload1
1643332101,4sleep2164330-21apt-get2
1554882101,3sleep20-21swapper/22
1442992100,1sleep00-21swapper/00
1361262101,4sleep012970-21diskmemload0
1305552101,3sleep212970-21diskmemload2
9242291,5sleep210917-21kthreadcore2
63766292,3sleep00-21swapper/00
61665290,5sleep00-21swapper/00
4192936291,4sleep2640-21tr2
4190153291,5sleep14190291-21/usr/sbin/munin1
4170598291,2sleep00-21swapper/00
4170298291,4sleep04170429-21latency_hist0
4144972290,5sleep14145701-21kthreadcore1
4139216290,3sleep10-21swapper/11
4134251291,4sleep04134459-21head0
4112661290,5sleep24112665-21modprobe2
4091340290,2sleep20-21swapper/22
4086722291,2sleep00-21swapper/00
355582291,4sleep0356335-21ssh0
306914291,5sleep0307131-21kthreadcore0
294776290,3sleep112970-21diskmemload1
271925291,2sleep20-21swapper/22
270301291,4sleep1270312-21sh1
263763291,3sleep2263501-21sendmail-msp2
230923291,3sleep00-21swapper/00
186692291,4sleep0186703-21ssh0
147654291,2sleep10-21swapper/11
119316291,3sleep012970-21diskmemload0
114308290,1sleep20-21swapper/22
102528291,2sleep10-21swapper/11
4186283280,3sleep24186289-21apt-get2
4175856281,3sleep04176079-21cstates0
4163620280,5sleep04164092-21needreboot0
4160696281,3sleep24161884-21grep2
4159704280,1sleep00-21swapper/00
4144415280,3sleep04144434-21apt-get0
4139194280,4sleep04139215-21dpkg0
4138920281,3sleep21-21systemd2
4124145281,3sleep14125743-21kthreadcore1
4118222280,2sleep10-21swapper/11
4112533280,1sleep00-21swapper/00
4109406281,3sleep14111141-21kthreadcore1
4102439281,2sleep10-21swapper/11
4102154280,1sleep10-21swapper/11
4102031280,1sleep20-21swapper/22
3487281,3sleep00-21swapper/00
4181082270,2sleep20-21swapper/22
4180595270,1sleep10-21swapper/11
4180102270,4sleep20-21swapper/22
4171596271,2sleep10-21swapper/11
4160123270,2sleep10-21swapper/11
4159541270,1sleep10-21swapper/11
4154355270,1sleep10-21swapper/11
4149975270,3sleep24150188-21if_enp1s02
4145703271,2sleep20-21swapper/22
4118805271,3sleep20-21swapper/22
4117651270,1sleep10-21swapper/11
4101939270,1sleep00-21swapper/00
4087295271,3sleep10-21swapper/11
4084338271,3sleep04084340-21modprobe0
2408270,1sleep10-21swapper/11
8013260,1sleep00-21swapper/00
4185980260,1sleep00-21swapper/00
4165383261,2sleep24165513-21cpuspeed_turbos2
4165361260,1sleep10-21swapper/11
4143827260,4sleep20-21swapper/22
4131780260,2sleep20-21swapper/22
4128747260,1sleep00-21swapper/00
4118514260,2sleep00-21swapper/00
4106054260,2sleep04106425-21strings0
4096147260,2sleep10-21swapper/11
4084145260,1sleep10-21swapper/11
8570250,2sleep10-21swapper/11
4173235250,1sleep20-21swapper/22
4091961250,1sleep20-21swapper/22
3055250,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional