You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 23:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Apr 01, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
989511212757,66sleep30-21swapper/33
107761621220,4sleep0221rcuc/00
989769211498,10sleep20-21swapper/22
98947928362,14sleep10-21swapper/11
98947827958,15sleep00-21swapper/00
989972993431,1cyclictest0-21swapper/33
98997299261,3cyclictest0-21swapper/33
10819172260,2sleep00-21swapper/00
98997299250,3cyclictest0-21swapper/33
98997299235,14cyclictest1354355-21taskset3
98997299231,18cyclictest66950irq/132-enp1s0-TxRx-33
98997299231,18cyclictest0-21swapper/33
98997299221,17cyclictest1470574-21taskset3
98997299221,17cyclictest1222008-21taskset3
98997299211,16cyclictest0-21swapper/33
989972992110,7cyclictest1120316-21turbostat3
98997299210,16cyclictest0-21swapper/33
13317842211,7sleep11332051-21df1
98997299201,7cyclictest1451383-21chrt3
989972992016,2cyclictest0-21swapper/33
98997299201,16cyclictest66950irq/132-enp1s0-TxRx-33
98997299201,15cyclictest66950irq/132-enp1s0-TxRx-33
98997299201,15cyclictest0-21swapper/33
98997299201,15cyclictest0-21swapper/33
989972991916,1cyclictest0-21swapper/33
989972991915,2cyclictest0-21swapper/33
989972991912,4cyclictest0-21swapper/33
98997299191,15cyclictest0-21swapper/33
98997299191,15cyclictest0-21swapper/33
98997299191,14cyclictest66950irq/132-enp1s0-TxRx-33
98997299191,14cyclictest0-21swapper/33
98997299191,14cyclictest0-21swapper/33
98997299191,14cyclictest0-21swapper/33
98997299191,14cyclictest0-21swapper/33
12926182191,14sleep21292843-21grep2
12325612191,3sleep20-21swapper/22
989972991815,2cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,14cyclictest0-21swapper/33
98997299181,13cyclictest0-21swapper/33
98997299181,13cyclictest0-21swapper/33
98997299180,15cyclictest0-21swapper/33
98997299180,14cyclictest0-21swapper/33
14695692181,7sleep11120300-21diskmemload1
14445602180,2sleep10-21swapper/11
11610952181,3sleep20-21swapper/22
11551762180,12sleep00-21swapper/00
98997299176,8cyclictest1056292-21turbostat3
98997299171,13cyclictest0-21swapper/33
98997299171,13cyclictest0-21swapper/33
98997299171,13cyclictest0-21swapper/33
98997299171,12cyclictest0-21swapper/33
13403972171,5sleep01340409-21sh0
98997299161,12cyclictest0-21swapper/33
98997299161,12cyclictest0-21swapper/33
98997299161,11cyclictest10872632sleep33
98997299160,13cyclictest0-21swapper/33
98997299160,13cyclictest0-21swapper/33
98997299160,13cyclictest0-21swapper/33
98997299160,12cyclictest0-21swapper/33
98997299160,12cyclictest0-21swapper/33
98997299160,12cyclictest0-21swapper/33
98997299160,12cyclictest0-21swapper/33
98997299160,12cyclictest0-21swapper/33
14682212161,4sleep21120300-21diskmemload2
13135822161,2sleep20-21swapper/22
12967832161,4sleep21297047-21df2
12104312160,4sleep2421rcuc/22
11610282161,3sleep00-21swapper/00
98997299151,11cyclictest0-21swapper/33
98997299150,12cyclictest0-21swapper/33
98997299150,12cyclictest0-21swapper/33
98997299150,12cyclictest0-21swapper/33
98997299150,12cyclictest0-21swapper/33
14234562151,9sleep11423511-21ssh1
12854882151,3sleep00-21swapper/00
12782192151,2sleep00-21swapper/00
12512442151,2sleep00-21swapper/00
12396332150,1sleep10-21swapper/11
12179562151,9sleep21218068-21perl2
12058322150,1sleep10-21swapper/11
11153842151,9sleep11115494-21diskstats1
10565382150,2sleep20-21swapper/22
98997299140,2cyclictest0-21swapper/33
98997299140,12cyclictest0-21swapper/33
98997299140,11cyclictest0-21swapper/33
98997299140,11cyclictest0-21swapper/33
98997299140,11cyclictest0-21swapper/33
14888012141,4sleep11488815-21ssh1
14417842140,2sleep00-21swapper/00
14240162141,7sleep01424191-21vmstat0
13753812140,2sleep20-21swapper/22
13369742141,4sleep21120300-21diskmemload2
13202772141,9sleep11120300-21diskmemload1
13105542143,2sleep1171rcu_preempt1
12500472140,2sleep10-21swapper/11
12230012141,4sleep11223012-21ssh1
12171042141,2sleep10-21swapper/11
11715622141,2sleep10-21swapper/11
9974882130,5sleep2997620-21cpuspeed_turbos2
14084372131,8sleep21408486-21apt-config2
13972652133,2sleep0171rcu_preempt0
13733972131,8sleep11120300-21diskmemload1
13642692131,8sleep11120300-21diskmemload1
13549362131,8sleep11120300-21diskmemload1
12757462131,8sleep11275760-21ssh1
12695942134,5sleep1331ktimers/11
12055442131,7sleep21120300-21diskmemload2
11984022131,8sleep01198416-21ssh0
11385892131,8sleep01138820-21ls0
11031882131,5sleep21103246-21ntpq2
10015092131,6sleep11001557-21ntpq1
14810632121,6sleep01481186-21grep0
14542102121,5sleep20-21swapper/22
14293052121,3sleep21430962-21kthreadcore2
14168102121,7sleep11416828-21ssh1
14165072121,2sleep00-21swapper/00
13911772121,7sleep01391189-21ssh0
13230732121,8sleep00-21swapper/00
13184942121,6sleep01318542-21ntpq0
13018872121,6sleep11120300-21diskmemload1
12922362121,6sleep01292284-21ntpq0
12616572122,5sleep01261687-21apt-get0
12492502121,7sleep21249264-21ssh2
12413942124,4sleep0151ktimers/00
12078672122,4sleep01120300-21diskmemload0
11974052120,1sleep20-21swapper/22
11878552121,6sleep21120300-21diskmemload2
11824992121,5sleep11120300-21diskmemload1
11687892121,5sleep21168843-21ntpq2
11530962121,3sleep11153106-21ssh1
10872102121,5sleep21087261-21ntpq2
10392952121,7sleep01034933-21/usr/sbin/munin0
10290352120,1sleep10-21swapper/11
10290352120,1sleep10-21swapper/11
10237092121,6sleep01023629-31find0
9977572111,5sleep0997966-21hddtemp_smartct0
9942222111,5sleep0994271-21ntpq0
14862162111,3sleep20-21swapper/22
14775022110,4sleep20-21swapper/22
14619982111,3sleep00-21swapper/00
14528242111,6sleep01452835-21ssh0
14507502111,4sleep21120300-21diskmemload2
14339012111,3sleep00-21swapper/00
14070992111,4sleep11120300-21diskmemload1
13842512111,5sleep11384283-21ssh1
13739392111,6sleep01373954-21ssh0
13301172111,4sleep21120300-21diskmemload2
13054642111,6sleep01305478-21ssh0
13036962111,3sleep00-21swapper/00
12853472111,4sleep11120300-21diskmemload1
12600892111,3sleep20-21swapper/22
12428412111,4sleep21120300-21diskmemload2
12335282111,4sleep11120300-21diskmemload1
12233422111,3sleep00-21swapper/00
11452912112,4sleep21120300-21diskmemload2
11432822110,7sleep11143297-21ssh1
11368802111,7sleep21136893-21ssh2
11349192111,3sleep10-21swapper/11
11274702111,4sleep10-21swapper/11
11045332111,5sleep01104792-21df_inode0
10765792111,5sleep11076625-21ntpq1
10237102111,5sleep11023629-31find1
10237052110,3sleep290-21kswapd02
10136232111,5sleep11013746-21diskstats1
10122192111,5sleep11012268-21ntpq1
9904232101,4sleep2990633-21hddtemp_smartct2
14791992101,3sleep10-21swapper/11
14774202101,3sleep00-21swapper/00
14684452101,4sleep01120300-21diskmemload0
14626182101,4sleep11462676-21munin-run1
14530842101,4sleep11120300-21diskmemload1
14124292101,6sleep20-21swapper/22
14074462101,3sleep00-21swapper/00
13991852101,3sleep10-21swapper/11
13981912101,4sleep21120300-21diskmemload2
13918142101,4sleep21120300-21diskmemload2
13832722101,4sleep11120300-21diskmemload1
13736142101,4sleep21120300-21diskmemload2
13634852101,4sleep20-21swapper/22
13622702101,4sleep01362340-21ssh0
13472332101,3sleep20-21swapper/22
13461952100,1sleep10-21swapper/11
13400962101,5sleep01340155-21munin-run0
13223272101,3sleep20-21swapper/22
12881422101,4sleep11120300-21diskmemload1
12859182101,3sleep21120300-21diskmemload2
12760642101,2sleep20-21swapper/22
12617672101,4sleep21261981-21diskstats2
12523212101,3sleep00-21swapper/00
11980082101,5sleep11120300-21diskmemload1
11879802101,5sleep01187992-21ssh0
11802952101,2sleep00-21swapper/00
11779502101,4sleep21178009-21ntpq2
11736932101,4sleep11173719-21apt-get1
11727712101,2sleep00-21swapper/00
11624062100,1sleep10-21swapper/11
11538722101,2sleep20-21swapper/22
11285132101,3sleep00-21swapper/00
10892412101,5sleep01090917-21cut0
10725482100,1sleep20-21swapper/22
10499382101,4sleep11049986-21ntpq1
10393992101,4sleep11039615-21users1
10078852101,4sleep01007933-21munin-run0
1442159291,3sleep20-21swapper/22
1435543291,3sleep11120300-21diskmemload1
1425717290,3sleep20-21swapper/22
1375526291,3sleep01375542-21apt-get0
1356111291,2sleep20-21swapper/22
1350225291,4sleep01351763-21ssh0
1329116291,3sleep11120300-21diskmemload1
1256854291,3sleep10-21swapper/11
1230602291,4sleep01230666-21ntp_states0
1213987292,1sleep00-21swapper/00
1126548291,2sleep20-21swapper/22
1093864291,4sleep11094084-21expr1
1072196291,3sleep172-21kcompactd01
1070950290,1sleep00-21swapper/00
1061843291,3sleep21061862-21apt-get2
1044296291,4sleep21044333-21ntpq2
1034932291,3sleep21034951-21apt-get2
1029626291,4sleep01029855-21expr0
1133146281,2sleep00-21swapper/00
1114053281,3sleep01114144-21ps0
1094359281,3sleep01094540-21cut0
1093531280,1sleep10-21swapper/11
1087373281,3sleep11087492-21find1
1082233281,3sleep11082245-21wc1
1055183280,4sleep00-21swapper/00
1051213280,2sleep10-21swapper/11
1008148280,3sleep01008167-21apt-get0
1003427281,4sleep21004933-21sort2
1002839280,4sleep10-21swapper/11
994512271,3sleep10-21swapper/11
1120291271,2sleep0563-21systemd-logind0
1114815270,1sleep10-21swapper/11
1109234270,1sleep10-21swapper/11
1087318270,4sleep01061564-21kworker/0:1+events0
1078142271,3sleep20-21swapper/22
1052624271,2sleep20-21swapper/22
1014036271,2sleep01015518-21kthreadcore0
1013058270,1sleep20-21swapper/22
1114997260,1sleep20-21swapper/22
1114839260,4sleep20-21swapper/22
1109582261,2sleep20-21swapper/22
1103009260,1sleep10-21swapper/11
1088829261,2sleep21089042-21/usr/sbin/munin2
1072250260,4sleep20-21swapper/22
1066554260,1sleep10-21swapper/11
1049988260,2sleep20-21swapper/22
1045874261,2sleep00-21swapper/00
1040239260,1sleep10-21swapper/11
1034370260,4sleep20-21swapper/22
1029161260,3sleep20-21swapper/22
1029161260,3sleep20-21swapper/22
1099357250,1sleep00-21swapper/00
1066553250,1sleep00-21swapper/00
1060423250,2sleep10-21swapper/11
1040530250,1sleep00-21swapper/00
1033977250,1sleep10-21swapper/11
1024858251,1sleep00-21swapper/00
1024858251,1sleep00-21swapper/00
1017297250,1sleep20-21swapper/22
1093834240,2sleep20-21swapper/22
1056820240,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional