You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 06:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Sep 08, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
141232722700,1sleep090-21kswapd00
975795213055,70sleep30-21swapper/33
103176521270,3sleep2421rcuc/22
97600028365,11sleep10-21swapper/11
97589128152,17sleep00-21swapper/00
97602727860,12sleep20-21swapper/22
10231352270,2sleep30-21swapper/33
97626599231,17cyclictest0-21swapper/33
97626599222,16cyclictest1439426-21taskset3
97626599222,16cyclictest1439426-21taskset3
97626599213,13cyclictest1122417-21taskset3
976265992118,1cyclictest0-21swapper/33
976265992117,2cyclictest0-21swapper/33
97626599211,3cyclictest0-21swapper/33
97626599211,16cyclictest1216907-21taskset3
97626599211,15cyclictest0-21swapper/33
11891112211,4sleep01104336-21diskmemload0
97626599209,6cyclictest66950irq/132-enp1s0-TxRx-33
97626599203,12cyclictest1228394-21taskset3
97626599201,5cyclictest1088201-21taskset3
97626599201,15cyclictest66950irq/132-enp1s0-TxRx-33
97626599201,15cyclictest0-21swapper/33
97626599201,15cyclictest0-21swapper/33
13342632201,9sleep11333954-21sendmail-msp1
11383182201,8sleep21138329-21ssh2
97626599192,5cyclictest13243012chrt3
976265991916,1cyclictest0-21swapper/33
976265991916,1cyclictest0-21swapper/33
97626599191,15cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,14cyclictest0-21swapper/33
97626599191,13cyclictest1469229-21chrt3
976265991910,5cyclictest1295165-21taskset3
97626599190,15cyclictest1072793-21turbostat3
97626599182,11cyclictest1131219-21taskset3
97626599182,10cyclictest1099411-21taskset3
97626599181,5cyclictest0-21swapper/33
976265991813,3cyclictest0-21swapper/33
976265991812,3cyclictest0-21swapper/33
97626599181,15cyclictest0-21swapper/33
97626599181,14cyclictest1339801-21taskset3
976265991811,4cyclictest0-21swapper/33
976265991811,4cyclictest0-21swapper/33
97626599181,14cyclictest0-21swapper/33
97626599181,14cyclictest0-21swapper/33
97626599181,14cyclictest0-21swapper/33
97626599181,14cyclictest0-21swapper/33
97626599181,13cyclictest0-21swapper/33
97626599181,13cyclictest0-21swapper/33
97626599181,13cyclictest0-21swapper/33
97626599181,13cyclictest0-21swapper/33
97626599180,2cyclictest0-21swapper/33
97626599180,14cyclictest0-21swapper/33
14643852181,6sleep11464537-21/usr/sbin/munin1
13761732181,3sleep10-21swapper/11
9815762171,5sleep1982474-21kthreadcore1
97626599171,13cyclictest0-21swapper/33
97626599171,13cyclictest0-21swapper/33
97626599171,13cyclictest0-21swapper/33
97626599171,13cyclictest0-21swapper/33
97626599171,12cyclictest0-21swapper/33
976265991710,5cyclictest0-21swapper/33
14153162170,1sleep20-21swapper/22
11281062171,4sleep11104336-21diskmemload1
97626599168,1cyclictest0-21swapper/33
97626599160,13cyclictest0-21swapper/33
97626599160,13cyclictest0-21swapper/33
14514672161,2sleep20-21swapper/22
14212562161,4sleep11104336-21diskmemload1
14038042161,2sleep00-21swapper/00
13923172160,6sleep10-21swapper/11
13657262160,2sleep00-21swapper/00
13512822161,3sleep20-21swapper/22
13137452161,2sleep20-21swapper/22
12592982169,3sleep21259508-21/usr/sbin/munin2
12422902161,4sleep11104336-21diskmemload1
12074522161,4sleep11104336-21diskmemload1
11582352161,10sleep01158315-21/usr/sbin/munin0
11484262161,3sleep00-21swapper/00
11481252161,10sleep11104336-21diskmemload1
10979712161,10sleep21093865-21sendmail-msp2
976265991512,2cyclictest0-21swapper/33
97626599150,12cyclictest0-21swapper/33
97626599150,12cyclictest0-21swapper/33
14762212151,9sleep11104336-21diskmemload1
14131882151,2sleep10-21swapper/11
13762712151,8sleep01104336-21diskmemload0
12250782151,3sleep20-21swapper/22
12079462150,2sleep20-21swapper/22
12002732151,3sleep21104336-21diskmemload2
11371422150,1sleep10-21swapper/11
10664562151,4sleep21066596-21ntp_192.168.1142
10664562151,4sleep21066596-21ntp_192.168.1142
97626599141,2cyclictest0-21swapper/33
976265991411,2cyclictest0-21swapper/33
976265991411,2cyclictest0-21swapper/33
976265991410,1cyclictest0-21swapper/33
97626599140,2cyclictest0-21swapper/33
97626599140,12cyclictest0-21swapper/33
97626599140,11cyclictest1093735-21chrt3
14692112142,6sleep21469213-21apt-get2
14221162141,7sleep21104336-21diskmemload2
13862772140,2sleep10-21swapper/11
13660682141,7sleep21104336-21diskmemload2
13046032141,3sleep01304614-21ssh0
12885682140,2sleep20-21swapper/22
12451322141,8sleep01104336-21diskmemload0
12174962141,4sleep01217509-21ssh0
11930282142,7sleep11193039-21ssh1
10316042141,9sleep11031763-21grep1
97626599139,2cyclictest0-21swapper/33
14557042131,6sleep10-21swapper/11
14420042131,8sleep11442017-21ssh1
14420042131,8sleep11442017-21ssh1
14310212131,8sleep21431036-21ssh2
14220442131,7sleep01104336-21diskmemload0
13952692131,5sleep00-21swapper/00
13853652132,6sleep01104336-21diskmemload0
13155092131,2sleep00-21swapper/00
13139502130,3sleep11313964-21ssh1
12732122131,7sleep11273853-21tr1
12454242130,8sleep21104336-21diskmemload2
12342722131,8sleep01234286-21ssh0
12263912131,2sleep10-21swapper/11
12020662131,8sleep01202221-21ssh0
11806322131,3sleep01180645-21ssh0
11804512131,3sleep21104336-21diskmemload2
11626852131,7sleep21157329-21sendmail-msp2
11544102130,2sleep10-21swapper/11
11135292131,7sleep21113532-21apt-get2
10526262131,8sleep11053357-21kthreadcore1
14680912121,4sleep20-21swapper/22
14656122121,7sleep01465619-21ssh0
14400262121,5sleep01104336-21diskmemload0
14400262121,5sleep01104336-21diskmemload0
13936082121,6sleep21104336-21diskmemload2
13432522121,6sleep01343417-21cstates0
13173202121,7sleep21317488-21tr2
12635742121,7sleep00-21swapper/00
12591162121,5sleep11259297-21mailstats1
12250512121,3sleep00-21swapper/00
11902932121,7sleep21190305-21sh2
11377122120,2sleep01104336-21diskmemload0
11283352121,5sleep21104336-21diskmemload2
11274632121,7sleep01127474-21ssh0
10991152121,7sleep01099142-21ls0
10468912121,5sleep11047094-21diskstats1
10264372121,7sleep01027977-21cut0
14572502110,4sleep00-21swapper/00
14482232111,4sleep01104336-21diskmemload0
13723232111,6sleep20-21swapper/22
13431242111,3sleep10-21swapper/11
13391952111,4sleep01333954-21sendmail-msp0
13321972111,4sleep11104336-21diskmemload1
13302822110,1sleep00-21swapper/00
13071182111,5sleep20-21swapper/22
12971462110,5sleep10-21swapper/11
12963972111,5sleep01104336-21diskmemload0
12814622111,6sleep01281677-21expr0
12778412111,4sleep20-21swapper/22
12637002111,5sleep11263735-21ssh1
12525382111,3sleep00-21swapper/00
12525382111,3sleep00-21swapper/00
12462182111,5sleep21246351-21diskstats2
12462182111,5sleep21246351-21diskstats2
12189452110,1sleep20-21swapper/22
12158702111,3sleep10-21swapper/11
11833492111,4sleep11104336-21diskmemload1
11729412111,6sleep01104336-21diskmemload0
11723632111,4sleep11104336-21diskmemload1
11645302111,3sleep10-21swapper/11
11558782110,1sleep00-21swapper/00
11488902111,3sleep20-21swapper/22
9931502101,4sleep1993208-21ntp_states1
9890672101,4sleep2988850-21sendmail-msp2
14765792101,3sleep01104336-21diskmemload0
14477622101,5sleep21447777-21ssh2
14395242101,4sleep21104336-21diskmemload2
14395242101,4sleep21104336-21diskmemload2
14320042101,4sleep01432018-21ssh0
14252992101,5sleep11425415-21modprobe1
14034652101,2sleep20-21swapper/22
13832682101,2sleep20-21swapper/22
13681212101,4sleep11104336-21diskmemload1
13567882100,6sleep21356803-21ssh2
13520162101,2sleep10-21swapper/11
13425472101,3sleep20-21swapper/22
13255162101,5sleep21325783-21df2
13246502101,2sleep10-21swapper/11
12943052101,4sleep21104336-21diskmemload2
12834382101,5sleep11285091-21tr1
12693182100,2sleep20-21swapper/22
12533872101,4sleep11104336-21diskmemload1
12533872101,4sleep11104336-21diskmemload1
12337032100,1sleep10-21swapper/11
12324992100,6sleep21228000-21sendmail-msp2
11988772101,3sleep01104336-21diskmemload0
11842862101,4sleep11104336-21diskmemload1
11745422101,4sleep21104336-21diskmemload2
11456502101,3sleep21104336-21diskmemload2
11196932101,4sleep11104336-21diskmemload1
11192432100,4sleep00-21swapper/00
11116892101,6sleep01111703-21ssh0
11032472101,5sleep11103381-21netstat1
10893862101,5sleep11089982-21cut1
10368152101,4sleep11036931-21irqstats1
10211472100,6sleep11021518-21cut1
10206352101,4sleep21020888-21df2
10153782101,4sleep11015397-21apt-get1
995553291,4sleep2997182-21sshd-session2
994390291,3sleep1994392-21apt-get1
985127290,5sleep0985903-21kthreadcore0
982478291,4sleep2983478-21df2
976441291,3sleep1976444-21apt-get1
1448394293,1sleep10-21swapper/11
1395953291,4sleep11395964-21ssh1
1351983290,4sleep01352012-21apt-get0
1323328291,2sleep00-21swapper/00
1299026291,2sleep10-21swapper/11
1277503291,3sleep00-21swapper/00
1255489291,4sleep01256913-21tr0
1106870291,4sleep11108378-21ssh1
1077731291,4sleep21072839-21sendmail-msp2
1068385291,4sleep01069647-21cut0
1056702291,4sleep21051838-21sendmail-msp2
1056468291,4sleep01056685-21users0
1030618290,1sleep20-21swapper/22
1014294291,3sleep21009855-21sendmail-msp2
1010848290,6sleep11012303-21cut1
1004580291,5sleep01004798-21latency_hist0
981439281,3sleep0981453-21latency_hist0
977505281,3sleep20-21swapper/22
1108541281,2sleep20-21swapper/22
1092923281,3sleep21093046-21mailstats2
1083601280,2sleep10-21swapper/11
1078048280,2sleep00-21swapper/00
1077753280,1sleep10-21swapper/11
1069648281,3sleep11069914-21kthreadcore1
1051466280,1sleep00-21swapper/00
1041594280,5sleep01041663-21cpuspeed_turbos0
1031389280,4sleep01031603-21basename0
1029991281,4sleep11030141-21ls1
1015400281,3sleep20-21swapper/22
1005602281,3sleep01007213-21kthreadcore0
999612270,2sleep10-21swapper/11
993733270,3sleep0926644-21kworker/u16:0+flush-8:00
1099390270,1sleep20-21swapper/22
1093453270,1sleep00-21swapper/00
1088231270,1sleep20-21swapper/22
1083290270,1sleep20-21swapper/22
1083078270,1sleep00-21swapper/00
1080326271,3sleep10-21swapper/11
1072587270,1sleep20-21swapper/22
1062091270,1sleep10-21swapper/11
1059570271,3sleep21061188-21latency2
1057338270,2sleep00-21swapper/00
1041185270,1sleep00-21swapper/00
1020939270,3sleep00-21swapper/00
1010075271,2sleep00-21swapper/00
1000194271,3sleep20-21swapper/22
1104182260,1sleep00-21swapper/00
1096331260,2sleep10-21swapper/11
1088462260,1sleep00-21swapper/00
1067386260,4sleep10-21swapper/11
1067386260,4sleep10-21swapper/11
1062301260,3sleep00-21swapper/00
1062301260,3sleep00-21swapper/00
1051407260,1sleep20-21swapper/22
1043361260,2sleep10-21swapper/11
1009751260,3sleep10-21swapper/11
1041861250,1sleep20-21swapper/22
1036353250,4sleep20-21swapper/22
1019716250,1sleep00-21swapper/00
994372240,1sleep00-21swapper/00
1004848240,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional