You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 16:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Wed Dec 03, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
38542539913070,1303cyclictest401772sleep33
38542539912854,1277cyclictest39505352sleep33
38542539912440,1241cyclictest40968172sleep33
3854253993650,362cyclictest4160973-21turbostat3
692482550,1sleep10-21swapper/11
3854253992111,8cyclictest0-21swapper/33
3854253992012,0cyclictest0-21swapper/33
3854253992010,1cyclictest0-21swapper/33
3854253991912,0cyclictest0-21swapper/33
385425399191,15cyclictest4177196-21turbostat3
385425399189,1cyclictest0-21swapper/33
3854253991815,1cyclictest0-21swapper/33
3854253991810,0cyclictest0-21swapper/33
41838322171,8sleep14183840-21ssh1
39182572174,10sleep13918304-21proc_pri1
3854253991714,1cyclictest0-21swapper/33
190898170,12rtkit-daemon3958519-21cut1
3854253991613,1cyclictest0-21swapper/33
385425399161,11cyclictest3891012-21taskset3
38483592161,5sleep03848475-21perl0
385425399155,7cyclictest3874824-21taskset3
3854253991513,1cyclictest0-21swapper/33
3854253991513,1cyclictest0-21swapper/33
3854253991513,1cyclictest0-21swapper/33
3854253991513,1cyclictest0-21swapper/33
3854253991513,1cyclictest0-21swapper/33
3854253991512,1cyclictest0-21swapper/33
3854253991512,1cyclictest0-21swapper/33
3854253991512,1cyclictest0-21swapper/33
3854253991512,1cyclictest0-21swapper/33
3854253991512,1cyclictest0-21swapper/33
3854253991511,2cyclictest0-21swapper/33
385425399151,11cyclictest3919140-21turbostat3
385425399151,10cyclictest4160967-21turbostat3
385425399151,10cyclictest4091193-21taskset3
385425399150,12cyclictest3860482-21turbostat3
385425399146,6cyclictest39266142sleep33
385425399145,6cyclictest40136732sleep33
385425399145,6cyclictest40136732sleep33
385425399144,6cyclictest41372132sleep33
385425399143,9cyclictest31844-21taskset3
3854253991413,0cyclictest0-21swapper/33
3854253991412,1cyclictest0-21swapper/33
3854253991412,1cyclictest0-21swapper/33
3854253991412,1cyclictest0-21swapper/33
3854253991412,1cyclictest0-21swapper/33
3854253991412,1cyclictest0-21swapper/33
3854253991412,0cyclictest0-21swapper/33
3854253991411,1cyclictest0-21swapper/33
3854253991411,1cyclictest0-21swapper/33
3854253991411,1cyclictest0-21swapper/33
385425399141,10cyclictest4191004-21chrt3
385425399141,10cyclictest4135385-21taskset3
385425399141,10cyclictest4116910-21taskset3
385425399141,10cyclictest4063154-21chrt3
385425399141,10cyclictest4007321-21taskset3
385425399141,10cyclictest3956389-21taskset3
385425399141,10cyclictest3938009-21turbostat3
385425399141,10cyclictest3867804-21taskset3
3854253991410,2cyclictest3975948-21turbostat3
385425399140,11cyclictest3963600-21taskset3
39954752135,6sleep13995570-21ps1
38756152130,1sleep03875814-21hddtemp_smartct0
385425399131,9cyclictest4193703-21taskset3
385425399131,9cyclictest3994455-21taskset3
385425399131,9cyclictest3978371-21taskset3
385425399131,9cyclictest3881108-21chrt3
385425399131,10cyclictest4029823-21chrt3
3854253991310,1cyclictest0-21swapper/33
3854253991310,1cyclictest0-21swapper/33
3854253991310,1cyclictest0-21swapper/33
385425399130,11cyclictest7806-21taskset3
385425399130,10cyclictest70474-21taskset3
385425399121,9cyclictest3906297-21taskset3
385425399121,8cyclictest102382-21taskset3
190898120,5rtkit-daemon1928-21sendmail-mta0
40650932110,4sleep04065972-21sed0
40154592110,8sleep24015508-21munin-run2
40154592110,8sleep24015508-21munin-run2
39887922111,5sleep13988814-21apt-get1
39371662111,7sleep03937267-21/usr/sbin/munin0
190898110,7rtkit-daemon4114343-21cut0
190898110,3rtkit-daemon667-21snmpd0
587282100,7sleep160744-21cut1
41535102101,5sleep14153684-21sed1
41020892101,5sleep04102174-21ntp_states0
40955272101,5sleep14095535-21ssh1
40166252104,3sleep24017530-21kthreadcore2
39761882101,5sleep13976221-21apt-get1
39449332101,7sleep00-21swapper/00
38688402101,6sleep23869050-21latency_hist2
190898100,6rtkit-daemon92122-21kthreadcore2
190898100,6rtkit-daemon436-21systemd-logind1
190898100,6rtkit-daemon4183501-21ssh0
190898100,6rtkit-daemon1928-21sendmail-mta1
190898100,6rtkit-daemon1495-21kthreadcore0
190898100,5rtkit-daemon4049694-21irqstats0
190898100,5rtkit-daemon3854168-21munin-run1
190898100,5rtkit-daemon150832-21taskset0
190898100,4rtkit-daemon62520-21ssh0
162552100,7sleep215938-21/usr/sbin/munin2
1373062101,5sleep2137328-21apt-get2
1054002101,6sleep0105996-21irqrtprio0
88419290,7sleep088484-40mandb0
47319290,6sleep047329-21ssh0
4169584291,4sleep14169606-21apt-get1
4112604290,6sleep14112612-21ssh1
4042051290,1sleep14044082-21kthreadcore1
4031554291,5sleep04031578-21apt0
3939030290,7sleep20-21swapper/22
3887927291,4sleep13887960-21apt-get1
3876396294,3sleep23876907-21kthreadcore2
21274291,4sleep121397-21python31
19089890,5rtkit-daemon87554-21ssh2
19089890,5rtkit-daemon667-21snmpd1
19089890,5rtkit-daemon667-21snmpd0
19089890,5rtkit-daemon438-21wpa_supplicant1
19089890,5rtkit-daemon4030991-21runrttasks1
19089890,4rtkit-daemon54287-21ssh0
19089890,4rtkit-daemon4104944-21ssh0
19089890,4rtkit-daemon4040074-21expr0
19089890,4rtkit-daemon3920359-21cut1
19089890,4rtkit-daemon3878394-21cut1
19089890,4rtkit-daemon24224-21df0
19089890,4rtkit-daemon136268-21ssh0
19089890,4rtkit-daemon118500-21python30
102718290,6sleep196963-21/usr/sbin/munin1
7867281,3sleep17904-21apt-get1
53458280,6sleep153582-21ssh1
4179591280,5sleep24181584-21cut2
4135042281,5sleep0443-21rs:main0
4098378280,5sleep14099923-21kthreadcore1
4053777281,4sleep24054002-21users2
4032975280,5sleep24034105-21tr2
4001332281,3sleep04001365-21apt-get0
4000160281,4sleep24000231-21ntp_states2
3965867281,5sleep03967920-21kthreadcore0
3956723280,1sleep10-21swapper/11
3951464281,4sleep03952054-21irqrtprio0
3950539281,4sleep23950560-21munin-run2
3937380281,4sleep23937624-21users2
3881671281,3sleep03881704-21apt-get0
3869121280,4sleep03869154-21apt-get0
3854417281,3sleep03854450-21apt-get0
3849627281,4sleep30-21swapper/33
19089880,5rtkit-daemon1928-21sendmail-mta1
19089880,5rtkit-daemon1928-21sendmail-mta0
19089880,5rtkit-daemon1928-21sendmail-mta0
19089880,4rtkit-daemon671-21ntpd1
19089880,4rtkit-daemon667-21snmpd0
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon421-21avahi-daemon1
19089880,4rtkit-daemon4144723-21ssh0
19089880,4rtkit-daemon4062413-21ssh0
19089880,4rtkit-daemon3860622-21munin-node1
19089880,4rtkit-daemon3860622-21munin-node1
19089880,4rtkit-daemon3860622-21munin-node1
19089880,4rtkit-daemon3860622-21munin-node0
19089880,4rtkit-daemon1928-21sendmail-mta1
19089880,3rtkit-daemon667-21snmpd0
19089880,3rtkit-daemon4094315-21swap0
19089880,3rtkit-daemon4015788-21/usr/sbin/munin1
19089880,1rtkit-daemon0-21swapper/11
19089880,1rtkit-daemon0-21swapper/00
121648281,4sleep0122647-21irqstats0
121079281,4sleep2121087-21ssh2
95716270,1sleep00-21swapper/00
9076271,3sleep210195-21kthreadcore2
70571270,1sleep00-21swapper/00
4194086271,3sleep2427-21sed2
4187109271,3sleep10-21swapper/11
4169053270,3sleep14159990-21kworker/u16:1+rpciod1
4160988270,4sleep24160964-21apt-get2
4150342270,4sleep14150479-21ssh1
4131367271,4sleep20-21swapper/22
4126394271,3sleep04126523-21python30
4121117271,3sleep24121154-21apt-get2
4073184270,4sleep14074104-21sed1
4048161272,3sleep1794406-21kworker/u16:0+nfsiod1
40223270,1sleep10-21swapper/11
3963614271,3sleep23963647-21apt-get2
3956263271,3sleep23956388-21python32
3920393270,4sleep03921465-21cut0
3893291271,4sleep00-21swapper/00
3848173270,2sleep20-21swapper/22
32117271,3sleep132149-21apt-get1
32117271,3sleep132149-21apt-get1
31793270,4sleep231842-21ssh2
19089870,4rtkit-daemon667-21snmpd0
19089870,4rtkit-daemon3860622-21munin-node1
19089870,3rtkit-daemon674-21lldpd1
19089870,3rtkit-daemon671-21ntpd1
19089870,3rtkit-daemon671-21ntpd1
19089870,3rtkit-daemon667-21snmpd1
19089870,3rtkit-daemon667-21snmpd0
19089870,3rtkit-daemon4120669-21ssh1
19089870,3rtkit-daemon3860622-21munin-node1
19089870,3rtkit-daemon3860622-21munin-node1
19089870,3rtkit-daemon3860622-21munin-node1
19089870,3rtkit-daemon1928-21sendmail-mta1
19089870,3rtkit-daemon14035-21kworker/u16:1+rpciod0
19089870,3rtkit-daemon139209-21kthreadcore0
153087270,1sleep20-21swapper/22
79440260,3sleep279448-21ssh2
48272261,3sleep248279-21sh2
4191723260,1sleep20-21swapper/22
4174753261,2sleep20-21swapper/22
4159646260,1sleep20-21swapper/22
4110831262,2sleep24110837-21ssh2
4102217260,4sleep24102345-21smartctl2
4081194260,4sleep20-21swapper/22
4074105261,2sleep24075145-21kthreadcore2
4065973260,3sleep14067049-21grep1
40638260,4sleep24120076-21kworker/u16:2+rpciod2
4046152260,4sleep24046160-21ssh2
4029961260,3sleep24029969-21ssh2
3982488261,2sleep00-21swapper/00
3957338261,2sleep20-21swapper/22
3943540260,4sleep03943670-21timerandwakeup0
3931428260,1sleep00-21swapper/00
3913071260,3sleep00-21swapper/00
3912816261,3sleep0423-21dbus-daemon0
3905954260,3sleep03906170-21users0
38190260,3sleep238198-21ssh2
38190260,3sleep238198-21ssh2
29462260,2sleep10-21swapper/11
136958260,3sleep2136966-21ssh2
130285260,4sleep1131383-21kthreadcore1
122648260,4sleep1123910-21kthreadcore1
111450260,1sleep20-21swapper/22
97157251,2sleep20-21swapper/22
79133250,3sleep00-21swapper/00
70715250,2sleep270723-21ssh2
4146495250,3sleep24147602-21kthreadcore2
4119447250,1sleep20-21swapper/22
4090042251,2sleep20-21swapper/22
4080993250,3sleep14081185-21ssh1
4067050250,3sleep24069092-21cut2
4063748251,2sleep20-21swapper/22
4063231250,3sleep10-21swapper/11
4012963252,1sleep00-21swapper/00
4012963252,1sleep00-21swapper/00
3990328250,3sleep23992321-21tr2
3989175251,2sleep00-21swapper/00
3988344250,2sleep23988345-21sshd2
3976174250,1sleep20-21swapper/22
3975745250,3sleep20-21swapper/22
3963194250,1sleep00-21swapper/00
3911782251,2sleep20-21swapper/22
3900604250,3sleep23900520-21/usr/sbin/munin2
3860249250,1sleep20-21swapper/22
151542251,2sleep10-21swapper/11
143106250,3sleep10-21swapper/11
114558251,2sleep20-21swapper/22
113097250,2sleep10-21swapper/11
110216250,3sleep10-21swapper/11
95570240,2sleep10-21swapper/11
63292240,2sleep263298-21ssh2
6305240,2sleep16313-21ssh1
4038324240,1sleep00-21swapper/00
4019483240,2sleep04020859-21memory0
4007105240,3sleep20-21swapper/22
3969946240,2sleep03970022-21expr0
3925693240,1sleep20-21swapper/22
3925108241,2sleep23854244-21cyclictest2
3895316241,1sleep00-21swapper/00
3860791240,2sleep03861120-21ps0
4137173230,1sleep20-21swapper/22
4000304230,1sleep00-21swapper/00
3976000230,1sleep00-21swapper/00
3913068230,1sleep20-21swapper/22
3894471230,1sleep20-21swapper/22
3889384230,1sleep20-21swapper/22
3881862230,1sleep20-21swapper/22
3869304230,1sleep20-21swapper/22
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional