You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Thu Dec 04, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29147239913282,1323cyclictest33841432sleep33
29147239911130,1110cyclictest30560282sleep33
2914723994090,406cyclictest2980257-21turbostat3
2914723992590,256cyclictest3310733-21turbostat3
29541232490,2sleep0221rcuc/00
291472399197,8cyclictest3084644-21turbostat3
31740852182,4sleep03174109-21apt-get0
2914723991815,1cyclictest0-21swapper/33
2914723991713,2cyclictest3092681-21turbostat3
2914723991614,1cyclictest0-21swapper/33
2914723991614,0cyclictest0-21swapper/33
2914723991613,1cyclictest0-21swapper/33
2914723991613,1cyclictest0-21swapper/33
2914723991613,1cyclictest0-21swapper/33
2914723991613,1cyclictest0-21swapper/33
2914723991613,1cyclictest0-21swapper/33
291472399161,12cyclictest3198204-21taskset3
291472399161,12cyclictest2922963-21taskset3
291472399161,12cyclictest2922963-21taskset3
190898160,1rtkit-daemon0-21swapper/00
291472399157,6cyclictest32625422sleep33
291472399155,7cyclictest30375032sleep33
291472399155,6cyclictest32384112sleep33
2914723991513,1cyclictest0-21swapper/33
2914723991513,1cyclictest0-21swapper/33
2914723991513,0cyclictest0-21swapper/33
2914723991512,1cyclictest0-21swapper/33
2914723991512,1cyclictest0-21swapper/33
2914723991512,1cyclictest0-21swapper/33
2914723991512,1cyclictest0-21swapper/33
291472399151,11cyclictest3259370-21taskset3
291472399151,11cyclictest2942071-21taskset3
30181182141,4sleep03018141-21munin-run0
29479652141,5sleep22948182-21users2
291472399141,9cyclictest30563222sleep33
291472399141,9cyclictest30246502sleep33
2914723991412,1cyclictest0-21swapper/33
2914723991412,1cyclictest0-21swapper/33
2914723991412,1cyclictest0-21swapper/33
2914723991411,1cyclictest0-21swapper/33
2914723991411,1cyclictest0-21swapper/33
2914723991411,1cyclictest0-21swapper/33
2914723991411,1cyclictest0-21swapper/33
291472399141,10cyclictest3359999-21chrt3
291472399141,10cyclictest3351743-21taskset3
291472399141,10cyclictest3319203-21chrt3
291472399141,10cyclictest3252788-21taskset3
291472399141,10cyclictest3236713-21taskset3
291472399141,10cyclictest3159599-21chrt3
291472399141,10cyclictest3005485-21taskset3
291472399141,10cyclictest2960127-21taskset3
190898140,3rtkit-daemon3068598-21diskmemload0
32384482130,9sleep23238472-21apt-get2
31340022131,2sleep20-21swapper/22
291472399134,6cyclictest33272922sleep33
291472399134,6cyclictest33272922sleep33
291472399131,9cyclictest3228144-21taskset3
291472399131,9cyclictest3141826-21taskset3
291472399131,9cyclictest3106576-21taskset3
291472399131,9cyclictest2929915-21taskset3
291472399131,8cyclictest29930172sleep33
2914723991311,1cyclictest0-21swapper/33
2914723991311,1cyclictest0-21swapper/33
2914723991311,1cyclictest0-21swapper/33
2914723991311,1cyclictest0-21swapper/33
2914723991311,1cyclictest0-21swapper/33
291472399131,10cyclictest3309637-21chrt3
291472399131,10cyclictest3294251-21taskset3
291472399131,10cyclictest3270651-21chrt3
291472399130,10cyclictest3397243-21taskset3
291472399130,10cyclictest2914869-21taskset3
190898130,4rtkit-daemon667-21snmpd2
31216662120,2sleep20-21swapper/22
30692512120,9sleep03069382-21sed0
29620722121,2sleep20-21swapper/22
291472399121,9cyclictest3350303-21taskset3
291472399121,8cyclictest30468852sleep33
291472399121,8cyclictest3030912-21taskset3
291472399120,9cyclictest3367940-21taskset3
291472399120,9cyclictest2948834-21taskset3
190898120,4rtkit-daemon3165708-21ssh0
29409782110,7sleep12941055-21ntp_states1
291472399111,8cyclictest3138807-21taskset3
291472399111,7cyclictest2961708-21taskset3
190898110,7rtkit-daemon3083040-21chrt0
190898110,4rtkit-daemon674-21lldpd2
190898110,3rtkit-daemon81364-21packagekitd0
32386442102,2sleep0171rcu_preempt0
31417592101,4sleep23141297-21apt-get2
30451962103,5sleep23046880-21kthreadcore2
30386562100,2sleep10-21swapper/11
29086382100,6sleep30-21swapper/33
190898100,6rtkit-daemon3237350-21ssh0
190898100,5rtkit-daemon3295102-21df0
190898100,5rtkit-daemon3277925-21ssh0
3398465290,6sleep13398473-21ssh1
3392772291,5sleep03393771-21irqstats0
3349234290,6sleep23349501-21grep2
3341557290,1sleep20-21swapper/22
3333801293,4sleep2421rcuc/22
3333801293,4sleep2421rcuc/22
3303154290,6sleep23303196-21fschecks_count2
3293041290,6sleep13293048-21sh1
3269311290,7sleep13269314-21sh1
3262574291,5sleep03262737-21diskstats0
3214330291,5sleep03214485-21df_inode0
3147063291,4sleep03147127-21ntp_states0
3084967292,3sleep03085004-21apt-get0
3076907291,4sleep13076942-21apt-get1
3062562291,4sleep23062595-21apt-get2
3061468291,6sleep03061549-21sendmail_mailqu0
3037200291,4sleep03037233-21apt-get0
3031120290,6sleep13031263-21grep1
3024831290,6sleep13024651-21/usr/sbin/munin1
3012265291,5sleep13012465-21hddtemp_smartct1
2999279291,4sleep12999312-21apt-get1
2989425291,6sleep02991585-21grep0
2980503291,4sleep12980536-21apt-get1
2957000290,1sleep12959027-21cut1
2908737291,5sleep12908838-21/usr/sbin/munin1
19089890,6rtkit-daemon433-21hwrng0
19089890,5rtkit-daemon3068598-21diskmemload0
19089890,5rtkit-daemon2979856-21users2
19089890,5rtkit-daemon1928-21sendmail-mta0
19089890,5rtkit-daemon1928-21sendmail-mta0
19089890,5rtkit-daemon1928-21sendmail-mta0
19089890,4rtkit-daemon438-21wpa_supplicant2
19089890,4rtkit-daemon438-21wpa_supplicant2
19089890,4rtkit-daemon3333880-21ssh0
19089890,4rtkit-daemon3333880-21ssh0
19089890,4rtkit-daemon3230335-21/usr/sbin/munin1
19089890,4rtkit-daemon3221830-21ssh1
19089890,4rtkit-daemon3166225-21diskstats0
19089890,4rtkit-daemon3136555-21grep0
19089890,4rtkit-daemon3083694-21ssh2
19089890,4rtkit-daemon3045215-21cut0
19089890,4rtkit-daemon3005307-21/usr/sbin/munin0
19089890,4rtkit-daemon2985495-21missed_timers0
3402518281,4sleep03404572-21kthreadcore0
3400427281,4sleep20-21swapper/22
3386759280,5sleep03388802-21cut0
3383886281,4sleep13383908-21munin-run1
3383877281,4sleep03383885-21ssh0
3376066281,3sleep23376099-21apt-get2
3309720281,4sleep13309723-21ssh1
3300096280,5sleep23300175-21ntp_states2
3270611281,3sleep23270648-21apt-get2
3228234281,5sleep03228242-21ssh0
3100509280,5sleep03100517-21ssh0
3100509280,5sleep03100517-21ssh0
3093223281,4sleep13093429-21/usr/sbin/munin1
3093223281,4sleep13093429-21/usr/sbin/munin1
3087695280,5sleep23089845-21cut2
3075169281,2sleep1331ktimers/11
3005562281,3sleep03005595-21apt-get0
2956108281,5sleep02956997-21kthreadcore0
2955340281,4sleep22955539-21hddtemp_smartct2
2935918281,4sleep22936070-21df_inode2
2929779280,5sleep22929911-21grep2
19089880,5rtkit-daemon667-21snmpd2
19089880,5rtkit-daemon2914340-21chrt2
19089880,5rtkit-daemon1928-21sendmail-mta2
19089880,4rtkit-daemon667-21snmpd1
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon421-21avahi-daemon2
19089880,4rtkit-daemon3302995-21apt-get0
19089880,4rtkit-daemon3255975-21irqstats0
19089880,4rtkit-daemon319-21systemd-udevd2
19089880,4rtkit-daemon319-21systemd-udevd2
19089880,4rtkit-daemon3185179-21cat0
19089880,4rtkit-daemon3157019-21ssh0
19089880,4rtkit-daemon2973054-21/usr/sbin/munin2
19089880,4rtkit-daemon2921077-21munin-node2
19089880,4rtkit-daemon2875608-21munin-node2
19089880,3rtkit-daemon671-21ntpd0
19089880,3rtkit-daemon3109585-21diskstats2
19089880,3rtkit-daemon3109585-21diskstats2
19089880,1rtkit-daemon0-21swapper/00
3397581271,4sleep23392281-21/usr/sbin/munin2
3391395270,4sleep13391403-21ssh1
3359017270,1sleep23359025-21ssh2
3339606270,5sleep13340417-21sed1
3334096271,3sleep13334104-21ssh1
3334096271,3sleep13334104-21ssh1
3294844270,5sleep13294852-21ssh1
3278694270,5sleep23278701-21sh2
3267547270,2sleep23267693-21strings2
3261072270,4sleep23261080-21ssh2
3255978270,4sleep13257276-21tr1
3253955270,4sleep23253963-21ssh2
3236679270,4sleep23236686-21sh2
3222331270,3sleep23222368-21apt-get2
3188957271,3sleep23188961-21ssh2
3174280270,4sleep23174456-21sed2
3167505271,3sleep10-21swapper/11
3163230271,3sleep20-21swapper/22
3101704270,5sleep13102110-21irqcore1
3091593270,1sleep10-21swapper/11
3050193271,3sleep20-21swapper/22
3049573271,4sleep1667-21snmpd1
3030964270,4sleep03031119-21diskstats0
2961161270,3sleep02961176-21apt-get0
2928646270,4sleep12928890-21users1
2928646270,4sleep12928890-21users1
2915553270,4sleep12916439-21tr1
19089870,4rtkit-daemon473-21gmain1
19089870,4rtkit-daemon2921077-21munin-node2
19089870,4rtkit-daemon2921077-21munin-node2
19089870,3rtkit-daemon671-21ntpd2
19089870,3rtkit-daemon3284647-21ssh0
19089870,3rtkit-daemon2921077-21munin-node0
19089870,3rtkit-daemon2921077-21munin-node0
19089870,3rtkit-daemon2921077-21munin-node0
19089870,3rtkit-daemon1928-21sendmail-mta0
19089870,3rtkit-daemon1223-21cron2
3375736260,1sleep20-21swapper/22
3325782260,1sleep20-21swapper/22
3325256260,1sleep10-21swapper/11
3284199260,3sleep13284414-21sed1
3246450261,2sleep10-21swapper/11
3212890260,4sleep10-21swapper/11
3212492260,1sleep00-21swapper/00
3196787260,1sleep00-21swapper/00
3163943260,1sleep10-21swapper/11
3148492260,1sleep10-21swapper/11
3124178260,1sleep00-21swapper/00
3123905260,4sleep13123913-21ssh1
3116547260,4sleep13116552-21sh1
3116547260,4sleep13116552-21sh1
3109814261,3sleep03110467-21irqrtprio0
3109814261,3sleep03110467-21irqrtprio0
3101500261,2sleep00-21swapper/00
3068358260,1sleep00-21swapper/00
3023586260,3sleep13023703-21python31
3006543260,3sleep13007537-21cat1
2998828260,1sleep20-21swapper/22
2980031260,1sleep10-21swapper/11
2974385260,3sleep02974430-21tune2fs0
2961871260,3sleep12962071-21hddtemp_smartct1
2926894260,4sleep02928225-21memory0
2926894260,4sleep02928225-21memory0
19089860,3rtkit-daemon424-21NetworkManager1
3406138250,3sleep10-21swapper/11
3368418250,3sleep10-21swapper/11
3365746250,3sleep13365754-21ssh1
3360636251,2sleep20-21swapper/22
3354252250,2sleep10-21swapper/11
3349865250,3sleep13349873-21ssh1
3312584251,2sleep10-21swapper/11
3277054250,3sleep10-21swapper/11
3238840251,2sleep10-21swapper/11
3183638251,2sleep10-21swapper/11
3175566250,2sleep13177335-21kthreadcore1
3140801250,3sleep13140809-21ssh1
3042919250,1sleep20-21swapper/22
3029950250,2sleep20-21swapper/22
3012466251,2sleep20-21swapper/22
2992641250,1sleep10-21swapper/11
2986257250,1sleep20-21swapper/22
2973870250,1sleep10-21swapper/11
2948832250,3sleep1667-21snmpd1
2935304250,1sleep10-21swapper/11
2920581250,1sleep00-21swapper/00
3317796240,2sleep23317804-21ssh2
3294182240,1sleep20-21swapper/22
3203463240,1sleep00-21swapper/00
3171785240,3sleep23102295-21kworker/u16:3+rpciod2
3156796240,2sleep23156804-21ssh2
3156062240,2sleep10-21swapper/11
3068441240,1sleep10-21swapper/11
3061550240,2sleep13061655-21switchtime1
3051096240,2sleep10-21swapper/11
3036816240,3sleep20-21swapper/22
3023086240,2sleep23023464-21strings2
3005282240,1sleep20-21swapper/22
2998919240,1sleep10-21swapper/11
2969966240,2sleep00-21swapper/00
2945123241,1sleep00-21swapper/00
2909620240,1sleep00-21swapper/00
3005559230,1sleep20-21swapper/22
2986755230,1sleep20-21swapper/22
2948274230,1sleep10-21swapper/11
2941221230,1sleep00-21swapper/00
2929589230,1sleep00-21swapper/00
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional