You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 18:26
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Sat Jul 12, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
48482324249,63sleep30-21swapper/33
48052320282,25sleep00-21swapper/00
47052306238,18sleep10-21swapper/11
31352291181,76sleep20-21swapper/22
509299228224,2cyclictest0-21swapper/22
509899187183,2cyclictest0-21swapper/33
508699180175,3cyclictest0-21swapper/11
508199141136,3cyclictest18362-21gzip0
3275521290,2sleep1508699cyclictest1
159152540,3sleep30-21swapper/33
199332500,3sleep20-21swapper/22
193152480,1sleep10-21swapper/11
218672460,1sleep00-21swapper/00
508199333,17cyclictest0-21swapper/00
509299311,27cyclictest28770-21cron2
5081993112,16cyclictest14265-21grep0
5081993014,13cyclictest13086-21sed0
5081993013,15cyclictest5381-21grep0
5081993012,16cyclictest17711-21ls0
508199292,24cyclictest21406-21gltestperf0
5081992913,14cyclictest6335-21seq0
5081992913,14cyclictest26833-21date0
5081992913,14cyclictest25334-21expr0
5081992913,14cyclictest12565-21seq0
5081992913,14cyclictest12565-21seq0
5081992912,15cyclictest29510-21grep0
5081992912,14cyclictest31757-21expr0
5081992911,15cyclictest31331-21which0
5081992911,15cyclictest10058-21seq0
508199290,4cyclictest19004-21diskstats0
5081992812,14cyclictest9626-21cat0
508199280,2cyclictest22842-21http0
508199278,16cyclictest26307-21cut0
508199277,17cyclictest17568-21processes0
508199276,18cyclictest16787-21cut0
5081992713,12cyclictest665-21runrttasks0
508199271,20cyclictest4612-21apt-get0
5081992710,14cyclictest10656-21df_inode0
508199270,3cyclictest7797-21diskstats0
508199270,3cyclictest32418-21taskset0
508199270,3cyclictest32418-21taskset0
508199270,3cyclictest27942-21gzip0
508199270,3cyclictest23073-21df_inode0
508199270,2cyclictest23544-21if_err_enp2s00
509299261,22cyclictest8161-21sendmail-mta2
508199269,14cyclictest25027-21sed0
508199266,18cyclictest7184-21ls0
5081992613,10cyclictest22763-21grep0
5081992612,12cyclictest18768-21/usr/sbin/munin0
5081992611,13cyclictest9407-21cat0
5081992611,13cyclictest25828-21awk0
5081992611,12cyclictest665-21runrttasks0
5081992611,12cyclictest2082-21systemd-journal0
508199260,2cyclictest2911-21cpuspeed_turbos0
508199260,2cyclictest11902-21find0
509299253,8cyclictest12492-21apt-get2
509299253,8cyclictest12492-21apt-get2
508199258,14cyclictest2082-21systemd-journal0
508199258,14cyclictest11012-21if_enp3s00
508199257,15cyclictest21255-21fw_packets0
508199256,17cyclictest28769-21munin-run0
508199256,16cyclictest9131-21proc_pri0
508199254,19cyclictest16297-21anacron0
508199254,18cyclictest7478-21latency0
508199254,18cyclictest29073-21cut0
508199254,18cyclictest29073-21cut0
508199252,18cyclictest32389-21apt-get0
508199251,20cyclictest18891-21apt-get0
508199251,19cyclictest19909-21apt-get0
508199250,2cyclictest4300-21seq0
509299245,17cyclictest24661-21cat2
509299244,18cyclictest0-21swapper/22
509299244,17cyclictest8589-21procmail2
509299242,14cyclictest29060-21taskset2
509299240,17cyclictest11229-1kworker/2:3H2
509299240,17cyclictest11229-1kworker/2:3H2
508199248,13cyclictest13322-21apt-key0
508199248,13cyclictest13322-21apt-key0
508199245,17cyclictest18708-21latency_hist0
508199245,16cyclictest12959-21gltestperf0
508199245,16cyclictest12959-21gltestperf0
508199243,19cyclictest2323-21turbostat0
508199243,18cyclictest18446-21cat0
5081992412,10cyclictest3723-21cron0
508199241,20cyclictest3316-21chrt0
508199241,20cyclictest2716-21munin-run0
508199241,19cyclictest28934-21apt-get0
508199240,22cyclictest7290-21irqstats0
509299239,12cyclictest22692-21latency_hist2
509299233,17cyclictest0-21swapper/22
5092992310,10cyclictest26771-21gltestperf2
508199237,14cyclictest5018-21cpuspeed_turbos0
508199233,18cyclictest28784-21idleruntime-cro0
508199231,19cyclictest13266-21grep0
508199231,19cyclictest13266-21grep0
508199231,18cyclictest2082-21systemd-journal0
28242230,4sleep121-21ksoftirqd/11
509899223,16cyclictest369-21cron3
509299225,14cyclictest0-21swapper/22
509299223,17cyclictest12662-21cat2
509299223,17cyclictest11268-21cat2
5092992216,4cyclictest11513-21switchtime2
5092992215,4cyclictest22562-21cron2
509299221,4cyclictest2737-21apt-get2
509299221,18cyclictest7073-21sed2
5086992216,4cyclictest3105-21df_inode1
509899212,16cyclictest27288-21cut3
5098992116,3cyclictest25517-21if_enp2s03
5098992116,3cyclictest11596-21memory3
5098992116,3cyclictest11596-21memory3
509299213,16cyclictest18534-21cat2
509299212,16cyclictest24686-21wc2
5092992116,3cyclictest25914-21pmu-power2
5092992116,3cyclictest15358-21ntp_kernel_pll_2
5092992115,4cyclictest14561-21ls2
5092992115,4cyclictest14561-21ls2
5092992114,5cyclictest32452-21seq2
5092992114,5cyclictest32452-21seq2
5092992114,4cyclictest4520-21ls2
5092992114,4cyclictest10068-21sed2
509299211,3cyclictest5071-21munin-run2
509299211,3cyclictest32187-1kworker/2:2H2
509299210,5cyclictest0-21swapper/22
509299210,4cyclictest10146-1kworker/2:0H2
509299210,19cyclictest0-21swapper/22
5086992116,3cyclictest18873-21/usr/sbin/munin1
5086992115,4cyclictest15636-21sed1
5086992115,4cyclictest12194-21cron1
5086992113,5cyclictest30096-21cron1
5086992113,5cyclictest30096-21cron1
203672210,2sleep30-21swapper/33
509899203,14cyclictest19057-21cut3
5098992016,2cyclictest780-21aten2_r9power_v3
5098992015,3cyclictest31567-21timerwakeupswit3
5098992015,3cyclictest30804-21sh3
5098992015,3cyclictest19312-21df_inode3
5098992014,4cyclictest5260-21cut3
5098992014,4cyclictest3333-21memory3
5098992014,4cyclictest24689-21diskstats3
5098992012,6cyclictest28318-21/usr/sbin/munin3
5098992012,6cyclictest28318-21/usr/sbin/munin3
509299209,9cyclictest28793-21cat2
509299202,16cyclictest684-21latency_hist2
509299202,16cyclictest6496-21cat2
5092992015,3cyclictest23994-21cron2
5092992014,4cyclictest17454-21if_err_enp3s02
5092992014,4cyclictest13759-21cat2
509299201,3cyclictest5703-21pmu-power2
509299201,3cyclictest10146-1kworker/2:0H2
5092992013,4cyclictest6099-21sh2
5092992013,4cyclictest21967-21grep2
509299201,2cyclictest0-21swapper/22
509299201,17cyclictest7113-21munin-run2
509299201,17cyclictest10465-21latency_hist2
509299201,16cyclictest12620-21cron2
509299201,16cyclictest12620-21cron2
5092992010,8cyclictest30765-21latency_hist2
509299200,3cyclictest21992-21apt-config2
509299200,2cyclictest15473-21ntp_states2
509299200,18cyclictest30824-21apt2
508699201,5cyclictest0-21swapper/11
5086992015,3cyclictest15034-21sed1
5086992015,3cyclictest13560-21swap1
5086992015,3cyclictest13560-21swap1
5086992014,4cyclictest3927-21awk1
5086992014,4cyclictest25220-21awk1
5086992014,3cyclictest26989-21acpi1
5086992013,4cyclictest505-21dbus-daemon1
5086992013,4cyclictest505-21dbus-daemon1
5086992013,4cyclictest1834-21grep1
5086992012,5cyclictest27654-21unixbench_singl1
509899193,10cyclictest5244-21apt-get3
5098991917,1cyclictest0-21swapper/33
5098991916,2cyclictest0-21swapper/33
5098991914,3cyclictest9525-21ntp_kernel_err3
5098991914,3cyclictest32642-21cut3
5098991914,3cyclictest32232-21ntp_kernel_pll_3
5098991914,3cyclictest30878-21date3
5098991914,3cyclictest29803-21expr3
5098991914,3cyclictest27323-21sh3
5098991914,3cyclictest24417-21munin-plugin-st3
5098991914,3cyclictest23534-21cat3
5098991914,3cyclictest18242-21expr3
5098991914,3cyclictest17328-21missed_timers3
5098991914,3cyclictest13377-21awk3
5098991914,3cyclictest13377-21awk3
5098991914,3cyclictest1272-21missed_timers3
5098991914,3cyclictest12673-21timerwakeupswit3
5098991914,3cyclictest12673-21timerwakeupswit3
5098991914,3cyclictest10978-21memory3
5098991913,4cyclictest9245-21entropy3
5098991913,4cyclictest7355-21find3
5098991913,4cyclictest28836-21cat3
5098991913,4cyclictest14915-21expr3
5098991913,3cyclictest7303-21kernelversion3
5098991913,3cyclictest7119-21cstates3
5098991913,3cyclictest15555-21missed_timers3
5092991917,1cyclictest0-21swapper/22
5092991917,1cyclictest0-21swapper/22
5092991917,1cyclictest0-21swapper/22
5092991915,2cyclictest17647-21cat2
5092991914,3cyclictest8968-21if_enp2s02
5092991914,3cyclictest18733-21ntp_states2
509299191,2cyclictest15586-21perf2
509299190,17cyclictest26658-21idleruntime-cro2
509299190,17cyclictest2521-21cat2
508699199,7cyclictest0-21swapper/11
508699199,7cyclictest0-21swapper/11
508699199,5cyclictest31374-21apt-get1
508699192,15cyclictest11316-21cat1
5086991915,2cyclictest6340-21/usr/sbin/munin1
5086991915,2cyclictest30944-21/usr/sbin/munin1
5086991915,2cyclictest30944-21/usr/sbin/munin1
5086991915,2cyclictest20788-21/usr/sbin/munin1
5086991914,3cyclictest8797-21/usr/sbin/munin1
5086991914,3cyclictest8668-21/usr/sbin/munin1
5086991914,3cyclictest28729-21seq1
5086991914,3cyclictest12806-21ls1
5086991913,4cyclictest29579-21expr1
5086991913,4cyclictest22196-21sh1
5086991913,4cyclictest15922-21cat1
5086991913,4cyclictest11722-21ntpq1
5086991913,4cyclictest11722-21ntpq1
5086991913,3cyclictest31293-21ls1
5086991912,4cyclictest7006-21sh1
5086991912,4cyclictest30693-21latency_hist1
5086991912,4cyclictest25611-21sed1
5086991912,4cyclictest16798-21cat1
5086991911,5cyclictest505-21dbus-daemon1
508699191,15cyclictest10737-21cut1
5086991911,4cyclictest505-21dbus-daemon1
5098991816,1cyclictest0-21swapper/33
5098991816,1cyclictest0-21swapper/33
5098991816,1cyclictest0-21swapper/33
5098991816,1cyclictest0-21swapper/33
5098991816,1cyclictest0-21swapper/33
5098991814,2cyclictest4080-21users3
5098991813,3cyclictest9376-21/usr/sbin/munin3
5098991813,3cyclictest9195-21rt-features3
5098991813,3cyclictest7194-21ntp_kernel_err3
5098991813,3cyclictest7142-21grep3
5098991813,3cyclictest4864-21idleruntime3
5098991813,3cyclictest26638-21expr3
5098991813,3cyclictest2408-21gltestperf3
5098991813,3cyclictest22703-21idleruntime-cro3
5098991813,3cyclictest22393-21latency_hist3
5098991813,3cyclictest13543-21sed3
5098991813,3cyclictest13543-21sed3
5098991813,3cyclictest13341-21ls3
5098991812,4cyclictest4836-21cut3
5098991812,4cyclictest23729-21ntpq3
5098991812,4cyclictest12511-21cron3
5098991811,4cyclictest31748-21cat3
5098991811,4cyclictest31748-21cat3
5092991816,1cyclictest0-21swapper/22
5092991814,2cyclictest1769-21cat2
5092991813,3cyclictest8946-21cat2
5092991813,3cyclictest7395-21missed_timers2
508699189,7cyclictest13252-21gltestperf1
508699189,7cyclictest13252-21gltestperf1
508699188,7cyclictest0-21swapper/11
508699184,8cyclictest10757-21apt-get1
508699182,10cyclictest24984-21apt-get1
5086991814,2cyclictest8377-21/usr/sbin/munin1
5086991814,2cyclictest26887-21/usr/sbin/munin1
5086991814,2cyclictest2285-21cstates1
5086991814,2cyclictest19890-21/usr/sbin/munin1
5086991814,2cyclictest15815-21/usr/sbin/munin1
5086991813,3cyclictest7794-21cat1
5086991813,3cyclictest665-21runrttasks1
5086991813,3cyclictest665-21runrttasks1
5086991813,3cyclictest4594-21/usr/sbin/munin1
5086991812,4cyclictest22834-21grep1
5086991812,4cyclictest1371-21grep1
5086991812,3cyclictest31176-21aten2.4-expect1
5086991812,3cyclictest2082-21systemd-journal1
5086991811,5cyclictest18786-21apt-get1
5086991811,4cyclictest3540-21grep1
5086991811,4cyclictest24974-21cut1
5086991811,4cyclictest23087-21cat1
508699180,15cyclictest28769-21cron1
5098991713,2cyclictest1049-21aten2_r9power_v3
5098991712,3cyclictest8790-21ls3
5098991712,3cyclictest25353-21expr3
5098991712,3cyclictest23345-21switchtime3
5098991711,3cyclictest10530-21aten2_r9power_p3
5092991715,1cyclictest0-21swapper/22
5092991711,3cyclictest23021-21cut2
509299170,16cyclictest0-21swapper/22
508699173,9cyclictest7370-21apt-get1
5086991713,2cyclictest22634-21/usr/sbin/munin1
5086991611,3cyclictest4727-21/usr/sbin/munin1
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional