You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 23:52
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot2.osadl.org (updated Wed Nov 19, 2025 12:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
268022333259,25sleep20-21swapper/22
268502294257,24sleep00-21swapper/00
266302289250,26sleep10-21swapper/11
27123992820,1cyclictest0-21swapper/22
266802265185,44sleep30-21swapper/33
2712899243240,2cyclictest0-21swapper/33
2712099241237,2cyclictest0-21swapper/11
2711599199194,3cyclictest7669-21popularity-cont0
2140621150,2sleep10-21swapper/11
323562470,1sleep10-21swapper/11
160132470,1sleep00-21swapper/00
71282450,2sleep30-21swapper/33
132022420,1sleep20-21swapper/22
2712899361,33cyclictest25492-21kworker/3:33
27115993613,20cyclictest31817-21cat0
27115993613,20cyclictest31731-21wc0
27115993412,19cyclictest25762-21seq0
2711599331,29cyclictest27883-21turbostat.cron0
27115993312,18cyclictest31783-21seq0
27115993312,18cyclictest31783-21seq0
27115993212,18cyclictest333-21cat0
27115993212,18cyclictest12498-21seq0
27115993112,17cyclictest20372-21sed0
27115993112,16cyclictest6401-21cat0
27115993111,17cyclictest24509-21seq0
27115993111,17cyclictest21371-21seq0
27115993110,18cyclictest2082-21systemd-journal0
2711599308,4cyclictest18208-21tail0
27115993015,13cyclictest4755-21apt-get0
27115993010,17cyclictest14758-21date0
27115992910,16cyclictest4701-21nfs4_client0
27115992910,16cyclictest28993-21seq0
2711599289,3cyclictest32261-21switchtime0
2711599282,9cyclictest28718-21apt-get0
2711599282,9cyclictest28718-21apt-get0
2711599279,3cyclictest17447-21sort0
2711599279,15cyclictest296092chrt0
2711599279,15cyclictest23091-21basename0
2711599278,3cyclictest3314-21cat0
2711599278,16cyclictest0-21swapper/00
2711599277,5cyclictest505-21dbus-daemon0
27115992712,12cyclictest665-21runrttasks0
2711599271,12cyclictest26426-21dump-pmu-power0
27115992711,13cyclictest17555-21if_enp3s00
27115992710,3cyclictest29038-21cat0
2711599268,4cyclictest15480-21cat0
2711599268,15cyclictest104622chrt0
2711599267,3cyclictest30479-21unixbench_multi0
27115992611,13cyclictest25083-21/usr/sbin/munin0
27115992611,12cyclictest18567-21aten2_r9power_c0
27115992610,14cyclictest22448-21latency_hist0
2712399256,16cyclictest0-21swapper/22
2711599259,3cyclictest8645-21if_enp2s00
2711599259,3cyclictest7583-21sh0
2711599258,2cyclictest14265-21cat0
2711599257,3cyclictest67562chrt0
2711599257,3cyclictest2940-21cpuspeed_turbos0
2711599256,3cyclictest17660-21cat0
2711599255,3cyclictest2845-21tr0
2711599254,8cyclictest665-21runrttasks0
2711599251,8cyclictest12010-21wc0
2712399244,18cyclictest28964-21cat2
27123992415,5cyclictest4510-21sh2
27120992416,5cyclictest4882-21sh1
2711599248,3cyclictest226592chrt0
2711599247,3cyclictest19093-21cpu0
2711599246,3cyclictest6554-21cut0
2711599245,4cyclictest22907-21sh0
2711599245,3cyclictest28115-21processes0
2711599245,3cyclictest13409-21aten2_r9power_v0
2711599245,3cyclictest13409-21aten2_r9power_v0
2711599242,4cyclictest12850-21apt-get0
2711599241,9cyclictest1811-21/usr/sbin/munin0
2711599241,9cyclictest13958-21gpgv0
2712399234,17cyclictest4945-21cat2
2712399234,16cyclictest611-21procmail2
2712399233,17cyclictest10727-21sendmail-mta2
27123992313,6cyclictest14911-21apt-get2
27120992312,7cyclictest12142-21apt-get1
2711599238,3cyclictest11412chrt0
2711599236,3cyclictest0-21swapper/00
2711599232,4cyclictest26671-21apt-get0
2711599232,4cyclictest14879-21apt-get0
27115992316,4cyclictest786-21apt-get0
27128992217,3cyclictest1595-21open_files3
2712399223,17cyclictest516-21cat2
2712399223,17cyclictest20761-21latency_hist2
2712399223,17cyclictest13902-21latency_hist2
2712399221,18cyclictest16655-21munin-run2
2712399221,17cyclictest27891-21munin-run2
27123992210,10cyclictest31926-21idleruntime-cro2
27123992210,10cyclictest31926-21idleruntime-cro2
2711599225,3cyclictest22728-21aten2.4-expect0
2711599225,3cyclictest10175-21awk0
2711599222,5cyclictest29119-21apt-get0
229072220,1sleep30-21swapper/33
2712899216,3cyclictest160142expr3
27128992116,3cyclictest23453-21gltestperf3
27128992115,4cyclictest27677-21seq3
2712399218,11cyclictest28-21ksoftirqd/22
2712399212,17cyclictest22359-21awk2
27123992116,3cyclictest1273-21munin-run2
27123992115,4cyclictest22749-21munin-run2
27123992115,4cyclictest20309-21cron2
27123992115,4cyclictest20060-21sed2
27123992115,4cyclictest10333-21gdbus2
27123992115,3cyclictest1501-21munin-run2
2712399211,4cyclictest505-21dbus-daemon2
27123992114,4cyclictest29586-21diskstats2
27123992114,4cyclictest28908-21sh2
2712399211,2cyclictest10607-21latency_hist2
27120992115,4cyclictest23029-21/usr/sbin/munin1
27120992115,3cyclictest6906-21cron1
27120992114,4cyclictest2170-21grep1
27120992114,4cyclictest10333-21gdbus1
2711599213,16cyclictest0-21swapper/00
2711599212,5cyclictest20984-21apt-get0
2711599212,4cyclictest704-21apt-get0
103032210,2sleep20-21swapper/22
27128992015,3cyclictest7166-21apt3
27128992015,3cyclictest3923-21cat3
27128992015,3cyclictest32755-21seq3
27128992015,3cyclictest28496-21vmstat3
27128992015,3cyclictest26853-21cron3
27128992015,3cyclictest22042-21mailstats3
27128992015,3cyclictest20140-21fw_packets3
27128992015,3cyclictest16201-21sed3
27128992014,4cyclictest4966-21ls3
27128992014,4cyclictest2938-21wc3
27128992014,4cyclictest2173-21sed3
27128992014,4cyclictest14798-21cat3
2712899200,18cyclictest20848-21sed3
2712399202,17cyclictest24411-21perf2
2712399202,14cyclictest9227-21apt-get2
27123992016,2cyclictest22723-21latency2
27123992015,3cyclictest31786-21latency_hist2
27123992015,3cyclictest29999-21ntp_states2
2712399201,4cyclictest25807-21cron2
27123992014,3cyclictest18250-21if_enp2s02
2712399201,3cyclictest3619-21perf2
27123992013,4cyclictest26814-21idleruntime-cro2
2712399201,17cyclictest6921-21perf2
2712399201,17cyclictest5863-21cat2
2712399201,17cyclictest30467-21cat2
2712399201,17cyclictest30467-21cat2
2712399201,17cyclictest26472-21cat2
2712399201,17cyclictest0-21swapper/22
2712099206,11cyclictest2082-21systemd-journal1
27120992015,3cyclictest22581-21/usr/sbin/munin1
27120992015,3cyclictest10811-21/usr/sbin/munin1
27120992014,4cyclictest8519-21grep1
27120992014,4cyclictest2457-21seq1
27120992014,3cyclictest13798-21gltestperf1
27120992014,3cyclictest13798-21gltestperf1
27120992011,3cyclictest30797-21irqstats1
27120992011,3cyclictest30797-21irqstats1
27128991916,2cyclictest0-21swapper/33
27128991916,2cyclictest0-21swapper/33
27128991915,2cyclictest6860-21df_inode3
27128991915,2cyclictest23064-21aten2_r9power_c3
27128991914,3cyclictest9501-21load3
27128991914,3cyclictest8843-21missed_timers3
27128991914,3cyclictest31300-21sed3
27128991914,3cyclictest31300-21sed3
27128991914,3cyclictest29860-21missed_timers3
27128991914,3cyclictest25017-21expr3
27128991914,3cyclictest20449-21sed3
27128991914,3cyclictest13926-21sendmail_mailst3
27128991914,3cyclictest13539-21sed3
27128991914,3cyclictest13385-21taskset3
27128991914,3cyclictest13385-21taskset3
27128991914,3cyclictest10089-21cat3
27128991913,4cyclictest6081-21ls3
27128991913,4cyclictest29874-21cat3
27128991913,4cyclictest28586-21grep3
27128991913,4cyclictest27536-21ntp_kernel_pll_3
27128991913,4cyclictest24507-21wc3
27128991913,4cyclictest21024-21fw_packets3
27128991913,4cyclictest11038-21cut3
27128991912,4cyclictest1110-21expr3
2712399192,2cyclictest28-21ksoftirqd/22
2712399192,15cyclictest2464-21latency_hist2
27123991915,2cyclictest23121-21aten2_r9power_v2
27123991915,2cyclictest16959-21/usr/sbin/munin2
27123991914,3cyclictest8581-21sendmail2
27123991914,3cyclictest6576-21sed2
27123991914,3cyclictest29113-21date2
27123991914,3cyclictest2587-21grep2
27123991914,3cyclictest16462-21nfs4_client2
27123991914,3cyclictest12234-21ps2
27123991913,3cyclictest13719-21forks2
27123991913,3cyclictest13719-21forks2
2712399190,17cyclictest3663-21cat2
2712399190,17cyclictest18314-21perf2
2712099194,3cyclictest111rcu_preempt1
27120991915,2cyclictest21011-21/usr/sbin/munin1
27120991915,2cyclictest14859-21/usr/sbin/munin1
27120991914,3cyclictest5451-21seq1
27120991914,3cyclictest32163-21/usr/sbin/munin1
27120991914,3cyclictest31129-21/usr/sbin/munin1
27120991914,3cyclictest29887-21/usr/sbin/munin1
27120991914,3cyclictest27074-21/usr/sbin/munin1
27120991914,3cyclictest15222-21sed1
27120991914,3cyclictest12871-21/usr/sbin/munin1
27120991913,4cyclictest26281-21sed1
27120991913,4cyclictest23670-21seq1
27120991913,4cyclictest21373-21cat1
27120991913,3cyclictest32049-21aten2.4-expect1
27120991913,3cyclictest21459-21latency_hist1
27120991912,4cyclictest2173-21expr1
27120991912,4cyclictest1580-21sh1
27120991911,5cyclictest13314-21grep1
2712099190,17cyclictest0-21swapper/11
2712099190,17cyclictest0-21swapper/11
2712099190,17cyclictest0-21swapper/11
27128991816,1cyclictest0-21swapper/33
27128991816,1cyclictest0-21swapper/33
27128991814,2cyclictest31773-21if_enp3s03
27128991814,2cyclictest18014-21sendmail_mailst3
27128991813,3cyclictest582-21ntp_kernel_pll_3
27128991813,3cyclictest32233-21expr3
27128991813,3cyclictest30560-21http3
27128991813,3cyclictest30560-21http3
27128991813,3cyclictest28402-21unixbench-2d3
27128991813,3cyclictest26918-21cat3
27128991813,3cyclictest18524-21ntp_kernel_pll_3
27128991813,3cyclictest12540-21expr3
27128991812,4cyclictest32011-21hostname3
27128991812,4cyclictest30369-21tr3
27128991812,4cyclictest17138-21sort3
2712899181,15cyclictest0-21swapper/33
27128991811,4cyclictest5457-21expr3
27128991811,4cyclictest2082-21systemd-journal3
27123991816,1cyclictest0-21swapper/22
27123991816,1cyclictest0-21swapper/22
27123991813,3cyclictest1661-21sh2
27123991813,3cyclictest11222-21expr2
27123991812,4cyclictest5461-21seq2
2712399180,2cyclictest0-21swapper/22
2712399180,17cyclictest0-21swapper/22
27120991814,2cyclictest31077-21/usr/sbin/munin1
27120991814,2cyclictest17661-21/usr/sbin/munin1
27120991814,2cyclictest14891-21/usr/sbin/munin1
27120991814,2cyclictest10086-21/usr/sbin/munin1
27120991813,3cyclictest8810-21seq1
27120991813,3cyclictest8801-21seq1
27120991813,3cyclictest6641-21sed1
27120991813,3cyclictest6223-21apt-config1
27120991813,3cyclictest6057-21date1
27120991813,3cyclictest3429-21seq1
27120991813,3cyclictest28399-21sed1
27120991813,3cyclictest25961-21ntpq1
27120991813,3cyclictest2082-21systemd-journal1
27120991813,3cyclictest18929-21seq1
27120991813,3cyclictest11240-21/usr/sbin/munin1
27120991812,4cyclictest30598-21seq1
27120991812,4cyclictest24210-21sed1
2712099180,16cyclictest0-21swapper/11
27128991715,1cyclictest0-21swapper/33
27128991712,3cyclictest4737-21cat3
27128991712,3cyclictest28419-21cut3
27128991712,3cyclictest10227-21vmstat3
2712899170,15cyclictest0-21swapper/33
27123991715,1cyclictest0-21swapper/22
27123991713,2cyclictest8058-21/usr/sbin/munin2
2712399171,14cyclictest24806-21cron2
2712399170,16cyclictest0-21swapper/22
2712099172,10cyclictest16913-21apt-get1
27120991713,2cyclictest665-21runrttasks1
27120991713,2cyclictest27111-21/usr/sbin/munin1
27120991712,3cyclictest25735-21seq1
27120991712,3cyclictest2082-21systemd-journal1
27120991712,3cyclictest18972-21seq1
27120991712,3cyclictest16008-21expr1
27120991712,3cyclictest12745-21latency_hist1
27120991712,3cyclictest10333-21gdbus1
27128991611,3cyclictest10858-21awk3
27120991611,3cyclictest26866-21missed_timers1
27120991611,3cyclictest18533-21/usr/sbin/munin1
2712099159,3cyclictest2801-21aten2_r9power_p1
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional