You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 05:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot2.osadl.org (updated Fri Nov 21, 2025 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14062279243,24sleep20-21swapper/22
16562268231,25sleep10-21swapper/11
15382268193,24sleep00-21swapper/00
13772241177,51sleep30-21swapper/33
184599205198,4cyclictest19741-21apt-get0
185599196192,2cyclictest0-21swapper/33
185199166160,4cyclictest0-21swapper/22
184799153145,5cyclictest19726-21munin-run1
40222590,2sleep10-21swapper/11
226012510,5sleep00-21swapper/00
4452500,2sleep10-21swapper/11
1845995018,8cyclictest8203-21/usr/sbin/munin0
1845995016,9cyclictest12844-21munin-run0
1845994816,18cyclictest19926-21diskstats0
1845994616,15cyclictest9449-21diskstats0
1845994516,16cyclictest28379-21sed0
1845994516,16cyclictest10366-21irqstats0
1845994515,9cyclictest6214-21latency_hist0
1845994512,8cyclictest3358-21latency_hist0
31492440,1sleep20-21swapper/22
188782440,1sleep10-21swapper/11
185599445,15cyclictest0-21swapper/33
1855994413,9cyclictest3914-21latency_hist3
1855994413,9cyclictest3914-21latency_hist3
1845994416,15cyclictest5107-21users0
1845994416,15cyclictest5107-21users0
1845994414,8cyclictest6764-21sed0
192622430,1sleep00-21swapper/00
1845994315,8cyclictest9600-21cut0
1845994315,8cyclictest31081-21latency_hist0
1845994314,16cyclictest26328-21memory0
1845994217,9cyclictest30860-21diskstats0
1845994216,10cyclictest7844-21aten2.4-expect0
1845994216,10cyclictest31813-21munin-run0
1845994215,15cyclictest21752-21sed0
1845994215,10cyclictest8915-21df_inode0
1845994214,8cyclictest28882-21latency_hist0
1845994211,15cyclictest0-21swapper/00
1845994117,7cyclictest16264-21df_inode0
1845994116,9cyclictest26803-21df0
1845994116,8cyclictest13257-21users0
1845994116,10cyclictest5505-21latency_hist0
1845994116,10cyclictest16423-21gltestperf0
1845994115,9cyclictest28027-21taskset0
1845994115,9cyclictest17821-21cat0
1845994115,8cyclictest30047-21cat0
1845994115,8cyclictest14259-21ntp_states0
1845994115,14cyclictest23760-21memory0
1845994110,15cyclictest0-21swapper/00
81922400,1sleep20-21swapper/22
1851994028,3cyclictest28-21ksoftirqd/22
1845994016,8cyclictest32115-21df_inode0
1845994015,9cyclictest12979-21diskstats0
1845994015,10cyclictest31934-21sed0
1845994015,10cyclictest16463-21sed0
185599397,9cyclictest13777-21sed3
185599397,8cyclictest31491-21latency_hist3
1855993911,9cyclictest5347-21sed3
1845993916,8cyclictest7130-21users0
1845993916,8cyclictest25730-21irqstats0
1845993916,7cyclictest24788-21df_inode0
1845993915,9cyclictest12641-21ntp_states0
1845993914,13cyclictest2645-21expr0
1855993812,9cyclictest11612-21turbostat3
1845993813,9cyclictest20306-21df0
1845993813,9cyclictest20306-21df0
1845993813,8cyclictest21125-21nfs4_client0
1845993813,10cyclictest3255-21gltestperf0
185599379,9cyclictest9970-21chrt3
185599378,17cyclictest2381-21awk3
1845993716,8cyclictest11177-21awk0
1845993716,7cyclictest21940-21df_inode0
1845993715,8cyclictest6522-21cat0
1845993715,8cyclictest32439-21cstates0
1845993715,8cyclictest1647-21/usr/sbin/munin0
1845993714,9cyclictest22155-21cpuspeed_turbos0
1845993714,9cyclictest15824-21cat0
1845993713,9cyclictest17366-21seq0
1845993713,8cyclictest29387-21ps0
185599368,15cyclictest18623-21cat3
185599368,10cyclictest9198-21chrt3
1855993610,15cyclictest27849-21date3
1845993614,7cyclictest23972-21apt-get0
185599359,9cyclictest27170-21chrt3
185599358,10cyclictest2533-21chrt3
185599355,18cyclictest0-21swapper/33
1855993510,10cyclictest15822-21sed3
1851993521,3cyclictest28-21ksoftirqd/22
1845993513,7cyclictest433-21df_inode0
1845993513,7cyclictest433-21df_inode0
1845993510,14cyclictest0-21swapper/00
1845993510,14cyclictest0-21swapper/00
185599349,9cyclictest30857-21chrt3
185599348,10cyclictest12791-21chrt3
185599339,9cyclictest6346-21chrt3
185599339,9cyclictest24781-21df3
185599339,9cyclictest15070-21chrt3
185599339,10cyclictest29466-21taskset3
185599338,9cyclictest30036-21taskset3
185599338,9cyclictest29225-21chrt3
185599338,9cyclictest23155-21chrt3
185599338,9cyclictest15416-21chrt3
185599337,10cyclictest11307-21chrt3
185599336,15cyclictest0-21swapper/33
185599336,15cyclictest0-21swapper/33
1845993311,13cyclictest0-21swapper/00
185599329,9cyclictest26725-21chrt3
185599329,8cyclictest25208-21chrt3
185599329,8cyclictest14380-21cat3
185599328,9cyclictest5314-21chrt3
185599328,9cyclictest5314-21chrt3
185599328,9cyclictest24027-21chrt3
185599328,9cyclictest1807-21latency_hist3
185599328,9cyclictest1807-21latency_hist3
185599328,8cyclictest20271-21chrt3
185599327,9cyclictest28852-21chrt3
185599327,8cyclictest31698-21apt-get3
1851993216,4cyclictest28-21ksoftirqd/22
1851993212,4cyclictest28-21ksoftirqd/22
185599319,9cyclictest10281-21sed3
185599318,9cyclictest25682-21chrt3
185599314,16cyclictest0-21swapper/33
185599314,16cyclictest0-21swapper/33
1851993118,3cyclictest28-21ksoftirqd/22
1845993116,12cyclictest1063-21tr0
1845993111,7cyclictest0-21swapper/00
1845993110,8cyclictest0-21swapper/00
185599308,9cyclictest17661-21chrt3
185599307,9cyclictest8517-21chrt3
185599307,9cyclictest7214-21chrt3
185599307,9cyclictest32071-21chrt3
185599306,15cyclictest31812-21awk3
185599305,9cyclictest0-21swapper/33
185599305,8cyclictest0-21swapper/33
185599305,14cyclictest0-21swapper/33
185599304,15cyclictest0-21swapper/33
1851993014,2cyclictest28-21ksoftirqd/22
184599308,8cyclictest0-21swapper/00
185599297,9cyclictest11993-21chrt3
185599294,22cyclictest189892chrt3
185599294,15cyclictest0-21swapper/33
1851992914,3cyclictest28-21ksoftirqd/22
185599286,9cyclictest4463-21chrt3
185599285,9cyclictest0-21swapper/33
185599285,8cyclictest0-21swapper/33
185599285,8cyclictest0-21swapper/33
1855992821,5cyclictest12397-21irqstats3
1855992811,14cyclictest8055-21tr3
1851992815,3cyclictest28-21ksoftirqd/22
1851992814,3cyclictest28-21ksoftirqd/22
1847992815,3cyclictest21-21ksoftirqd/11
185199276,19cyclictest0-21swapper/22
1851992723,2cyclictest271rcuc/22
1845992719,5cyclictest10504-21/usr/sbin/munin0
185599262,11cyclictest0-21swapper/33
1851992620,3cyclictest271rcuc/22
1851992614,3cyclictest28-21ksoftirqd/22
47012250,2sleep20-21swapper/22
47012250,2sleep20-21swapper/22
185199258,7cyclictest28-21ksoftirqd/22
185199253,10cyclictest2563-21dump-pmu-power2
185199251,9cyclictest28-21ksoftirqd/22
1851992514,3cyclictest111rcu_preempt2
1851992511,3cyclictest28-21ksoftirqd/22
1851992510,4cyclictest28-21ksoftirqd/22
1845992517,5cyclictest6329-21sed0
1855992415,5cyclictest7466-21sendmail-mta3
185199247,8cyclictest28-21ksoftirqd/22
1851992417,4cyclictest23289-21munin-run2
185199240,9cyclictest28-21ksoftirqd/22
185199240,6cyclictest28-21ksoftirqd/22
1847992410,2cyclictest111rcu_preempt1
1845992416,5cyclictest23904-21/usr/sbin/munin0
185599233,7cyclictest0-21swapper/33
1855992315,5cyclictest488-21grep3
1855992314,6cyclictest28172-21munin-run3
185199233,6cyclictest28-21ksoftirqd/22
1851992313,3cyclictest28-21ksoftirqd/22
1851992311,3cyclictest28-21ksoftirqd/22
1851992310,3cyclictest28-21ksoftirqd/22
1851992310,3cyclictest111rcu_preempt2
184799238,2cyclictest111rcu_preempt1
1847992318,3cyclictest21-21ksoftirqd/11
1855992214,5cyclictest31845-21tr3
185199229,4cyclictest28-21ksoftirqd/22
185199229,3cyclictest28-21ksoftirqd/22
185199229,3cyclictest28-21ksoftirqd/22
185199229,3cyclictest28-21ksoftirqd/22
185199228,4cyclictest28-21ksoftirqd/22
185199226,7cyclictest28-21ksoftirqd/22
185199223,11cyclictest28-21ksoftirqd/22
185199222,10cyclictest111rcu_preempt2
1851992214,5cyclictest1715-21grep2
1851992213,2cyclictest111rcu_preempt2
1851992212,3cyclictest28-21ksoftirqd/22
1851992211,3cyclictest28-21ksoftirqd/22
1851992211,3cyclictest28-21ksoftirqd/22
1851992211,2cyclictest111rcu_preempt2
1851992210,3cyclictest28-21ksoftirqd/22
1847992216,4cyclictest5482-21sh1
1847992216,4cyclictest5482-21sh1
185199219,3cyclictest28-21ksoftirqd/22
185199215,8cyclictest28-21ksoftirqd/22
185199214,5cyclictest0-21swapper/22
1851992116,3cyclictest32563-21vmstat2
1851992114,4cyclictest2116-21sed2
1851992114,4cyclictest2116-21sed2
185199211,18cyclictest31777-1kworker/2:0H2
1851992111,2cyclictest111rcu_preempt2
1851992111,2cyclictest111rcu_preempt2
1851992111,2cyclictest111rcu_preempt2
1851992110,2cyclictest111rcu_preempt2
185199210,18cyclictest9375-21ntp_states2
185199210,10cyclictest28-21ksoftirqd/22
1847992115,4cyclictest8628-21sh1
1847992115,4cyclictest3904-21seq1
1847992114,4cyclictest30846-21df1
1847992114,4cyclictest21473-21seq1
1847992114,4cyclictest21473-21seq1
1855992011,7cyclictest23845-21needreboot3
185199208,3cyclictest28-21ksoftirqd/22
185199208,2cyclictest111rcu_preempt2
185199207,3cyclictest28-21ksoftirqd/22
1851992014,4cyclictest3562-21date2
1851992014,3cyclictest6649-21cpuspeed_turbos2
1851992011,2cyclictest111rcu_preempt2
1851992010,3cyclictest111rcu_preempt2
185199200,6cyclictest16639-21apt-get2
184799208,3cyclictest21-21ksoftirqd/11
1847992018,1cyclictest0-21swapper/11
184799201,6cyclictest0-21swapper/11
1847992015,3cyclictest3614-21/usr/sbin/munin1
1847992015,3cyclictest31313-21seq1
1847992015,3cyclictest25969-21/usr/sbin/munin1
1847992015,3cyclictest17290-21seq1
1847992015,3cyclictest11545-21processes1
1847992014,4cyclictest5867-21seq1
1847992014,4cyclictest22972-21seq1
1847992014,4cyclictest19400-21sshd1
1847992014,4cyclictest13160-21cat1
1847992014,3cyclictest10227-21sh1
1847992013,5cyclictest21648-21grep1
1847992013,5cyclictest21401-21sed1
1847992013,4cyclictest1297-21seq1
185199190,3cyclictest27402-21ls2
184799196,11cyclictest0-21swapper/11
1847991915,2cyclictest23952-21/usr/sbin/munin1
1847991914,3cyclictest6974-21/usr/sbin/munin1
1847991914,3cyclictest4824-21/usr/sbin/munin1
1847991914,3cyclictest30793-21seq1
1847991914,3cyclictest30658-21cat1
1847991914,3cyclictest2857-21ntpq1
1847991914,3cyclictest26010-21df_inode1
1847991914,3cyclictest2516-21ntp_kernel_err1
1847991914,3cyclictest23401-21seq1
1847991914,3cyclictest21686-21sed1
1847991914,3cyclictest17118-21seq1
1847991914,3cyclictest16621-21/usr/sbin/munin1
1847991914,3cyclictest15908-21seq1
1847991914,3cyclictest15611-21processes1
1847991914,3cyclictest13372-21/usr/sbin/munin1
1847991914,3cyclictest11747-21/usr/sbin/munin1
1847991914,3cyclictest10256-21latency_hist1
1847991913,4cyclictest8208-21sh1
1847991913,4cyclictest3931-21cat1
1847991913,4cyclictest11591-21cut1
184799183,9cyclictest776-21apt-get1
1847991814,2cyclictest5628-21/usr/sbin/munin1
1847991814,2cyclictest31152-21/usr/sbin/munin1
1847991814,2cyclictest26175-21switchtime1
1847991813,3cyclictest665-21runrttasks1
1847991813,3cyclictest665-21runrttasks1
1847991813,3cyclictest28388-21seq1
1847991813,3cyclictest25455-21ls1
1847991813,3cyclictest21-21ksoftirqd/11
1847991813,3cyclictest13856-21/usr/sbin/munin1
1847991812,4cyclictest28821-21cat1
1847991812,4cyclictest11973-21cut1
184799180,6cyclictest28167-21perf1
1847991713,2cyclictest17456-21/usr/sbin/munin1
1847991712,3cyclictest4988-21if_enp3s01
1847991712,3cyclictest23910-21seq1
1847991712,3cyclictest1265-21cat1
1847991712,3cyclictest1265-21cat1
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional