You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 01:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Sat Nov 08, 2025 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2089374997669,3cyclictest0-21swapper/008:30:030
2089382997028,12cyclictest235-21systemd-journal10:29:262
2089382997028,12cyclictest235-21systemd-journal10:29:262
2089390996630,12cyclictest37-21ksoftirqd/309:20:003
208937799660,45cyclictest0-21swapper/110:55:031
208939099657,11cyclictest37-21ksoftirqd/308:35:013
2089390996335,22cyclictest2102365-21sendmail11:20:013
208939099614,10cyclictest37-21ksoftirqd/307:45:013
2089390995915,4cyclictest131rcu_preempt11:02:263
208939099571,14cyclictest2102292-21latency_hist11:15:023
208939099568,7cyclictest37-21ksoftirqd/311:50:023
208939099562,7cyclictest37-21ksoftirqd/309:50:023
208939099562,7cyclictest37-21ksoftirqd/309:50:013
208939099558,12cyclictest37-21ksoftirqd/310:35:013
2089390995539,12cyclictest235-21systemd-journal08:20:003
208939099551,16cyclictest2095081-21cut09:00:003
208939099547,9cyclictest37-21ksoftirqd/310:10:023
208939099542,12cyclictest37-21ksoftirqd/308:25:023
2089382995437,12cyclictest272-21dbus-daemon07:55:022
208939099536,12cyclictest37-21ksoftirqd/309:40:033
208939099533,12cyclictest37-21ksoftirqd/310:20:003
2089374995348,2cyclictest0-21swapper/008:35:010
208939099527,9cyclictest37-21ksoftirqd/312:20:013
208939099527,9cyclictest37-21ksoftirqd/312:20:013
208939099525,14cyclictest37-21ksoftirqd/311:25:003
208939099523,7cyclictest37-21ksoftirqd/308:45:013
2089390995210,8cyclictest37-21ksoftirqd/309:30:023
2089374995220,11cyclictest12-21ksoftirqd/012:35:030
208939099516,9cyclictest37-21ksoftirqd/311:35:023
208939099516,8cyclictest37-21ksoftirqd/311:55:023
208939099512,8cyclictest1-21systemd09:05:003
208939099511,20cyclictest0-21swapper/310:45:023
208939099511,20cyclictest0-21swapper/310:45:013
208939099511,12cyclictest37-21ksoftirqd/312:32:253
2089382995137,10cyclictest272-21dbus-daemon11:20:002
2089374995121,24cyclictest12-21ksoftirqd/008:12:280
208939099507,6cyclictest37-21ksoftirqd/308:50:013
208939099506,11cyclictest37-21ksoftirqd/307:30:023
208939099502,12cyclictest37-21ksoftirqd/307:40:003
208937499505,3cyclictest131rcu_preempt08:00:000
208939099496,9cyclictest37-21ksoftirqd/307:35:013
2089390994927,16cyclictest235-21systemd-journal10:05:003
208939099491,5cyclictest0-21swapper/309:35:013
2089390994913,12cyclictest37-21ksoftirqd/307:50:013
208939099487,7cyclictest37-21ksoftirqd/308:40:013
208939099487,7cyclictest37-21ksoftirqd/307:10:023
208939099486,8cyclictest37-21ksoftirqd/308:00:003
208939099486,7cyclictest37-21ksoftirqd/311:10:013
208939099486,13cyclictest37-21ksoftirqd/309:25:023
208939099485,12cyclictest37-21ksoftirqd/311:40:013
2089390994827,14cyclictest272-21dbus-daemon10:25:003
208939099481,9cyclictest2106014-21modprobe12:30:003
208939099481,7cyclictest2098196-21cron10:00:003
2089382994832,11cyclictest235-21systemd-journal07:58:262
2089377994833,11cyclictest23-21ksoftirqd/109:00:001
208937499483,38cyclictest2102372-21systemctl11:20:010
208939099476,9cyclictest37-21ksoftirqd/311:59:593
208939099475,11cyclictest37-21ksoftirqd/312:40:013
208939099474,8cyclictest37-21ksoftirqd/308:55:023
208939099473,8cyclictest37-21ksoftirqd/308:30:033
2089390994732,9cyclictest306-21in:imuxsock11:45:013
208939099473,10cyclictest37-21ksoftirqd/307:25:033
208939099471,13cyclictest37-21ksoftirqd/312:25:003
2089382994731,11cyclictest235-21systemd-journal11:30:002
208938299471,4cyclictest131rcu_preempt09:10:012
208938299471,4cyclictest131rcu_preempt09:10:012
2089374994737,8cyclictest12-21ksoftirqd/008:06:140
208939099469,5cyclictest131rcu_preempt12:15:003
208939099463,9cyclictest37-21ksoftirqd/309:10:013
208939099463,9cyclictest37-21ksoftirqd/309:10:013
208939099461,6cyclictest0-21swapper/309:55:013
2089390994616,5cyclictest131rcu_preempt07:50:263
2089390994610,5cyclictest131rcu_preempt11:00:003
2089390994610,5cyclictest131rcu_preempt11:00:003
2089382994630,12cyclictest1-21systemd11:45:002
2089377994633,9cyclictest1-21systemd12:35:011
2089377994630,12cyclictest2098206-21systemctl10:00:011
2089374994629,13cyclictest2096083-21sshd09:15:260
208937499462,3cyclictest131rcu_preempt10:55:020
208939099458,8cyclictest37-21ksoftirqd/310:30:013
208939099458,8cyclictest37-21ksoftirqd/310:30:003
208939099455,8cyclictest37-21ksoftirqd/311:30:023
208939099453,7cyclictest37-21ksoftirqd/308:10:013
2089390994537,5cyclictest37-21ksoftirqd/309:15:023
208939099452,7cyclictest37-21ksoftirqd/307:15:023
208939099452,7cyclictest37-21ksoftirqd/307:15:023
2089382994536,5cyclictest2089384-21cat07:10:012
2089382994526,13cyclictest1-21systemd11:40:002
208938299452,18cyclictest0-21swapper/209:20:012
2089377994530,11cyclictest235-21systemd-journal08:15:251
2089374994528,13cyclictest2094279-21sshd08:40:260
208937499451,39cyclictest2097130-21sshd09:35:540
208939099445,8cyclictest37-21ksoftirqd/310:55:013
208939099444,9cyclictest37-21ksoftirqd/310:35:023
2089390994431,10cyclictest37-21ksoftirqd/312:10:023
208939099442,7cyclictest37-21ksoftirqd/308:05:013
2089390994413,10cyclictest37-21ksoftirqd/307:20:013
2089382994430,10cyclictest306-21in:imuxsock10:30:022
2089377994424,13cyclictest272-21dbus-daemon08:10:001
208937499449,8cyclictest131rcu_preempt10:15:020
208937499441,39cyclictest2095837-21sshd09:11:270
208937499441,39cyclictest0-21swapper/011:25:010
2089382994330,10cyclictest30-21ksoftirqd/208:05:012
2089382994327,12cyclictest272-21dbus-daemon10:00:002
2089382994315,18cyclictest1-21systemd10:25:002
2089377994333,5cyclictest2094565-21munin-plugin-st08:50:011
2089377994331,9cyclictest23-21ksoftirqd/112:14:591
2089377994331,8cyclictest235-21systemd-journal08:25:011
2089377994317,16cyclictest131rcu_preempt09:54:591
2089374994328,12cyclictest2090588-21sshd07:30:110
2089374994327,12cyclictest235-21systemd-journal08:40:000
2089374994323,16cyclictest235-21systemd-journal10:45:010
2089374994323,16cyclictest235-21systemd-journal10:45:000
208937499431,7cyclictest2092190-21munin-run08:05:010
208937499431,3cyclictest131rcu_preempt09:05:000
208939099427,9cyclictest37-21ksoftirqd/312:05:013
208939099425,8cyclictest37-21ksoftirqd/308:15:003
208939099425,10cyclictest37-21ksoftirqd/310:15:023
2089382994225,13cyclictest1-21systemd08:40:012
2089382994224,13cyclictest235-21systemd-journal07:15:002
2089382994224,13cyclictest235-21systemd-journal07:15:002
2089377994229,9cyclictest2105468-21cron12:20:011
2089377994229,9cyclictest2105468-21cron12:20:001
2089377994222,14cyclictest1-21systemd08:05:001
2089374994235,4cyclictest16950irq/224-800000012:10:010
2089374994235,4cyclictest16950irq/224-800000011:10:010
208937499423,3cyclictest131rcu_preempt10:05:030
208937499422,4cyclictest0-21swapper/011:37:390
2089374994223,14cyclictest272-21dbus-daemon10:00:000
208937499421,3cyclictest131rcu_preempt07:30:020
2089382994126,9cyclictest272-21dbus-daemon11:15:012
208938299411,4cyclictest131rcu_preempt07:25:002
208937799414,6cyclictest131rcu_preempt11:25:011
2089377994126,12cyclictest23-21ksoftirqd/109:10:011
2089377994126,12cyclictest23-21ksoftirqd/109:10:011
2089377994123,14cyclictest272-21dbus-daemon07:15:011
2089377994123,14cyclictest272-21dbus-daemon07:15:011
208937499419,6cyclictest131rcu_preempt09:25:000
2089374994134,4cyclictest16950irq/224-800000012:30:010
2089374994131,8cyclictest12-21ksoftirqd/007:40:000
2089374994130,8cyclictest12-21ksoftirqd/009:10:010
2089374994130,8cyclictest12-21ksoftirqd/009:10:000
2089374994125,12cyclictest272-21dbus-daemon08:55:020
208937499412,34cyclictest2093113-21latency_hist08:20:010
2089374994123,14cyclictest235-21systemd-journal09:54:590
208938299404,3cyclictest131rcu_preempt12:35:012
2089382994028,7cyclictest306-21in:imuxsock09:05:002
2089382994027,9cyclictest1-21systemd09:35:002
208937799405,6cyclictest131rcu_preempt10:05:001
2089377994031,7cyclictest23-21ksoftirqd/108:00:031
2089377994029,7cyclictest2095336-21cron09:05:011
2089377994019,6cyclictest23-21ksoftirqd/109:50:021
2089377994019,6cyclictest23-21ksoftirqd/109:50:011
2089377994019,4cyclictest23-21ksoftirqd/110:40:011
208937499404,3cyclictest131rcu_preempt07:10:020
208937499402,8cyclictest1-21systemd10:35:010
2089374994027,10cyclictest12-21ksoftirqd/011:35:010
2089374994026,10cyclictest2095078-21run-parts09:00:000
2089374994024,12cyclictest2105471-21modprobe12:20:010
2089374994024,12cyclictest2105471-21modprobe12:20:000
208937499401,4cyclictest131rcu_preempt10:49:590
208937499401,3cyclictest131rcu_preempt12:35:020
208937499401,35cyclictest12-21ksoftirqd/008:45:030
2089390993927,7cyclictest2100799-21cron10:49:593
2089382993928,6cyclictest2090623-21munin-plugin-st07:35:002
2089382993924,12cyclictest30-21ksoftirqd/210:05:012
2089382993912,19cyclictest0-21swapper/209:00:022
2089377993928,8cyclictest235-21systemd-journal11:00:011
2089377993928,8cyclictest235-21systemd-journal11:00:001
2089377993916,15cyclictest131rcu_preempt09:20:001
208937799391,3cyclictest131rcu_preempt12:10:011
2089374993929,6cyclictest306-21in:imuxsock09:50:020
2089374993929,6cyclictest306-21in:imuxsock09:50:010
2089374993926,9cyclictest235-21systemd-journal10:20:260
208937499392,5cyclictest2102894-21munin-plugin-st11:30:010
208937499392,3cyclictest131rcu_preempt11:50:000
2089374993923,12cyclictest235-21systemd-journal11:45:020
2089374993921,14cyclictest1-21systemd12:05:000
2089390993819,14cyclictest235-21systemd-journal09:45:003
208938299384,5cyclictest131rcu_preempt09:27:262
208938299383,8cyclictest30-21ksoftirqd/210:20:032
2089382993828,8cyclictest30-21ksoftirqd/207:45:022
2089382993828,7cyclictest30-21ksoftirqd/211:55:012
208938299382,7cyclictest272-21dbus-daemon10:55:002
2089382993824,10cyclictest306-21in:imuxsock08:45:022
2089382993823,12cyclictest30-21ksoftirqd/212:40:002
2089382993823,11cyclictest30-21ksoftirqd/211:06:082
2089382993817,15cyclictest272-21dbus-daemon10:50:012
2089382993817,15cyclictest235-21systemd-journal11:04:592
208937799384,3cyclictest131rcu_preempt10:20:001
208937799383,3cyclictest131rcu_preempt11:15:011
208937799383,3cyclictest131rcu_preempt07:40:021
2089377993829,6cyclictest2101860-21systemctl11:10:001
2089377993827,7cyclictest235-21systemd-journal11:55:001
2089377993825,10cyclictest23-21ksoftirqd/110:10:561
2089377993822,11cyclictest272-21dbus-daemon09:25:021
2089377993822,10cyclictest0-21swapper/107:50:001
2089377993814,6cyclictest131rcu_preempt11:20:001
208937799381,3cyclictest131rcu_preempt07:10:021
2089377993813,4cyclictest23-21ksoftirqd/111:50:011
2089377993813,20cyclictest272-21dbus-daemon09:30:001
208937499383,5cyclictest0-21swapper/011:00:000
208937499383,5cyclictest0-21swapper/011:00:000
2089374993832,4cyclictest16950irq/224-800000007:20:000
2089374993831,4cyclictest16950irq/224-800000007:45:010
208937499382,6cyclictest131rcu_preempt09:35:020
208937499381,4cyclictest131rcu_preempt10:40:000
208937499381,2cyclictest0-21swapper/011:15:000
2089382993724,10cyclictest30-21ksoftirqd/207:20:002
2089382993723,11cyclictest30-21ksoftirqd/210:42:262
2089382993723,11cyclictest30-21ksoftirqd/210:42:262
2089382993719,13cyclictest272-21dbus-daemon10:15:002
2089382993715,10cyclictest30-21ksoftirqd/210:09:592
208937799375,8cyclictest23-21ksoftirqd/110:26:001
208937799375,8cyclictest23-21ksoftirqd/110:25:591
208937799374,4cyclictest131rcu_preempt09:15:011
2089377993728,5cyclictest306-21in:imuxsock12:40:001
2089377993726,8cyclictest2103136-21cron11:35:001
2089377993726,7cyclictest23-21ksoftirqd/110:50:001
2089377993726,6cyclictest2092877-21latency_hist08:15:021
2089377993724,10cyclictest2103545-21sed11:40:031
208937799372,3cyclictest131rcu_preempt08:55:021
208937799371,5cyclictest2091153-21modprobe07:45:001
208937499374,3cyclictest131rcu_preempt10:09:590
2089374993730,4cyclictest1-21systemd10:30:010
2089374993730,4cyclictest1-21systemd10:30:000
2089374993727,7cyclictest2101562-21sshd11:00:260
2089374993722,11cyclictest235-21systemd-journal07:50:010
2089374993722,10cyclictest12-21ksoftirqd/012:14:590
208938299366,3cyclictest131rcu_preempt11:35:022
2089382993628,5cyclictest2090371-21munin-plugin-st07:30:012
2089382993627,6cyclictest30-21ksoftirqd/212:10:292
2089382993625,8cyclictest2100268-21cron10:40:012
2089382993625,7cyclictest30-21ksoftirqd/212:00:262
2089382993624,9cyclictest2094596-21sed08:50:012
2089382993621,11cyclictest30-21ksoftirqd/212:24:272
2089382993617,14cyclictest272-21dbus-daemon12:00:012
2089382993616,15cyclictest272-21dbus-daemon07:50:002
2089377993628,4cyclictest2089808-21cron07:17:011
2089377993623,10cyclictest23-21ksoftirqd/110:25:021
2089377993614,4cyclictest23-21ksoftirqd/107:25:021
208937799361,3cyclictest131rcu_preempt09:40:001
2089374993625,8cyclictest235-21systemd-journal07:12:550
2089374993625,8cyclictest235-21systemd-journal07:12:550
2089374993624,7cyclictest306-21in:imuxsock07:55:000
2089374993623,10cyclictest12-21ksoftirqd/012:00:020
208937499361,3cyclictest131rcu_preempt11:53:470
208938299354,3cyclictest131rcu_preempt11:00:022
208938299354,3cyclictest131rcu_preempt11:00:012
2089382993523,8cyclictest30-21ksoftirqd/212:10:012
208938299352,13cyclictest2096489-21latency_hist09:25:022
2089382993521,10cyclictest30-21ksoftirqd/209:15:012
208938299352,10cyclictest363-21snmpd12:30:012
2089382993520,11cyclictest1-21systemd08:15:012
208937799352,5cyclictest131rcu_preempt08:45:011
2089377993524,7cyclictest2104697-21sshd12:03:271
2089377993523,7cyclictest306-21in:imuxsock11:30:011
2089377993523,5cyclictest23-21ksoftirqd/107:55:011
2089377993522,9cyclictest235-21systemd-journal12:25:011
2089377993516,15cyclictest23-21ksoftirqd/111:04:591
208937499356,3cyclictest131rcu_preempt08:25:000
2089374993527,5cyclictest2096656-21latency_hist09:30:010
2089374993522,10cyclictest12-21ksoftirqd/012:25:010
2089374993519,12cyclictest235-21systemd-journal07:20:110
208937499351,30cyclictest0-21swapper/009:40:200
2089382993424,7cyclictest2090862-21cron07:40:002
2089382993422,9cyclictest30-21ksoftirqd/208:20:012
2089382993421,9cyclictest30-21ksoftirqd/209:40:002
2089382993420,11cyclictest235-21systemd-journal09:50:012
2089382993420,11cyclictest235-21systemd-journal09:50:012
208938299341,9cyclictest2105184-21kworker/u8:212:20:032
208938299341,9cyclictest2105184-21kworker/u8:212:20:022
208938299341,7cyclictest0-21swapper/209:45:022
2089382993417,12cyclictest1-21systemd08:10:012
208938299341,3cyclictest131rcu_preempt08:25:022
208937799342,14cyclictest2100051-21latency_hist10:35:011
2089377993418,12cyclictest1-21systemd08:30:011
208938299333,5cyclictest2094797-21modprobe08:53:262
2089382993323,8cyclictest30-21ksoftirqd/211:50:022
208938299331,12cyclictest235-21systemd-journal08:30:002
208937799336,3cyclictest131rcu_preempt12:30:001
208937799336,3cyclictest131rcu_preempt09:35:021
208937799335,6cyclictest131rcu_preempt07:32:261
2089377993317,12cyclictest2103855-21wc11:45:021
208937799331,4cyclictest131rcu_preempt10:45:021
208937799331,4cyclictest131rcu_preempt10:45:011
2089377993310,6cyclictest131rcu_preempt07:30:021
208938299323,8cyclictest0-21swapper/208:35:012
2089382993218,10cyclictest235-21systemd-journal11:21:072
208937799326,22cyclictest1-21systemd10:10:011
2089377993222,6cyclictest2094038-21latency_hist08:40:001
208937499321,27cyclictest0-21swapper/010:15:560
208937799313,9cyclictest23-21ksoftirqd/108:30:261
2089377993126,3cyclictest231ksoftirqd/112:00:001
2089377993121,7cyclictest23-21ksoftirqd/109:45:011
2089382993019,8cyclictest2097909-21sshd09:50:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional