You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 14:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Nov 13, 2025 00:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2645003999910,3cyclictest131rcu_preempt21:50:020
264500399933,3cyclictest131rcu_preempt20:35:020
264500399933,3cyclictest131rcu_preempt20:35:010
2645003997737,14cyclictest235-21systemd-journal20:50:010
2645016996151,6cyclictest2647545-21sendmail20:00:013
264501699603,9cyclictest37-21ksoftirqd/322:15:013
264501699603,9cyclictest37-21ksoftirqd/322:15:003
2645007996015,7cyclictest23-21ksoftirqd/100:40:001
264501699591,29cyclictest0-21swapper/300:00:003
264500799597,9cyclictest23-21ksoftirqd/120:05:001
2645007995710,6cyclictest2659177-21latency_hist23:40:031
2645007995710,6cyclictest2659177-21latency_hist23:40:031
2645003995750,5cyclictest19050irq/34-mmc123:35:020
2645016995645,7cyclictest37-21ksoftirqd/321:30:023
2645016995645,7cyclictest37-21ksoftirqd/321:30:013
264501699558,12cyclictest37-21ksoftirqd/320:50:013
2645016995542,9cyclictest37-21ksoftirqd/300:05:003
2645016995541,8cyclictest37-21ksoftirqd/321:10:013
2645016995514,9cyclictest37-21ksoftirqd/322:30:023
264500799559,12cyclictest2652912-21latency_hist21:40:021
264500799555,10cyclictest23-21ksoftirqd/121:35:011
2645007995516,10cyclictest23-21ksoftirqd/121:10:011
2645007995513,11cyclictest23-21ksoftirqd/100:05:001
2645007995512,7cyclictest2647601-21cat20:00:011
2645007995511,6cyclictest2658431-21sed23:25:031
2645007995511,6cyclictest2658431-21sed23:25:021
264501699532,10cyclictest37-21ksoftirqd/323:45:003
264501699531,9cyclictest37-21ksoftirqd/321:45:023
264501699531,9cyclictest37-21ksoftirqd/321:45:013
2645011995312,9cyclictest131rcu_preempt19:30:012
2645003995318,31cyclictest2647804-21sshd20:04:410
264501699528,11cyclictest37-21ksoftirqd/321:05:023
264501699521,8cyclictest37-21ksoftirqd/319:10:023
2645016995213,5cyclictest131rcu_preempt00:15:023
264500799525,10cyclictest23-21ksoftirqd/120:40:021
2645007995213,6cyclictest131rcu_preempt00:00:021
264500799520,16cyclictest2653222-21cat21:45:031
2645003995233,13cyclictest235-21systemd-journal19:20:010
2645003995233,13cyclictest235-21systemd-journal19:20:000
264501699517,10cyclictest37-21ksoftirqd/320:45:033
264501699516,12cyclictest37-21ksoftirqd/320:40:013
264501699515,8cyclictest37-21ksoftirqd/319:25:013
264501699514,7cyclictest37-21ksoftirqd/322:00:023
264501699512,9cyclictest37-21ksoftirqd/320:05:013
264501699511,9cyclictest37-21ksoftirqd/300:15:023
264501699511,9cyclictest37-21ksoftirqd/300:15:013
264500799518,9cyclictest2650939-21/usr/sbin/munin21:05:021
2645007995133,12cyclictest1-21systemd19:35:011
2645007995113,9cyclictest23-21ksoftirqd/100:35:021
2645007995113,9cyclictest23-21ksoftirqd/100:35:011
264501699509,9cyclictest37-21ksoftirqd/322:25:033
264501699506,9cyclictest37-21ksoftirqd/300:10:023
264501699506,9cyclictest37-21ksoftirqd/300:10:013
264501699506,8cyclictest37-21ksoftirqd/322:40:023
264501699506,8cyclictest37-21ksoftirqd/322:40:013
264501699503,13cyclictest37-21ksoftirqd/322:59:593
264501699501,10cyclictest2651727-21munin-run21:20:003
2645011995031,13cyclictest235-21systemd-journal23:40:012
2645011995031,13cyclictest235-21systemd-journal23:40:002
264501699497,8cyclictest37-21ksoftirqd/321:40:023
264501699496,11cyclictest37-21ksoftirqd/323:10:033
264501699496,11cyclictest37-21ksoftirqd/323:10:023
264501699494,13cyclictest37-21ksoftirqd/319:39:593
264501699493,10cyclictest37-21ksoftirqd/322:50:023
2645011994932,12cyclictest235-21systemd-journal22:26:422
264501199492,5cyclictest131rcu_preempt21:40:012
2645011994913,8cyclictest131rcu_preempt00:15:022
2645011994913,8cyclictest131rcu_preempt00:15:022
264500799495,10cyclictest23-21ksoftirqd/123:15:011
2645007994938,8cyclictest23-21ksoftirqd/123:45:011
2645007994911,9cyclictest23-21ksoftirqd/120:54:591
264500799491,10cyclictest23-21ksoftirqd/100:30:001
264501699485,7cyclictest37-21ksoftirqd/320:20:033
264501699485,7cyclictest37-21ksoftirqd/320:20:023
2645016994830,13cyclictest272-21dbus-daemon19:50:013
264501699482,10cyclictest37-21ksoftirqd/322:35:013
264501699481,7cyclictest1-21systemd21:35:003
264500799483,12cyclictest2653115-21sed21:45:011
2645007994814,6cyclictest23-21ksoftirqd/123:30:011
2645007994810,7cyclictest0-21swapper/100:10:011
2645007994810,7cyclictest0-21swapper/100:10:001
264501699479,4cyclictest131rcu_preempt23:20:003
264501699478,8cyclictest37-21ksoftirqd/323:35:013
264501699476,8cyclictest37-21ksoftirqd/300:40:013
264501699474,8cyclictest37-21ksoftirqd/322:55:033
264501699474,8cyclictest37-21ksoftirqd/322:55:023
2645016994730,11cyclictest272-21dbus-daemon23:50:003
264501699471,6cyclictest37-21ksoftirqd/319:55:013
2645011994733,10cyclictest272-21dbus-daemon23:20:012
2645011994727,9cyclictest30-21ksoftirqd/222:35:002
2645011994713,6cyclictest131rcu_preempt00:00:002
264500799477,8cyclictest23-21ksoftirqd/122:50:011
2645007994730,13cyclictest1-21systemd22:45:001
2645007994712,5cyclictest23-21ksoftirqd/122:05:001
2645007994711,7cyclictest0-21swapper/119:39:421
2645003994736,7cyclictest19050irq/34-mmc121:10:010
2645003994728,13cyclictest272-21dbus-daemon23:15:010
2645003994728,13cyclictest272-21dbus-daemon22:30:010
264501699467,7cyclictest37-21ksoftirqd/320:30:013
264501699463,10cyclictest37-21ksoftirqd/319:45:023
264501699461,7cyclictest37-21ksoftirqd/321:15:013
264501699461,7cyclictest37-21ksoftirqd/320:25:003
2645016994611,6cyclictest131rcu_preempt22:05:023
2645011994636,6cyclictest30-21ksoftirqd/222:15:022
2645011994636,6cyclictest30-21ksoftirqd/222:15:012
2645011994629,13cyclictest235-21systemd-journal23:45:012
264500799464,9cyclictest23-21ksoftirqd/100:15:021
2645007994630,12cyclictest2651728-21systemctl21:20:001
264500399461,26cyclictest0-21swapper/019:28:410
264501699459,6cyclictest131rcu_preempt20:55:013
264501699459,5cyclictest131rcu_preempt21:47:203
264501699457,7cyclictest37-21ksoftirqd/322:45:023
264501699456,9cyclictest37-21ksoftirqd/319:15:003
264501699456,9cyclictest37-21ksoftirqd/300:30:013
264501699456,7cyclictest37-21ksoftirqd/323:40:013
264501699456,7cyclictest37-21ksoftirqd/323:40:003
264501699456,7cyclictest37-21ksoftirqd/300:25:023
264501699456,10cyclictest37-21ksoftirqd/321:25:013
264501699455,8cyclictest37-21ksoftirqd/323:05:023
264501699455,11cyclictest37-21ksoftirqd/319:20:013
264501699455,11cyclictest37-21ksoftirqd/319:20:003
264501699454,11cyclictest37-21ksoftirqd/319:35:013
264501699452,11cyclictest37-21ksoftirqd/321:55:003
264501699451,6cyclictest37-21ksoftirqd/320:15:023
264501699451,10cyclictest2646167-21date19:30:023
2645011994523,17cyclictest272-21dbus-daemon19:35:012
2645003994536,6cyclictest16950irq/224-800000000:10:010
2645003994536,6cyclictest16950irq/224-800000000:10:010
264500399453,36cyclictest0-21swapper/000:00:000
264501699448,6cyclictest131rcu_preempt23:55:033
264501699448,6cyclictest131rcu_preempt23:55:023
264501699446,7cyclictest37-21ksoftirqd/322:20:023
2645016994433,6cyclictest306-21in:imuxsock23:30:003
264501199448,10cyclictest2645000-21cyclictest22:53:412
264501199448,10cyclictest2645000-21cyclictest22:53:402
2645011994434,5cyclictest306-21in:imuxsock19:55:002
2645011994433,7cyclictest30-21ksoftirqd/222:10:022
2645011994431,9cyclictest30-21ksoftirqd/220:00:002
2645011994410,9cyclictest131rcu_preempt21:15:022
2645011994410,6cyclictest131rcu_preempt21:55:012
264500799443,5cyclictest23-21ksoftirqd/121:55:411
2645007994429,12cyclictest23-21ksoftirqd/119:17:011
2645007994429,12cyclictest23-21ksoftirqd/119:17:001
2645007994428,12cyclictest272-21dbus-daemon19:25:011
2645007994413,19cyclictest0-21swapper/119:15:011
264500799441,10cyclictest2651602-21sed21:15:031
2645007994410,8cyclictest0-21swapper/120:30:411
2645007994410,8cyclictest0-21swapper/120:30:411
2645003994428,11cyclictest235-21systemd-journal21:35:000
2645003994423,15cyclictest272-21dbus-daemon00:30:010
264501699438,12cyclictest37-21ksoftirqd/320:35:023
264501699438,12cyclictest37-21ksoftirqd/320:35:013
264501699434,7cyclictest37-21ksoftirqd/320:06:413
264501699432,9cyclictest37-21ksoftirqd/323:15:023
2645016994327,13cyclictest37-21ksoftirqd/300:31:423
2645016994327,13cyclictest37-21ksoftirqd/300:31:413
264501199434,5cyclictest0-21swapper/200:30:012
2645011994331,9cyclictest30-21ksoftirqd/220:50:002
2645011994331,8cyclictest30-21ksoftirqd/200:40:002
2645011994327,12cyclictest272-21dbus-daemon21:20:002
264500799434,9cyclictest23-21ksoftirqd/122:35:011
2645007994331,10cyclictest23-21ksoftirqd/120:30:011
264500399437,8cyclictest2662157-21cron00:39:590
264500399432,4cyclictest0-21swapper/021:20:000
264500399431,3cyclictest131rcu_preempt23:10:030
264500399431,3cyclictest131rcu_preempt23:10:020
264501699424,7cyclictest37-21ksoftirqd/322:10:013
264501199427,6cyclictest131rcu_preempt20:55:002
2645011994229,11cyclictest2658760-21sshd23:32:412
2645011994219,20cyclictest30-21ksoftirqd/222:25:012
2645011994211,4cyclictest40850lldpd00:09:092
2645011994211,4cyclictest40850lldpd00:09:082
264500799426,6cyclictest23-21ksoftirqd/119:50:001
264500799425,9cyclictest23-21ksoftirqd/123:50:011
2645007994226,11cyclictest272-21dbus-daemon21:30:011
2645007994226,11cyclictest272-21dbus-daemon21:30:001
2645007994225,12cyclictest23-21ksoftirqd/123:20:001
2645003994235,4cyclictest16950irq/224-800000021:45:020
2645003994235,4cyclictest16950irq/224-800000021:45:010
2645003994227,11cyclictest272-21dbus-daemon23:50:000
264500399422,6cyclictest2654894-21munin-run22:20:000
264500399422,4cyclictest131rcu_preempt19:50:020
264501699415,5cyclictest131rcu_preempt21:00:013
264501199415,6cyclictest131rcu_preempt20:05:012
2645011994133,5cyclictest2648588-21sendmail20:20:002
2645011994133,5cyclictest2648588-21sendmail20:20:002
2645011994131,6cyclictest30-21ksoftirqd/200:13:402
2645011994131,6cyclictest271-21cron19:39:592
2645011994127,10cyclictest30-21ksoftirqd/221:00:022
264501199412,12cyclictest30-21ksoftirqd/222:48:422
264501199411,7cyclictest2662056-21wc00:35:032
264501199411,7cyclictest2662056-21wc00:35:022
2645011994112,5cyclictest131rcu_preempt19:45:012
264500799418,8cyclictest0-21swapper/121:22:411
264500799415,9cyclictest23-21ksoftirqd/121:48:411
2645007994130,9cyclictest23-21ksoftirqd/120:45:011
264500799411,7cyclictest2656955-21latency_hist23:00:021
2645003994129,9cyclictest12-21ksoftirqd/023:25:020
2645003994129,9cyclictest12-21ksoftirqd/023:25:020
2645003994123,14cyclictest363-21snmpd19:56:400
264500399411,35cyclictest2656304-21latency_hist22:45:020
264501699405,7cyclictest37-21ksoftirqd/323:25:033
264501699405,7cyclictest37-21ksoftirqd/323:25:033
264501199408,6cyclictest131rcu_preempt21:25:022
264501199404,7cyclictest30-21ksoftirqd/222:40:012
264501199404,7cyclictest30-21ksoftirqd/222:40:012
2645011994029,8cyclictest30-21ksoftirqd/223:25:032
2645011994029,8cyclictest30-21ksoftirqd/223:25:022
2645011994029,7cyclictest2656203-21systemd-run22:45:002
2645011994028,8cyclictest272-21dbus-daemon19:15:012
2645011994024,12cyclictest2657473-21cron23:10:012
2645011994024,12cyclictest2657473-21cron23:10:002
264500799406,5cyclictest2654710-21latency_hist22:15:031
264500799406,5cyclictest2654710-21latency_hist22:15:021
264500799404,9cyclictest23-21ksoftirqd/122:10:021
2645007994028,9cyclictest23-21ksoftirqd/120:50:031
2645007994028,8cyclictest272-21dbus-daemon19:30:011
264500799402,7cyclictest23-21ksoftirqd/122:55:031
264500799402,7cyclictest23-21ksoftirqd/122:55:021
264500799401,7cyclictest0-21swapper/123:01:301
264500399405,10cyclictest0-21swapper/023:40:030
264500399405,10cyclictest0-21swapper/023:40:020
264500399404,3cyclictest131rcu_preempt19:55:010
2645003994030,5cyclictest2650685-21sendmail-msp21:00:000
2645003994024,11cyclictest235-21systemd-journal19:35:010
264500399402,34cyclictest2660378-21grep00:05:020
264500399402,33cyclictest2661137-21munin-run00:20:010
264500399402,33cyclictest2661137-21munin-run00:20:000
264500399401,14cyclictest363-21snmpd23:40:520
264501199397,5cyclictest131rcu_preempt21:35:012
264501199395,7cyclictest30-21ksoftirqd/220:15:002
264501199392,6cyclictest131rcu_preempt20:45:012
2645011993915,19cyclictest131rcu_preempt19:25:012
2645011993915,15cyclictest131rcu_preempt19:10:012
2645007993924,11cyclictest235-21systemd-journal21:00:001
264500399394,4cyclictest131rcu_preempt00:25:010
2645003993932,4cyclictest16950irq/224-800000019:11:480
2645003993931,5cyclictest19050irq/34-mmc121:59:360
264501199384,9cyclictest30-21ksoftirqd/219:15:412
264501199384,9cyclictest30-21ksoftirqd/219:15:412
264501199382,5cyclictest131rcu_preempt20:39:592
2645011993824,10cyclictest235-21systemd-journal22:56:422
264501199381,6cyclictest0-21swapper/223:48:452
264500799389,6cyclictest0-21swapper/100:17:411
264500799389,6cyclictest0-21swapper/100:17:401
264500799386,7cyclictest131rcu_preempt20:20:021
264500799386,7cyclictest131rcu_preempt20:20:011
264500799383,7cyclictest23-21ksoftirqd/119:55:001
2645007993827,8cyclictest2658766-21sshd23:34:411
2645007993826,10cyclictest23-21ksoftirqd/121:50:411
2645007993822,12cyclictest272-21dbus-daemon22:40:011
2645007993822,12cyclictest272-21dbus-daemon22:40:001
2645003993829,6cyclictest12-21ksoftirqd/020:15:030
2645003993828,6cyclictest2646761-21sshd19:42:050
2645003993827,7cyclictest12-21ksoftirqd/023:25:020
2645003993824,10cyclictest2652235-21sshd21:29:420
2645003993824,10cyclictest2652235-21sshd21:29:410
2645003993821,12cyclictest408-21lldpd20:24:590
2645003993819,15cyclictest0-21swapper/000:34:180
2645003993819,15cyclictest0-21swapper/000:34:170
264500399381,3cyclictest131rcu_preempt22:25:010
264500399381,3cyclictest131rcu_preempt21:25:000
264501199378,5cyclictest131rcu_preempt23:05:012
264501199376,5cyclictest131rcu_preempt19:50:022
2645011993725,8cyclictest2653250-21sshd21:45:412
264501199371,7cyclictest131rcu_preempt23:14:592
264501199371,7cyclictest0-21swapper/223:28:412
264500799375,6cyclictest23-21ksoftirqd/122:17:081
264500799373,7cyclictest23-21ksoftirqd/123:50:031
264500799373,7cyclictest23-21ksoftirqd/123:50:021
2645007993724,10cyclictest2648398-21sed20:15:011
264500799371,11cyclictest2648103-21latency_hist20:10:011
264500399378,3cyclictest131rcu_preempt00:15:010
2645003993731,3cyclictest16950irq/224-800000021:03:410
2645003993728,5cyclictest2650029-21latency_hist20:45:030
264500399372,4cyclictest131rcu_preempt20:20:010
264500399372,4cyclictest131rcu_preempt20:20:000
2645003993721,12cyclictest2652720-21sshd21:35:410
264500399371,9cyclictest2659900-21latency_hist23:55:020
264500399371,9cyclictest2659900-21latency_hist23:55:010
264500399371,5cyclictest2657248-21munin-run23:05:010
2645003993714,3cyclictest131rcu_preempt22:02:060
264500399371,31cyclictest0-21swapper/022:15:010
264500399371,31cyclictest0-21swapper/022:15:000
2645003993711,5cyclictest131rcu_preempt23:20:010
2645011993619,13cyclictest272-21dbus-daemon21:30:012
2645011993619,13cyclictest272-21dbus-daemon21:30:012
264501199361,5cyclictest131rcu_preempt23:55:012
264501199361,5cyclictest131rcu_preempt23:55:002
264501199361,5cyclictest131rcu_preempt22:00:012
2645007993624,8cyclictest2655135-21sshd22:22:211
264500399364,4cyclictest131rcu_preempt22:35:010
264500399362,6cyclictest2645525-21kworker/0:120:10:010
264500399361,3cyclictest131rcu_preempt22:37:120
264500399361,3cyclictest131rcu_preempt22:37:110
264501199357,5cyclictest131rcu_preempt22:05:002
264501199356,5cyclictest131rcu_preempt21:45:012
264501199356,5cyclictest131rcu_preempt21:45:002
264501199355,5cyclictest131rcu_preempt20:35:032
264501199355,5cyclictest131rcu_preempt20:35:022
264501199353,6cyclictest131rcu_preempt20:30:002
2645011993524,8cyclictest30-21ksoftirqd/220:25:032
2645011993515,13cyclictest131rcu_preempt00:25:012
264500799354,6cyclictest23-21ksoftirqd/120:25:001
2645003993523,8cyclictest12-21ksoftirqd/021:15:020
2645003993522,8cyclictest12-21ksoftirqd/020:55:030
264500399351,31cyclictest2656422-21sshd22:47:240
264500399351,30cyclictest0-21swapper/022:51:240
264500399351,30cyclictest0-21swapper/022:51:230
264501199343,27cyclictest2650940-21grep21:05:002
2645011993427,4cyclictest363-21snmpd20:06:412
2645011993426,5cyclictest2655078-21latency_hist22:20:012
2645011993421,10cyclictest235-21systemd-journal00:00:422
264500799349,6cyclictest0-21swapper/100:21:051
2645007993423,8cyclictest23-21ksoftirqd/123:07:261
2645007993423,8cyclictest23-21ksoftirqd/123:07:251
2645003993427,4cyclictest16950irq/224-800000019:22:130
2645003993425,6cyclictest408-21lldpd21:53:070
2645003993422,9cyclictest235-21systemd-journal20:26:410
2645003993422,10cyclictest12-21ksoftirqd/020:39:590
264501199332,4cyclictest131rcu_preempt21:10:012
2645007993322,8cyclictest23-21ksoftirqd/122:27:091
2645007993315,12cyclictest272-21dbus-daemon19:45:001
264500399331,28cyclictest0-21swapper/019:35:050
264500399325,4cyclictest131rcu_preempt19:10:010
2645003993222,7cyclictest1-21systemd22:05:400
264500399321,28cyclictest0-21swapper/022:56:200
264500799256,14cyclictest2645009-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional