You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 04:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Jan 05, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2893898997524,46cyclictest1-21systemd20:15:022
2893898996920,15cyclictest238-21systemd-journal22:20:002
2893900996811,11cyclictest37-21ksoftirqd/322:10:013
2893890996861,3cyclictest0-21swapper/019:25:010
2893900996718,11cyclictest37-21ksoftirqd/323:20:023
2893900996712,13cyclictest2899299-21latency_hist20:55:023
289389899652,16cyclictest2900649-21unin-run21:20:022
289389899652,16cyclictest2900649-21unin-run21:20:012
289389899635,33cyclictest291rcuc/221:00:022
2893898996324,36cyclictest30-21ksoftirqd/200:20:022
2893898996222,12cyclictest30-21ksoftirqd/223:44:392
2893900996110,12cyclictest2907855-21latency_hist23:40:013
2893898996114,9cyclictest2899070-21latency_hist20:50:022
2893900996013,10cyclictest37-21ksoftirqd/320:05:023
2893900996010,9cyclictest37-21ksoftirqd/321:35:003
289389899604,16cyclictest2897083-21wc20:10:022
289389899603,18cyclictest238-21systemd-journal19:15:002
289389899592,16cyclictest2894336-21modprobe19:19:592
2893898995921,4cyclictest2898983-21latency_hist20:45:022
2893898995818,7cyclictest2897502-21cut20:19:592
289390099576,30cyclictest2910426-21kworker/u8:300:35:023
2893900995710,16cyclictest2909922-21cron00:20:003
289389899572,17cyclictest2907618-21unin-run23:35:012
2893900995533,16cyclictest275-21dbus-daemon19:40:003
2893900995511,9cyclictest37-21ksoftirqd/323:45:003
289389899553,14cyclictest2906050-21modprobe23:05:012
289389899553,14cyclictest2906050-21modprobe23:05:002
2893898995515,5cyclictest2904833-21latency_hist22:40:022
2893898995513,5cyclictest0-21swapper/222:45:012
2893890995532,19cyclictest16950irq/224-800000022:39:590
2893898995420,12cyclictest275-21dbus-daemon19:35:022
2893898995420,12cyclictest275-21dbus-daemon19:35:012
2893898995413,8cyclictest2911190-21cat00:40:022
2893900995311,9cyclictest37-21ksoftirqd/319:45:023
2893898995335,12cyclictest1-21systemd21:40:012
289389899532,16cyclictest238-21systemd-journal00:05:012
2893898995320,14cyclictest30-21ksoftirqd/220:54:592
289389899531,17cyclictest2908905-21munin-run00:00:002
2893890995331,17cyclictest2897494-21sh20:20:000
2893900995235,14cyclictest238-21systemd-journal20:15:033
2893900995211,8cyclictest37-21ksoftirqd/320:15:013
2893898995217,4cyclictest2903037-21latency_hist22:05:012
2893898995134,12cyclictest275-21dbus-daemon20:40:002
2893898995116,11cyclictest2902658-21cron22:00:002
2893894995134,12cyclictest1-21systemd23:55:011
2893894995131,15cyclictest275-21dbus-daemon23:30:011
289389099518,26cyclictest151rcuc/000:20:020
2893890995136,11cyclictest238-21systemd-journal00:35:010
2893890995133,13cyclictest275-21dbus-daemon20:55:010
2893890995122,17cyclictest151rcuc/021:45:020
289389899502,16cyclictest2900334-21cron21:15:002
2893898995015,6cyclictest2902268-21cat21:50:022
2893894995029,17cyclictest1-21systemd22:40:011
2893890995033,12cyclictest238-21systemd-journal00:10:000
2893890995028,16cyclictest0-21swapper/000:00:010
289390099495,11cyclictest2901134-21latency_hist21:30:023
2893900994936,7cyclictest2908384-21munin-plugin-st23:50:003
2893900994911,8cyclictest0-21swapper/300:25:013
289389899497,10cyclictest311-21rs:main2
289389899494,13cyclictest2907843-21modprobe23:40:012
2893890994936,10cyclictest12-21ksoftirqd/020:40:000
2893890994918,28cyclictest12-21ksoftirqd/023:35:230
2893900994829,13cyclictest275-21dbus-daemon22:45:013
2893898994838,7cyclictest30-21ksoftirqd/220:30:022
2893898994838,7cyclictest30-21ksoftirqd/220:30:022
289389899481,18cyclictest2906582-21wc23:15:012
289389099484,7cyclictest2903726-21sendmail22:20:000
2893890994832,11cyclictest1-21systemd19:45:020
2893890994830,13cyclictest238-21systemd-journal19:15:000
289390099471,16cyclictest0-21swapper/323:15:013
2893898994734,10cyclictest238-21systemd-journal21:00:032
2893898994734,10cyclictest238-21systemd-journal21:00:022
2893898994716,7cyclictest30-21ksoftirqd/219:10:022
2893898994715,8cyclictest30-21ksoftirqd/220:25:012
2893898994713,10cyclictest30-21ksoftirqd/221:44:592
2893894994735,8cyclictest238-21systemd-journal21:09:591
289389099473,9cyclictest2896479-21sendmail-msp19:59:590
2893890994736,7cyclictest19350irq/34-mmc123:10:000
289390099465,11cyclictest37-21ksoftirqd/319:25:003
2893898994635,6cyclictest309-21in:imuxsock21:25:002
2893898994628,14cyclictest1-21systemd19:30:002
2893898994619,7cyclictest30-21ksoftirqd/200:25:012
2893894994629,13cyclictest238-21systemd-journal23:49:591
289389099465,37cyclictest2906824-21sh23:20:010
2893890994634,8cyclictest12-21ksoftirqd/021:20:020
2893890994634,8cyclictest12-21ksoftirqd/021:20:010
2893900994533,7cyclictest2899543-21cron21:00:003
2893900994527,14cyclictest2905506-21sshd22:53:413
2893894994532,7cyclictest2898018-21munin-plugin-st20:30:011
2893894994532,7cyclictest2898018-21munin-plugin-st20:30:001
2893890994528,12cyclictest238-21systemd-journal21:45:020
2893890994527,15cyclictest2902417-21latency_hist21:55:010
2893890994527,15cyclictest2902417-21latency_hist21:55:010
2893890994526,13cyclictest238-21systemd-journal22:05:010
2893890994521,19cyclictest1-21systemd22:10:010
2893900994424,15cyclictest2894305-21cron19:17:013
2893898994433,8cyclictest2905498-21sshd22:52:242
2893898994431,9cyclictest2895673-21sshd19:41:592
2893898994420,4cyclictest2901388-21latency_hist21:35:022
2893894994430,8cyclictest275-21dbus-daemon21:45:001
2893894994424,14cyclictest275-21dbus-daemon22:35:001
2893890994437,4cyclictest16950irq/224-800000020:50:000
289389099442,8cyclictest1-21systemd21:35:000
2893890994426,14cyclictest275-21dbus-daemon19:50:000
2893900994329,9cyclictest2903721-21run-parts22:20:003
2893900994328,10cyclictest2900087-21sh21:09:593
289389899432,3cyclictest131rcu_preempt20:00:002
289389899432,19cyclictest238-21systemd-journal20:05:002
2893894994325,12cyclictest275-21dbus-daemon22:00:011
2893894994324,14cyclictest2907871-21tr23:40:011
2893894994324,13cyclictest238-21systemd-journal19:50:011
2893890994336,4cyclictest16950irq/224-800000019:35:020
2893890994336,4cyclictest16950irq/224-800000019:35:010
2893890994333,6cyclictest2902685-21sendmail21:59:590
2893890994331,9cyclictest12-21ksoftirqd/000:40:000
2893890994330,9cyclictest12-21ksoftirqd/020:30:020
2893890994330,9cyclictest12-21ksoftirqd/020:30:010
2893890994326,14cyclictest2897771-21munin-run20:25:000
289389099432,10cyclictest2905789-21munin-run23:00:000
2893900994228,7cyclictest309-21in:imuxsock21:40:013
2893900994225,12cyclictest238-21systemd-journal22:35:013
2893900994224,14cyclictest2909639-21sshd00:10:413
2893900994221,15cyclictest37-21ksoftirqd/320:39:593
2893894994222,13cyclictest238-21systemd-journal20:34:591
2893890994228,11cyclictest12-21ksoftirqd/020:08:400
2893890994224,13cyclictest1-21systemd19:20:010
289389099422,35cyclictest0-21swapper/021:00:010
289389099421,36cyclictest2901734-21cat21:40:020
289390099417,7cyclictest0-21swapper/323:56:183
289390099417,6cyclictest2902905-21sed22:00:013
289390099414,7cyclictest37-21ksoftirqd/320:32:033
2893900994131,6cyclictest2903978-21cron22:25:013
2893900994127,10cyclictest2909390-21sshd00:05:393
2893900994125,12cyclictest238-21systemd-journal21:15:003
2893900994122,13cyclictest275-21dbus-daemon19:30:003
2893894994132,6cyclictest2895698-21munin-plugin-st19:45:001
2893894994128,10cyclictest1-21systemd22:25:001
289389099414,13cyclictest12-21ksoftirqd/022:25:010
2893890994132,7cyclictest12-21ksoftirqd/021:10:010
2893890994130,8cyclictest12-21ksoftirqd/022:45:020
2893890994128,8cyclictest12-21ksoftirqd/022:30:020
2893890994128,8cyclictest12-21ksoftirqd/022:30:010
2893890994128,10cyclictest2906544-21sshd23:14:410
289389099412,35cyclictest2897204-21sshd20:11:590
289389099411,7cyclictest151rcuc/019:55:010
289389099411,7cyclictest151rcuc/019:55:000
2893900994030,5cyclictest2903440-21cron22:15:003
2893900994027,9cyclictest238-21systemd-journal22:02:153
2893900994025,11cyclictest2896425-21sshd19:56:593
2893900994025,11cyclictest275-21dbus-daemon23:35:013
2893900994022,14cyclictest238-21systemd-journal00:35:013
289389899403,4cyclictest131rcu_preempt23:50:002
2893894994028,9cyclictest23-21ksoftirqd/122:50:011
2893894994028,9cyclictest23-21ksoftirqd/122:50:001
2893894994026,8cyclictest309-21in:imuxsock21:50:011
2893894994022,13cyclictest2909922-21cron00:19:591
2893894994020,14cyclictest2908901-21cat00:00:011
2893894994019,14cyclictest275-21dbus-daemon19:55:011
2893894994019,14cyclictest275-21dbus-daemon19:55:001
2893890994031,5cyclictest16950irq/224-800000023:50:010
289389099402,6cyclictest131rcu_preempt00:05:000
2893890994022,13cyclictest275-21dbus-daemon00:25:000
289389099401,5cyclictest131rcu_preempt19:39:590
2893900993930,5cyclictest2896970-21cron20:10:013
2893900993927,8cyclictest37-21ksoftirqd/322:50:023
2893900993926,10cyclictest37-21ksoftirqd/319:35:013
2893900993926,10cyclictest37-21ksoftirqd/319:35:003
2893900993925,11cyclictest2901898-21sshd21:43:403
289390099391,7cyclictest37-21ksoftirqd/323:06:403
289389899398,7cyclictest131rcu_preempt23:55:002
2893898993912,5cyclictest131rcu_preempt22:12:332
2893894993929,7cyclictest23-21ksoftirqd/123:35:011
2893894993929,5cyclictest2905765-21cron23:00:001
2893894993927,7cyclictest2894654-21cron19:24:591
2893894993927,10cyclictest23-21ksoftirqd/123:05:021
2893894993923,11cyclictest238-21systemd-journal20:55:011
2893894993920,14cyclictest275-21dbus-daemon21:00:001
2893890993931,5cyclictest2898270-21munin-plugin-st20:34:590
2893890993923,12cyclictest1-21systemd23:25:000
289390099385,6cyclictest37-21ksoftirqd/321:02:073
289390099385,6cyclictest37-21ksoftirqd/321:02:063
2893900993828,6cyclictest309-21in:imuxsock23:55:003
2893900993826,9cyclictest2898764-21sshd20:40:403
2893900993825,10cyclictest2904244-21munin-run22:30:013
2893900993825,10cyclictest2904244-21munin-run22:30:003
2893900993825,10cyclictest275-21dbus-daemon00:05:013
2893900993824,11cyclictest2905743-21sshd22:55:403
2893900993824,10cyclictest2910428-21sshd00:25:403
2893900993824,10cyclictest2910428-21sshd00:25:393
2893900993824,10cyclictest2900871-21sshd21:24:393
2893900993823,10cyclictest1-21systemd19:55:023
2893900993823,10cyclictest1-21systemd19:55:023
289389899387,8cyclictest30-21ksoftirqd/221:25:412
289389899387,7cyclictest30-21ksoftirqd/223:30:002
2893898993828,7cyclictest2896172-21sshd19:52:002
2893898993828,7cyclictest2896172-21sshd19:51:592
2893898993827,6cyclictest2903199-21munin-plugin-st22:10:012
2893898993824,10cyclictest1-21systemd23:10:012
2893898993823,10cyclictest30-21ksoftirqd/219:40:012
2893894993827,8cyclictest2905510-21sshd22:54:401
2893894993824,10cyclictest2905002-21sshd22:42:411
289389499382,28cyclictest221rcuc/100:15:021
2893894993821,12cyclictest275-21dbus-daemon23:15:011
2893890993829,6cyclictest12-21ksoftirqd/021:27:410
289389099382,5cyclictest2893981-21latency_hist19:10:010
289389099381,2cyclictest0-21swapper/021:15:000
289390099374,9cyclictest37-21ksoftirqd/322:48:393
2893900993724,10cyclictest2902400-21sshd21:53:403
2893900993724,10cyclictest2902400-21sshd21:53:403
2893900993724,10cyclictest1-21systemd21:50:003
2893900993720,13cyclictest37-21ksoftirqd/321:20:013
2893900993720,13cyclictest37-21ksoftirqd/321:20:003
289389899372,7cyclictest0-21swapper/219:50:032
2893898993724,9cyclictest238-21systemd-journal00:05:402
2893898993724,10cyclictest30-21ksoftirqd/222:30:002
2893898993724,10cyclictest30-21ksoftirqd/222:30:002
2893894993727,7cyclictest2910444-21sshd00:29:411
2893894993727,7cyclictest2910444-21sshd00:29:401
2893894993724,9cyclictest23-21ksoftirqd/123:25:031
2893894993724,10cyclictest275-21dbus-daemon19:15:011
289389499372,3cyclictest131rcu_preempt22:20:021
289389499372,3cyclictest131rcu_preempt20:05:011
2893894993719,13cyclictest238-21systemd-journal22:10:031
2893894993716,15cyclictest1-21systemd00:35:001
289390099363,9cyclictest37-21ksoftirqd/323:25:413
2893900993623,9cyclictest2897734-21sshd20:20:403
2893900993611,21cyclictest2898221-21latency_hist20:30:033
2893898993624,8cyclictest2907096-21cron23:25:012
2893894993628,5cyclictest2904419-21latency_hist22:30:031
2893894993628,5cyclictest2904419-21latency_hist22:30:021
2893894993625,9cyclictest23-21ksoftirqd/121:30:011
2893894993625,9cyclictest23-21ksoftirqd/121:20:011
2893894993625,9cyclictest23-21ksoftirqd/121:20:011
2893894993619,12cyclictest23-21ksoftirqd/120:45:021
289389499361,3cyclictest131rcu_preempt00:40:001
2893890993630,3cyclictest16950irq/224-800000022:35:000
2893890993629,5cyclictest12-21ksoftirqd/021:24:390
2893890993626,6cyclictest12-21ksoftirqd/022:15:000
2893890993624,9cyclictest12-21ksoftirqd/020:05:020
2893890993620,12cyclictest275-21dbus-daemon00:15:020
289389099361,5cyclictest2905275-21munin-run22:50:010
289389099361,5cyclictest2905275-21munin-run22:50:000
2893900993523,9cyclictest2899025-21sshd20:47:403
2893900993520,10cyclictest2904731-21cron22:39:593
2893900993519,13cyclictest364-21snmpd19:49:423
289389899354,7cyclictest30-21ksoftirqd/200:33:402
2893898993524,8cyclictest30-21ksoftirqd/220:35:002
2893898993522,10cyclictest30-21ksoftirqd/222:50:012
2893898993522,10cyclictest30-21ksoftirqd/222:50:002
289389899351,4cyclictest131rcu_preempt23:00:022
2893894993525,7cyclictest23-21ksoftirqd/121:55:021
2893894993525,7cyclictest23-21ksoftirqd/121:55:011
2893894993525,6cyclictest2901733-21sort21:40:031
2893894993524,8cyclictest2897206-21sshd20:12:001
2893894993518,12cyclictest1-21systemd23:45:011
289389499351,31cyclictest23-21ksoftirqd/119:35:011
289389499351,31cyclictest23-21ksoftirqd/119:35:011
2893890993524,8cyclictest12-21ksoftirqd/023:45:010
2893890993511,6cyclictest131rcu_preempt22:55:000
2893900993422,9cyclictest2906233-21wc23:05:033
2893900993422,9cyclictest2906233-21wc23:05:033
2893900993419,11cyclictest2898020-21/usr/sbin/munin20:30:013
2893900993415,14cyclictest2907077-21sshd23:22:393
2893898993426,6cyclictest2906825-21sendmail23:20:012
2893898993415,14cyclictest1-21systemd21:10:012
2893894993426,5cyclictest2903197-21latency_hist22:10:011
2893894993423,7cyclictest2906476-21latency_hist23:10:021
289389499342,28cyclictest1-21systemd00:05:001
2893894993421,8cyclictest309-21in:imuxsock19:16:401
2893894993421,10cyclictest23-21ksoftirqd/120:25:001
2893890993426,5cyclictest2899833-21munin-plugin-st21:05:010
2893890993426,5cyclictest2899833-21munin-plugin-st21:05:000
2893890993425,5cyclictest16950irq/224-800000023:55:000
2893890993421,10cyclictest12-21ksoftirqd/019:25:390
2893890993418,13cyclictest1-21systemd20:45:010
289390099338,22cyclictest2893949-21latency_hist19:10:013
2893900993320,9cyclictest2894053-21sshd19:12:393
2893898993321,9cyclictest30-21ksoftirqd/219:21:152
2893894993323,7cyclictest238-21systemd-journal21:32:391
2893894993323,7cyclictest23-21ksoftirqd/120:16:591
2893894993323,4cyclictest23-21ksoftirqd/123:20:001
2893894993321,9cyclictest23-21ksoftirqd/119:30:011
2893894993321,9cyclictest23-21ksoftirqd/100:25:031
2893894993319,10cyclictest309-21in:imuxsock21:20:401
289389099331,8cyclictest12-21ksoftirqd/000:28:400
289389099331,8cyclictest12-21ksoftirqd/000:28:400
289389099331,29cyclictest0-21swapper/023:30:010
289389899327,5cyclictest131rcu_preempt00:15:012
2893898993224,6cyclictest30-21ksoftirqd/222:25:032
289389499327,5cyclictest131rcu_preempt20:38:301
2893894993223,5cyclictest2909600-21latency_hist00:10:031
2893894993220,9cyclictest23-21ksoftirqd/123:03:411
2893894993220,10cyclictest23-21ksoftirqd/121:05:021
2893894993220,10cyclictest23-21ksoftirqd/121:05:021
289389499321,4cyclictest41050lldpd20:07:121
289389099323,6cyclictest131rcu_preempt23:05:020
289389099323,6cyclictest131rcu_preempt23:05:020
2893890993223,6cyclictest2907616-21systemctl23:35:010
289389899314,6cyclictest30-21ksoftirqd/200:26:242
289389899314,6cyclictest30-21ksoftirqd/200:26:232
2893894993122,7cyclictest23-21ksoftirqd/119:40:021
2893894993117,11cyclictest2896435-21sshd19:59:411
2893894993115,7cyclictest0-21swapper/119:10:011
289389499305,3cyclictest131rcu_preempt21:15:021
2893898992921,6cyclictest30-21ksoftirqd/221:55:022
2893898992921,6cyclictest30-21ksoftirqd/221:55:012
2893894992822,4cyclictest2902674-21latency_hist22:00:021
2893894992717,7cyclictest2899030-21sshd20:48:401
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional