OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails

Click here to display the system's profile data.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rackeslot8s.osadl.org (updated Tue Nov 04, 2025 11:11:15)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2778646	99	750	747,2	cyclictest	2784961	-21	kworker/1:0+events	08:30:23	1
2778646	99	746	735,5	cyclictest	2784961	-21	kworker/1:0+events	07:55:22	1
2778652	99	745	737,7	cyclictest	2950100	-21	kworker/3:2+events	10:20:22	3
2778652	99	744	735,7	cyclictest	3043843	-21	kworker/3:0+events	10:55:53	3
2778646	99	744	738,5	cyclictest	2784961	-21	kworker/1:0+events	09:50:20	1
2778646	99	743	738,4	cyclictest	2784961	-21	kworker/1:0+events	09:59:10	1
2778646	99	742	737,4	cyclictest	2784961	-21	kworker/1:0+events	11:06:07	1
2778652	99	738	731,6	cyclictest	2950100	-21	kworker/3:2+events	10:25:32	3
2778646	99	737	732,4	cyclictest	2784961	-21	kworker/1:0+events	10:01:58	1
2778652	99	735	730,4	cyclictest	2950100	-21	kworker/3:2+events	09:46:37	3
2778652	99	735	729,5	cyclictest	3044022	-21	turbostat	10:44:55	3
2778652	99	734	729,4	cyclictest	2792778	-21	kworker/3:2+events	07:50:45	3
2778652	99	734	729,4	cyclictest	2792778	-21	kworker/3:2+events	07:31:46	3
2778652	99	731	725,5	cyclictest	2950100	-21	kworker/3:2+events	10:30:21	3
2778652	99	728	723,4	cyclictest	2856254	-21	turbostat	08:14:54	3
2778652	99	724	719,4	cyclictest	2950100	-21	kworker/3:2+events	10:12:29	3
2778652	99	724	719,4	cyclictest	2950100	-21	kworker/3:2+events	09:39:35	3
2778652	99	724	716,7	cyclictest	2792778	-21	kworker/3:2+events	07:25:21	3
2778652	99	723	719,3	cyclictest	2950100	-21	kworker/3:2+events	09:28:47	3
2778652	99	721	718,2	cyclictest	2792778	-21	kworker/3:2+events	08:00:44	3
2778652	99	721	717,3	cyclictest	2862437	-21	kworker/3:0+events	08:26:06	3
2778652	99	721	716,4	cyclictest	2862437	-21	kworker/3:0+events	08:37:55	3
2778652	99	720	716,3	cyclictest	2792778	-21	kworker/3:2+events	07:22:31	3
2778652	99	720	714,5	cyclictest	2792778	-21	kworker/3:2+events	07:43:44	3
2778652	99	720	712,6	cyclictest	2844037	-21	kworker/3:1+events	08:05:21	3
2778652	99	720	712,5	cyclictest	2906203	-21	kworker/3:1+events	08:58:08	3
2778652	99	719	716,2	cyclictest	3043843	-21	kworker/3:0+events	11:01:13	3
2778652	99	719	713,5	cyclictest	2950100	-21	kworker/3:2+events	09:30:21	3
2778652	99	719	713,3	cyclictest	2950100	-21	kworker/3:2+events	10:17:08	3
2778652	99	718	715,2	cyclictest	2862437	-21	kworker/3:0+events	08:15:23	3
2778652	99	718	711,6	cyclictest	2862437	-21	kworker/3:0+events	08:45:21	3
2778652	99	717	714,2	cyclictest	3043843	-21	kworker/3:0+events	10:45:23	3
2778652	99	717	714,2	cyclictest	2906203	-21	kworker/3:1+events	09:00:22	3
2778652	99	717	713,3	cyclictest	2950100	-21	kworker/3:2+events	10:05:42	3
2778652	99	717	712,4	cyclictest	2862437	-21	kworker/3:0+events	08:42:44	3
2778652	99	716	712,3	cyclictest	737060	-21	kworker/3:1+events	07:12:31	3
2778652	99	716	712,3	cyclictest	3043843	-21	kworker/3:0+events	10:54:52	3
2778652	99	716	711,4	cyclictest	2950100	-21	kworker/3:2+events	09:40:44	3
2778652	99	716	711,4	cyclictest	2906203	-21	kworker/3:1+events	08:53:59	3
2778652	99	716	711,4	cyclictest	2792778	-21	kworker/3:2+events	07:39:40	3
2778652	99	715	711,3	cyclictest	2950100	-21	kworker/3:2+events	10:37:58	3
2778652	99	714	710,3	cyclictest	2862437	-21	kworker/3:0+events	08:22:21	3
2778652	99	713	707,5	cyclictest	2994019	-21	turbostat	10:04:55	3
2778652	99	713	703,4	cyclictest	19	1	rcu_preempt	07:45:22	3
2778652	99	712	709,2	cyclictest	737060	-21	kworker/3:1+events	07:15:42	3
2778652	99	712	708,3	cyclictest	2906203	-21	kworker/3:1+events	09:09:39	3
2778642	99	687	684,2	cyclictest	2925203	-21	turbostat	09:09:55	0
2778642	99	680	673,6	cyclictest	2714830	-21	kworker/0:2+events	09:20:22	0
2778646	99	669	666,2	cyclictest	2831059	-21	turbostat	07:54:55	1
2778642	99	669	666,2	cyclictest	2778189	-21	kworker/0:0+events	09:11:46	0
2778642	99	669	665,3	cyclictest	2714830	-21	kworker/0:2+events	09:18:59	0
2778652	99	664	660,3	cyclictest	2943870	-21	turbostat	09:24:56	3
2778652	99	664	659,4	cyclictest	2881689	-21	turbostat	08:34:55	3
2778646	99	652	646,5	cyclictest	3068844	-21	turbostat	11:04:56	1
2778652	99	629	626,2	cyclictest	2837268	-21	turbostat	07:59:55	3
2778646	99	605	601,3	cyclictest	2974909	-21	turbostat	09:49:55	1
2778646	99	603	600,2	cyclictest	2875470	-21	turbostat	08:29:55	1
2778646	99	25	22,2	cyclictest	0	-21	swapper/1	08:20:00	1
2957	98	23	0,10	rtkit-daemon	734	-21	sshd	09:36:33	0
2778642	99	23	20,2	cyclictest	0	-21	swapper/0	08:19:59	0
2778650	99	22	20,1	cyclictest	2919006	-21	idleruntime-cro	09:00:01	2
2778650	99	21	18,2	cyclictest	2987561	-21	idleruntime-cro	09:55:00	2
2778650	99	20	17,2	cyclictest	2837300	-21	systemd-run	07:55:00	2
2778650	99	20	15,4	cyclictest	2950636	-21	gltestperf	09:25:15	2
2778642	99	20	17,2	cyclictest	2784797	-21	latency_hist	07:15:01	0
2778642	99	20	17,2	cyclictest	0	-21	swapper/0	10:40:00	0
2778642	99	20	15,4	cyclictest	2882288	-21	cut	08:30:18	0
2778642	99	20	15,3	cyclictest	2802602	-21	ps	07:25:18	0
2778642	99	20	0,2	cyclictest	2894131	-21	munin-run	08:40:00	0
2778650	99	19	0,12	cyclictest	3062630	-21	munin-plugin-st	10:55:01	2
2778642	99	19	15,3	cyclictest	3068383	-21	unixbench_multi	10:55:25	0
2778650	99	18	15,2	cyclictest	493	-21	dbus-daemon	08:20:00	2
2778650	99	18	15,2	cyclictest	2856433	-21	latency_hist	08:10:01	2
2778650	99	18	14,3	cyclictest	2907241	-21	sed	08:50:17	2
2778646	99	18	4,2	cyclictest	19	1	rcu_preempt	08:05:19	1
2778646	99	18	15,2	cyclictest	0	-21	swapper/1	07:15:01	1
2778642	99	18	16,1	cyclictest	3062637	-21	munin-plugin-st	10:55:00	0
2778642	99	18	16,1	cyclictest	2961393	-21	latency	09:30:21	0
2778642	99	18	15,2	cyclictest	2873	-21	cron	07:50:00	0
2778652	99	17	14,2	cyclictest	3075670	-21	gltestperf	11:05:15	3
2778652	99	17	14,2	cyclictest	2943855	-21	ls	09:20:00	3
2778650	99	17	5,2	cyclictest	19	1	rcu_preempt	10:15:01	2
2778650	99	17	14,2	cyclictest	3081276	-21	munin-plugin-st	11:10:01	2
2778650	99	17	14,2	cyclictest	2894737	-21	gltestperf	08:40:16	2
2778650	99	17	14,2	cyclictest	2844417	-21	gltestperf	08:00:16	2
2778650	99	17	13,2	cyclictest	2940917	-21	kthreadcore	09:15:18	2
2778650	99	17	13,2	cyclictest	0	-21	swapper/2	10:39:59	2
2778646	99	17	14,2	cyclictest	1	-21	systemd	10:51:32	1
2778646	99	17	14,2	cyclictest	0	-21	swapper/1	09:00:01	1
2778646	99	17	14,2	cyclictest	0	-21	swapper/1	07:45:00	1
2778646	99	17	13,3	cyclictest	3006044	-21	unixbench_singl	10:05:26	1
2778646	99	17	13,3	cyclictest	0	-21	swapper/1	10:20:00	1
2778646	99	17	13,2	cyclictest	0	-21	swapper/1	07:35:00	1
2778642	99	17	9,7	cyclictest	3012704	-21	http	10:15:00	0
2778642	99	17	14,2	cyclictest	2975491	-21	gltestperf	09:45:15	0
2778650	99	16	13,2	cyclictest	3031152	-21	unixbench_singl	10:25:32	2
2778650	99	16	13,2	cyclictest	3000607	-21	cpuspeed_turbos	10:05:15	2
2778650	99	16	13,2	cyclictest	2792651	-21	unixbench_multi	07:15:24	2
2778650	99	16	12,3	cyclictest	3062965	-21	cpuspeed_turbos	10:55:13	2
2778650	99	16	12,3	cyclictest	2869813	-21	gltestperf	08:20:15	2
2778650	99	16	12,2	cyclictest	0	-21	swapper/2	11:05:01	2
2778646	99	16	5,2	cyclictest	19	1	rcu_preempt	08:10:15	1
2778646	99	16	13,2	cyclictest	2895369	-21	tr	08:40:17	1
2778646	99	16	12,3	cyclictest	69	50	irq/9-acpi	09:35:13	1
2778642	99	16	7,2	cyclictest	0	-21	swapper/0	08:55:16	0
2778642	99	16	3,2	cyclictest	19	1	rcu_preempt	07:45:00	0
2778642	99	16	13,2	cyclictest	3005490	-21	needreboot	10:05:22	0
2778642	99	16	13,2	cyclictest	0	-21	swapper/0	08:10:00	0
2772599	2	16	2,7	sleep0	0	-21	swapper/0	07:05:13	0
2778650	99	15	9,5	cyclictest	0	-21	swapper/2	07:35:16	2
2778650	99	15	12,2	cyclictest	2975293	-21	cpuspeed_turbos	09:45:14	2
2778650	99	15	12,2	cyclictest	2830642	-21	unixbench_singl	07:45:23	2
2778650	99	15	12,2	cyclictest	0	-21	swapper/2	08:40:00	2
2778650	99	15	11,3	cyclictest	3018297	-21	ntpq	10:15:21	2
2778650	99	15	11,3	cyclictest	2944598	-21	sed	09:20:15	2
2778650	99	15	11,3	cyclictest	2805901	-21	cpuspeed_turbos	07:30:13	2
2778646	99	15	3,2	cyclictest	19	1	rcu_preempt	09:20:13	1
2778646	99	15	12,2	cyclictest	2787211	-21	cpuspeed_turbos	07:15:13	1
2778646	99	15	12,2	cyclictest	0	-21	swapper/1	10:40:20	1
2778646	99	15	0,3	cyclictest	2868990	-21	apt-get	08:20:02	1
2778642	99	15	2,2	cyclictest	2957	98	rtkit-daemon	08:01:49	0
2778642	99	15	12,2	cyclictest	491	-21	avahi-daemon	09:55:00	0
2778642	99	15	12,2	cyclictest	2956258	-21	cron	09:30:00	0
2778642	99	15	12,2	cyclictest	2831404	-21	cpuspeed_turbos	07:50:16	0
2778642	99	15	12,2	cyclictest	2778189	-21	kworker/0:0+kacpid	08:20:19	0
2778642	99	15	12,2	cyclictest	0	-21	swapper/0	10:20:00	0
2778642	99	15	12,2	cyclictest	0	-21	swapper/0	10:00:00	0
2778650	99	14	6,7	cyclictest	3032044	-21	uniq	10:30:17	2
2778650	99	14	6,3	cyclictest	2805557	-21	munin-plugin-st	07:30:00	2
2778650	99	14	12,1	cyclictest	2861672	-21	latency	08:10:20	2
2778650	99	14	11,2	cyclictest	0	-21	swapper/2	10:45:17	2
2778650	99	14	10,3	cyclictest	2882427	-21	irqcore	08:30:19	2
2778646	99	14	9,3	cyclictest	2805530	-21	cron	07:30:00	1
2778646	99	14	4,2	cyclictest	19	1	rcu_preempt	07:45:15	1
2778646	99	14	11,2	cyclictest	3039703	-21	kthreadcore	10:35:21	1
2778646	99	14	11,2	cyclictest	2920224	-21	irqrtprio	09:00:17	1
2778646	99	14	11,2	cyclictest	0	-21	swapper/1	07:40:00	1
2778642	99	14	9,4	cyclictest	3763191	-21	systemd-journal	08:42:32	0
2778642	99	14	4,2	cyclictest	0	-21	swapper/0	10:45:20	0
2778642	99	14	1,3	cyclictest	2956	-21	rtkit-daemon	07:23:55	0
2778642	99	14	11,2	cyclictest	2805886	-21	cpuspeed_turbos	07:30:13	0
2778642	99	14	11,2	cyclictest	2778189	-21	kworker/0:0+kacpid	07:15:18	0
2778642	99	14	11,2	cyclictest	0	-21	swapper/0	09:00:15	0
2778642	99	14	10,3	cyclictest	2875812	-21	cpuspeed_turbos	08:25:14	0
2778652	99	13	9,3	cyclictest	2937198	-21	unixbench_multi	09:10:26	3
2778650	99	13	8,4	cyclictest	3049859	-21	seq	10:40:27	2
2778650	99	13	8,4	cyclictest	2819141	-21	sed	07:40:17	2
2778650	99	13	8,4	cyclictest	0	-21	swapper/2	07:20:22	2
2778650	99	13	10,2	cyclictest	0	-21	swapper/2	09:34:59	2
2778650	99	13	10,2	cyclictest	0	-21	swapper/2	07:15:01	2
2778646	99	13	9,3	cyclictest	31	1	ktimers/1	09:10:14	1
2778646	99	13	7,5	cyclictest	523	-21	gdbus	09:45:00	1
2778646	99	13	2,2	cyclictest	19	1	rcu_preempt	09:34:59	1
2778646	99	13	10,2	cyclictest	3065897	-21	kthreadcore	10:55:18	1
2778646	99	13	10,2	cyclictest	2900670	-21	cpuspeed_turbos	08:45:15	1
2778646	99	13	0,4	cyclictest	3019673	-21	fschecks_time	10:20:19	1
2778642	99	13	9,3	cyclictest	3080797	-21	sed	11:05:24	0
2778642	99	13	6,4	cyclictest	3074102	-21	cut	11:00:20	0
2778642	99	13	5,1	cyclictest	2974927	-21	awk	09:45:00	0
2778642	99	13	12,1	cyclictest	0	-21	swapper/0	07:55:20	0
2778642	99	13	11,1	cyclictest	0	-21	swapper/0	10:00:18	0
2778642	99	13	10,2	cyclictest	2907242	-21	grep	08:50:18	0
2778642	99	13	0,2	cyclictest	19	1	rcu_preempt	07:35:13	0
2778652	99	12	9,2	cyclictest	0	-21	swapper/3	09:55:23	3
2778650	99	12	9,2	cyclictest	2969094	-21	cpuspeed_turbos	09:40:13	2
2778650	99	12	9,2	cyclictest	0	-21	swapper/2	09:10:16	2
2778650	99	12	9,2	cyclictest	0	-21	swapper/2	08:45:15	2
2778650	99	12	7,4	cyclictest	493	-21	dbus-daemon	09:55:24	2
2778650	99	12	7,4	cyclictest	2930362	-21	sed	09:05:23	2
2778650	99	12	1,2	cyclictest	19	1	rcu_preempt	10:05:00	2
2778650	99	12	10,1	cyclictest	3023499	-21	kthreadcore	10:20:23	2
2778650	99	12	10,1	cyclictest	0	-21	swapper/2	08:25:16	2
2778650	99	12	0,2	cyclictest	19	1	rcu_preempt	09:35:25	2
2778646	99	12	9,2	cyclictest	3032165	-21	gltestperf	10:30:18	1
2778646	99	12	8,3	cyclictest	2793471	-21	cpuspeed_turbos	07:20:14	1
2778646	99	12	7,4	cyclictest	2937919	-21	apt-get	09:15:12	1
2778646	99	12	0,3	cyclictest	0	-21	swapper/1	08:53:15	1
2778646	99	12	0,12	cyclictest	0	-21	swapper/1	09:05:20	1
2778642	99	12	9,2	cyclictest	2714830	-21	kworker/0:2+kacpid	10:40:18	0
2778642	99	12	9,2	cyclictest	2714830	-21	kworker/0:2+kacpid	10:30:21	0
2778642	99	12	4,2	cyclictest	21	1	rcuc/0	08:10:14	0
2778642	99	12	10,1	cyclictest	2778189	-21	kworker/0:0+kacpid	10:29:12	0
2777671	2	12	2,4	sleep2	0	-21	swapper/2	07:05:22	2
2778652	99	11	5,4	cyclictest	493	-21	dbus-daemon	09:55:00	3
2778650	99	11	8,2	cyclictest	2843788	-21	systemctl	08:00:00	2
2778650	99	11	7,3	cyclictest	2924525	-21	sed	09:00:23	2
2778646	99	11	9,1	cyclictest	0	-21	swapper/1	10:45:17	1
2778646	99	11	5,4	cyclictest	2888411	-21	cut	08:35:16	1
2778646	99	11	2,2	cyclictest	2957	98	rtkit-daemon	08:03:01	1
2778646	99	11	0,2	cyclictest	19	1	rcu_preempt	10:10:14	1
2778646	99	11	0,10	cyclictest	0	-21	swapper/1	10:28:53	1
2778642	99	11	10,1	cyclictest	2714830	-21	kworker/0:2+kacpid	10:20:07	0
2778642	99	11	0,3	cyclictest	17	1	ktimers/0	08:45:24	0
2772263	2	11	0,6	sleep1	2772099	-21	apt-get	07:05:02	1
2778646	99	10	0,2	cyclictest	19	1	rcu_preempt	09:25:18	1
2778426	2	9	5,2	sleep3	0	-21	swapper/3	07:09:13	3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2022-03-28 - 20:20