OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot4s.osadl.org (updated Tue Sep 01, 2026 12:44:06)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2985365	99	46	41,3	cyclictest	3329246	-21	base64	12:40:00	3
2985363	99	43	38,3	cyclictest	3110709	-21	cat	09:10:01	2
2985363	99	39	34,3	cyclictest	3250782	-21	gpgv	11:25:00	2
2985365	99	38	32,4	cyclictest	3173333	-21	apt-get	10:10:12	3
2985363	99	32	27,4	cyclictest	3011129	-21	kworker/u8:1+flush-259:0	08:00:26	2
2985359	99	32	3,28	cyclictest	872	-21	lldpd	07:40:11	1
2985355	99	31	24,5	cyclictest	3121060	-21	latency_hist	09:20:01	0
2985365	99	30	22,6	cyclictest	3016926	-21	cat	07:40:01	3
2985355	99	30	26,3	cyclictest	3229637	-21	kworker/u8:3+flush-259:0	11:35:00	0
2985355	99	28	24,3	cyclictest	3147292	-21	kworker/u8:2+flush-259:0	09:50:26	0
2985355	99	28	24,3	cyclictest	2995617	-21	kworker/u8:0+flush-259:0	07:25:25	0
2985355	99	28	22,4	cyclictest	3271935	-21	apt-get	11:45:12	0
2985365	99	27	21,4	cyclictest	3251270	-21	apt-get	11:25:11	3
2985363	99	27	24,2	cyclictest	3193947	-21	kworker/u8:4+flush-259:0	10:35:26	2
2985355	99	27	23,3	cyclictest	3162779	-21	kworker/u8:1+events_unbound	10:50:01	0
2985355	99	27	23,3	cyclictest	21	1	rcuc/0	10:20:21	0
2985363	99	26	19,5	cyclictest	335557	-21	systemd-journal	07:25:00	2
2985355	99	25	21,3	cyclictest	3229637	-21	kworker/u8:3+events_unbound	11:40:13	0
2985355	99	25	20,4	cyclictest	3021917	-21	latency_hist	07:45:01	0
2985355	99	25	17,3	cyclictest	3089865	-21	kworker/u8:0+events_unbound	10:20:01	0
2985355	99	25	1,21	cyclictest	335557	-21	systemd-journal	11:19:58	0
2985365	99	24	21,2	cyclictest	3307991	-21	apt-key	12:20:00	3
2985365	99	24	18,4	cyclictest	3111130	-21	diskstats	09:10:14	3
2985363	99	24	19,4	cyclictest	3079473	-21	/usr/sbin/munin	08:40:15	2
2985359	99	24	20,3	cyclictest	3266258	-40	mandb	11:37:10	1
2985359	99	24	18,4	cyclictest	3313558	-21	latency_hist	12:25:01	1
2985355	99	24	18,4	cyclictest	3079327	-21	base64	08:40:01	0
2985365	99	23	18,3	cyclictest	3001011	-21	/usr/sbin/munin	07:25:22	3
2985365	99	23	16,5	cyclictest	3239294	-21	ntpq	11:10:22	3
2985363	99	23	18,4	cyclictest	335557	-21	systemd-journal	11:29:59	2
2985363	99	23	18,4	cyclictest	3126648	-21	latency_hist	09:25:01	2
2985359	99	23	20,2	cyclictest	3203668	-21	kworker/u8:2+flush-259:0	10:45:16	1
2985359	99	23	18,3	cyclictest	3136939	-21	latency_hist	09:35:01	1
2985359	99	23	17,4	cyclictest	3084438	-21	latency_hist	08:45:01	1
2985355	99	23	20,2	cyclictest	3203668	-21	kworker/u8:2+events_unbound	10:59:22	0
2985355	99	23	20,2	cyclictest	3037274	-21	kworker/u8:3+events_unbound	08:35:21	0
2985355	99	23	20,1	cyclictest	3068717	-21	apt-get	08:30:12	0
2985355	99	22	18,3	cyclictest	2995617	-21	kworker/u8:0+events_unbound	07:55:16	0
2985355	99	22	18,3	cyclictest	2995617	-21	kworker/u8:0+events_unbound	07:35:00	0
2985365	99	21	15,4	cyclictest	3156763	-21	ntpq	09:50:23	3
2985365	99	21	14,5	cyclictest	3057167	-21	ntpq	08:15:23	3
2985363	99	21	17,3	cyclictest	2990697	-21	/usr/sbin/munin	07:15:21	2
2985363	99	21	17,3	cyclictest	2990697	-21	/usr/sbin/munin	07:15:20	2
2985359	99	21	9,7	cyclictest	3209232	-21	munin-run	10:45:01	1
2985359	99	21	15,4	cyclictest	3281168	-21	ntpq	11:50:21	1
2985359	99	21	15,4	cyclictest	3037692	-21	apt-get	08:00:11	1
2985355	99	21	18,2	cyclictest	3245425	-21	kworker/u8:1+events_unbound	12:30:17	0
2985355	99	21	17,3	cyclictest	3089865	-21	kworker/u8:0+events_unbound	09:42:27	0
2985365	99	20	12,6	cyclictest	3125726	-21	ntpq	09:20:22	3
2985363	99	20	17,2	cyclictest	3089865	-21	kworker/u8:0+flush-259:0	09:10:23	2
2985359	99	20	16,3	cyclictest	2995617	-21	kworker/u8:0+flush-259:0	08:00:01	1
2985359	99	20	14,4	cyclictest	335557	-21	systemd-journal	10:59:55	1
2985359	99	20	11,7	cyclictest	335557	-21	systemd-journal	07:50:00	1
2985355	99	20	17,2	cyclictest	902910	-21	kworker/u8:5+events_unbound	07:35:22	0
2985355	99	20	17,2	cyclictest	3089865	-21	kworker/u8:0+flush-259:0	11:00:15	0
2985363	99	19	14,3	cyclictest	3094764	-21	cron	08:55:00	2
2985363	99	19	13,4	cyclictest	335557	-21	systemd-journal	12:31:21	2
2985363	99	19	13,4	cyclictest	3116146	-21	apt-get	09:15:11	2
2985363	99	19	12,5	cyclictest	3235041	-21	latency_hist	11:10:01	2
2985359	99	19	13,4	cyclictest	3178199	-21	latency_hist	10:15:01	1
2985355	99	19	16,2	cyclictest	3282281	-21	kworker/u8:2+flush-259:0	12:07:41	0
2985355	99	19	16,2	cyclictest	3162779	-21	kworker/u8:1+flush-259:0	10:25:19	0
2985355	99	19	16,2	cyclictest	3162779	-21	kworker/u8:1+events_unbound	10:31:34	0
2985355	99	19	16,2	cyclictest	3089865	-21	kworker/u8:0+events_unbound	08:50:12	0
2985355	99	19	16,2	cyclictest	3052934	-21	kworker/u8:2+events_unbound	08:25:20	0
2985355	99	19	15,3	cyclictest	3089865	-21	kworker/u8:0+flush-259:0	09:35:16	0
2985355	99	19	15,3	cyclictest	3089865	-21	kworker/u8:0+flush-259:0	09:35:15	0
2985355	99	19	13,4	cyclictest	335557	-21	systemd-journal	09:56:11	0
2985365	99	18	15,2	cyclictest	3319400	-21	irqcore	12:30:17	3
2985365	99	18	12,4	cyclictest	3105092	-21	latency_hist	09:05:00	3
2985363	99	18	15,2	cyclictest	3266503	-21	idleruntime-cro	11:40:00	2
2985363	99	18	15,2	cyclictest	3048044	-21	apt-get	08:10:12	2
2985363	99	18	15,2	cyclictest	2995617	-21	kworker/u8:0+flush-259:0	08:20:00	2
2985363	99	18	14,2	cyclictest	3239333	-21	ntp_kernel_pll_	11:10:23	2
2985363	99	18	14,2	cyclictest	0	-21	swapper/2	07:15:00	2
2985359	99	18	11,5	cyclictest	335557	-21	systemd-journal	08:30:01	1
2985355	99	18	15,2	cyclictest	3162779	-21	kworker/u8:1+events_unbound	10:40:22	0
2985355	99	18	15,2	cyclictest	3052934	-21	kworker/u8:2+flush-259:0	08:15:20	0
2985355	99	18	14,3	cyclictest	3271520	-21	kworker/u8:0+events_unbound	11:55:08	0
2985355	99	18	14,3	cyclictest	3037274	-21	kworker/u8:3+flush-259:0	08:10:01	0
2985359	99	17	14,2	cyclictest	3153141	-21	irqcore	09:50:18	1
2985359	99	17	14,2	cyclictest	3078844	-21	apt-config	08:40:01	1
2985359	99	17	14,2	cyclictest	0	-21	swapper/1	09:20:00	1
2985359	99	17	13,3	cyclictest	335557	-21	systemd-journal	10:37:22	1
2985359	99	17	13,3	cyclictest	3297831	-21	kworker/u8:3+flush-259:0	12:25:28	1
2985359	99	17	13,3	cyclictest	3250574	-21	ls	11:25:00	1
2985359	99	17	13,3	cyclictest	3058360	-21	apt-get	08:20:12	1
2985359	99	17	11,4	cyclictest	3157822	-21	/usr/sbin/munin	09:55:14	1
2985355	99	17	13,3	cyclictest	3089865	-21	kworker/u8:0+events_unbound	09:45:17	0
2985355	99	17	13,3	cyclictest	3017393	-21	kworker/u8:2+events_unbound	07:50:15	0
2985365	99	16	15,1	cyclictest	2985351	-21	cyclictest	07:15:01	3
2985363	99	16	14,1	cyclictest	3219528	-21	sh	10:55:01	2
2985363	99	16	13,2	cyclictest	3318705	-21	idleruntime-cro	12:30:00	2
2985359	99	16	9,5	cyclictest	335557	-21	systemd-journal	08:58:31	1
2985359	99	16	13,2	cyclictest	3245425	-21	kworker/u8:1+flush-259:0	12:00:36	1
2985359	99	16	12,2	cyclictest	3261318	-21	sh	11:35:00	1
2985359	99	16	12,2	cyclictest	0	-21	swapper/1	12:15:00	1
2985359	99	16	10,4	cyclictest	3016317	-21	latency_hist	07:40:00	1
2985355	99	16	9,5	cyclictest	2999887	-21	ntpq	07:20:22	0
2985355	99	16	14,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	11:35:24	0
2985355	99	16	14,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	09:00:18	0
2985355	99	16	13,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	07:10:14	0
2985355	99	16	13,2	cyclictest	3245425	-21	kworker/u8:1+events_unbound	12:15:14	0
2985355	99	16	13,2	cyclictest	3037274	-21	kworker/u8:3+events_unbound	08:20:16	0
2985355	99	16	12,3	cyclictest	3162779	-21	kworker/u8:1+events_unbound	10:06:20	0
2985355	99	16	11,4	cyclictest	3199162	-21	apt-get	10:35:14	0
2985365	99	15	10,3	cyclictest	3052934	-21	kworker/u8:2+flush-259:0	08:50:00	3
2985363	99	15	9,1	cyclictest	0	-21	swapper/2	09:29:59	2
2985359	99	15	9,4	cyclictest	3329095	-21	apt-get	12:40:00	1
2985359	99	15	11,3	cyclictest	3052934	-21	kworker/u8:2+flush-259:0	08:15:18	1
2985359	99	15	10,4	cyclictest	3000780	-21	latency_hist	07:25:00	1
2985355	99	15	9,4	cyclictest	509	-21	rs:main		0
2985355	99	15	9,4	cyclictest	3132058	-21	apt-get	09:30:12	0
2985355	99	15	8,5	cyclictest	3312658	-21	ntpq	12:20:19	0
2985355	99	15	14,1	cyclictest	3099861	-21	kworker/u8:1+events_unbound	09:05:07	0
2985355	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	11:20:23	0
2985355	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	10:10:22	0
2985355	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	09:24:50	0
2985355	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	08:10:24	0
2985355	99	15	13,1	cyclictest	17	1	ktimers/0	08:55:28	0
2985355	99	15	12,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	12:25:13	0
2985355	99	15	12,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	08:45:26	0
2985355	99	15	12,2	cyclictest	3245425	-21	kworker/u8:1+events_unbound	11:50:45	0
2985355	99	15	12,2	cyclictest	3089865	-21	kworker/u8:0+events_unbound	11:10:23	0
2985355	99	15	11,3	cyclictest	603	50	irq/132-enp0s29f1:rx-0	07:50:00	0
2985355	99	15	11,3	cyclictest	3245425	-21	kworker/u8:1+events_unbound	12:00:12	0
2985365	99	14	12,1	cyclictest	3047785	-21	sh	08:10:00	3
2985365	99	14	12,1	cyclictest	0	-21	swapper/3	12:15:00	3
2985363	99	14	12,1	cyclictest	0	-21	swapper/2	12:20:00	2
2985363	99	14	11,2	cyclictest	3067979	-21	unixbench_multi	08:25:26	2
2985363	99	14	11,2	cyclictest	0	-21	swapper/2	09:55:19	2
2985363	99	14	10,2	cyclictest	0	-21	swapper/2	11:44:59	2
2985359	99	14	8,4	cyclictest	335557	-21	systemd-journal	10:16:18	1
2985359	99	14	12,1	cyclictest	0	-21	swapper/1	10:30:01	1
2985359	99	14	11,2	cyclictest	3073596	-21	sh	08:35:00	1
2985355	99	14	8,4	cyclictest	3255287	-21	ntpq	11:25:20	0
2985355	99	14	7,5	cyclictest	3041779	-21	ntpq	08:00:23	0
2985355	99	14	12,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	11:05:18	0
2985355	99	14	11,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	09:10:40	0
2985355	99	14	10,3	cyclictest	902910	-21	kworker/u8:5+flush-259:0	07:15:21	0
2985355	99	14	10,3	cyclictest	902910	-21	kworker/u8:5+flush-259:0	07:15:20	0
2985355	99	14	10,3	cyclictest	3203668	-21	kworker/u8:2+events_unbound	10:50:15	0
2985365	99	13	9,3	cyclictest	3162779	-21	kworker/u8:1+flush-259:0	10:35:19	3
2985365	99	13	9,2	cyclictest	0	-21	swapper/3	09:34:59	3
2985365	99	13	8,3	cyclictest	335557	-21	systemd-journal	12:20:02	3
2985365	99	13	8,3	cyclictest	335557	-21	systemd-journal	10:07:19	3
2985365	99	13	11,1	cyclictest	3244449	-21	ntp_kernel_err	11:15:20	3
2985365	99	13	10,2	cyclictest	0	-21	swapper/3	07:40:00	3
2985363	99	13	9,3	cyclictest	3162779	-21	kworker/u8:1+flush-259:0	10:30:16	2
2985363	99	13	6,5	cyclictest	3276029	-21	ntpq	11:45:23	2
2985363	99	13	11,1	cyclictest	3031325	-21	grep	07:50:20	2
2985363	99	13	11,1	cyclictest	0	-21	swapper/2	09:45:19	2
2985363	99	13	10,2	cyclictest	3072766	-21	ntp_offset	08:30:22	2
2985359	99	13	9,3	cyclictest	335557	-21	systemd-journal	10:34:21	1
2985359	99	13	8,4	cyclictest	335557	-21	systemd-journal	08:10:00	1
2985359	99	13	8,3	cyclictest	3276858	-21	kworker/u8:4+events_unbound	11:55:18	1
2985359	99	13	6,5	cyclictest	3308076	-21	apt-key	12:20:00	1
2985359	99	13	10,2	cyclictest	2971	-21	cron	12:35:00	1
2985355	99	13	11,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	10:01:21	0
2985355	99	13	10,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	09:27:10	0
2985355	99	13	10,2	cyclictest	3245425	-21	kworker/u8:1+flush-259:0	12:35:21	0
2985365	99	12	9,2	cyclictest	3266951	-21	fschecks_count	11:40:16	3
2985365	99	12	9,2	cyclictest	3225143	-21	forks	11:00:14	3
2985365	99	12	9,2	cyclictest	3183732	-21	cpuspeed_turbos	10:20:10	3
2985365	99	12	9,2	cyclictest	3031365	-21	ntp_kernel_err	07:50:21	3
2985365	99	12	10,1	cyclictest	0	-21	swapper/3	08:45:01	3
2985363	99	12	9,2	cyclictest	3099944	-21	cron	09:00:00	2
2985363	99	12	9,2	cyclictest	3089865	-21	kworker/u8:0+flush-259:0	09:00:14	2
2985363	99	12	8,3	cyclictest	3089865	-21	kworker/u8:0+flush-259:0	11:00:25	2
2985359	99	12	3,8	cyclictest	0	-21	swapper/1	10:04:59	1
2985365	99	11	8,2	cyclictest	3284647	-21	kthreadcore	11:55:17	3
2985365	99	11	8,2	cyclictest	3256517	-21	cpuspeed_turbos	11:30:14	3
2985365	99	11	8,2	cyclictest	0	-21	swapper/3	09:40:17	3
2985365	99	11	7,3	cyclictest	3109660	-21	sed	09:05:26	3
2985365	99	11	6,4	cyclictest	3190477	-21	cut	10:25:18	3
2985365	99	11	6,3	cyclictest	335557	-21	systemd-journal	09:15:00	3
2985365	99	11	5,4	cyclictest	3261592	-21	apt-get	11:35:14	3
2985365	99	11	0,3	cyclictest	3000780	-21	latency_hist	07:25:01	3
2985363	99	11	8,2	cyclictest	3297831	-21	kworker/u8:3+flush-259:0	12:10:21	2
2985363	99	11	8,2	cyclictest	3162779	-21	kworker/u8:1+flush-259:0	10:25:19	2
2985363	99	11	8,2	cyclictest	0	-21	swapper/2	12:05:23	2
2985363	99	11	8,1	cyclictest	0	-21	swapper/2	10:10:20	2
2985363	99	11	7,2	cyclictest	3027112	-21	awk	07:50:01	2
2985363	99	11	1,6	cyclictest	490	-21	dbus-daemon	12:25:00	2
2985363	99	11	1,2	cyclictest	3157573	-21	awk	09:54:59	2
2985363	99	11	0,8	cyclictest	3188537	-21	latency_hist	10:25:01	2
2985359	99	11	9,1	cyclictest	0	-21	swapper/1	10:55:01	1
2985359	99	11	8,2	cyclictest	3147713	-21	fschecks_count	09:45:15	1
2985359	99	11	8,2	cyclictest	3006283	-21	cpuspeed_turbos	07:30:13	1
2985359	99	11	8,2	cyclictest	0	-21	swapper/1	09:29:59	1
2985359	99	11	7,3	cyclictest	3001211	-21	fschecks_count	07:25:14	1
2985359	99	11	0,4	cyclictest	3271662	-21	apt	11:45:00	1
2985365	99	10	7,2	cyclictest	3272195	-21	fschecks_time	11:45:15	3
2985365	99	10	7,2	cyclictest	3223694	-21	needreboot	10:55:20	3
2985365	99	10	7,2	cyclictest	3037759	-21	cpuspeed_turbos	08:00:14	3
2985365	99	10	5,4	cyclictest	490	-21	dbus-daemon	10:55:01	3
2985365	99	10	5,3	cyclictest	3297698	-21	cat	12:10:00	3
2985365	99	10	5,3	cyclictest	3095332	-21	fw_forwarded_lo	08:55:16	3
2985365	99	10	3,4	cyclictest	490	-21	dbus-daemon	11:10:01	3
2985365	99	10	1,7	cyclictest	3292442	-21	latency_hist	12:05:00	3
2985365	99	10	0,2	cyclictest	3068456	-21	munin-run	08:30:01	3
2985363	99	10	7,2	cyclictest	3209898	-21	irqcore	10:45:15	2
2985363	99	10	6,3	cyclictest	2995617	-21	kworker/u8:0+flush-259:0	08:09:49	2
2985363	99	10	6,2	cyclictest	3006633	-21	irqcore	07:30:18	2
2985359	99	10	7,2	cyclictest	0	-21	swapper/1	09:04:59	1
2985359	99	10	5,4	cyclictest	3107587	-21	cut	09:05:20	1
2985359	99	10	4,4	cyclictest	3114996	-21	gcc	09:10:20	1
2985365	99	9	5,3	cyclictest	49	1	ktimers/3	09:45:14	3
2985363	99	9	3,4	cyclictest	3142504	-21	df	09:40:12	2
2985359	99	9	7,1	cyclictest	0	-21	swapper/1	11:45:18	1
2985365	99	8	5,2	cyclictest	3137281	-21	cpuspeed_turbos	09:35:14	3
2985365	99	8	5,2	cyclictest	3137281	-21	cpuspeed_turbos	09:35:13	3
2985363	99	8	5,2	cyclictest	3173093	-21	kworker/u8:3+events_unbound	10:15:14	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:34