OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rackeslot8s.osadl.org (updated Sun Nov 01, 2026 12:43:54)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3218721	2	126	0,1	sleep1	33	1	rcuc/1		1
2767516	99	28	24,1	cyclictest	0	-21	swapper/3		3
2767516	99	24	20,1	cyclictest	0	-21	swapper/3		3
2793990	2	22	0,2	sleep0	0	-21	swapper/0		0
2767516	99	20	17,1	cyclictest	0	-21	swapper/3		3
2767516	99	20	17,1	cyclictest	0	-21	swapper/3		3
2767516	99	20	1,13	cyclictest	3158611	-21	taskset		3
3191575	2	19	0,7	sleep2	3192569	-21	tr		2
2767516	99	19	16,1	cyclictest	0	-21	swapper/3		3
2767516	99	18	2,12	cyclictest	0	-21	swapper/3		3
2767516	99	18	15,1	cyclictest	0	-21	swapper/3		3
2767516	99	18	15,1	cyclictest	0	-21	swapper/3		3
2767516	99	18	15,1	cyclictest	0	-21	swapper/3		3
2767516	99	18	1,12	cyclictest	3090665	-21	taskset		3
2767516	99	18	0,13	cyclictest	3058785	-21	taskset		3
2767516	99	17	8,6	cyclictest	2839433	-21	turbostat		3
2767516	99	17	6,8	cyclictest	2788926	2	sleep3		3
2767516	99	17	3,11	cyclictest	0	-21	swapper/3		3
2767516	99	17	14,1	cyclictest	0	-21	swapper/3		3
2767516	99	17	14,1	cyclictest	0	-21	swapper/3		3
2767516	99	17	14,1	cyclictest	0	-21	swapper/3		3
2767516	99	17	14,1	cyclictest	0	-21	swapper/3		3
2767516	99	17	1,12	cyclictest	3228130	-21	taskset		3
2767516	99	17	1,12	cyclictest	3118806	-21	taskset		3
2767516	99	17	1,11	cyclictest	3167181	-21	taskset		3
2767516	99	17	1,11	cyclictest	3066006	-21	taskset		3
2767516	99	17	1,11	cyclictest	3050213	-21	taskset		3
2767516	99	17	1,11	cyclictest	3036826	-21	turbostat		3
2767516	99	17	1,11	cyclictest	2989578	-21	taskset		3
2767516	99	17	1,10	cyclictest	2767717	2	sleep3		3
2767516	99	16	7,7	cyclictest	3006080	2	sleep3		3
2767516	99	16	7,7	cyclictest	3006080	2	sleep3		3
2767516	99	16	2,11	cyclictest	0	-21	swapper/3		3
2767516	99	16	2,10	cyclictest	3042464	-21	taskset		3
2767516	99	16	2,10	cyclictest	2959922	-21	taskset		3
2767516	99	16	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	16	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	16	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	16	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	16	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	16	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	16	1,11	cyclictest	2982148	-21	taskset		3
2767516	99	16	1,11	cyclictest	2974494	-21	taskset		3
2767516	99	16	1,11	cyclictest	2883424	-21	taskset		3
2767516	99	16	1,11	cyclictest	2826381	-21	taskset		3
2767516	99	16	1,11	cyclictest	2813197	-21	taskset		3
2767516	99	16	1,10	cyclictest	3145668	2	sleep3		3
2767516	99	16	1,10	cyclictest	3129095	-21	taskset		3
2767516	99	16	1,10	cyclictest	3111836	-21	taskset		3
2767516	99	16	1,10	cyclictest	3098286	-21	taskset		3
2767516	99	16	1,10	cyclictest	3098286	-21	taskset		3
2767516	99	16	1,10	cyclictest	3067772	-21	taskset		3
2767516	99	16	1,10	cyclictest	2927453	-21	taskset		3
2767516	99	16	1,10	cyclictest	2890439	-21	taskset		3
2767516	99	16	1,10	cyclictest	2832817	-21	taskset		3
3044815	2	15	1,4	sleep1	3044818	-21	apt-get		1
2767516	99	15	1,9	cyclictest	3121369	-21	taskset		3
2767516	99	15	1,9	cyclictest	2858737	-21	taskset		3
2767516	99	15	13,1	cyclictest	0	-21	swapper/3		3
2767516	99	15	12,1	cyclictest	0	-21	swapper/3		3
2767516	99	15	1,10	cyclictest	3236356	-21	taskset		3
2767516	99	15	1,10	cyclictest	3213093	-21	taskset		3
2767516	99	15	1,10	cyclictest	3203388	-21	taskset		3
2767516	99	15	1,10	cyclictest	3203388	-21	taskset		3
2767516	99	15	1,10	cyclictest	3028041	-21	taskset		3
2767516	99	15	1,10	cyclictest	2951274	-21	taskset		3
2767516	99	15	1,10	cyclictest	2943481	-21	taskset		3
2767516	99	15	1,10	cyclictest	2921568	-21	taskset		3
2767516	99	15	1,10	cyclictest	2889864	-21	taskset		3
2767516	99	15	1,10	cyclictest	2870852	-21	taskset		3
2767516	99	15	1,10	cyclictest	2858322	-21	taskset		3
2767516	99	15	1,10	cyclictest	2844908	-21	taskset		3
2767516	99	15	1,10	cyclictest	2807334	-21	taskset		3
3143093	2	14	4,4	sleep1	35	-21	ksoftirqd/1		1
3091823	2	14	0,3	sleep2	43	1	rcuc/2		2
2767516	99	14	1,9	cyclictest	2782756	-21	taskset		3
3213241	2	13	0,4	sleep0	3213372	-21	diskstats		0
3011469	2	13	0,9	sleep0	3011615	-21	ntpq		0
2935013	2	13	1,9	sleep0	2935059	-21	ntpq		0
3158344	2	12	1,8	sleep1	3158348	-21	ssh		1
2960597	2	12	0,3	sleep2	0	-21	swapper/2		2
2952482	2	12	1,6	sleep2	2952519	-21	munin-run		2
2879129	2	12	1,9	sleep2	2880216	-21	kthreadcore		2
2787761	2	12	0,4	sleep0	2787792	-21	ntpq		0
3234435	2	11	0,3	sleep1	3234437	-21	ssh		1
3172346	2	11	1,6	sleep0	3172391	-21	ntpq		0
2959397	2	11	1,7	sleep1	0	-21	swapper/1		1
2943369	2	11	0,4	sleep1	2943375	-21	ssh		1
2929469	2	11	1,8	sleep2	0	-21	swapper/2		2
2916741	2	11	0,8	sleep1	2917791	-21	cut		1
2860543	2	11	1,7	sleep1	2862308	-21	cut		1
3211288	2	10	0,2	sleep1	3211294	-21	ssh		1
3210632	2	10	0,3	sleep0	3210676	-21	ntpq		0
3150088	2	10	0,7	sleep2	3150092	-21	ssh		2
3138033	2	10	5,3	sleep0	3139047	-21	kthreadcore		0
3121327	2	10	2,4	sleep2	3121364	-21	apt-get		2
3111151	2	10	1,4	sleep2	3111203	-21	ssh		2
3097059	2	10	6,2	sleep0	15	1	ktimers/0		0
3065435	2	10	1,5	sleep0	3065461	-21	ntpq		0
3022528	2	10	0,7	sleep0	3023311	-21	kthreadcore		0
3005819	2	10	0,7	sleep0	2921854	-21	diskmemload		0
3005819	2	10	0,7	sleep0	2921854	-21	diskmemload		0
2990596	2	10	0,8	sleep1	2921854	-21	diskmemload		1
2981103	2	10	0,6	sleep1	2981150	-21	ntpq		1
2951595	2	10	0,3	sleep0	2921854	-21	diskmemload		0
2800361	2	10	0,2	sleep1	0	-21	swapper/1		1
2768893	2	10	0,3	sleep0	0	-21	swapper/0		0
3202992	2	9	0,6	sleep2	3203034	-21	ntpq		2
3202992	2	9	0,6	sleep2	3203034	-21	ntpq		2
3197454	2	9	1,4	sleep0	3197486	-21	munin-run		0
3170178	2	9	0,6	sleep1	3171999	-21	kthreadcore		1
3164721	2	9	1,5	sleep0	3164767	-21	ntpq		0
3158187	2	9	1,6	sleep0	2921854	-21	diskmemload		0
3127449	2	9	0,6	sleep1	3127453	-21	ssh		1
3112078	2	9	1,2	sleep1	0	-21	swapper/1		1
3104875	2	9	0,6	sleep0	2921854	-21	diskmemload		0
3104875	2	9	0,6	sleep0	2921854	-21	diskmemload		0
3103613	2	9	1,4	sleep2	3103644	-21	ntpq		2
3103613	2	9	1,4	sleep2	3103644	-21	ntpq		2
3090644	2	9	1,5	sleep1	2921854	-21	diskmemload		1
3076849	2	9	0,6	sleep2	3077230	-21	grep		2
3075433	2	9	0,5	sleep1	3075439	-21	ssh		1
3075393	2	9	0,1	sleep0	0	-21	swapper/0		0
3073752	2	9	0,6	sleep1	3073756	-21	ssh		1
3059679	2	9	0,6	sleep1	2921854	-21	diskmemload		1
3044048	2	9	1,4	sleep2	3044052	-21	ssh		2
3042323	2	9	1,4	sleep0	3042364	-21	ntpq		0
3014167	2	9	1,5	sleep1	3014324	-21	ssh		1
3004719	2	9	0,7	sleep2	3004725	-21	ssh		2
3004719	2	9	0,7	sleep2	3004725	-21	ssh		2
2957981	2	9	0,6	sleep0	2958027	-21	cat		0
2889230	2	9	1,4	sleep1	2889277	-21	ntp_states		1
2882724	2	9	1,4	sleep1	2882824	-21	ntpq		1
2875820	2	9	0,6	sleep2	2876364	-21	wc		2
2845994	2	9	0,5	sleep2	2846027	-21	apt-get		2
2776630	2	9	1,5	sleep2	2777441	-21	irqrtprio		2
3241439	2	8	0,5	sleep0	3241486	-21	ntpq		0
3233566	2	8	1,4	sleep0	3233660	-21	ntpq		0
3227238	2	8	0,4	sleep1	3227244	-21	ssh		1
3187804	2	8	3,2	sleep0	3187932	-21	sensors_temp		0
3159493	2	8	1,3	sleep1	3159530	-21	apt-get		1
3157115	2	8	0,4	sleep2	3157150	-21	ntpq		2
3120343	2	8	1,3	sleep1	3120347	-21	ssh		1
3116152	2	8	0,5	sleep0	3117844	-21	kthreadcore		0
3090088	2	8	0,6	sleep2	3090094	-21	ssh		2
3060668	2	8	1,5	sleep1	0	-21	swapper/1		1
3034677	2	8	0,5	sleep2	2921854	-21	diskmemload		2
2996811	2	8	0,5	sleep1	2996926	-21	ssh		1
2944180	2	8	1,4	sleep2	2944184	-21	ssh		2
2914624	2	8	1,4	sleep0	2914652	-21	ntpq		0
2833368	2	8	1,4	sleep1	2833440	-21	cpuspeed_turbos		1
2794332	2	8	0,5	sleep1	2794576	-21	users		1
2776370	2	8	0,5	sleep0	2776422	-21	fschecks_count		0
2766463	2	8	0,4	sleep2	2766512	-21	ntpq		2
3236190	2	7	1,2	sleep1	0	-21	swapper/1		1
3220840	2	7	1,3	sleep0	0	-21	swapper/0		0
3197765	2	7	1,2	sleep0	3197480	-21	apt-get		0
3197765	2	7	1,2	sleep0	3197480	-21	apt-get		0
3190152	2	7	1,2	sleep1	0	-21	swapper/1		1
3174827	2	7	1,2	sleep1	0	-21	swapper/1		1
3166526	2	7	0,1	sleep2	3166532	-21	ssh		2
3150122	2	7	4,1	sleep0	0	-21	swapper/0		0
3142758	2	7	1,3	sleep2	3142764	-21	ssh		2
3105390	2	7	0,5	sleep1	0	-21	swapper/1		1
3105390	2	7	0,5	sleep1	0	-21	swapper/1		1
3083017	2	7	1,3	sleep0	3083050	-21	apt-get		0
3069284	2	7	0,5	sleep2	3070959	-21	kthreadcore		2
3052489	2	7	1,2	sleep0	0	-21	swapper/0		0
3050336	2	7	0,1	sleep0	223	-21	plymouthd		0
3034778	2	7	0,3	sleep0	3034811	-21	ntpq		0
3021204	2	7	0,3	sleep2	2921854	-21	diskmemload		2
2992821	2	7	0,5	sleep0	2994515	-21	cut		0
2986569	2	7	1,3	sleep0	2988123	-21	cut		0
2975836	2	7	1,3	sleep2	2975840	-21	apt-get		2
2973277	2	7	0,4	sleep1	2973325	-21	ntpq		1
2959431	2	7	0,2	sleep2	0	-21	swapper/2		2
2944586	2	7	0,1	sleep0	0	-21	swapper/0		0
2936555	2	7	0,1	sleep2	0	-21	swapper/2		2
2922208	2	7	0,4	sleep0	2922231	-21	cpu		0
2908167	2	7	0,1	sleep0	0	-21	swapper/0		0
2908167	2	7	0,1	sleep0	0	-21	swapper/0		0
2901930	2	7	0,4	sleep1	2901976	-21	ntpq		1
2896516	2	7	1,3	sleep2	0	-21	swapper/2		2
2890136	2	7	0,4	sleep2	2890269	-21	cat		2
2877701	2	7	1,3	sleep0	2877734	-21	apt-get		0
2865087	2	7	1,3	sleep0	2865120	-21	apt-get		0
2858720	2	7	0,5	sleep2	223	-21	plymouthd		2
2858366	2	7	0,1	sleep0	0	-21	swapper/0		0
2832171	2	7	0,4	sleep2	2832206	-21	ntpq		2
2825949	2	7	1,4	sleep0	2826032	-21	swap		0
2825834	2	7	0,4	sleep2	2825869	-21	ntpq		2
2806744	2	7	0,4	sleep0	2806784	-21	ntpq		0
2795681	2	7	0,4	sleep0	2796640	-21	irqstats		0
2773192	2	7	0,4	sleep1	2773436	-21	users		1
2767371	2	7	0,2	sleep3	0	-21	swapper/3		3
3242641	2	6	1,2	sleep2	0	-21	swapper/2		2
3231230	2	6	0,3	sleep2	0	-21	swapper/2		2
3228152	2	6	0,3	sleep2	3228153	-21	sh		2
3213579	2	6	0,3	sleep2	3213730	-21	cut		2
3197858	2	6	0,2	sleep1	3197866	-21	ssh		1
3197858	2	6	0,2	sleep1	3197866	-21	ssh		1
3187662	2	6	0,3	sleep1	3187705	-21	ntpq		1
3185564	2	6	1,3	sleep2	0	-21	swapper/2		2
3180997	2	6	0,1	sleep0	0	-21	swapper/0		0
3173392	2	6	2,1	sleep2	0	-21	swapper/2		2
3149479	2	6	0,3	sleep1	3149511	-21	ssh		1
3129402	2	6	1,2	sleep1	0	-21	swapper/1		1
3128661	2	6	0,4	sleep2	3128919	-21	ssh		2
3126726	2	6	0,2	sleep0	2921854	-21	diskmemload		0
3113664	2	6	1,2	sleep2	0	-21	swapper/2		2
3105971	2	6	1,2	sleep0	2921854	-21	diskmemload		0
3090302	2	6	1,2	sleep1	2921854	-21	diskmemload		1
3073847	2	6	0,1	sleep0	0	-21	swapper/0		0
3060137	2	6	0,1	sleep2	0	-21	swapper/2		2
3057671	2	6	0,3	sleep2	3057717	-21	ntpq		2
3050893	2	6	1,2	sleep2	0	-21	swapper/2		2
3027005	2	6	0,3	sleep1	3027055	-21	ntpq		1
3011980	2	6	0,4	sleep2	3012230	-21	sed		2
2999572	2	6	0,3	sleep1	3000340	-21	kthreadcore		1
2999572	2	6	0,3	sleep1	3000340	-21	kthreadcore		1
2991129	2	6	0,3	sleep2	2991162	-21	apt-get		2
2989544	2	6	0,4	sleep2	2989550	-21	ssh		2
2974612	2	6	0,3	sleep2	2974618	-21	ssh		2
2968139	2	6	1,3	sleep0	0	-21	swapper/0		0
2967579	2	6	0,1	sleep0	0	-21	swapper/0		0
2965615	2	6	1,3	sleep1	2965661	-21	ssh		1
2951426	2	6	1,2	sleep1	223	-21	plymouthd		1
2935501	2	6	0,3	sleep1	2935615	-21	ssh		1
2928057	2	6	0,1	sleep1	0	-21	swapper/1		1
2915498	2	6	0,3	sleep1	0	-21	swapper/1		1
2910316	2	6	0,3	sleep2	0	-21	swapper/2		2
2901886	2	6	0,3	sleep0	2901928	-21	gcc		0
2896790	2	6	0,3	sleep2	2896823	-21	apt-get		2
2872856	2	6	0,3	sleep0	2873947	-21	kthreadcore		0
2870824	2	6	0,1	sleep2	0	-21	swapper/2		2
2863856	2	6	0,3	sleep0	2863896	-21	ntpq		0
2853756	2	6	1,2	sleep2	2854795	-21	grep		2
2851178	2	6	0,3	sleep1	2851219	-21	ntpq		1
2847458	2	6	0,3	sleep0	2849071	-21	cut		0
2845714	2	6	0,1	sleep2	0	-21	swapper/2		2
2844821	2	6	0,3	sleep1	2844868	-21	ntp_states		1
2841105	2	6	1,2	sleep0	0	-21	swapper/0		0
2833343	2	6	0,3	sleep0	0	-21	swapper/0		0
2826441	2	6	0,1	sleep1	0	-21	swapper/1		1
2808397	2	6	1,2	sleep0	0	-21	swapper/0		0
2806786	2	6	0,3	sleep1	0	-21	swapper/1		1
2801995	2	6	0,3	sleep2	2802476	-21	cut		2
2793026	2	6	1,2	sleep2	0	-21	swapper/2		2
2782591	2	6	0,2	sleep1	2782608	-21	apt-get		1
2773601	2	6	0,1	sleep2	0	-21	swapper/2		2
2762312	2	6	0,3	sleep1	2763219	-21	cut		1
2761344	2	6	0,2	sleep0	0	-21	swapper/0		0
3205657	2	5	0,2	sleep2	0	-21	swapper/2		2
3181011	2	5	0,2	sleep2	2921854	-21	diskmemload		2
3135417	2	5	0,1	sleep0	0	-21	swapper/0		0
3037100	2	5	0,1	sleep1	3037106	-21	ssh		1
3029525	2	5	0,2	sleep1	3029522	-21	apt-get		1
3028962	2	5	0,3	sleep2	3028964	-21	ssh		2
3014160	2	5	0,2	sleep0	0	-21	swapper/0		0
3006425	2	5	0,3	sleep1	0	-21	swapper/1		1
2977402	2	5	0,3	sleep0	2979058	-21	cut		0
2915793	2	5	0,2	sleep0	0	-21	swapper/0		0
2909155	2	5	0,2	sleep2	1	-21	systemd		2
2909155	2	5	0,2	sleep2	1	-21	systemd		2
2904569	2	5	0,2	sleep1	33	1	rcuc/1		1
2904569	2	5	0,2	sleep1	33	1	rcuc/1		1
2892227	2	5	0,2	sleep0	2894000	-21	kthreadcore		0
2890605	2	5	0,2	sleep1	0	-21	swapper/1		1
2873951	2	5	0,3	sleep1	2875817	-21	kthreadcore		1
2865123	2	5	0,2	sleep1	0	-21	swapper/1		1
2839084	2	5	0,1	sleep2	0	-21	swapper/2		2
2827952	2	5	0,2	sleep0	2828437	-21	kthreadcore		0
2815176	2	5	0,3	sleep2	2815838	-21	cut		2
2814738	2	5	1,2	sleep1	0	-21	swapper/1		1
2809230	2	5	0,2	sleep1	2809823	-21	kthreadcore		1
2808138	2	5	0,3	sleep2	0	-21	swapper/2		2
2798022	2	5	0,2	sleep2	2799922	-21	kthreadcore		2
2787664	2	5	0,3	sleep2	0	-21	swapper/2		2
2885490	2	4	0,2	sleep0	0	-21	swapper/0		0
2852364	2	4	0,2	sleep1	0	-21	swapper/1		1
2828942	2	4	0,2	sleep1	0	-21	swapper/1		1
2820252	2	4	0,3	sleep0	0	-21	swapper/0		0
2915649	2	3	0,1	sleep2	0	-21	swapper/2		2
2774322	2	3	0,1	sleep1	0	-21	swapper/1		1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2022-03-28 - 20:20