OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot8s.osadl.org (updated Fri Nov 06, 2026 12:43:53)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3246206	2	136	0,1	sleep2	43	1	rcuc/2		2
3052418	99	23	21,0	cyclictest	0	-21	swapper/3		3
3052418	99	23	20,1	cyclictest	0	-21	swapper/3		3
3052418	99	23	19,2	cyclictest	0	-21	swapper/3		3
3052418	99	23	1,18	cyclictest	3505406	-21	turbostat		3
3052418	99	23	1,18	cyclictest	3505406	-21	turbostat		3
3413979	2	22	0,10	sleep1	3414406	-21	sed		1
3395850	2	22	0,1	sleep1	0	-21	swapper/1		1
3052418	99	21	18,1	cyclictest	0	-21	swapper/3		3
3052418	99	21	18,1	cyclictest	0	-21	swapper/3		3
3052418	99	20	17,1	cyclictest	0	-21	swapper/3		3
3052418	99	20	1,15	cyclictest	3130481	-21	turbostat		3
3052418	99	20	1,15	cyclictest	0	-21	swapper/3		3
3052418	99	19	1,13	cyclictest	3313527	-21	turbostat		3
3052418	99	18	8,7	cyclictest	3458606	2	sleep3		3
3052418	99	18	15,1	cyclictest	0	-21	swapper/3		3
3052418	99	18	1,11	cyclictest	3439427	-21	taskset		3
3052418	99	17	5,9	cyclictest	3229177	-21	sleep		3
3052418	99	17	2,3	cyclictest	0	-21	swapper/3		3
3052418	99	17	1,4	cyclictest	0	-21	swapper/3		3
3052418	99	17	14,1	cyclictest	0	-21	swapper/3		3
3052418	99	17	14,1	cyclictest	0	-21	swapper/3		3
3052418	99	17	14,1	cyclictest	0	-21	swapper/3		3
3052418	99	17	14,1	cyclictest	0	-21	swapper/3		3
3052418	99	17	14,1	cyclictest	0	-21	swapper/3		3
3052418	99	17	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	17	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	17	13,2	cyclictest	0	-21	swapper/3		3
3472659	2	16	0,8	sleep1	3467308	-21	/usr/sbin/munin		1
3357447	2	16	0,7	sleep0	3357491	-21	ntp_states		0
3052418	99	16	8,5	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,9	cyclictest	3305884	-21	sleep		3
3052418	99	16	1,5	cyclictest	3508758	-21	taskset		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	16	13,1	cyclictest	0	-21	swapper/3		3
3052418	99	16	13,1	cyclictest	0	-21	swapper/3		3
3052418	99	16	13,1	cyclictest	0	-21	swapper/3		3
3052418	99	16	1,10	cyclictest	3117863	-21	turbostat		3
3052418	99	16	1,10	cyclictest	3117863	-21	turbostat		3
3052418	99	16	0,7	cyclictest	3058609	-21	taskset		3
3052418	99	16	0,13	cyclictest	3067037	2	sleep3		3
3450600	2	15	0,1	sleep0	0	-21	swapper/0		0
3419222	2	15	0,6	sleep0	72	-21	kcompactd0		0
3314493	2	15	1,2	sleep1	0	-21	swapper/1		1
3111815	2	15	0,7	sleep2	3111971	-21	diskstats		2
3111815	2	15	0,7	sleep2	3111971	-21	diskstats		2
3052418	99	15	1,5	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,2	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,2	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,2	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,2	cyclictest	0	-21	swapper/3		3
3052418	99	15	11,2	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,11	cyclictest	0	-21	swapper/3		3
3052418	99	15	1,10	cyclictest	0	-21	swapper/3		3
3052418	99	15	0,4	cyclictest	0	-21	swapper/3		3
3052418	99	15	0,2	cyclictest	0	-21	swapper/3		3
3311701	2	14	0,4	sleep0	3311949	-21	users		0
3150002	2	14	4,4	sleep2	3149697	-21	/usr/sbin/munin		2
3052418	99	14	1,9	cyclictest	0	-21	swapper/3		3
3052418	99	14	1,3	cyclictest	0	-21	swapper/3		3
3052418	99	14	1,3	cyclictest	0	-21	swapper/3		3
3521134	2	13	1,4	sleep2	3521145	-21	ssh		2
3099148	2	13	2,6	sleep1	3099145	-21	apt-get		1
3046468	2	13	1,9	sleep0	0	-21	swapper/0		0
3503455	2	12	1,7	sleep0	3503563	-21	/usr/sbin/munin		0
3503455	2	12	1,7	sleep0	3503563	-21	/usr/sbin/munin		0
3297217	2	12	1,8	sleep1	3297223	-21	ssh		1
3052418	99	12	1,2	cyclictest	0	-21	swapper/3		3
3474974	2	11	0,8	sleep1	3206339	-21	diskmemload		1
3444024	2	11	0,7	sleep0	3444031	-21	ssh		0
3436707	2	11	0,7	sleep1	3436702	-21	apt-get		1
3436331	2	11	0,3	sleep2	3436408	-21	munin-run		2
3336046	2	11	0,2	sleep2	0	-21	swapper/2		2
3298002	2	11	0,8	sleep2	3298008	-21	ssh		2
3270223	2	11	0,8	sleep1	3271816	-21	tr		1
3228385	2	11	0,4	sleep1	0	-21	swapper/1		1
3174912	2	11	0,1	sleep0	0	-21	swapper/0		0
3107885	2	11	0,8	sleep0	3109688	-21	cut		0
3046310	2	11	1,4	sleep2	0	-21	swapper/2		2
3513350	2	10	1,4	sleep0	3513356	-21	ssh		0
3389973	2	10	0,6	sleep0	3206339	-21	diskmemload		0
3362214	2	10	0,7	sleep2	3363908	-21	kthreadcore		2
3349942	2	10	0,2	sleep2	3350053	-21	ssh		2
3334531	2	10	1,5	sleep0	3334563	-21	ntpq		0
3326594	2	10	0,7	sleep0	3206339	-21	diskmemload		0
3290830	2	10	1,5	sleep0	3206339	-21	diskmemload		0
3288458	2	10	1,4	sleep1	3288486	-21	ssh		1
3280832	2	10	1,6	sleep0	3280959	-21	python3		0
3226852	2	10	0,7	sleep2	3226998	-21	ntpq		2
3206672	2	10	1,5	sleep2	3206683	-21	ssh		2
3162312	2	10	1,5	sleep1	3162344	-21	apt-get		1
3092484	2	10	0,7	sleep0	611	-21	dbus-daemon		0
3081516	2	10	0,7	sleep0	3083136	-21	grep		0
3078773	2	10	0,7	sleep0	3078817	-21	ntpq		0
3484067	2	9	0,2	sleep2	3484158	-21	kthreadcore		2
3475029	2	9	0,5	sleep0	3475186	-21	df_inode		0
3466404	2	9	1,5	sleep1	3466407	-21	sh		1
3465775	2	9	0,6	sleep2	3465781	-21	ssh		2
3458364	2	9	0,1	sleep1	3458366	-21	ssh		1
3436011	2	9	0,7	sleep1	3436016	-21	sh		1
3427540	2	9	0,3	sleep0	0	-21	swapper/0		0
3405969	2	9	1,4	sleep2	3405991	-21	apt-get		2
3389030	2	9	0,1	sleep1	0	-21	swapper/1		1
3313853	2	9	1,4	sleep0	3313877	-21	apt-get		0
3258780	2	9	0,6	sleep2	3258784	-21	ssh		2
3189103	2	9	4,3	sleep1	0	-21	swapper/1		1
3187835	2	9	0,6	sleep2	3187999	-21	head		2
3186288	2	9	1,4	sleep2	3186419	-21	ntpq		2
3137039	2	9	1,5	sleep2	3137072	-21	apt-get		2
3105438	2	9	1,4	sleep1	3105471	-21	apt-get		1
3526583	2	8	0,6	sleep0	3526828	-21	/usr/sbin/munin		0
3510849	2	8	0,4	sleep1	3510899	-21	ntpq		1
3504610	2	8	1,3	sleep2	3206339	-21	diskmemload		2
3504610	2	8	1,3	sleep2	3206339	-21	diskmemload		2
3496965	2	8	0,5	sleep0	3206339	-21	diskmemload		0
3465202	2	8	1,4	sleep0	3465412	-21	users		0
3396941	2	8	0,1	sleep0	0	-21	swapper/0		0
3367541	2	8	1,4	sleep2	3367543	-21	apt-get		2
3358357	2	8	0,1	sleep2	0	-21	swapper/2		2
3349554	2	8	0,4	sleep1	3349696	-21	ntpq		1
3344581	2	8	1,3	sleep0	3344584	-21	apt-get		0
3342133	2	8	0,5	sleep2	3342178	-21	ntpq		2
3342025	2	8	0,4	sleep1	3342129	-21	ntpq		1
3329700	2	8	1,4	sleep1	3330665	-21	cut		1
3327384	2	8	0,5	sleep1	3327390	-21	ssh		1
3319727	2	8	1,2	sleep2	0	-21	swapper/2		2
3304755	2	8	0,5	sleep1	0	-21	swapper/1		1
3267075	2	8	0,2	sleep0	0	-21	swapper/0		0
3260199	2	8	1,3	sleep1	3260239	-21	sh		1
3248695	2	8	0,5	sleep0	3249847	-21	/usr/sbin/munin		0
3243169	2	8	0,5	sleep0	3206339	-21	diskmemload		0
3229945	2	8	0,4	sleep0	3230869	-21	cut		0
3219419	2	8	1,3	sleep0	0	-21	swapper/0		0
3214278	2	8	1,3	sleep2	3214313	-21	apt-get		2
3176391	2	8	0,4	sleep0	3177430	-21	grep		0
3143965	2	8	0,5	sleep2	3144845	-21	ps		2
3130376	2	8	1,4	sleep0	959	-21	snmpd		0
3129571	2	8	0,4	sleep1	3129612	-21	ntpq		1
3079997	2	8	0,3	sleep1	0	-21	swapper/1		1
3061864	2	8	0,5	sleep0	3062699	-21	cut		0
3059063	2	8	0,4	sleep1	3059315	-21	ntpq		1
3046273	2	8	0,3	sleep1	0	-21	swapper/1		1
3518604	2	7	0,5	sleep1	3518647	-21	ntp_kernel_pll_		1
3513334	2	7	0,2	sleep2	3513340	-21	ssh		2
3510908	2	7	1,3	sleep2	3510949	-21	ntpq		2
3467722	2	7	0,5	sleep0	3468066	-21	cut		0
3453156	2	7	0,4	sleep2	3454026	-21	cut		2
3452065	2	7	0,3	sleep0	3452102	-21	apt-get		0
3444419	2	7	1,4	sleep1	0	-21	swapper/1		1
3438124	2	7	0,4	sleep2	3439426	-21	kthreadcore		2
3422767	2	7	1,3	sleep2	0	-21	swapper/2		2
3420579	2	7	0,1	sleep2	3420581	-21	ssh		2
3400128	2	7	2,3	sleep2	3401704	-21	kthreadcore		2
3398253	2	7	0,3	sleep0	3398255	-21	apt-get		0
3389635	2	7	0,2	sleep2	0	-21	swapper/2		2
3381459	2	7	1,3	sleep2	3381463	-21	ssh		2
3374416	2	7	0,1	sleep1	0	-21	swapper/1		1
3366251	2	7	1,2	sleep1	0	-21	swapper/1		1
3359058	2	7	1,2	sleep1	0	-21	swapper/1		1
3328904	2	7	0,4	sleep2	3329025	-21	date		2
3312145	2	7	0,5	sleep2	3206339	-21	diskmemload		2
3305852	2	7	1,3	sleep2	0	-21	swapper/2		2
3303750	2	7	0,3	sleep0	3303792	-21	ntp_states		0
3283122	2	7	0,4	sleep0	0	-21	swapper/0		0
3274506	2	7	1,2	sleep2	0	-21	swapper/2		2
3267868	2	7	0,3	sleep0	3268024	-21	df_inode		0
3257892	2	7	1,3	sleep0	3258019	-21	python3		0
3252187	2	7	0,2	sleep1	0	-21	swapper/1		1
3243933	2	7	1,3	sleep2	3243938	-21	sh		2
3236310	2	7	0,5	sleep1	3236316	-21	ssh		1
3231841	2	7	0,4	sleep2	3233519	-21	cut		2
3167482	2	7	0,4	sleep0	3167522	-21	ntpq		0
3148290	2	7	0,3	sleep1	3148445	-21	/usr/sbin/munin		1
3117041	2	7	0,4	sleep1	3111778	-21	/usr/sbin/munin		1
3117041	2	7	0,4	sleep1	3111778	-21	/usr/sbin/munin		1
3110693	2	7	0,4	sleep2	3110797	-21	awk		2
3092767	2	7	0,4	sleep1	0	-21	swapper/1		1
3080414	2	7	1,3	sleep2	3080544	-21	ps		2
3059316	2	7	0,4	sleep2	3060696	-21	kthreadcore		2
3527271	2	6	1,2	sleep1	223	-21	plymouthd		1
3503564	2	6	0,2	sleep1	3503805	-21	ssh		1
3503564	2	6	0,2	sleep1	3503805	-21	ssh		1
3491519	2	6	0,3	sleep2	3492442	-21	cat		2
3490407	2	6	0,2	sleep1	0	-21	swapper/1		1
3488976	2	6	0,2	sleep0	3206339	-21	diskmemload		0
3482703	2	6	1,2	sleep1	0	-21	swapper/1		1
3480307	2	6	0,3	sleep2	3480337	-21	ntpq		2
3472517	2	6	0,2	sleep2	3472548	-21	ssh		2
3450462	2	6	0,2	sleep2	0	-21	swapper/2		2
3434964	2	6	0,2	sleep0	3206339	-21	diskmemload		0
3427595	2	6	0,3	sleep1	3427601	-21	ssh		1
3412507	2	6	0,4	sleep1	3412513	-21	ssh		1
3405587	2	6	0,3	sleep0	3405894	-21	tar		0
3390628	2	6	0,2	sleep2	0	-21	swapper/2		2
3375207	2	6	0,2	sleep1	3375210	-21	apt-get		1
3369414	2	6	0,3	sleep0	0	-21	swapper/0		0
3343646	2	6	0,1	sleep0	0	-21	swapper/0		0
3312627	2	6	0,3	sleep1	3312633	-21	ssh		1
3283217	2	6	0,3	sleep2	0	-21	swapper/2		2
3282477	2	6	0,4	sleep2	3206339	-21	diskmemload		2
3275441	2	6	0,3	sleep1	3275447	-21	ssh		1
3260240	2	6	1,2	sleep2	223	-21	plymouthd		2
3244854	2	6	0,2	sleep1	0	-21	swapper/1		1
3242377	2	6	0,3	sleep1	3242422	-21	ntpq		1
3217222	2	6	1,2	sleep1	0	-21	swapper/1		1
3212110	2	6	0,2	sleep0	3206339	-21	diskmemload		0
3211896	2	6	0,3	sleep1	3206339	-21	diskmemload		1
3206077	2	6	0,1	sleep2	0	-21	swapper/2		2
3199779	2	6	0,1	sleep1	0	-21	swapper/1		1
3194397	2	6	0,4	sleep2	3195164	-21	irqrtprio		2
3187677	2	6	1,2	sleep0	0	-21	swapper/0		0
3185444	2	6	1,2	sleep1	0	-21	swapper/1		1
3181489	2	6	0,4	sleep0	3181557	-21	/usr/sbin/munin		0
3180815	2	6	0,1	sleep1	0	-21	swapper/1		1
3174963	2	6	1,2	sleep2	3174996	-21	apt-get		2
3167601	2	6	1,3	sleep2	0	-21	swapper/2		2
3161834	2	6	0,1	sleep1	223	-21	plymouthd		1
3154811	2	6	0,3	sleep0	3154853	-21	ntpq		0
3148548	2	6	0,3	sleep0	3148586	-21	ntp_states		0
3138471	2	6	1,2	sleep0	0	-21	swapper/0		0
3136423	2	6	0,1	sleep0	0	-21	swapper/0		0
3130107	2	6	0,1	sleep2	0	-21	swapper/2		2
3124142	2	6	0,3	sleep2	3124293	-21	cat		2
3105026	2	6	0,1	sleep0	0	-21	swapper/0		0
3095413	2	6	1,2	sleep0	0	-21	swapper/0		0
3092803	2	6	0,3	sleep2	0	-21	swapper/2		2
3091494	2	6	0,2	sleep1	3091540	-21	ntpq		1
3087849	2	6	1,2	sleep2	0	-21	swapper/2		2
3079679	2	6	0,1	sleep1	0	-21	swapper/1		1
3067421	2	6	0,2	sleep1	0	-21	swapper/1		1
3505705	2	5	0,2	sleep0	0	-21	swapper/0		0
3404377	2	5	0,2	sleep1	0	-21	swapper/1		1
3359886	2	5	0,2	sleep0	0	-21	swapper/0		0
3229091	2	5	0,2	sleep0	0	-21	swapper/0		0
3200683	2	5	0,2	sleep1	3200927	-21	irqcore		1
3169976	2	5	0,2	sleep1	3170953	-21	kthreadcore		1
3158681	2	5	1,2	sleep2	0	-21	swapper/2		2
3142961	2	5	0,1	sleep1	0	-21	swapper/1		1
3132749	2	5	1,2	sleep1	0	-21	swapper/1		1
3130722	2	5	1,2	sleep2	3130734	-21	acpi		2
3124040	2	5	0,1	sleep1	0	-21	swapper/1		1
3118112	2	5	0,2	sleep0	0	-21	swapper/0		0
3117527	2	5	0,4	sleep0	0	-21	swapper/0		0
3117527	2	5	0,4	sleep0	0	-21	swapper/0		0
3073227	2	5	0,3	sleep0	0	-21	swapper/0		0
3067382	2	5	0,4	sleep2	0	-21	swapper/2		2
3052630	2	5	1,2	sleep1	0	-21	swapper/1		1
3052594	2	5	0,2	sleep0	0	-21	swapper/0		0
3381091	2	4	0,1	sleep0	0	-21	swapper/0		0
3205442	2	4	0,1	sleep0	0	-21	swapper/0		0
3196194	2	4	0,2	sleep0	0	-21	swapper/0		0
3156055	2	4	0,1	sleep0	0	-21	swapper/0		0
3152425	2	4	0,1	sleep1	0	-21	swapper/1		1
3099149	2	4	0,1	sleep2	0	-21	swapper/2		2
3078869	2	4	0,2	sleep2	0	-21	swapper/2		2
3057964	2	4	1,1	sleep2	223	-21	plymouthd		2
3168832	2	3	0,1	sleep2	0	-21	swapper/2		2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2022-03-28 - 20:20