OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rackeslot8s.osadl.org (updated Thu Dec 04, 2025 12:43:50)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2443905	99	1232	0,1228	cyclictest	2799397	2	sleep3		3
2443905	99	1183	0,1180	cyclictest	2527713	2	sleep3		3
2443905	99	1170	0,1166	cyclictest	2597349	2	sleep3		3
2443905	99	651	0,647	cyclictest	2661826	2	chrt		3
2554900	2	31	0,2	sleep0	14	-21	ksoftirqd/0		0
2443905	99	21	16,2	cyclictest	2508953	-21	sleep		3
2443905	99	18	8,7	cyclictest	2477342	-21	turbostat		3
2553550	2	17	0,6	sleep2	2553994	-21	cut		2
2490055	2	17	1,5	sleep0	2490074	-21	munin-run		0
2443905	99	17	8,7	cyclictest	2905333	2	sleep3		3
2443905	99	17	14,1	cyclictest	0	-21	swapper/3		3
2443905	99	17	14,1	cyclictest	0	-21	swapper/3		3
2443905	99	17	1,12	cyclictest	2905044	-21	turbostat		3
2443905	99	17	1,12	cyclictest	2629660	-21	turbostat		3
2443905	99	16	9,4	cyclictest	2534336	-21	taskset		3
2443905	99	16	13,1	cyclictest	0	-21	swapper/3		3
2443905	99	16	13,1	cyclictest	0	-21	swapper/3		3
2443905	99	16	1,11	cyclictest	2497075	-21	taskset		3
2443905	99	15	3,9	cyclictest	2937630	-21	taskset		3
2443905	99	15	13,1	cyclictest	0	-21	swapper/3		3
2443905	99	15	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	15	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	15	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	15	1,11	cyclictest	2759423	-21	taskset		3
2443905	99	15	1,11	cyclictest	2726531	-21	turbostat		3
1908	98	15	0,4	rtkit-daemon	2441866	-21	cut		0
2865068	2	14	1,4	sleep2	2865975	-21	sed		2
2718769	2	14	2,8	sleep0	2718806	-21	apt-get		0
2443905	99	14	5,6	cyclictest	2735336	2	sleep3		3
2443905	99	14	4,7	cyclictest	2717025	-21	taskset		3
2443905	99	14	4,7	cyclictest	2651249	2	sleep3		3
2443905	99	14	2,10	cyclictest	2775177	-21	turbostat		3
2443905	99	14	13,0	cyclictest	0	-21	swapper/3		3
2443905	99	14	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	12,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	14	1,10	cyclictest	2921770	-21	taskset		3
2443905	99	14	1,10	cyclictest	2888888	-21	turbostat		3
2443905	99	14	1,10	cyclictest	2750737	-21	turbostat		3
2443905	99	14	1,10	cyclictest	2668235	-21	taskset		3
2443905	99	14	1,10	cyclictest	2591548	-21	taskset		3
2443905	99	14	1,10	cyclictest	2540585	-21	taskset		3
2443905	99	14	1,10	cyclictest	2490344	-21	taskset		3
2443905	99	14	1,10	cyclictest	2449919	-21	taskset		3
1908	98	14	0,7	rtkit-daemon	2831531	-21	ssh		0
2647659	2	13	1,9	sleep0	0	-21	swapper/0		0
2455091	2	13	1,2	sleep0	0	-21	swapper/0		0
2443905	99	13	9,1	cyclictest	0	-21	swapper/3		3
2443905	99	13	5,6	cyclictest	2751732	2	sleep3		3
2443905	99	13	5,6	cyclictest	2733077	2	sleep3		3
2443905	99	13	4,7	cyclictest	2484028	-21	taskset		3
2443905	99	13	4,6	cyclictest	2637956	2	sleep3		3
2443905	99	13	1,9	cyclictest	2862358	-21	taskset		3
2443905	99	13	1,9	cyclictest	2848475	-21	taskset		3
2443905	99	13	1,9	cyclictest	2565108	-21	taskset		3
2443905	99	13	1,9	cyclictest	2450660	-21	taskset		3
2443905	99	13	12,0	cyclictest	0	-21	swapper/3		3
2443905	99	13	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	13	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	13	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	13	11,1	cyclictest	0	-21	swapper/3		3
2443905	99	13	1,10	cyclictest	2830301	-21	taskset		3
2443905	99	13	10,1	cyclictest	0	-21	swapper/3		3
2443905	99	13	0,10	cyclictest	2527964	-21	taskset		3
2443905	99	13	0,10	cyclictest	2471065	-21	taskset		3
2635358	2	12	0,1	sleep2	2635540	-21	sort		2
2527717	2	12	1,8	sleep2	2527725	-21	turbostat.cron		2
2465236	2	12	0,5	sleep2	2465436	-21	hddtemp_smartct		2
2443905	99	12	3,7	cyclictest	2840254	-21	taskset		3
2443905	99	12	1,8	cyclictest	2783563	-21	taskset		3
2443905	99	12	10,1	cyclictest	0	-21	swapper/3		3
2443905	99	12	10,1	cyclictest	0	-21	swapper/3		3
2443905	99	12	0,10	cyclictest	2678237	-21	taskset		3
2437801	2	12	1,6	sleep3	0	-21	swapper/3		3
1908	98	12	0,8	rtkit-daemon	2896147	-21	ssh		2
1908	98	12	0,7	rtkit-daemon	2887430	-21	sh		2
2839287	2	11	0,9	sleep0	2839289	-21	ssh		0
2838919	2	11	0,1	sleep2	0	-21	swapper/2		2
2544327	2	11	0,1	sleep2	0	-21	swapper/2		2
1908	98	11	0,7	rtkit-daemon	2749898	-21	ssh		0
1908	98	11	0,6	rtkit-daemon	2808373	-21	irqcore		0
1908	98	11	0,4	rtkit-daemon	2732665	-21	ssh		0
2927221	2	10	1,2	sleep1	0	-21	swapper/1		1
2921356	2	10	0,7	sleep1	2921362	-21	ssh		1
2878860	2	10	0,7	sleep1	2878868	-21	ssh		1
2790942	2	10	0,7	sleep0	2790950	-21	ssh		0
2774578	2	10	1,6	sleep2	2774583	-21	sh		2
2768902	2	10	0,4	sleep1	2770291	-21	cut		1
2734870	2	10	1,7	sleep2	2734876	-21	ssh		2
2726845	2	10	2,4	sleep2	2726882	-21	apt-get		2
2571968	2	10	2,4	sleep1	2572001	-21	apt-get		1
2509386	2	10	1,7	sleep2	2509517	-21	grep		2
2457235	2	10	0,7	sleep1	2457527	-21	grep		1
1908	98	10	0,6	rtkit-daemon	1928	-21	sendmail-mta		1
1908	98	10	0,5	rtkit-daemon	2853772	-21	ntp_states		0
1908	98	10	0,5	rtkit-daemon	2806680	-21	ssh		0
1908	98	10	0,5	rtkit-daemon	2710921	-21	df		0
1908	98	10	0,5	rtkit-daemon	2659950	-21	ssh		1
1908	98	10	0,5	rtkit-daemon	2614212	-21	sh		0
1908	98	10	0,3	rtkit-daemon	667	-21	snmpd		0
2889198	2	9	1,5	sleep1	2889236	-21	ssh		1
2840210	2	9	1,4	sleep2	2840249	-21	apt-get		2
2717316	2	9	1,6	sleep2	2717314	-21	ssh		2
2703031	2	9	0,1	sleep1	2703466	-21	irqcore		1
2691682	2	9	1,4	sleep0	2691759	-21	ntp_states		0
2667567	2	9	1,5	sleep1	2667682	-21	find		1
2653244	2	9	0,6	sleep2	2653251	-21	sh		2
2591277	2	9	1,5	sleep1	0	-21	swapper/1		1
2559452	2	9	2,3	sleep0	2559485	-21	apt-get		0
2514314	2	9	1,4	sleep0	2514373	-21	ntp_states		0
2479449	2	9	0,6	sleep2	2481506	-21	tr		2
1908	98	9	0,5	rtkit-daemon	2908791	-21	cut		2
1908	98	9	0,5	rtkit-daemon	2724743	-21	kworker/u16:3-rpciod		1
1908	98	9	0,5	rtkit-daemon	2700868	-21	ssh		0
1908	98	9	0,5	rtkit-daemon	1928	-21	sendmail-mta		2
1908	98	9	0,3	rtkit-daemon	674	-21	lldpd		0
1908	98	9	0,1	rtkit-daemon	0	-21	swapper/0		0
2927879	2	8	0,6	sleep0	2927886	-21	sh		0
2919571	2	8	3,3	sleep2	2636973	-21	kworker/u16:2+nfsiod		2
2881077	2	8	1,3	sleep1	2881114	-21	apt-get		1
2872999	2	8	0,4	sleep2	2873034	-21	apt-get		2
2824047	2	8	1,3	sleep1	2824081	-21	apt-get		1
2799782	2	8	1,3	sleep1	2799816	-21	apt-get		1
2799421	2	8	0,6	sleep2	2799447	-21	awk		2
2761292	2	8	1,5	sleep0	2763355	-21	kthreadcore		0
2706388	2	8	0,5	sleep0	2707676	-21	memory		0
2700066	2	8	0,5	sleep2	2700302	-21	unixbench_singl		2
2677745	2	8	1,4	sleep1	2677747	-21	sh		1
2662374	2	8	1,4	sleep0	2662429	-21	fschecks_count		0
2636490	2	8	0,1	sleep0	2636498	-21	ssh		0
2635233	2	8	0,4	sleep1	2635357	-21	python3		1
2627260	2	8	0,1	sleep2	0	-21	swapper/2		2
2607893	2	8	1,4	sleep2	2609891	-21	cut		2
2590572	2	8	1,4	sleep2	2590645	-21	apt-config		2
2572004	2	8	1,4	sleep2	2572147	-21	df		2
2546899	2	8	0,4	sleep0	2546932	-21	apt-get		0
2546489	2	8	0,5	sleep0	72	-21	kcompactd0		0
2515506	2	8	0,4	sleep2	2515661	-21	diskstats		2
2496780	2	8	0,5	sleep1	2496961	-21	grep		1
2484449	2	8	0,5	sleep2	2485124	-21	cut		2
2477324	2	8	0,2	sleep2	0	-21	swapper/2		2
2452447	2	8	1,3	sleep2	2452480	-21	apt-get		2
2444057	2	8	1,4	sleep0	2444089	-21	apt		0
1908	98	8	0,5	rtkit-daemon	674	-21	lldpd		0
1908	98	8	0,5	rtkit-daemon	1928	-21	sendmail-mta		0
1908	98	8	0,5	rtkit-daemon	1928	-21	sendmail-mta		0
1908	98	8	0,4	rtkit-daemon	667	-21	snmpd		2
1908	98	8	0,4	rtkit-daemon	667	-21	snmpd		2
1908	98	8	0,4	rtkit-daemon	667	-21	snmpd		0
1908	98	8	0,4	rtkit-daemon	2902531	-21	ntp_states		2
1908	98	8	0,4	rtkit-daemon	2644512	-21	ssh		0
1908	98	8	0,4	rtkit-daemon	2603590	-21	ssh		0
1908	98	8	0,4	rtkit-daemon	2597114	-21	diskmemload		0
1908	98	8	0,4	rtkit-daemon	2534333	-21	apt-get		0
1908	98	8	0,4	rtkit-daemon	2450266	-21	munin-node		0
1908	98	8	0,3	rtkit-daemon	2782937	-21	ssh		0
1908	98	8	0,2	rtkit-daemon	2772477	-21	ssh		0
1908	98	8	0,1	rtkit-daemon	0	-21	swapper/0		0
1908	98	8	0,1	rtkit-daemon	0	-21	swapper/0		0
2911427	2	7	0,4	sleep0	2911435	-21	ssh		0
2863120	2	7	0,4	sleep2	2863128	-21	ssh		2
2848970	2	7	0,4	sleep1	2849229	-21	cut		1
2767069	2	7	1,3	sleep2	2767302	-21	latency_hist		2
2628383	2	7	1,4	sleep1	2628385	-21	sh		1
2604786	2	7	0,1	sleep2	0	-21	swapper/2		2
2578686	2	7	1,2	sleep2	0	-21	swapper/2		2
2578686	2	7	1,2	sleep2	0	-21	swapper/2		2
2560906	2	7	1,3	sleep1	2562222	-21	kthreadcore		1
2533190	2	7	0,4	sleep0	2533312	-21	grep		0
2531682	2	7	0,4	sleep1	2532981	-21	memory		1
2471691	2	7	0,4	sleep1	2471829	-21	sed		1
2471520	2	7	0,4	sleep0	2471687	-21	sed		0
1908	98	7	0,4	rtkit-daemon	667	-21	snmpd		0
1908	98	7	0,4	rtkit-daemon	473	-21	gmain		0
1908	98	7	0,4	rtkit-daemon	2539970	-21	kworker/0:1+kacpid		0
1908	98	7	0,4	rtkit-daemon	1928	-21	sendmail-mta		1
1908	98	7	0,3	rtkit-daemon	667	-21	snmpd		0
1908	98	7	0,3	rtkit-daemon	667	-21	snmpd		0
1908	98	7	0,3	rtkit-daemon	473	-21	gmain		2
1908	98	7	0,3	rtkit-daemon	438	-21	wpa_supplicant		0
1908	98	7	0,3	rtkit-daemon	2645084	-21	ssh		2
1908	98	7	0,3	rtkit-daemon	2597114	-21	diskmemload		0
1908	98	7	0,3	rtkit-daemon	2450266	-21	munin-node		0
1908	98	7	0,3	rtkit-daemon	2450266	-21	munin-node		0
1908	98	7	0,3	rtkit-daemon	2450266	-21	munin-node		0
1908	98	7	0,3	rtkit-daemon	1928	-21	sendmail-mta		0
1908	98	7	0,3	rtkit-daemon	0	-21	swapper/2		2
2919588	2	6	0,1	sleep0	0	-21	swapper/0		0
2912877	2	6	0,1	sleep1	0	-21	swapper/1		1
2904349	2	6	0,1	sleep0	0	-21	swapper/0		0
2897235	2	6	0,1	sleep1	0	-21	swapper/1		1
2894207	2	6	0,3	sleep0	2894283	-21	missed_timers		0
2849230	2	6	0,4	sleep2	2850155	-21	kthreadcore		2
2830096	2	6	0,1	sleep2	0	-21	swapper/2		2
2822375	2	6	0,1	sleep2	0	-21	swapper/2		2
2816586	2	6	0,3	sleep1	2817527	-21	cut		1
2814475	2	6	0,4	sleep1	0	-21	swapper/1		1
2781999	2	6	0,1	sleep1	0	-21	swapper/1		1
2775477	2	6	1,2	sleep2	0	-21	swapper/2		2
2759376	2	6	0,1	sleep1	0	-21	swapper/1		1
2758694	2	6	0,1	sleep2	0	-21	swapper/2		2
2758624	2	6	0,1	sleep1	0	-21	swapper/1		1
2742993	2	6	0,1	sleep2	0	-21	swapper/2		2
2724022	2	6	0,1	sleep1	0	-21	swapper/1		1
2715727	2	6	1,2	sleep1	0	-21	swapper/1		1
2709392	2	6	0,1	sleep2	0	-21	swapper/2		2
2699070	2	6	1,2	sleep1	0	-21	swapper/1		1
2691760	2	6	0,4	sleep1	2691839	-21	ps		1
2680722	2	6	2,1	sleep0	0	-21	swapper/0		0
2660949	2	6	0,1	sleep2	0	-21	swapper/2		2
2654191	2	6	0,3	sleep0	2654307	-21	grep		0
2610899	2	6	0,3	sleep1	2610962	-21	ntp_states		1
2565690	2	6	0,2	sleep1	2565723	-21	apt-get		1
2552671	2	6	0,1	sleep1	0	-21	swapper/1		1
2521093	2	6	0,1	sleep1	0	-21	swapper/1		1
2502907	2	6	0,3	sleep1	2502940	-21	apt-get		1
2496244	2	6	1,3	sleep2	667	-21	snmpd		2
2488350	2	6	0,4	sleep1	2489058	-21	grep		1
2478551	2	6	0,3	sleep1	2479448	-21	cat		1
1908	98	6	0,3	rtkit-daemon	671	-21	ntpd		1
1908	98	6	0,3	rtkit-daemon	473	-21	gmain		2
1908	98	6	0,3	rtkit-daemon	2450266	-21	munin-node		0
2881018	2	5	0,3	sleep0	2881022	-21	ssh		0
2879790	2	5	0,1	sleep0	0	-21	swapper/0		0
2838370	2	5	0,3	sleep1	0	-21	swapper/1		1
2810633	2	5	0,2	sleep2	2812768	-21	kthreadcore		2
2806655	2	5	0,3	sleep2	0	-21	swapper/2		2
2797706	2	5	0,1	sleep1	0	-21	swapper/1		1
2789408	2	5	0,3	sleep2	0	-21	swapper/2		2
2744576	2	5	0,2	sleep1	0	-21	swapper/1		1
2742234	2	5	0,2	sleep1	2742242	-21	ssh		1
2721619	2	5	0,3	sleep2	2723725	-21	cut		2
2685496	2	5	0,3	sleep1	2685504	-21	ssh		1
2674484	2	5	0,2	sleep0	0	-21	swapper/0		0
2644413	2	5	0,2	sleep1	2644421	-21	ssh		1
2604944	2	5	1,2	sleep1	2597114	-21	diskmemload		1
2596870	2	5	0,1	sleep2	0	-21	swapper/2		2
2565357	2	5	0,1	sleep2	0	-21	swapper/2		2
2534300	2	5	0,2	sleep2	0	-21	swapper/2		2
2524165	2	5	0,3	sleep1	2526281	-21	cut		1
2510660	2	5	0,3	sleep1	2512466	-21	cut		1
2508776	2	5	0,1	sleep2	0	-21	swapper/2		2
2495285	2	5	0,2	sleep1	0	-21	swapper/1		1
2465123	2	5	1,2	sleep1	32	1	rcuc/1		1
2444933	2	5	0,3	sleep1	2445828	-21	cat		1
2443752	2	5	0,2	sleep2	0	-21	swapper/2		2
2443641	2	5	0,1	sleep1	0	-21	swapper/1		1
2931141	2	4	0,1	sleep1	0	-21	swapper/1		1
2867105	2	4	0,2	sleep0	0	-21	swapper/0		0
2783315	2	4	0,2	sleep1	2783520	-21	ssh		1
2620627	2	4	0,2	sleep1	2620635	-21	ssh		1
2614154	2	4	0,2	sleep2	0	-21	swapper/2		2
2583568	2	4	0,2	sleep1	2583681	-21	timerandwakeup		1
2583568	2	4	0,2	sleep1	2583681	-21	timerandwakeup		1
2567143	2	4	0,2	sleep2	0	-21	swapper/2		2
2559101	2	4	0,3	sleep1	0	-21	swapper/1		1
2549572	2	4	0,2	sleep2	0	-21	swapper/2		2
2542308	2	4	0,2	sleep1	2544322	-21	kthreadcore		1
2528369	2	4	0,2	sleep2	0	-21	swapper/2		2
2496586	2	4	0,1	sleep2	0	-21	swapper/2		2
2459752	2	4	0,2	sleep2	0	-21	swapper/2		2
2445830	2	4	0,2	sleep2	0	-21	swapper/2		2
2727117	2	3	0,1	sleep1	0	-21	swapper/1		1
2584734	2	3	0,1	sleep1	0	-21	swapper/1		1
2539242	2	3	0,1	sleep1	0	-21	swapper/1		1
2458771	2	3	0,1	sleep1	2458804	-21	apt-get		1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2022-03-28 - 20:20