OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #9, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack9slot2.osadl.org (updated Sat Nov 29, 2025 12:43:25)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
15281	2	323	262,49	sleep2	0	-21	swapper/2		2
15380	2	272	207,33	sleep0	0	-21	swapper/0		0
15660	2	256	200,43	sleep3	0	-21	swapper/3		3
13888	2	248	209,26	sleep1	0	-21	swapper/1		1
15841	99	223	219,2	cyclictest	0	-21	swapper/3		3
15825	99	214	208,4	cyclictest	4053	-21	strings		0
15836	99	206	202,2	cyclictest	0	-21	swapper/2		2
15833	99	183	179,2	cyclictest	0	-21	swapper/1		1
32039	2	157	0,2	sleep3	0	-21	swapper/3		3
15825	99	50	1,34	cyclictest	11270	-21	apt-get		0
15825	99	47	2,29	cyclictest	17802	-21	apt-get		0
15825	99	46	11,24	cyclictest	9	-21	ksoftirqd/0		0
5212	2	45	0,1	sleep3	0	-21	swapper/3		3
30336	2	45	0,1	sleep0	0	-21	swapper/0		0
15825	99	45	2,26	cyclictest	31619	-21	apt-get		0
15825	99	45	2,26	cyclictest	31619	-21	apt-get		0
15825	99	43	1,3	cyclictest	21898	1	latency_hist		0
16622	2	41	0,1	sleep0	0	-21	swapper/0		0
16622	2	41	0,1	sleep0	0	-21	swapper/0		0
15836	99	39	0,25	cyclictest	6831	-21	taskset		2
15825	99	38	0,3	cyclictest	8942	-21	apt-get		0
15825	99	37	1,3	cyclictest	27723	-21	sed		0
15825	99	37	0,3	cyclictest	30453	-21	ntpq		0
15841	99	35	13,5	cyclictest	505	-21	dbus-daemon		3
15841	99	35	13,4	cyclictest	21870	-21	expr		3
15836	99	35	0,25	cyclictest	0	-21	swapper/2		2
15825	99	35	3,26	cyclictest	5285	-21	apt-get		0
15825	99	35	2,26	cyclictest	24909	-21	apt-get		0
15825	99	34	2,26	cyclictest	15350	-21	apt-get		0
15825	99	34	12,20	cyclictest	4368	-21	seq		0
15825	99	34	12,20	cyclictest	16694	-21	sed		0
15825	99	34	11,20	cyclictest	27873	-21	cat		0
15825	99	34	0,4	cyclictest	25635	-21	taskset		0
15836	99	33	11,9	cyclictest	0	-21	swapper/2		2
15836	99	33	0,30	cyclictest	10922	-21	cat		2
15825	99	33	2,25	cyclictest	8716	-21	apt-get		0
15825	99	33	1,28	cyclictest	6430	-21	perf		0
15825	99	33	12,18	cyclictest	23512	-21	aten2_r9power_c		0
15825	99	33	11,19	cyclictest	9636	-21	grep		0
15825	99	33	10,20	cyclictest	28170	-21	grep		0
15841	99	32	14,3	cyclictest	22241	-21	sed		3
15841	99	32	12,4	cyclictest	7172	-21	date		3
15836	99	32	15,14	cyclictest	10486	-21	munin-run		2
15836	99	32	0,3	cyclictest	29655	-21	/usr/sbin/munin		2
15836	99	32	0,29	cyclictest	6063	-21	latency_hist		2
15833	99	32	29,2	cyclictest	20	1	rcuc/1		1
15825	99	32	2,24	cyclictest	28253	-21	apt-get		0
15825	99	32	2,23	cyclictest	21411	-21	/usr/sbin/munin		0
15825	99	32	1,26	cyclictest	6688	-21	apt-get		0
15825	99	32	1,25	cyclictest	18899	-21	apt-get		0
15825	99	32	12,18	cyclictest	17383	-21	/usr/sbin/munin		0
15825	99	32	12,18	cyclictest	17383	-21	/usr/sbin/munin		0
15825	99	32	10,19	cyclictest	21159	-21	irqstats		0
15841	99	31	1,9	cyclictest	28003	-21	munin-run		3
15841	99	31	12,3	cyclictest	17579	-21	cpuspeed		3
15841	99	31	12,3	cyclictest	17579	-21	cpuspeed		3
15841	99	31	12,3	cyclictest	11644	-21	expr		3
15841	99	31	0,29	cyclictest	23877	-21	kworker/3:2		3
15836	99	31	15,14	cyclictest	22811	-21	irqstats		2
15836	99	31	15,13	cyclictest	1248	-21	aten2_r9power_e		2
15836	99	31	13,15	cyclictest	2352	-21	munin-run		2
15836	99	31	1,27	cyclictest	21191	-21	munin-run		2
15836	99	31	1,23	cyclictest	21477	-21	apt-get		2
15836	99	31	0,19	cyclictest	0	-21	swapper/2		2
15836	99	31	0,18	cyclictest	0	-21	swapper/2		2
15825	99	31	8,20	cyclictest	2499	-21	grep		0
15825	99	31	4,24	cyclictest	11397	-21	mailstats		0
15825	99	31	2,26	cyclictest	29198	-21	cpuspeed_turbos		0
15825	99	31	2,24	cyclictest	807	-21	apt-get		0
15825	99	31	2,24	cyclictest	7304	-21	apt-get		0
15825	99	31	2,23	cyclictest	6909	-21	apt-get		0
15825	99	31	2,23	cyclictest	26230	-21	apt-get		0
15825	99	31	2,23	cyclictest	16871	-21	apt-get		0
15825	99	31	2,23	cyclictest	11373	-21	apt-get		0
15825	99	31	1,4	cyclictest	18502	-21	apt-get		0
15825	99	31	1,25	cyclictest	22883	-21	apt-get		0
15825	99	31	1,23	cyclictest	22180	-21	apt-get		0
15825	99	31	0,3	cyclictest	2788	-21	taskset		0
15825	99	31	0,1	cyclictest	32627	-21	apt-get		0
15825	99	31	0,1	cyclictest	32627	-21	apt-get		0
15841	99	30	13,4	cyclictest	1183	-21	/usr/sbin/munin		3
15841	99	30	12,4	cyclictest	505	-21	dbus-daemon		3
15841	99	30	12,3	cyclictest	18843	-21	expr		3
15841	99	30	11,4	cyclictest	25766	-21	cpuspeed_turbos		3
15841	99	30	11,3	cyclictest	28012	-21	awk		3
15836	99	30	1,26	cyclictest	302	-21	munin-run		2
15836	99	30	12,15	cyclictest	6001	-21	sed		2
15836	99	30	12,15	cyclictest	14696	-21	grep		2
15836	99	30	0,3	cyclictest	13073	-21	munin-run		2
15825	99	30	3,22	cyclictest	20926	-21	apt-get		0
15825	99	30	3,21	cyclictest	12375	-21	apt-get		0
15825	99	30	2,23	cyclictest	6061	-21	sendmail-mta		0
15825	99	30	2,23	cyclictest	14429	-21	apt-get		0
15825	99	30	2,23	cyclictest	13326	-21	apt-get		0
15825	99	30	2,22	cyclictest	25527	-21	apt-get		0
15825	99	30	2,22	cyclictest	18623	-21	apt-get		0
15825	99	30	0,3	cyclictest	1620	-21	gzip		0
15841	99	29	13,3	cyclictest	9372	-21	sh		3
15836	99	29	12,15	cyclictest	17763	-21	/usr/sbin/munin		2
15836	99	29	12,14	cyclictest	5716	-21	nfs4_client		2
15836	99	29	0,4	cyclictest	29592	-21	irqstats		2
15836	99	29	0,4	cyclictest	25328	-21	sh		2
15836	99	29	0,3	cyclictest	29283	-21	taskset		2
15833	99	29	3,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	29	2,7	cyclictest	21	-21	ksoftirqd/1		1
15825	99	29	8,18	cyclictest	19660	-21	grep		0
15825	99	29	8,18	cyclictest	18237	-21	latency_hist		0
15825	99	29	4,22	cyclictest	6245	-21	sed		0
15825	99	29	2,22	cyclictest	29208	-21	apt-get		0
15825	99	29	2,22	cyclictest	22999	-21	apt-get		0
15825	99	29	1,24	cyclictest	8081	-21	awk		0
15841	99	28	11,3	cyclictest	826	-21	cpuspeed_turbos		3
15836	99	28	4,21	cyclictest	0	-21	swapper/2		2
15836	99	28	2,21	cyclictest	12772	-21	apt-get		2
15836	99	28	2,21	cyclictest	12772	-21	apt-get		2
15836	99	28	1,24	cyclictest	24706	-21	grep		2
15836	99	28	12,13	cyclictest	23299	-21	sh		2
15836	99	28	11,14	cyclictest	8075	-21	uptime		2
15836	99	28	11,14	cyclictest	19275	-21	expr		2
15836	99	28	0,3	cyclictest	3570	-21	memory		2
15836	99	28	0,3	cyclictest	27965	-21	rt-features		2
15836	99	28	0,3	cyclictest	16251	-21	ntp_states		2
15836	99	28	0,2	cyclictest	17354	-21	/usr/sbin/munin		2
15825	99	28	7,18	cyclictest	15005	-21	cpuspeed		0
15825	99	28	1,24	cyclictest	25321	-21	cpuspeed_turbos		0
15841	99	27	3,8	cyclictest	15761	-21	apt-get		3
15841	99	27	3,8	cyclictest	15761	-21	apt-get		3
15841	99	27	2,8	cyclictest	28959	-21	apt-get		3
15836	99	27	6,18	cyclictest	3986	-21	sed		2
15836	99	27	1,21	cyclictest	31225	-21	apt-get		2
15836	99	27	1,20	cyclictest	9339	-21	apt-get		2
15836	99	27	1,20	cyclictest	25030	-21	apt-get		2
15836	99	27	11,14	cyclictest	7671	-21	cat		2
15836	99	27	11,14	cyclictest	17735	-21	ls		2
15836	99	27	11,14	cyclictest	17735	-21	ls		2
15836	99	27	11,13	cyclictest	20437	-21	cut		2
15836	99	27	10,14	cyclictest	24054	-21	if_enp2s0		2
15836	99	27	10,14	cyclictest	10405	-21	cut		2
15836	99	27	0,3	cyclictest	29332	-21	aten2.4-expect		2
15836	99	27	0,3	cyclictest	13007	-21	irqstats		2
15836	99	27	0,2	cyclictest	22590	2	sleep2		2
15836	99	27	0,24	cyclictest	12498	-21	switchtime		2
15833	99	27	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	27	1,7	cyclictest	21	-21	ksoftirqd/1		1
15833	99	27	1,7	cyclictest	21	-21	ksoftirqd/1		1
15825	99	27	9,15	cyclictest	665	-21	runrttasks		0
15825	99	27	9,15	cyclictest	665	-21	runrttasks		0
15825	99	27	4,19	cyclictest	11306	-21	latency_hist		0
15841	99	26	5,6	cyclictest	19394	-21	gltestperf		3
15836	99	26	9,15	cyclictest	30640	-21	systemctl		2
15836	99	26	5,19	cyclictest	20441	-21	expr		2
15836	99	26	2,21	cyclictest	32260	-21	uname		2
15836	99	26	2,21	cyclictest	32260	-21	uname		2
15836	99	26	10,13	cyclictest	1040	-21	ntp_offset		2
15836	99	26	10,13	cyclictest	1040	-21	ntp_offset		2
15836	99	26	0,2	cyclictest	0	-21	swapper/2		2
15836	99	26	0,23	cyclictest	30737	-21	/usr/sbin/munin		2
15836	99	26	0,23	cyclictest	25661	-21	cpu		2
15836	99	26	0,23	cyclictest	18671	-21	ntp_states		2
15833	99	26	23,2	cyclictest	20	1	rcuc/1		1
15825	99	26	1,23	cyclictest	32500	-21	grep		0
15825	99	26	1,21	cyclictest	11990	-21	timerwakeupswit		0
15836	99	25	9,14	cyclictest	11891	-21	threads		2
15836	99	25	7,16	cyclictest	14551	-21	aten2_r9power_v		2
15836	99	25	1,20	cyclictest	16474	-21	apt-get		2
15836	99	25	0,4	cyclictest	22374	-21	cat		2
15836	99	25	0,3	cyclictest	24597	-21	taskset		2
15836	99	25	0,2	cyclictest	27994	-21	pmu-power		2
15836	99	25	0,22	cyclictest	7214	-21	cut		2
15833	99	25	22,2	cyclictest	20	1	rcuc/1		1
15833	99	25	21,3	cyclictest	20	1	rcuc/1		1
15833	99	25	0,6	cyclictest	20	1	rcuc/1		1
15825	99	25	5,17	cyclictest	0	-21	swapper/0		0
15825	99	25	1,6	cyclictest	21752	-21	tr		0
15825	99	25	0,22	cyclictest	32343	-21	tail		0
7356	2	24	0,2	sleep3	0	-21	swapper/3		3
29407	2	24	0,2	sleep1	0	-21	swapper/1		1
15841	99	24	3,6	cyclictest	4246	-21	apt-get		3
15841	99	24	16,1	cyclictest	0	-21	swapper/3		3
15836	99	24	1,20	cyclictest	26321	-21	aten2.4-expect		2
15836	99	24	1,19	cyclictest	8319	-21	apt-get		2
15836	99	24	0,1	cyclictest	0	-21	swapper/2		2
15833	99	24	3,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	24	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	24	21,2	cyclictest	20	1	rcuc/1		1
15833	99	24	21,2	cyclictest	20	1	rcuc/1		1
15833	99	24	20,3	cyclictest	20	1	rcuc/1		1
15833	99	24	0,6	cyclictest	21	-21	ksoftirqd/1		1
15836	99	23	1,19	cyclictest	19358	-21	wc		2
15836	99	23	1,19	cyclictest	19358	-21	wc		2
15833	99	23	6,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	23	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	23	2,5	cyclictest	21	-21	ksoftirqd/1		1
15833	99	23	20,2	cyclictest	20	1	rcuc/1		1
15833	99	23	20,2	cyclictest	20	1	rcuc/1		1
15833	99	23	1,6	cyclictest	21	-21	ksoftirqd/1		1
15841	99	22	1,8	cyclictest	22333	-21	sendmail		3
15841	99	22	17,3	cyclictest	26815	-21	perf		3
15833	99	22	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	4,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	3,11	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	3,11	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	2,5	cyclictest	21	-21	ksoftirqd/1		1
15833	99	22	19,2	cyclictest	20	1	rcuc/1		1
15833	99	22	18,2	cyclictest	20	1	rcuc/1		1
15833	99	22	0,13	cyclictest	21	-21	ksoftirqd/1		1
15841	99	21	14,4	cyclictest	32081	-21	basename		3
15841	99	21	12,5	cyclictest	505	-21	dbus-daemon		3
15833	99	21	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	5,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	4,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	4,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	4,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	4,9	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	18,2	cyclictest	20	1	rcuc/1		1
15833	99	21	1,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	1,5	cyclictest	21	-21	ksoftirqd/1		1
15833	99	21	14,4	cyclictest	2244	-21	seq		1
15833	99	21	14,4	cyclictest	17924	-21	seq		1
15833	99	21	12,2	cyclictest	11	1	rcu_preempt		1
15833	99	21	12,2	cyclictest	11	1	rcu_preempt		1
15833	99	21	0,18	cyclictest	6296	-21	seq		1
15841	99	20	15,3	cyclictest	604	-21	sendmail-mta		3
15841	99	20	15,3	cyclictest	5651	-21	missed_timers		3
15841	99	20	14,4	cyclictest	505	-21	dbus-daemon		3
15841	99	20	10,6	cyclictest	6293	-21	apt-get		3
15833	99	20	3,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	20	2,7	cyclictest	21	-21	ksoftirqd/1		1
15833	99	20	2,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	20	2,5	cyclictest	21	-21	ksoftirqd/1		1
15833	99	20	2,5	cyclictest	0	-21	swapper/1		1
15833	99	20	17,2	cyclictest	20	1	rcuc/1		1
15833	99	20	17,2	cyclictest	20	1	rcuc/1		1
15833	99	20	17,2	cyclictest	20	1	rcuc/1		1
15833	99	20	1,6	cyclictest	21	-21	ksoftirqd/1		1
15833	99	20	15,3	cyclictest	20003	-21	seq		1
15833	99	20	14,4	cyclictest	22365	-21	cat		1
15833	99	20	11,2	cyclictest	11	1	rcu_preempt		1
15833	99	20	0,7	cyclictest	13780	-21	runrttasks		1
15841	99	19	16,2	cyclictest	0	-21	swapper/3		3
15841	99	19	16,2	cyclictest	0	-21	swapper/3		3
15841	99	19	15,2	cyclictest	8185	-21	ntp_states		3
15841	99	19	14,3	cyclictest	300	-21	latency_hist		3
15841	99	19	14,3	cyclictest	27204	-21	expr		3
15841	99	19	14,3	cyclictest	2598	-21	/usr/sbin/munin		3
15841	99	19	14,3	cyclictest	1489	-21	sed		3
15841	99	19	14,3	cyclictest	13190	-21	sed		3
15841	99	19	14,2	cyclictest	31601	-21	/usr/sbin/munin		3
15841	99	19	14,2	cyclictest	31601	-21	/usr/sbin/munin		3
15841	99	19	13,3	cyclictest	13954	-21	interrupts		3
15841	99	19	13,3	cyclictest	0	-21	swapper/3		3
15841	99	19	12,4	cyclictest	6925	-21	cut		3
15841	99	19	11,4	cyclictest	22493	-21	cpuspeed_turbos		3
15841	99	19	10,5	cyclictest	14825	-21	apt-get		3
15833	99	19	14,3	cyclictest	9135	-21	sh		1
15833	99	19	0,6	cyclictest	31229	-21	runrttasks		1
15841	99	18	3,9	cyclictest	10751	-21	apt-get		3
15841	99	18	16,1	cyclictest	0	-21	swapper/3		3
15841	99	18	16,1	cyclictest	0	-21	swapper/3		3
15841	99	18	16,1	cyclictest	0	-21	swapper/3		3
15841	99	18	14,2	cyclictest	482	-21	cstates		3
15841	99	18	14,2	cyclictest	482	-21	cstates		3
15841	99	18	13,3	cyclictest	9197	-21	ntp_states		3
15841	99	18	13,3	cyclictest	25218	-21	grep		3
15841	99	18	13,3	cyclictest	22865	-21	/usr/sbin/munin		3
15841	99	18	13,3	cyclictest	20763	-21	irqstats		3
15841	99	18	13,3	cyclictest	16724	-21	sed		3
15841	99	18	12,3	cyclictest	24194	-21	acpi		3
15841	99	18	12,3	cyclictest	18468	-21	processes		3
15841	99	18	11,4	cyclictest	22715	-21	ls		3
15841	99	18	11,4	cyclictest	16765	-21	users		3
15841	99	18	11,4	cyclictest	10333	-21	gdbus		3
15841	99	18	10,5	cyclictest	18090	-21	tr		3
15841	99	18	0,16	cyclictest	25410	-21	sed		3
15841	99	18	0,16	cyclictest	0	-21	swapper/3		3
15833	99	18	0,16	cyclictest	0	-21	swapper/1		1
15841	99	17	15,1	cyclictest	0	-21	swapper/3		3
15841	99	17	12,3	cyclictest	13361	-21	cat		3
15841	99	17	12,3	cyclictest	13361	-21	cat		3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-04-10 - 20:27