OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack3slot6.osadl.org (updated Wed Sep 09, 2026 00:45:04)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
26661	2	125	0,8	sleep1	25	1	rcuc/1	21:25:34	1
10018	99	99	20,50	cyclictest	0	-21	swapper/0	23:50:12	0
10018	99	94	12,74	cyclictest	0	-21	swapper/0	20:10:16	0
9642	2	93	21,27	sleep0	0	-21	swapper/0	19:07:59	0
9642	2	93	21,27	sleep0	0	-21	swapper/0	19:07:58	0
10018	99	93	50,35	cyclictest	14	-21	ksoftirqd/0	21:20:15	0
10018	99	92	46,35	cyclictest	14	-21	ksoftirqd/0	20:40:13	0
10018	99	90	16,60	cyclictest	0	-21	swapper/0	19:40:16	0
10018	99	89	9,71	cyclictest	20577	-21	apt-get	21:10:14	0
10018	99	89	15,59	cyclictest	0	-21	swapper/0	21:00:16	0
10018	99	88	25,56	cyclictest	13920	-21	cron	19:20:00	0
9661	2	87	30,47	sleep1	0	-21	swapper/1	19:08:12	1
9661	2	87	30,47	sleep1	0	-21	swapper/1	19:08:12	1
10018	99	87	15,61	cyclictest	0	-21	swapper/0	23:25:13	0
10018	99	87	14,59	cyclictest	0	-21	swapper/0	22:25:13	0
10018	99	87	13,65	cyclictest	0	-21	swapper/0	00:25:14	0
10018	99	86	15,61	cyclictest	0	-21	swapper/0	00:00:14	0
10018	99	86	15,58	cyclictest	0	-21	swapper/0	00:35:16	0
10018	99	86	13,62	cyclictest	0	-21	swapper/0	21:50:16	0
10018	99	85	6,20	cyclictest	467	-21	apt-get	23:15:12	0
10018	99	85	22,51	cyclictest	0	-21	swapper/0	21:40:22	0
10018	99	85	14,59	cyclictest	0	-21	swapper/0	21:55:15	0
10018	99	85	13,61	cyclictest	0	-21	swapper/0	23:30:13	0
10018	99	85	13,60	cyclictest	0	-21	swapper/0	00:05:14	0
10018	99	84	5,14	cyclictest	9843	-21	apt-get	22:10:11	0
10018	99	84	3,74	cyclictest	26690	-21	aten2.4_r3power	19:55:16	0
10018	99	84	16,61	cyclictest	0	-21	swapper/0	22:50:19	0
10018	99	84	13,59	cyclictest	0	-21	swapper/0	21:25:14	0
10018	99	84	13,59	cyclictest	0	-21	swapper/0	20:45:15	0
10018	99	84	12,60	cyclictest	0	-21	swapper/0	22:55:12	0
10018	99	83	7,21	cyclictest	8256	-21	apt-get	20:35:12	0
10018	99	83	6,60	cyclictest	4777	-21	apt-get	20:25:14	0
10018	99	83	22,55	cyclictest	453	-21	in:imuxsock	00:20:20	0
10018	99	83	20,56	cyclictest	481	-21	gdbus	21:50:00	0
10018	99	83	15,59	cyclictest	0	-21	swapper/0	22:15:15	0
10018	99	83	15,59	cyclictest	0	-21	swapper/0	19:20:13	0
10018	99	83	15,58	cyclictest	0	-21	swapper/0	23:10:11	0
10018	99	83	15,58	cyclictest	0	-21	swapper/0	23:10:11	0
10018	99	83	14,58	cyclictest	0	-21	swapper/0	19:45:11	0
10018	99	82	9,62	cyclictest	0	-21	swapper/0	23:35:15	0
10018	99	82	15,59	cyclictest	243	-21	systemd-journal	20:20:59	0
10018	99	82	13,59	cyclictest	0	-21	swapper/0	22:40:15	0
10018	99	82	13,59	cyclictest	0	-21	swapper/0	00:30:13	0
10018	99	82	11,61	cyclictest	0	-21	swapper/0	22:45:11	0
10018	99	81	7,19	cyclictest	17596	-21	apt-get	19:30:13	0
10018	99	81	6,21	cyclictest	18865	-21	apt-get	22:35:14	0
10018	99	81	6,19	cyclictest	17124	-21	apt-get	22:30:11	0
10018	99	81	6,19	cyclictest	17124	-21	apt-get	22:30:11	0
10018	99	81	6,19	cyclictest	11040	-21	apt-get	23:45:14	0
10018	99	81	32,41	cyclictest	22768	-21	ls	00:15:45	0
10018	99	81	29,46	cyclictest	8390	-21	tail	22:05:26	0
10018	99	81	15,58	cyclictest	0	-21	swapper/0	23:05:13	0
10018	99	81	14,59	cyclictest	0	-21	swapper/0	00:10:13	0
10018	99	81	13,55	cyclictest	0	-21	swapper/0	23:40:15	0
10018	99	81	13,53	cyclictest	0	-21	swapper/0	23:55:14	0
10018	99	81	12,58	cyclictest	0	-21	swapper/0	19:10:15	0
12951	2	80	0,7	chrt	0	-21	swapper/1	20:45:39	1
10018	99	80	7,23	cyclictest	28430	-21	apt-get	20:00:13	0
10018	99	80	7,23	cyclictest	28430	-21	apt-get	20:00:13	0
10018	99	80	7,19	cyclictest	27653	-21	apt-get	23:00:14	0
10018	99	80	7,19	cyclictest	18783	-21	apt-get	21:05:11	0
10018	99	80	24,46	cyclictest	10009	-21	cyclictest	20:50:14	0
10018	99	80	13,56	cyclictest	0	-21	swapper/0	19:25:14	0
10018	99	79	8,22	cyclictest	19638	-21	apt-get	19:35:23	0
10018	99	79	7,22	cyclictest	6014	-21	apt-get	22:00:14	0
10018	99	79	6,20	cyclictest	22315	-21	apt-get	21:15:15	0
10018	99	79	6,19	cyclictest	30168	-21	apt-get	20:05:12	0
10018	99	79	19,54	cyclictest	29705	-21	cat	21:35:23	0
10018	99	79	19,54	cyclictest	29705	-21	cat	21:35:23	0
10018	99	79	15,57	cyclictest	27647	-21	aten2.4_r3power	21:30:20	0
10020	99	78	3,45	cyclictest	0	-21	swapper/1	23:45:22	1
10018	99	78	8,20	cyclictest	15286	-21	apt-get	20:55:14	0
10018	99	78	6,24	cyclictest	24929	-21	apt-get	19:50:14	0
10018	99	78	6,19	cyclictest	6517	-21	apt-get	20:30:15	0
10018	99	77	7,20	cyclictest	1269	-21	apt-get	20:15:15	0
10018	99	77	6,22	cyclictest	2264	-21	apt-get	23:20:14	0
10018	99	75	15,48	cyclictest	0	-21	swapper/0	22:20:13	0
10020	99	74	24,21	cyclictest	20404	-21	diskmemload	21:10:32	1
10020	99	72	27,39	cyclictest	503	-21	snmpd	23:36:42	1
10020	99	71	29,36	cyclictest	5626	-21	seq	21:58:00	1
10020	99	71	22,21	cyclictest	16101	-21	ntpq	22:25:28	1
10020	99	70	30,20	cyclictest	30201	-21	seq	00:37:44	1
10020	99	70	28,37	cyclictest	16407	-21	telnet	00:00:17	1
10020	99	66	36,24	cyclictest	455	-21	rs:main		1
10020	99	66	29,32	cyclictest	30200	-21	python3	21:35:35	1
10020	99	66	29,32	cyclictest	30200	-21	python3	21:35:35	1
10020	99	66	22,23	cyclictest	24887	-21	ntp_states	22:50:34	1
10020	99	66	21,18	cyclictest	25260	-21	sshd	19:50:20	1
10020	99	65	45,15	cyclictest	13629	-21	python3	23:50:34	1
10020	99	65	26,32	cyclictest	26945	-21	latency_hist	00:30:01	1
10020	99	65	2,34	cyclictest	0	-21	swapper/1	21:30:28	1
10020	99	63	29,28	cyclictest	22324	-21	ls	22:45:01	1
10020	99	62	8,7	cyclictest	15	1	rcu_preempt	22:00:37	1
10020	99	61	35,20	cyclictest	3541	-21	grep	21:50:38	1
10020	99	61	34,21	cyclictest	12864	-21	ls	22:15:38	1
10020	99	61	32,23	cyclictest	26241	-21	taskset	22:55:21	1
10020	99	61	31,24	cyclictest	12364	-21	chrt	19:15:16	1
10020	99	60	34,21	cyclictest	18700	-21	unixbench_singl	19:30:35	1
10020	99	60	34,21	cyclictest	1149	-21	sort	21:45:22	1
10020	99	60	34,20	cyclictest	23118	-21	ntpq	22:45:27	1
10020	99	60	34,20	cyclictest	16965	-21	unixbench_singl	19:25:36	1
10020	99	60	33,21	cyclictest	562	-21	runrttasks	19:55:33	1
10020	99	60	32,22	cyclictest	28216	-21	unixbench_singl	00:30:34	1
10020	99	60	32,22	cyclictest	18533	-21	sed	00:05:26	1
10020	99	60	31,23	cyclictest	6037	-21	cstates	23:30:19	1
10020	99	60	30,24	cyclictest	15733	-21	taskset	23:56:21	1
10020	99	60	27,26	cyclictest	2999	-21	latency_hist	20:20:01	1
10020	99	60	26,29	cyclictest	4898	-21	aten2.4_r3power	20:25:18	1
10020	99	59	34,19	cyclictest	4705	-21	ntp_kernel_pll_	23:25:29	1
10020	99	59	33,20	cyclictest	23360	-21	unixbench_multi	21:15:38	1
10020	99	59	32,22	cyclictest	503	-21	snmpd	19:10:42	1
10020	99	59	32,21	cyclictest	24550	-21	runrttasks	21:20:27	1
10020	99	59	32,21	cyclictest	21351	-21	latency_hist	19:40:01	1
10020	99	59	32,21	cyclictest	10180	-21	cut	20:40:17	1
10020	99	59	31,22	cyclictest	10000	-21	grep	23:40:29	1
10020	99	59	31,21	cyclictest	24396	-21	seq	19:46:13	1
10020	99	59	30,23	cyclictest	8991	-21	chrt	20:35:30	1
10020	99	59	30,23	cyclictest	29728	-21	df	23:05:19	1
10020	99	59	30,23	cyclictest	19115	-21	chrt	21:05:19	1
10020	99	59	29,24	cyclictest	17121	-21	/usr/sbin/munin	22:30:24	1
10020	99	59	29,24	cyclictest	17121	-21	/usr/sbin/munin	22:30:24	1
10020	99	59	27,27	cyclictest	2993	-21	ntp_states	23:20:32	1
10020	99	58	32,20	cyclictest	3650	-21	tr	20:20:26	1
10020	99	58	32,20	cyclictest	32245	-21	cstates	20:10:20	1
10020	99	58	32,20	cyclictest	31178	-21	/usr/sbin/munin	23:10:22	1
10020	99	58	32,20	cyclictest	31178	-21	/usr/sbin/munin	23:10:22	1
10020	99	58	31,22	cyclictest	28495	-21	aten2.4-expect	20:00:17	1
10020	99	58	31,22	cyclictest	28495	-21	aten2.4-expect	20:00:17	1
10020	99	58	31,21	cyclictest	7485	-21	cat	20:30:37	1
10020	99	58	31,21	cyclictest	20181	-21	seq	22:37:23	1
10020	99	58	31,21	cyclictest	14844	-21	taskset	22:21:05	1
10020	99	58	30,22	cyclictest	562	-21	runrttasks	00:14:50	1
10020	99	58	30,22	cyclictest	31636	-21	aten2.4_r3power	21:40:26	1
10020	99	58	30,22	cyclictest	25232	-21	ls	00:25:01	1
10020	99	58	30,21	cyclictest	31847	-21	date	20:10:01	1
10020	99	58	30,21	cyclictest	15632	-21	chrt	19:24:31	1
10020	99	58	29,22	cyclictest	562	-21	runrttasks	22:13:02	1
10020	99	58	28,23	cyclictest	8171	-21	if_enp3s0	22:05:22	1
10020	99	57	6,45	cyclictest	0	-21	swapper/1	20:50:16	1
10020	99	57	32,19	cyclictest	28628	-21	timerwakeupswit	23:00:36	1
10020	99	57	30,22	cyclictest	562	-21	runrttasks	21:00:30	1
10020	99	57	30,21	cyclictest	21772	-21	touch	00:15:05	1
10020	99	57	30,21	cyclictest	21731	-21	cat	19:40:19	1
10020	99	57	28,23	cyclictest	16267	-21	cat	20:55:36	1
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17