OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack3slot6.osadl.org (updated Mon Apr 13, 2026 00:45:01)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
12315	99	84	67,10	cyclictest	14	-21	ksoftirqd/0	21:35:24	0
12315	99	81	11,62	cyclictest	0	-21	swapper/0	23:35:12	0
12315	99	78	7,20	cyclictest	0	-21	swapper/0	21:25:02	0
12315	99	78	60,10	cyclictest	14	-21	ksoftirqd/0	23:25:12	0
12315	99	77	61,11	cyclictest	12406	-21	latency_hist	19:10:02	0
12315	99	77	49,13	cyclictest	14	-21	ksoftirqd/0	20:35:14	0
12315	99	76	27,42	cyclictest	0	-21	swapper/0	23:40:13	0
12315	99	76	15,19	cyclictest	0	-21	swapper/0	19:45:15	0
12024	2	75	16,50	sleep0	0	-21	swapper/0	19:08:54	0
12024	2	75	16,50	sleep0	0	-21	swapper/0	19:08:54	0
12317	99	74	15,18	cyclictest	14257	-21	cron	19:15:00	1
12315	99	72	48,14	cyclictest	14	-21	ksoftirqd/0	20:30:15	0
12317	99	71	12,29	cyclictest	12930	-21	/usr/sbin/munin	23:40:37	1
12315	99	71	11,23	cyclictest	24492	-21	apt-get	21:15:13	0
12315	99	71	11,23	cyclictest	24492	-21	apt-get	21:15:13	0
12317	99	70	7,23	cyclictest	26	-21	ksoftirqd/1	20:39:00	1
12317	99	70	17,15	cyclictest	26	-21	ksoftirqd/1	22:36:52	1
12315	99	70	41,17	cyclictest	14	-21	ksoftirqd/0	19:55:13	0
12315	99	70	38,18	cyclictest	0	-21	swapper/0	19:30:12	0
14958	2	69	0,7	sleep1	25	1	rcuc/1	20:45:34	1
12315	99	69	9,19	cyclictest	16941	-21	apt-get	23:55:13	0
12315	99	69	12,18	cyclictest	562	-21	runrttasks	19:39:03	0
12025	2	69	22,37	sleep1	0	-21	swapper/1	19:08:55	1
12025	2	69	22,37	sleep1	0	-21	swapper/1	19:08:55	1
12317	99	68	13,36	cyclictest	0	-21	swapper/1	22:15:15	1
12315	99	68	41,15	cyclictest	14	-21	ksoftirqd/0	21:05:15	0
12315	99	68	39,15	cyclictest	0	-21	swapper/0	20:15:12	0
12315	99	68	19,42	cyclictest	28404	-21	aten2.4-expect	00:25:17	0
12315	99	67	7,19	cyclictest	0	-21	swapper/0	20:10:01	0
12315	99	67	5,20	cyclictest	0	-21	swapper/0	23:50:15	0
12315	99	67	5,20	cyclictest	0	-21	swapper/0	21:10:18	0
12315	99	67	5,20	cyclictest	0	-21	swapper/0	21:10:18	0
12315	99	67	39,17	cyclictest	0	-21	swapper/0	22:55:11	0
12317	99	66	7,24	cyclictest	26	-21	ksoftirqd/1	23:55:37	1
12315	99	66	41,15	cyclictest	0	-21	swapper/0	19:20:11	0
12317	99	65	9,8	cyclictest	15	1	rcu_preempt	22:25:37	1
12315	99	65	7,20	cyclictest	12364	-21	apt-get	20:40:12	0
12315	99	65	6,10	cyclictest	0	-21	swapper/0	20:59:18	0
12315	99	65	42,12	cyclictest	1941	-21	apt-get	21:40:14	0
12315	99	65	37,15	cyclictest	0	-21	swapper/0	22:00:12	0
12315	99	65	33,23	cyclictest	19343	-21	apt-get	21:00:13	0
12315	99	65	33,20	cyclictest	10650	-21	apt-get	22:05:14	0
12315	99	65	30,25	cyclictest	0	-21	swapper/0	23:30:13	0
12317	99	64	17,42	cyclictest	0	-21	swapper/1	23:15:29	1
12315	99	64	35,15	cyclictest	14183	-21	apt-get	22:15:13	0
12315	99	64	30,24	cyclictest	0	-21	swapper/0	22:10:12	0
12317	99	63	13,24	cyclictest	3184	-21	runrttasks	20:12:50	1
12317	99	63	12,30	cyclictest	0	-21	swapper/1	23:12:22	1
12315	99	63	39,14	cyclictest	0	-21	swapper/0	19:50:13	0
12315	99	63	37,14	cyclictest	28766	-21	apt-get	21:25:23	0
12315	99	63	32,19	cyclictest	21246	-21	apt-get	00:05:23	0
12317	99	62	15,10	cyclictest	0	-21	swapper/1	19:40:20	1
12317	99	62	14,42	cyclictest	25	1	rcuc/1	22:50:41	1
12317	99	62	14,19	cyclictest	25	1	rcuc/1	20:44:07	1
12317	99	62	10,29	cyclictest	0	-21	swapper/1	20:30:42	1
12315	99	62	40,11	cyclictest	3678	-21	apt-get	21:45:14	0
12315	99	62	39,11	cyclictest	30250	-21	apt-get	23:00:13	0
12315	99	62	37,17	cyclictest	14	-21	ksoftirqd/0	20:50:21	0
12315	99	62	33,13	cyclictest	14103	-21	apt-get	20:45:13	0
12315	99	62	31,22	cyclictest	0	-21	swapper/0	21:55:13	0
12315	99	62	30,23	cyclictest	0	-21	swapper/0	22:20:14	0
12315	99	62	30,18	cyclictest	13631	-21	apt-get	23:45:13	0
12317	99	61	8,14	cyclictest	26	-21	ksoftirqd/1	19:50:29	1
12317	99	61	13,20	cyclictest	25	1	rcuc/1	21:22:50	1
12315	99	61	7,42	cyclictest	19209	-21	apt-get	00:00:13	0
12315	99	61	7,19	cyclictest	0	-21	swapper/0	22:45:15	0
12315	99	61	6,46	cyclictest	1829	-21	apt-get	20:10:13	0
12315	99	61	39,14	cyclictest	31753	-21	apt-get	00:35:13	0
12315	99	61	38,19	cyclictest	14	-21	ksoftirqd/0	21:50:17	0
12315	99	61	36,15	cyclictest	18456	-21	apt-get	19:25:12	0
12315	99	61	32,22	cyclictest	0	-21	swapper/0	23:15:16	0
12315	99	61	30,20	cyclictest	30015	-21	apt-get	00:30:13	0
12315	99	61	11,19	cyclictest	5986	-21	runrttasks	20:20:27	0
12317	99	60	12,36	cyclictest	29068	-21	cstates	21:25:29	1
12315	99	60	9,20	cyclictest	32703	-21	grep	23:05:29	0
12315	99	60	39,11	cyclictest	26755	-21	apt-get	22:50:14	0
12315	99	60	34,14	cyclictest	17984	-21	apt-get	22:25:23	0
12315	99	60	30,23	cyclictest	0	-21	swapper/0	22:30:13	0
12315	99	60	28,18	cyclictest	21466	-21	apt-get	22:35:12	0
12315	99	60	11,25	cyclictest	24602	-21	cron	00:15:00	0
12315	99	60	10,25	cyclictest	23842	-21	tail	19:40:15	0
12317	99	59	8,24	cyclictest	26	-21	ksoftirqd/1	21:30:12	1
12317	99	59	40,8	cyclictest	26	-21	ksoftirqd/1	23:28:53	1
12317	99	59	30,12	cyclictest	26	-21	ksoftirqd/1	20:05:26	1
12317	99	59	26,13	cyclictest	26	-21	ksoftirqd/1	19:55:55	1
12317	99	59	16,19	cyclictest	25	1	rcuc/1	20:52:19	1
12317	99	59	14,40	cyclictest	0	-21	swapper/1	22:20:30	1
12317	99	59	14,20	cyclictest	25	1	rcuc/1	23:50:22	1
12317	99	59	14,18	cyclictest	25	1	rcuc/1	23:24:23	1
12317	99	59	13,18	cyclictest	25	1	rcuc/1	22:12:27	1
12317	99	59	12,19	cyclictest	25	1	rcuc/1	23:36:15	1
12315	99	59	9,20	cyclictest	27691	-21	vmstat	00:20:39	0
12315	99	59	6,21	cyclictest	0	-21	swapper/0	23:20:21	0
12315	99	59	6,21	cyclictest	0	-21	swapper/0	20:25:19	0
12315	99	59	6,21	cyclictest	0	-21	swapper/0	19:15:08	0
12315	99	59	26,24	cyclictest	0	-21	swapper/0	22:40:14	0
12315	99	59	11,19	cyclictest	30725	-21	taskset	20:00:12	0
12315	99	59	10,24	cyclictest	30566	-21	basename	21:30:16	0
12317	99	58	7,33	cyclictest	17582	-21	apt-get	20:55:12	1
12317	99	58	7,24	cyclictest	26	-21	ksoftirqd/1	21:40:31	1
12317	99	58	28,11	cyclictest	26	-21	ksoftirqd/1	00:00:13	1
12317	99	58	20,15	cyclictest	26	-21	ksoftirqd/1	00:20:16	1
12317	99	58	18,19	cyclictest	9930	-21	unixbench-2d	22:00:35	1
12317	99	58	17,23	cyclictest	243	-21	systemd-journal	19:39:00	1
12317	99	58	15,22	cyclictest	562	-21	runrttasks	00:06:50	1
12317	99	58	14,18	cyclictest	25	1	rcuc/1	21:57:51	1
12315	99	58	33,11	cyclictest	1357	-21	apt-get	23:10:12	0
12315	99	58	32,14	cyclictest	0	-21	swapper/0	00:15:11	0
12317	99	57	17,19	cyclictest	29398	-21	sed	00:25:37	1
12317	99	57	17,19	cyclictest	16855	-21	cut	19:20:16	1
12317	99	57	16,23	cyclictest	26	-21	ksoftirqd/1	23:30:22	1
12317	99	57	15,21	cyclictest	23650	-21	ntp_offset	21:10:30	1
12317	99	57	15,21	cyclictest	23650	-21	ntp_offset	21:10:30	1
12317	99	57	12,39	cyclictest	25	1	rcuc/1	00:16:50	1
12317	99	56	9,14	cyclictest	26	-21	ksoftirqd/1	22:30:22	1
12317	99	56	8,14	cyclictest	26	-21	ksoftirqd/1	21:15:13	1
12317	99	56	8,14	cyclictest	26	-21	ksoftirqd/1	21:15:13	1
12317	99	56	4,15	cyclictest	26	-21	ksoftirqd/1	21:35:31	1
12317	99	56	3,19	cyclictest	26	-21	ksoftirqd/1	00:35:35	1
12317	99	56	17,18	cyclictest	21810	-21	ntp_states	21:05:30	1
12317	99	56	16,19	cyclictest	26593	-21	munin-run	22:50:00	1
12317	99	56	13,36	cyclictest	25	1	rcuc/1	22:40:43	1
12317	99	56	11,19	cyclictest	25	1	rcuc/1	23:08:02	1
12317	99	56	11,13	cyclictest	26	-21	ksoftirqd/1	20:15:17	1
15691	2	55	0,6	sleep1	25	1	rcuc/1	19:15:31	1
12317	99	55	9,41	cyclictest	0	-21	swapper/1	19:30:25	1
12317	99	55	9,40	cyclictest	0	-21	swapper/1	21:50:14	1
12317	99	55	7,16	cyclictest	26	-21	ksoftirqd/1	22:55:20	1
12317	99	55	4,23	cyclictest	26	-21	ksoftirqd/1	23:45:12	1
12317	99	55	15,19	cyclictest	7799	-21	ntpq	20:25:30	1
12317	99	55	15,19	cyclictest	30593	-21	df_inode	23:00:20	1
12317	99	55	15,19	cyclictest	18801	-21	diskstats	19:25:20	1
12317	99	55	14,21	cyclictest	10670	-21	/usr/sbin/munin	22:05:15	1
12317	99	54	16,18	cyclictest	4418	-21	ntp_states	21:45:34	1
12317	99	54	14,16	cyclictest	15	1	rcu_preempt	00:30:36	1
12317	99	54	13,25	cyclictest	5666	-21	df	20:20:20	1
12317	99	54	13,20	cyclictest	25	1	rcuc/1	19:45:36	1
12317	99	54	13,16	cyclictest	15	1	rcu_preempt	00:10:20	1
12317	99	54	12,16	cyclictest	15	1	rcu_preempt	21:00:24	1
12317	99	53	15,18	cyclictest	31055	-21	df	20:00:20	1
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17