OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack3slot6.osadl.org (updated Fri Sep 11, 2026 00:45:02)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
31883	99	109	11,88	cyclictest	0	-21	swapper/0	00:00:13	0
31887	99	107	76,18	cyclictest	26	-21	ksoftirqd/1	19:30:12	1
31887	99	103	69,20	cyclictest	26	-21	ksoftirqd/1	20:55:13	1
31723	2	102	39,18	sleep1	0	-21	swapper/1	19:09:44	1
31887	99	100	67,20	cyclictest	26	-21	ksoftirqd/1	21:41:59	1
31887	99	98	64,20	cyclictest	26	-21	ksoftirqd/1	19:25:13	1
31887	99	98	61,21	cyclictest	26	-21	ksoftirqd/1	21:35:14	1
31887	99	96	56,26	cyclictest	26	-21	ksoftirqd/1	21:00:12	1
31887	99	93	60,20	cyclictest	26	-21	ksoftirqd/1	21:55:14	1
31887	99	93	57,23	cyclictest	26	-21	ksoftirqd/1	20:00:13	1
31887	99	92	57,22	cyclictest	26	-21	ksoftirqd/1	23:20:13	1
31887	99	92	51,28	cyclictest	26	-21	ksoftirqd/1	23:40:12	1
31887	99	91	57,21	cyclictest	26	-21	ksoftirqd/1	21:30:14	1
31887	99	91	56,23	cyclictest	26	-21	ksoftirqd/1	00:10:16	1
31887	99	90	57,21	cyclictest	26	-21	ksoftirqd/1	22:25:14	1
31887	99	90	56,23	cyclictest	26	-21	ksoftirqd/1	19:50:22	1
31887	99	89	56,21	cyclictest	26	-21	ksoftirqd/1	20:10:12	1
31883	99	89	7,57	cyclictest	10755	-21	diskmemload	21:10:10	0
31887	99	88	54,22	cyclictest	26	-21	ksoftirqd/1	22:40:24	1
31887	99	88	50,25	cyclictest	26	-21	ksoftirqd/1	22:00:15	1
31887	99	88	12,61	cyclictest	0	-21	swapper/1	22:10:12	1
31883	99	88	11,57	cyclictest	0	-21	swapper/0	20:45:12	0
31883	99	88	11,57	cyclictest	0	-21	swapper/0	20:45:12	0
31887	99	87	48,27	cyclictest	26	-21	ksoftirqd/1	19:10:14	1
31883	99	87	9,58	cyclictest	0	-21	swapper/0	21:50:12	0
31887	99	86	48,25	cyclictest	26	-21	ksoftirqd/1	23:00:13	1
31887	99	86	43,31	cyclictest	26	-21	ksoftirqd/1	22:45:14	1
31883	99	86	15,52	cyclictest	0	-21	swapper/0	23:25:13	0
31887	99	85	9,60	cyclictest	23571	-21	taskset	23:15:13	1
31887	99	85	55,21	cyclictest	26	-21	ksoftirqd/1	19:35:23	1
31887	99	85	48,24	cyclictest	26	-21	ksoftirqd/1	23:35:12	1
31887	99	85	47,26	cyclictest	26	-21	ksoftirqd/1	19:45:14	1
31883	99	85	12,24	cyclictest	0	-21	swapper/0	00:10:17	0
31887	99	84	8,60	cyclictest	10755	-21	diskmemload	21:10:13	1
31887	99	84	44,28	cyclictest	26	-21	ksoftirqd/1	00:20:12	1
31887	99	84	18,21	cyclictest	0	-21	swapper/1	00:00:15	1
31887	99	84	12,34	cyclictest	0	-21	swapper/1	22:15:11	1
31887	99	84	12,34	cyclictest	0	-21	swapper/1	22:15:11	1
31887	99	83	8,61	cyclictest	12213	-21	taskset	00:15:12	1
31887	99	83	8,60	cyclictest	8689	-21	seq	00:05:13	1
31887	99	83	27,42	cyclictest	26	-21	ksoftirqd/1	23:10:15	1
31887	99	83	10,58	cyclictest	0	-21	swapper/1	19:15:12	1
31887	99	82	7,59	cyclictest	7193	-21	sshd	22:30:24	1
31887	99	82	39,30	cyclictest	26	-21	ksoftirqd/1	20:20:12	1
31887	99	82	25,25	cyclictest	562	-21	runrttasks	22:35:34	1
31887	99	82	20,50	cyclictest	0	-21	swapper/1	23:45:11	1
31887	99	82	20,50	cyclictest	0	-21	swapper/1	22:05:14	1
31887	99	82	20,50	cyclictest	0	-21	swapper/1	20:15:13	1
31887	99	82	19,52	cyclictest	0	-21	swapper/1	19:55:11	1
31887	99	82	19,51	cyclictest	0	-21	swapper/1	23:30:16	1
31887	99	82	19,51	cyclictest	0	-21	swapper/1	22:20:12	1
31887	99	82	19,51	cyclictest	0	-21	swapper/1	21:50:14	1
31887	99	82	19,50	cyclictest	0	-21	swapper/1	21:20:16	1
31887	99	82	19,50	cyclictest	0	-21	swapper/1	21:15:14	1
31887	99	82	18,53	cyclictest	0	-21	swapper/1	20:50:14	1
31887	99	82	18,52	cyclictest	0	-21	swapper/1	20:40:13	1
31887	99	82	18,49	cyclictest	0	-21	swapper/1	19:40:12	1
31887	99	81	18,51	cyclictest	0	-21	swapper/1	23:50:13	1
31887	99	81	18,51	cyclictest	0	-21	swapper/1	22:50:16	1
31887	99	81	18,50	cyclictest	0	-21	swapper/1	23:25:14	1
31887	99	81	17,53	cyclictest	0	-21	swapper/1	19:20:22	1
31883	99	81	12,47	cyclictest	0	-21	swapper/0	22:00:13	0
31887	99	80	18,51	cyclictest	0	-21	swapper/1	23:05:23	1
31887	99	80	17,49	cyclictest	0	-21	swapper/1	22:55:13	1
31887	99	80	16,51	cyclictest	0	-21	swapper/1	20:05:16	1
31887	99	80	16,51	cyclictest	0	-21	swapper/1	20:05:16	1
31887	99	79	20,49	cyclictest	0	-21	swapper/1	00:35:16	1
31887	99	79	19,48	cyclictest	0	-21	swapper/1	00:25:15	1
31887	99	79	19,47	cyclictest	0	-21	swapper/1	00:30:13	1
31883	99	79	34,26	cyclictest	0	-21	swapper/0	23:55:17	0
31883	99	79	34,26	cyclictest	0	-21	swapper/0	23:55:16	0
31883	99	79	11,51	cyclictest	0	-21	swapper/0	19:25:12	0
31887	99	78	24,24	cyclictest	31039	-21	runrttasks	20:35:23	1
31887	99	78	19,49	cyclictest	0	-21	swapper/1	20:25:12	1
31887	99	78	18,49	cyclictest	0	-21	swapper/1	21:45:25	1
31887	99	78	18,49	cyclictest	0	-21	swapper/1	20:45:15	1
31887	99	78	18,49	cyclictest	0	-21	swapper/1	20:45:15	1
31887	99	78	17,49	cyclictest	0	-21	swapper/1	20:30:14	1
31883	99	78	6,50	cyclictest	20213	-21	chrt	20:05:14	0
31883	99	78	6,50	cyclictest	20213	-21	chrt	20:05:13	0
31883	99	77	41,21	cyclictest	14	-21	ksoftirqd/0	22:20:16	0
31883	99	77	40,21	cyclictest	14	-21	ksoftirqd/0	21:31:38	0
31887	99	76	29,31	cyclictest	26	-21	ksoftirqd/1	23:55:15	1
31887	99	76	29,31	cyclictest	26	-21	ksoftirqd/1	23:55:15	1
31887	99	76	18,45	cyclictest	0	-21	swapper/1	21:05:15	1
31883	99	75	43,15	cyclictest	14	-21	ksoftirqd/0	19:55:20	0
31883	99	75	10,45	cyclictest	0	-21	swapper/0	20:40:12	0
31578	2	74	15,50	sleep0	0	-21	swapper/0	19:08:42	0
31883	99	73	43,16	cyclictest	14	-21	ksoftirqd/0	21:45:39	0
31883	99	72	4,52	cyclictest	0	-21	swapper/0	23:10:12	0
31887	99	70	20,20	cyclictest	16273	-21	aten2.4_r3power	21:25:16	1
31883	99	69	42,11	cyclictest	14	-21	ksoftirqd/0	00:05:34	0
31883	99	69	31,31	cyclictest	14	-21	ksoftirqd/0	00:25:18	0
31883	99	67	7,19	cyclictest	3890	-21	apt-get	20:50:13	0
31883	99	66	8,19	cyclictest	23563	-21	apt-get	23:15:11	0
31883	99	66	7,19	cyclictest	30524	-21	apt-get	22:05:14	0
31883	99	65	35,25	cyclictest	13665	-21	ntp_states	19:45:31	0
31883	99	65	33,24	cyclictest	14	-21	ksoftirqd/0	23:05:15	0
31883	99	65	2,57	cyclictest	1690	-21	aten2.4-expect	22:15:14	0
31883	99	65	2,57	cyclictest	1690	-21	aten2.4-expect	22:15:14	0
31883	99	64	7,19	cyclictest	9128	-21	apt-get	21:05:12	0
31883	99	64	30,29	cyclictest	503	-21	snmpd	19:20:12	0
31883	99	64	23,34	cyclictest	15020	-21	aten2.4-expect	19:50:26	0
31883	99	64	19,14	cyclictest	14	-21	ksoftirqd/0	20:35:05	0
31883	99	64	12,31	cyclictest	0	-21	swapper/0	19:30:11	0
31883	99	63	5,20	cyclictest	21997	-21	apt-get	20:10:12	0
31883	99	63	36,20	cyclictest	14	-21	ksoftirqd/0	00:15:37	0
31883	99	63	12,32	cyclictest	0	-21	swapper/0	00:35:13	0
31883	99	62	20,37	cyclictest	17604	-21	tail	00:30:17	0
31883	99	62	15,12	cyclictest	23064	-21	cron	21:45:00	0
31883	99	62	14,27	cyclictest	0	-21	swapper/0	22:35:13	0
31883	99	61	24,18	cyclictest	14	-21	ksoftirqd/0	20:18:24	0
31883	99	61	13,31	cyclictest	0	-21	swapper/0	20:00:12	0
31883	99	60	38,9	cyclictest	14	-21	ksoftirqd/0	22:45:18	0
31883	99	60	34,20	cyclictest	14	-21	ksoftirqd/0	19:40:25	0
31883	99	60	22,32	cyclictest	30613	-21	aten2.4-expect	23:35:16	0
31883	99	60	21,33	cyclictest	5194	-21	aten2.4-expect	22:25:16	0
31883	99	59	36,18	cyclictest	14	-21	ksoftirqd/0	19:10:17	0
31883	99	59	3,11	cyclictest	12590	-21	munin-run	22:45:01	0
31883	99	59	29,26	cyclictest	14	-21	ksoftirqd/0	20:25:00	0
31883	99	59	28,9	cyclictest	14	-21	ksoftirqd/0	23:20:32	0
31883	99	59	22,19	cyclictest	14	-21	ksoftirqd/0	19:17:24	0
31883	99	59	17,17	cyclictest	16319	-21	aten2.4-expect	21:25:17	0
31883	99	58	37,16	cyclictest	14	-21	ksoftirqd/0	20:25:26	0
31883	99	58	32,20	cyclictest	14	-21	ksoftirqd/0	21:00:16	0
31883	99	57	8,11	cyclictest	0	-21	swapper/0	22:55:33	0
31883	99	57	6,44	cyclictest	0	-21	swapper/0	20:30:27	0
31883	99	57	3,17	cyclictest	14852	-21	runrttasks	22:50:25	0
31883	99	56	39,11	cyclictest	14	-21	ksoftirqd/0	21:55:32	0
31883	99	56	39,10	cyclictest	14	-21	ksoftirqd/0	23:41:18	0
31883	99	56	30,12	cyclictest	14	-21	ksoftirqd/0	19:40:01	0
31883	99	56	17,24	cyclictest	0	-21	swapper/0	23:00:14	0
31883	99	56	12,28	cyclictest	0	-21	swapper/0	22:30:25	0
31883	99	55	38,9	cyclictest	14	-21	ksoftirqd/0	23:30:19	0
31883	99	55	30,20	cyclictest	14	-21	ksoftirqd/0	21:21:38	0
31883	99	55	19,31	cyclictest	5691	-21	aten2.4-expect	20:55:16	0
31883	99	55	17,31	cyclictest	32411	-21	aten2.4-expect	22:10:16	0
31883	99	55	15,23	cyclictest	0	-21	swapper/0	23:45:14	0
31883	99	54	37,9	cyclictest	14	-21	ksoftirqd/0	21:40:00	0
31883	99	54	17,31	cyclictest	243	-21	systemd-journal	00:21:26	0
31883	99	54	12,27	cyclictest	0	-21	swapper/0	23:50:14	0
31883	99	53	38,9	cyclictest	14	-21	ksoftirqd/0	21:15:33	0
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17