OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #e, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot6.osadl.org (updated Sun Mar 22, 2026 00:44:54)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2013604	99	154	134,15	cyclictest	2108299	-40	logrotate	00:00:00	0
2013612	99	111	105,4	cyclictest	2013393	-21	kworker/u8:1+events_unbound	19:50:01	1
2013612	99	99	89,7	cyclictest	2117918	-21	latency_hist	00:30:00	1
2013612	99	94	86,5	cyclictest	2031974	-21	latency_hist	20:05:00	1
2013612	99	89	82,5	cyclictest	266	-21	systemd-journal	23:40:01	1
2013604	99	88	75,10	cyclictest	2051082	-21	apt-get	21:05:00	0
2013612	99	84	75,6	cyclictest	2080476	-21	latency_hist	22:35:01	1
2013612	99	82	73,7	cyclictest	2038577	-21	/usr/sbin/munin	20:25:23	1
2013612	99	82	70,9	cyclictest	2027431	-21	kworker/u8:0+events_unbound	20:40:16	1
2013612	99	82	68,4	cyclictest	2100383	-21	kworker/u8:3+events_unbound	00:10:24	1
2013612	99	81	73,5	cyclictest	2098500	-21	latency_hist	23:30:01	1
2013612	99	81	70,9	cyclictest	2091877	-21	latency_hist	23:10:01	1
2013604	99	79	71,6	cyclictest	2100383	-21	kworker/u8:3+events_unbound	00:10:01	0
2013604	99	79	70,7	cyclictest	2030405	-21	latency_hist	20:00:02	0
2013604	99	79	70,7	cyclictest	2030405	-21	latency_hist	20:00:01	0
2013612	99	78	66,10	cyclictest	266	-21	systemd-journal	22:20:00	1
2013612	99	76	67,6	cyclictest	2018725	-21	latency_hist	19:25:01	1
2013612	99	75	69,4	cyclictest	2062693	-21	kworker/u8:2+flush-179:0	21:50:23	1
2013612	99	75	65,8	cyclictest	2056347	-21	latency_hist	21:20:00	1
2013612	99	74	65,6	cyclictest	2105361	-21	/usr/sbin/munin	23:50:23	1
2013612	99	73	66,5	cyclictest	2013393	-21	kworker/u8:1+flush-179:0	19:15:15	1
2013612	99	73	65,6	cyclictest	266	-21	systemd-journal	23:00:00	1
2013604	99	73	69,3	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	23:00:23	0
2013604	99	70	63,5	cyclictest	2074109	-21	latency_hist	22:15:01	0
2013612	99	69	62,5	cyclictest	2013393	-21	kworker/u8:1+flush-179:0	19:50:02	1
2013612	99	68	56,8	cyclictest	2015272	-21	latency_hist	19:15:02	1
2013612	99	66	55,8	cyclictest	2049493	-21	latency_hist	21:00:00	1
2013612	99	64	57,4	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	23:10:20	1
2013612	99	64	53,9	cyclictest	2041767	-21	apt-get	20:35:12	1
2013604	99	64	58,4	cyclictest	2080152	-21	kworker/u8:2+flush-179:0	22:35:24	0
2013604	99	64	52,6	cyclictest	2088454	-21	latency_hist	23:00:01	0
2013612	99	63	57,5	cyclictest	2018760	-21	kworker/u8:0+flush-179:0	19:35:23	1
2013612	99	63	57,4	cyclictest	2100383	-21	kworker/u8:3+flush-179:0	00:10:01	1
2013612	99	63	55,6	cyclictest	2121114	-21	latency_hist	00:40:01	1
2013604	99	63	54,7	cyclictest	2076498	-21	kworker/u8:1+events_unbound	22:25:00	0
2013612	99	62	56,5	cyclictest	2025042	-21	kworker/u8:2+flush-179:0	19:55:21	1
2013612	99	62	56,5	cyclictest	2025042	-21	kworker/u8:2+flush-179:0	19:55:21	1
2013604	99	62	56,5	cyclictest	2027431	-21	kworker/u8:0+events_unbound	20:45:02	0
2013604	99	62	56,4	cyclictest	2027431	-21	kworker/u8:0+flush-179:0	20:20:18	0
2013604	99	62	54,6	cyclictest	2119512	-21	latency_hist	00:35:02	0
2013604	99	61	52,6	cyclictest	2080152	-21	kworker/u8:2+flush-179:0	22:40:21	0
2013604	99	61	51,8	cyclictest	2025606	-21	apt-get	19:45:11	0
2013612	99	60	55,3	cyclictest	2027431	-21	kworker/u8:0+flush-179:0	20:30:21	1
2013612	99	60	53,5	cyclictest	266	-21	systemd-journal	00:00:48	1
2013612	99	60	0,57	cyclictest	1	-21	systemd	23:20:00	1
2013612	99	59	53,4	cyclictest	2067721	-21	kworker/u8:1+flush-179:0	21:55:23	1
2013612	99	59	53,4	cyclictest	2033814	-21	apt-get	20:10:12	1
2013604	99	59	51,6	cyclictest	2098996	-21	kworker/u8:0+flush-179:0	23:30:25	0
2013612	99	58	53,4	cyclictest	2013393	-21	kworker/u8:1+flush-179:0	19:30:20	1
2013612	99	58	51,5	cyclictest	2044741	-21	kworker/u8:3+events_unbound	21:15:01	1
2013612	99	58	51,5	cyclictest	2018760	-21	kworker/u8:0+flush-179:0	19:40:23	1
2013612	99	58	49,7	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	22:30:01	1
2013604	99	58	52,5	cyclictest	1252047	-21	kworker/u8:2+events_unbound	19:10:24	0
2013612	99	57	1,53	cyclictest	1	-21	systemd	22:55:00	1
2013612	99	57	1,53	cyclictest	1	-21	systemd	22:55:00	1
2013604	99	57	44,11	cyclictest	714	-21	Xorg	00:21:59	0
2013612	99	56	51,4	cyclictest	2044696	-21	kworker/u8:2+flush-179:0	20:45:18	1
2013612	99	56	50,4	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	22:15:01	1
2013604	99	56	46,8	cyclictest	2113139	-21	latency_hist	00:15:01	0
2013612	99	55	49,4	cyclictest	2080152	-21	kworker/u8:2+flush-179:0	22:40:15	1
2013612	99	53	48,4	cyclictest	2044741	-21	kworker/u8:3+flush-179:0	21:25:22	1
2013612	99	53	48,4	cyclictest	2044696	-21	kworker/u8:2+events_unbound	20:50:24	1
2013612	99	53	47,5	cyclictest	2111590	-21	kworker/u8:0+flush-179:0	00:15:15	1
2013604	99	53	48,4	cyclictest	2095289	-21	kworker/u8:2+flush-179:0	23:20:24	0
2013612	99	52	45,5	cyclictest	2080152	-21	kworker/u8:2+flush-179:0	22:45:19	1
2013612	99	52	45,5	cyclictest	2077304	-21	latency_hist	22:25:01	1
2013612	99	52	43,7	cyclictest	2082084	-21	latency_hist	22:40:01	1
2013612	99	51	43,4	cyclictest	2055696	-21	kworker/u8:0+flush-179:0	21:20:16	1
2013612	99	51	41,8	cyclictest	2102211	-21	apt-get	23:40:11	1
2013612	99	51	41,8	cyclictest	2102211	-21	apt-get	23:40:11	1
2013612	99	50	45,4	cyclictest	2098996	-21	kworker/u8:0+flush-179:0	23:30:17	1
2013612	99	50	44,4	cyclictest	2067721	-21	kworker/u8:1+events_unbound	22:05:01	1
2013604	99	50	42,6	cyclictest	2044741	-21	kworker/u8:3+flush-179:0	21:15:01	0
2013604	99	50	40,7	cyclictest	2020312	-21	latency_hist	19:30:01	0
2013612	99	49	44,4	cyclictest	2116367	-21	kworker/u8:2+events_unbound	00:30:25	1
2013612	99	49	42,5	cyclictest	2108289	-21	kworker/u8:1+flush-179:0	00:20:23	1
2013604	99	49	44,4	cyclictest	2044696	-21	kworker/u8:2+flush-179:0	20:55:18	0
2013604	99	49	38,9	cyclictest	2030641	-21	apt-get	20:00:13	0
45770	91	48	0,13	ptp4l	44	1	ktimers/3	19:10:01	3
2013612	99	48	43,4	cyclictest	2027431	-21	kworker/u8:0+flush-179:0	20:05:17	1
2013612	99	48	34,10	cyclictest	2072275	-21	apt-get	22:10:00	1
2013604	99	48	41,5	cyclictest	2055696	-21	kworker/u8:0+flush-179:0	21:25:18	0
2013612	99	47	41,4	cyclictest	2030644	-21	kworker/u8:1+flush-179:0	20:15:21	1
2013604	99	47	42,4	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	23:10:21	0
2013604	99	47	40,5	cyclictest	2015787	-21	apt-get	19:15:11	0
2013612	99	46	42,2	cyclictest	2055696	-21	kworker/u8:0+events_unbound	21:40:02	1
2013612	99	46	42,2	cyclictest	2055696	-21	kworker/u8:0+events_unbound	21:40:01	1
2013612	99	46	41,4	cyclictest	2062693	-21	kworker/u8:2+flush-179:0	21:44:55	1
2013612	99	46	39,5	cyclictest	2044741	-21	kworker/u8:3+events_unbound	21:05:00	1
2013604	99	46	35,9	cyclictest	2051870	-21	diskstats	21:05:16	0
2013604	99	46	35,9	cyclictest	2040467	-21	diskstats	20:30:14	0
2013614	99	45	41,2	cyclictest	0	-21	swapper/2	00:06:59	2
2013612	99	45	39,5	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	23:00:21	1
2013604	99	45	40,4	cyclictest	2067721	-21	kworker/u8:1+flush-179:0	21:55:24	0
2013604	99	45	35,9	cyclictest	0	-21	swapper/0	22:04:56	0
2013614	99	44	41,2	cyclictest	0	-21	swapper/2	20:53:58	2
2013604	99	44	38,5	cyclictest	2072242	-21	kworker/u8:0+flush-179:0	22:15:18	0
2013604	99	44	37,5	cyclictest	2100383	-21	kworker/u8:3+flush-179:0	23:50:17	0
2013604	99	44	35,5	cyclictest	2027431	-21	kworker/u8:0+flush-179:0	20:45:23	0
2013615	99	43	40,2	cyclictest	0	-21	swapper/3	00:06:59	3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2020-10-04 - 12:21