OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of all highest latencies:

System rack3slot6.osadl.org (updated Thu Oct 08, 2026 12:44:58)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
16236	99	100	72,18	cyclictest	14	-21	ksoftirqd/0	08:10:11	0
16236	99	92	62,20	cyclictest	14	-21	ksoftirqd/0	11:35:23	0
16236	99	90	56,20	cyclictest	0	-21	swapper/0	10:55:15	0
16236	99	89	8,67	cyclictest	15272	-21	apt-get	11:40:11	0
16236	99	88	3,75	cyclictest	22204	-21	grep	10:25:37	0
16236	99	88	19,61	cyclictest	453	-21	in:imuxsock	07:52:51	0
16236	99	87	10,61	cyclictest	27505	-21	apt-get	07:40:13	0
16241	99	86	11,22	cyclictest	16228	-21	cyclictest	09:20:12	1
16236	99	86	13,62	cyclictest	0	-21	swapper/0	09:35:13	0
16241	99	85	9,55	cyclictest	0	-21	swapper/1	12:20:13	1
16241	99	85	7,59	cyclictest	0	-21	swapper/1	09:50:11	1
16236	99	85	3,20	cyclictest	24394	-21	awk	10:35:01	0
16236	99	85	2,75	cyclictest	18359	-21	sed	07:15:06	0
16236	99	85	14,56	cyclictest	0	-21	swapper/0	12:15:14	0
16241	99	83	9,24	cyclictest	0	-21	swapper/1	08:00:14	1
16236	99	83	5,70	cyclictest	0	-21	swapper/0	08:41:26	0
16236	99	83	4,71	cyclictest	455	-21	rs:main		0
16236	99	83	3,19	cyclictest	17948	-21	munin-node	07:15:02	0
16236	99	83	11,59	cyclictest	0	-21	swapper/0	07:55:12	0
16241	99	82	7,55	cyclictest	0	-21	swapper/1	10:50:10	1
16236	99	82	8,18	cyclictest	23397	-21	apt-get	09:00:12	0
16236	99	82	3,71	cyclictest	29974	-21	basename	10:50:15	0
16236	99	82	23,17	cyclictest	27763	-21	sort	12:15:02	0
16241	99	81	7,56	cyclictest	0	-21	swapper/1	09:45:12	1
16236	99	80	6,25	cyclictest	28660	-21	apt-get	09:15:12	0
16236	99	80	20,19	cyclictest	457	-21	dbus-daemon	12:35:00	0
16236	99	80	17,51	cyclictest	19896	-21	apt-get	08:50:13	0
16236	99	80	16,18	cyclictest	457	-21	dbus-daemon	08:39:00	0
16236	99	80	15,52	cyclictest	0	-21	swapper/0	10:20:11	0
16241	99	79	8,53	cyclictest	0	-21	swapper/1	11:05:15	1
16241	99	79	7,55	cyclictest	0	-21	swapper/1	08:45:12	1
16236	99	79	9,62	cyclictest	18222	-21	aten2.4-expect	08:45:15	0
16236	99	79	21,18	cyclictest	16409	-21	sort	10:10:27	0
16241	99	78	9,19	cyclictest	0	-21	swapper/1	10:30:13	1
16241	99	78	7,51	cyclictest	0	-21	swapper/1	08:35:15	1
16241	99	78	7,22	cyclictest	0	-21	swapper/1	08:30:14	1
16236	99	78	9,19	cyclictest	14042	-21	apt-get	10:05:10	0
16236	99	78	8,25	cyclictest	17019	-21	apt-get	11:45:13	0
16236	99	78	8,20	cyclictest	7943	-21	apt-get	11:20:14	0
16236	99	78	5,66	cyclictest	19142	-21	cron	10:20:01	0
16236	99	78	4,65	cyclictest	20325	-21	munin-run	11:55:01	0
16236	99	78	27,44	cyclictest	243	-21	systemd-journal	11:08:53	0
16236	99	78	23,47	cyclictest	1	-21	systemd	07:38:39	0
16236	99	78	12,54	cyclictest	0	-21	swapper/0	11:55:13	0
16241	99	77	8,52	cyclictest	0	-21	swapper/1	08:40:14	1
16236	99	77	7,19	cyclictest	3796	-21	apt-get	08:05:12	0
16236	99	77	19,18	cyclictest	7414	-21	diskstats	09:45:19	0
16236	99	77	13,56	cyclictest	21667	-21	aten2.4-expect	08:55:15	0
16236	99	77	12,59	cyclictest	453	-21	in:imuxsock	08:02:51	0
16236	99	77	12,51	cyclictest	0	-21	swapper/0	12:05:15	0
15844	2	77	17,49	sleep0	0	-21	swapper/0	07:07:47	0
16241	99	76	8,52	cyclictest	0	-21	swapper/1	10:00:14	1
16241	99	76	8,49	cyclictest	0	-21	swapper/1	08:50:11	1
16241	99	76	7,54	cyclictest	0	-21	swapper/1	11:10:12	1
16241	99	76	12,35	cyclictest	8790	-21	python3	11:20:33	1
16236	99	76	8,22	cyclictest	31329	-21	apt-get	12:25:13	0
16236	99	76	7,61	cyclictest	22206	-21	aten2.4-expect	07:25:16	0
16236	99	76	4,24	cyclictest	8039	-21	ethtool	08:15:34	0
16236	99	76	3,20	cyclictest	243	-21	systemd-journal	10:43:24	0
16236	99	76	26,43	cyclictest	403	-21	taskset	09:25:28	0
16241	99	75	8,49	cyclictest	0	-21	swapper/1	11:40:12	1
16241	99	75	7,51	cyclictest	0	-21	swapper/1	07:30:14	1
16236	99	75	9,21	cyclictest	11124	-21	apt-get	08:25:13	0
16236	99	75	7,21	cyclictest	29243	-21	apt-get	07:45:12	0
16236	99	75	7,11	cyclictest	0	-21	swapper/0	07:32:46	0
16236	99	75	5,62	cyclictest	0	-21	swapper/0	11:10:10	0
16236	99	75	4,64	cyclictest	509	-21	sshd	12:39:41	0
16236	99	75	4,63	cyclictest	243	-21	systemd-journal	09:12:52	0
16236	99	75	24,17	cyclictest	24899	-21	diskstats	10:35:21	0
16236	99	74	8,18	cyclictest	12304	-21	apt-get	10:00:12	0
16236	99	74	7,17	cyclictest	12862	-21	apt-get	08:30:12	0
16236	99	74	4,18	cyclictest	243	-21	systemd-journal	09:23:08	0
16236	99	74	4,16	cyclictest	1311	-21	apt-get	09:30:11	0
16236	99	74	3,63	cyclictest	30479	-21	apache2	12:20:28	0
16236	99	74	24,43	cyclictest	12993	-21	apt-get	11:35:01	0
16236	99	74	22,46	cyclictest	10280	-21	runrttasks	09:54:03	0
16241	99	73	8,48	cyclictest	0	-21	swapper/1	07:45:13	1
16241	99	73	8,20	cyclictest	0	-21	swapper/1	08:15:23	1
16241	99	73	7,50	cyclictest	0	-21	swapper/1	07:50:12	1
16236	99	72	8,19	cyclictest	9683	-21	apt-get	11:25:15	0
16236	99	72	8,18	cyclictest	914	-21	apt-get	11:00:14	0
16236	99	72	7,17	cyclictest	5331	-21	apt-get	09:40:12	0
16236	99	72	2,36	cyclictest	7047	-21	/usr/sbin/munin	11:15:35	0
16241	99	71	7,49	cyclictest	0	-21	swapper/1	09:00:15	1
16236	99	71	7,18	cyclictest	9386	-21	apt-get	08:20:13	0
16236	99	71	6,15	cyclictest	25135	-21	apt-get	09:05:11	0
16236	99	71	6,15	cyclictest	25135	-21	apt-get	09:05:11	0
16236	99	71	5,60	cyclictest	0	-21	swapper/0	09:55:17	0
16236	99	71	13,17	cyclictest	22018	-21	sort	07:25:02	0
14653	2	71	30,28	sleep1	0	-21	swapper/1	07:05:16	1
16241	99	69	8,39	cyclictest	10951	-21	latency_hist	08:25:02	1
16241	99	69	7,47	cyclictest	0	-21	swapper/1	08:55:14	1
16241	99	69	6,46	cyclictest	0	-21	swapper/1	12:10:21	1
16241	99	69	26,25	cyclictest	0	-21	swapper/1	12:35:12	1
16236	99	69	7,18	cyclictest	28103	-21	apt-get	10:45:13	0
16241	99	66	6,46	cyclictest	0	-21	swapper/1	11:30:12	1
16241	99	66	6,46	cyclictest	0	-21	swapper/1	10:15:14	1
16241	99	66	4,43	cyclictest	31584	-21	runrttasks	10:55:12	1
16241	99	66	10,38	cyclictest	0	-21	swapper/1	07:40:12	1
16241	99	64	3,44	cyclictest	26750	-21	diskmemload	09:10:15	1
16241	99	64	16,9	cyclictest	0	-21	swapper/1	11:25:21	1
16241	99	64	1,47	cyclictest	0	-21	swapper/1	12:25:13	1
16241	99	63	6,28	cyclictest	2280	-21	cat	12:35:01	1
16241	99	63	5,44	cyclictest	0	-21	swapper/1	10:20:12	1
16241	99	62	21,29	cyclictest	26	-21	ksoftirqd/1	11:45:13	1
16241	99	61	20,27	cyclictest	26	-21	ksoftirqd/1	09:55:11	1
16241	99	60	6,23	cyclictest	17402	-21	latency_hist	10:15:02	1
16241	99	60	20,25	cyclictest	0	-21	swapper/1	09:40:09	1
16241	99	60	14,40	cyclictest	0	-21	swapper/1	07:38:11	1
16241	99	59	20,26	cyclictest	0	-21	swapper/1	07:20:12	1
16241	99	58	25,28	cyclictest	0	-21	swapper/1	09:35:14	1
16241	99	58	12,41	cyclictest	0	-21	swapper/1	10:40:20	1
16241	99	57	3,39	cyclictest	0	-21	swapper/1	10:05:11	1
16241	99	57	12,39	cyclictest	0	-21	swapper/1	08:25:26	1
16241	99	57	12,28	cyclictest	0	-21	swapper/1	08:05:11	1
16241	99	56	3,39	cyclictest	0	-21	swapper/1	07:25:14	1
16241	99	56	17,27	cyclictest	26	-21	ksoftirqd/1	10:45:14	1
16241	99	56	10,29	cyclictest	0	-21	swapper/1	09:30:22	1
16241	99	55	37,11	cyclictest	0	-21	swapper/1	10:35:30	1
16241	99	55	14,25	cyclictest	0	-21	swapper/1	09:15:10	1
16241	99	55	10,27	cyclictest	0	-21	swapper/1	11:15:12	1
16241	99	54	9,20	cyclictest	17437	-21	unixbench_singl	07:10:36	1
16241	99	54	8,29	cyclictest	0	-21	swapper/1	11:50:10	1
16241	99	54	6,42	cyclictest	0	-21	swapper/1	11:01:58	1
16241	99	54	3,37	cyclictest	0	-21	swapper/1	10:25:13	1
16241	99	54	12,37	cyclictest	0	-21	swapper/1	08:10:12	1
16241	99	54	10,38	cyclictest	0	-21	swapper/1	09:25:26	1
16241	99	53	14,22	cyclictest	0	-21	swapper/1	12:15:12	1
16241	99	53	10,13	cyclictest	13465	-21	apt-get	11:35:14	1
16241	99	52	5,19	cyclictest	26	-21	ksoftirqd/1	07:55:11	1
16241	99	52	19,13	cyclictest	26	-21	ksoftirqd/1	12:00:30	1
16241	99	52	14,8	cyclictest	0	-21	swapper/1	12:05:26	1
16241	99	51	9,17	cyclictest	26260	-21	grep	09:05:36	1
16241	99	51	9,17	cyclictest	26260	-21	grep	09:05:36	1
16241	99	50	7,18	cyclictest	19524	-21	ls	07:15:35	1
16241	99	50	10,16	cyclictest	21349	-21	runrttasks	11:55:35	1
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17