OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rack3slot6.osadl.org (updated Tue May 05, 2026 00:45:07)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
15112	2	100	32,19	sleep1	0	-21	swapper/1	19:05:07	1
16897	99	91	18,17	cyclictest	0	-21	swapper/1	22:35:10	1
16897	99	91	16,64	cyclictest	0	-21	swapper/1	19:50:13	1
16897	99	88	6,20	cyclictest	2337	-21	apt-get	23:00:16	1
16897	99	88	14,17	cyclictest	0	-21	swapper/1	22:10:14	1
16897	99	88	14,17	cyclictest	0	-21	swapper/1	22:10:14	1
16897	99	88	14,16	cyclictest	0	-21	swapper/1	21:05:15	1
16897	99	87	7,25	cyclictest	27728	-21	apt-get	21:10:14	1
16897	99	87	7,25	cyclictest	27728	-21	apt-get	21:10:14	1
16897	99	87	7,25	cyclictest	19510	-21	apt-get	19:15:15	1
16897	99	87	15,17	cyclictest	0	-21	swapper/1	19:25:22	1
16897	99	87	14,17	cyclictest	0	-21	swapper/1	21:30:22	1
16891	99	87	6,26	cyclictest	29520	-21	apt-get	22:45:13	0
16891	99	87	6,26	cyclictest	29520	-21	apt-get	22:45:13	0
16897	99	86	6,19	cyclictest	6686	-21	apt-get	20:10:12	1
16897	99	86	14,16	cyclictest	0	-21	swapper/1	23:05:15	1
16897	99	86	14,16	cyclictest	0	-21	swapper/1	23:05:15	1
16891	99	86	6,24	cyclictest	6440	-21	apt-get	21:40:12	0
16897	99	85	7,21	cyclictest	26791	-21	apt-get	19:35:14	1
16891	99	85	5,17	cyclictest	31231	-21	apt-get	21:20:12	0
16897	99	84	7,20	cyclictest	28601	-21	apt-get	19:40:12	1
16897	99	84	6,22	cyclictest	5863	-21	apt-get	23:10:13	1
16897	99	84	15,15	cyclictest	0	-21	swapper/1	00:15:13	1
16891	99	84	5,24	cyclictest	29466	-21	apt-get	21:15:14	0
16891	99	84	5,10	cyclictest	0	-21	swapper/0	19:35:19	0
16897	99	83	7,22	cyclictest	25051	-21	apt-get	19:30:13	1
16897	99	83	15,17	cyclictest	0	-21	swapper/1	21:20:12	1
16897	99	83	14,17	cyclictest	0	-21	swapper/1	19:20:13	1
16897	99	83	12,17	cyclictest	0	-21	swapper/1	22:40:13	1
16891	99	83	6,28	cyclictest	4076	-21	apt-get	23:05:14	0
16891	99	83	6,28	cyclictest	4076	-21	apt-get	23:05:14	0
16897	99	82	8,24	cyclictest	24200	-21	apt-get	21:00:13	1
16897	99	82	6,20	cyclictest	1397	-21	apt-get	19:55:12	1
16897	99	82	15,16	cyclictest	0	-21	swapper/1	20:35:11	1
16897	99	81	7,20	cyclictest	23411	-21	apt-get	00:00:13	1
16897	99	81	6,21	cyclictest	9917	-21	apt-get	21:50:13	1
16897	99	81	6,18	cyclictest	25456	-21	apt-get	00:05:17	1
16897	99	81	17,54	cyclictest	0	-21	swapper/1	21:40:12	1
16897	99	81	13,16	cyclictest	0	-21	swapper/1	22:20:12	1
16891	99	81	4,19	cyclictest	28977	-21	apt-get	00:15:13	0
16891	99	81	23,51	cyclictest	22892	-21	cat	20:55:23	0
16897	99	80	25,21	cyclictest	243	-21	systemd-journal	21:40:01	1
16897	99	80	14,15	cyclictest	0	-21	swapper/1	20:05:14	1
16891	99	80	6,23	cyclictest	21275	-21	apt-get	19:20:14	0
16891	99	80	5,23	cyclictest	15403	-21	apt-get	20:35:13	0
16891	99	80	13,58	cyclictest	0	-21	swapper/0	00:20:11	0
16509	2	80	20,51	sleep0	0	-21	swapper/0	19:08:08	0
16897	99	79	7,22	cyclictest	567	-21	apt-get	22:55:14	1
16897	99	79	7,22	cyclictest	11925	-21	apt-get	20:25:14	1
16897	99	79	5,19	cyclictest	30735	-21	apt-get	00:20:12	1
16897	99	79	20,48	cyclictest	0	-21	swapper/1	22:45:13	1
16897	99	79	20,48	cyclictest	0	-21	swapper/1	22:45:13	1
16897	99	79	12,14	cyclictest	0	-21	swapper/1	00:10:14	1
16891	99	79	9,62	cyclictest	0	-21	swapper/0	22:55:14	0
16891	99	79	5,10	cyclictest	0	-21	swapper/0	21:55:15	0
16891	99	79	26,47	cyclictest	23476	-21	aten2.4_r3power	19:25:27	0
16891	99	79	11,61	cyclictest	0	-21	swapper/0	20:30:14	0
16897	99	78	7,21	cyclictest	10188	-21	apt-get	20:20:13	1
16897	99	78	6,20	cyclictest	15455	-21	apt-get	22:05:12	1
16897	99	78	6,19	cyclictest	19872	-21	apt-get	23:50:11	1
16897	99	78	6,19	cyclictest	14576	-21	apt-get	23:35:11	1
16897	99	78	6,18	cyclictest	13664	-21	apt-get	20:30:14	1
16897	99	78	15,16	cyclictest	0	-21	swapper/1	21:15:13	1
16891	99	78	5,23	cyclictest	1397	-21	apt-get	19:55:13	0
16891	99	78	5,22	cyclictest	17254	-21	apt-get	19:10:25	0
16891	99	78	2,69	cyclictest	2025	-21	awk	00:30:17	0
16891	99	78	23,49	cyclictest	29310	-21	ntp_kernel_pll_	19:40:27	0
16897	99	77	9,21	cyclictest	16882	-21	cyclictest	23:15:13	1
16897	99	77	6,19	cyclictest	12840	-21	apt-get	23:30:13	1
16897	99	77	19,17	cyclictest	503	-21	snmpd	22:32:04	1
16897	99	77	13,16	cyclictest	0	-21	swapper/1	19:10:26	1
16891	99	77	7,10	cyclictest	23449	-21	aten2.4-expect	00:00:16	0
16891	99	77	5,63	cyclictest	0	-21	swapper/0	21:00:13	0
16891	99	77	23,48	cyclictest	12737	-21	date	23:30:01	0
16891	99	77	22,49	cyclictest	11007	-21	processes	20:20:32	0
16891	99	77	2,17	cyclictest	2147	-21	ls	21:30:01	0
16891	99	77	18,52	cyclictest	7599	-21	awk	20:10:34	0
16891	99	77	12,57	cyclictest	0	-21	swapper/0	23:20:14	0
4251	2	76	0,6	sleep0	4248	-21	apt-config	21:35:01	0
16897	99	76	9,20	cyclictest	1787	-21	apt-get	00:30:11	1
16897	99	76	7,22	cyclictest	21610	-21	apt-get	23:55:12	1
16897	99	76	7,21	cyclictest	16391	-21	apt-get	23:40:12	1
16897	99	76	7,21	cyclictest	16391	-21	apt-get	23:40:12	1
16897	99	76	7,20	cyclictest	11655	-21	apt-get	21:55:13	1
16897	99	76	19,18	cyclictest	3685	-21	head	00:35:17	1
16897	99	76	19,18	cyclictest	3685	-21	head	00:35:17	1
16891	99	76	6,18	cyclictest	4898	-21	apt-get	20:05:14	0
16891	99	76	5,9	cyclictest	0	-21	swapper/0	21:50:24	0
16891	99	76	2,68	cyclictest	19367	-21	if_enp2s0	19:15:07	0
16891	99	76	2,21	cyclictest	24551	-21	ls	22:30:17	0
16891	99	76	20,51	cyclictest	4723	-21	seq	20:05:00	0
16891	99	76	17,54	cyclictest	29114	-21	seq	22:42:34	0
16891	99	76	11,56	cyclictest	0	-21	swapper/0	23:50:11	0
16891	99	76	10,58	cyclictest	0	-21	swapper/0	21:45:14	0
16891	99	76	10,57	cyclictest	0	-21	swapper/0	20:40:13	0
16897	99	75	6,21	cyclictest	17220	-21	apt-get	20:40:13	1
16891	99	75	6,11	cyclictest	0	-21	swapper/0	23:30:13	0
16891	99	75	5,65	cyclictest	0	-21	swapper/0	21:10:21	0
16891	99	75	5,65	cyclictest	0	-21	swapper/0	21:10:21	0
16891	99	75	23,46	cyclictest	4955	-21	cat	21:35:28	0
16891	99	75	20,49	cyclictest	26905	-21	cat	21:05:38	0
16891	99	75	13,54	cyclictest	0	-21	swapper/0	00:25:15	0
16897	99	74	3,22	cyclictest	510	-21	latency_hist	22:55:02	1
16897	99	74	13,20	cyclictest	4718	-21	cron	20:05:00	1
16891	99	74	9,57	cyclictest	0	-21	swapper/0	00:35:11	0
16891	99	74	9,57	cyclictest	0	-21	swapper/0	00:35:11	0
16891	99	74	7,16	cyclictest	7663	-21	aten2.4-expect	23:15:16	0
16891	99	74	5,23	cyclictest	32086	-21	apt-get	19:50:15	0
16891	99	74	2,64	cyclictest	8420	-21	/usr/sbin/munin	20:15:32	0
16891	99	74	22,47	cyclictest	16615	-21	cpuspeed	23:40:17	0
16891	99	74	22,47	cyclictest	16615	-21	cpuspeed	23:40:17	0
16891	99	74	20,48	cyclictest	23409	-21	timerandwakeup	22:25:36	0
16891	99	74	10,57	cyclictest	0	-21	swapper/0	23:55:11	0
16891	99	74	10,56	cyclictest	0	-21	swapper/0	23:45:11	0
16897	99	73	8,19	cyclictest	32477	-21	apt-get	00:25:16	1
16897	99	73	6,23	cyclictest	22440	-21	apt-get	20:55:12	1
16897	99	73	6,20	cyclictest	18961	-21	apt-get	20:45:12	1
16897	99	73	4,23	cyclictest	21059	-21	cat	20:50:23	1
16897	99	73	4,23	cyclictest	21059	-21	cat	20:50:23	1
16891	99	73	7,12	cyclictest	0	-21	swapper/0	20:25:14	0
16891	99	73	6,23	cyclictest	20754	-21	apt-get	22:20:12	0
16891	99	73	16,50	cyclictest	15909	-21	cat	22:05:24	0
16891	99	73	12,53	cyclictest	0	-21	swapper/0	19:45:14	0
16891	99	73	11,55	cyclictest	0	-21	swapper/0	19:30:13	0
16897	99	72	7,19	cyclictest	8177	-21	apt-get	21:45:12	1
16897	99	72	7,19	cyclictest	11101	-21	apt-get	23:25:15	1
16897	99	72	7,18	cyclictest	536	-21	apt-get	21:25:13	1
16897	99	72	4,25	cyclictest	14627	-21	switchtime	22:00:45	1
16891	99	72	6,10	cyclictest	0	-21	swapper/0	23:10:12	0
16891	99	72	3,63	cyclictest	243	-21	systemd-journal	20:46:41	0
16891	99	72	2,13	cyclictest	26194	-21	ntp_states	00:05:35	0
16891	99	72	17,49	cyclictest	3130	-21	processes	23:00:33	0
16891	99	72	16,51	cyclictest	21440	-21	sh	20:50:32	0
16891	99	72	16,51	cyclictest	21440	-21	sh	20:50:32	0
16897	99	71	7,19	cyclictest	9362	-21	apt-get	23:20:13	1
16897	99	71	6,18	cyclictest	22493	-21	apt-get	22:25:14	1
16891	99	71	5,19	cyclictest	27238	-21	apt-get	00:10:14	0
16891	99	71	19,47	cyclictest	14905	-21	df	23:35:18	0
16891	99	70	7,12	cyclictest	0	-21	swapper/0	22:10:14	0
16891	99	70	7,12	cyclictest	0	-21	swapper/0	22:10:14	0
16891	99	70	5,12	cyclictest	0	-21	swapper/0	22:35:11	0
16891	99	70	17,49	cyclictest	32398	-21	uptime	22:50:38	0
16897	99	69	7,19	cyclictest	8423	-21	apt-get	20:15:13	1
16897	99	69	3,27	cyclictest	18130	-21	/usr/sbin/munin	23:45:25	1
16891	99	69	21,42	cyclictest	19069	-21	cut	22:15:15	0
16891	99	69	16,48	cyclictest	0	-21	swapper/0	22:00:34	0
16897	99	68	7,19	cyclictest	30341	-21	apt-get	19:45:14	1
16897	99	68	6,14	cyclictest	0	-21	swapper/1	22:15:35	1
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17