OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rack3slot6.osadl.org (updated Tue Apr 28, 2026 00:45:03)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
560	99	109	6,82	cyclictest	0	-21	swapper/0	19:55:22	0
560	99	109	6,82	cyclictest	0	-21	swapper/0	19:55:22	0
560	99	104	7,75	cyclictest	0	-21	swapper/0	21:05:12	0
560	99	91	6,66	cyclictest	0	-21	swapper/0	22:15:14	0
560	99	91	6,65	cyclictest	0	-21	swapper/0	22:00:24	0
565	99	90	28,23	cyclictest	457	-21	dbus-daemon	00:40:00	1
565	99	90	28,23	cyclictest	457	-21	dbus-daemon	00:40:00	1
560	99	90	20,36	cyclictest	11796	-21	aten2.4_r3power	21:10:17	0
27820	2	88	0,5	chrt	0	-21	swapper/0	20:20:56	0
565	99	87	7,21	cyclictest	18736	-21	apt-get	21:30:14	1
565	99	86	3,19	cyclictest	7303	-21	ls	00:00:02	1
565	99	85	25,19	cyclictest	243	-21	systemd-journal	23:36:42	1
560	99	85	20,48	cyclictest	9787	-21	latency_hist	21:05:01	0
565	99	84	7,19	cyclictest	7434	-21	apt-get	00:00:14	1
565	99	84	36,42	cyclictest	0	-21	swapper/1	20:43:03	1
565	99	84	20,20	cyclictest	25741	-21	seq	20:15:45	1
565	99	83	7,21	cyclictest	17545	-21	apt-get	19:55:24	1
565	99	83	7,21	cyclictest	17545	-21	apt-get	19:55:24	1
565	99	83	27,17	cyclictest	29382	-21	turbostat	22:00:00	1
565	99	83	25,17	cyclictest	14714	-21	sort	19:45:36	1
565	99	83	21,20	cyclictest	243	-21	systemd-journal	22:41:32	1
565	99	82	7,21	cyclictest	6732	-21	apt-get	19:25:14	1
565	99	82	7,19	cyclictest	17021	-21	apt-get	22:55:13	1
565	99	82	3,44	cyclictest	8249	-21	apt-get	22:30:12	1
565	99	82	21,22	cyclictest	457	-21	dbus-daemon	21:55:01	1
560	99	82	19,26	cyclictest	969	-21	apt-get	19:10:12	0
565	99	81	9,22	cyclictest	21048	-21	apt-get	20:05:13	1
565	99	81	7,24	cyclictest	20475	-21	apt-get	21:35:12	1
565	99	81	18,20	cyclictest	243	-21	systemd-journal	22:46:33	1
565	99	81	16,22	cyclictest	503	-21	snmpd	19:10:49	1
560	99	81	28,23	cyclictest	562	-21	runrttasks	23:20:19	0
565	99	80	6,26	cyclictest	6462	-21	apt-get	20:55:13	1
560	99	80	20,37	cyclictest	24093	-21	aten2.4_r3power	23:15:15	0
560	99	80	18,38	cyclictest	12727	-21	grep	19:40:29	0
565	99	79	7,19	cyclictest	2988	-21	apt-get	20:45:13	1
565	99	79	3,38	cyclictest	13331	-21	awk	21:15:01	1
565	99	79	25,10	cyclictest	0	-21	swapper/1	23:50:16	1
565	99	79	20,18	cyclictest	918	-21	grep	22:10:02	1
565	99	79	20,18	cyclictest	918	-21	grep	22:10:02	1
565	99	79	19,18	cyclictest	243	-21	systemd-journal	22:39:42	1
560	99	79	9,50	cyclictest	0	-21	swapper/0	20:50:11	0
560	99	79	40,23	cyclictest	0	-21	swapper/0	23:50:12	0
565	99	78	8,25	cyclictest	26639	-21	apt-get	20:20:12	1
565	99	78	7,25	cyclictest	25812	-21	apt-get	23:20:13	1
565	99	78	3,22	cyclictest	24549	-21	apt-config	20:15:02	1
565	99	78	23,20	cyclictest	243	-21	systemd-journal	19:34:56	1
565	99	78	19,16	cyclictest	13023	-21	sort	19:40:36	1
565	99	77	6,19	cyclictest	30119	-21	apt-get	20:30:14	1
565	99	77	6,19	cyclictest	16993	-21	apt-get	21:25:12	1
565	99	77	3,21	cyclictest	8306	-21	aten2.4_r3power	21:00:14	1
565	99	77	20,19	cyclictest	18586	-21	seq	00:30:21	1
565	99	77	20,19	cyclictest	18586	-21	seq	00:30:21	1
565	99	77	17,18	cyclictest	15479	-21	sort	21:20:17	1
560	99	77	26,44	cyclictest	23821	-21	seq	23:14:31	0
560	99	77	26,23	cyclictest	15231	-21	uname	00:20:25	0
32733	2	77	19,48	sleep0	0	-21	swapper/0	19:08:59	0
565	99	76	7,18	cyclictest	10960	-21	apt-get	00:10:14	1
565	99	76	7,17	cyclictest	4722	-21	apt-get	20:50:12	1
565	99	76	6,18	cyclictest	28378	-21	apt-get	20:25:16	1
565	99	76	3,14	cyclictest	17217	-21	apt-config	19:55:01	1
565	99	76	20,13	cyclictest	15191	-21	grep	21:20:01	1
31321	2	76	38,23	sleep1	0	-21	swapper/1	19:05:17	1
565	99	75	7,22	cyclictest	795	-21	apt-get	22:10:16	1
565	99	75	7,18	cyclictest	14766	-21	apt-get	00:20:14	1
565	99	75	6,21	cyclictest	3232	-21	apt-get	19:15:16	1
565	99	75	6,19	cyclictest	16504	-21	apt-get	00:25:15	1
560	99	75	19,23	cyclictest	0	-21	swapper/0	23:00:34	0
565	99	74	8,19	cyclictest	31857	-21	apt-get	20:35:15	1
565	99	74	7,21	cyclictest	22310	-21	apt-get	23:10:13	1
565	99	74	18,21	cyclictest	8075	-21	cron	22:30:01	1
565	99	74	17,20	cyclictest	457	-21	dbus-daemon	21:10:00	1
560	99	74	8,14	cyclictest	0	-21	swapper/0	00:35:12	0
560	99	74	8,14	cyclictest	0	-21	swapper/0	00:35:12	0
560	99	74	26,23	cyclictest	25237	-21	runrttasks	20:15:32	0
560	99	74	26,21	cyclictest	8420	-21	runrttasks	00:00:39	0
560	99	74	26,20	cyclictest	14502	-21	runrttasks	19:45:32	0
560	99	74	24,21	cyclictest	22106	-21	munin-run	21:40:00	0
560	99	74	24,20	cyclictest	27335	-21	turbostat	21:55:01	0
560	99	74	23,22	cyclictest	562	-21	runrttasks	22:55:26	0
560	99	74	22,27	cyclictest	6564	-21	idleruntime-cro	19:25:00	0
560	99	74	22,27	cyclictest	15814	-21	chrt	21:20:25	0
560	99	74	20,29	cyclictest	11176	-21	timerwakeupswit	19:35:35	0
560	99	74	18,28	cyclictest	0	-21	swapper/0	19:50:37	0
565	99	73	7,23	cyclictest	4772	-21	apt-get	22:20:14	1
565	99	73	6,18	cyclictest	9171	-21	apt-get	00:05:13	1
565	99	73	4,27	cyclictest	12526	-21	apt-get	00:15:14	1
565	99	73	3,19	cyclictest	4661	-21	latency_hist	22:20:01	1
560	99	73	24,30	cyclictest	20758	-21	ls	23:05:18	0
560	99	73	24,20	cyclictest	8092	-21	sleep	22:30:01	0
560	99	73	24,20	cyclictest	0	-21	swapper/0	23:45:36	0
560	99	73	23,25	cyclictest	16582	-21	cut	00:25:17	0
560	99	73	21,25	cyclictest	7266	-40	systemctl	00:00:02	0
560	99	73	21,19	cyclictest	32649	-21	munin-run	23:40:01	0
560	99	73	19,32	cyclictest	3007	-21	cut	20:45:16	0
560	99	73	19,30	cyclictest	12524	-21	munin-run	00:15:01	0
565	99	72	7,18	cyclictest	10212	-21	apt-get	19:35:14	1
565	99	72	7,17	cyclictest	374	-21	apt-get	23:40:11	1
565	99	72	19,14	cyclictest	0	-21	swapper/1	21:40:32	1
560	99	72	25,20	cyclictest	1252	-21	runrttasks	20:40:13	0
560	99	72	23,21	cyclictest	8315	-21	awk	19:30:01	0
560	99	72	22,24	cyclictest	29382	-21	turbostat	22:00:00	0
560	99	72	22,20	cyclictest	13611	-21	aten2.4-expect	22:45:19	0
560	99	72	22,18	cyclictest	16849	-21	munin-run	22:55:00	0
560	99	72	21,20	cyclictest	2849	-21	sleep	22:15:00	0
560	99	72	21,19	cyclictest	25595	-21	turbostat	21:50:00	0
560	99	72	18,29	cyclictest	1104	-21	nfs4_client	23:40:27	0
560	99	72	18,21	cyclictest	0	-21	swapper/0	20:10:34	0
565	99	71	9,18	cyclictest	4992	-21	apt-get	19:20:12	1
565	99	71	8,19	cyclictest	19310	-21	apt-get	20:00:14	1
565	99	71	6,17	cyclictest	29861	-21	apt-get	22:00:24	1
560	99	71	26,21	cyclictest	13012	-21	runrttasks	00:15:18	0
560	99	71	24,25	cyclictest	5607	-21	ps	22:20:32	0
560	99	71	24,23	cyclictest	30584	-21	runrttasks	20:30:24	0
560	99	71	24,18	cyclictest	0	-21	swapper/0	19:30:33	0
560	99	71	22,25	cyclictest	21081	-21	aten2.4_r3power	20:05:16	0
560	99	71	22,20	cyclictest	29497	-21	head	20:25:37	0
560	99	71	21,26	cyclictest	22926	-21	head	21:40:27	0
560	99	71	21,24	cyclictest	9869	-21	ntpq	00:05:30	0
560	99	71	21,21	cyclictest	15068	-21	munin-run	21:20:00	0
560	99	71	19,23	cyclictest	18570	-21	munin-run	21:30:01	0
560	99	71	19,22	cyclictest	27889	2	sleep0	23:25:21	0
560	99	71	18,29	cyclictest	31600	-21	apt-get	22:05:11	0
560	99	71	18,29	cyclictest	31600	-21	apt-get	22:05:11	0
560	99	71	16,25	cyclictest	0	-21	swapper/0	20:35:31	0
560	99	71	13,31	cyclictest	0	-21	swapper/0	20:55:32	0
565	99	70	19,19	cyclictest	28478	-21	cat	23:25:36	1
560	99	70	25,21	cyclictest	3576	-21	runrttasks	19:15:23	0
560	99	70	24,17	cyclictest	0	-21	swapper/0	22:35:16	0
560	99	70	23,20	cyclictest	29932	-21	needreboot	23:30:25	0
560	99	70	22,20	cyclictest	0	-21	swapper/0	00:30:31	0
560	99	70	22,20	cyclictest	0	-21	swapper/0	00:30:31	0
565	99	69	3,26	cyclictest	30856	-21	munin-run	23:35:00	1
565	99	69	18,17	cyclictest	20742	-21	sort	23:05:18	1
560	99	69	7,47	cyclictest	0	-21	swapper/0	22:30:15	0
560	99	69	23,18	cyclictest	0	-21	swapper/0	21:30:27	0
560	99	69	19,20	cyclictest	0	-21	swapper/0	20:01:00	0
565	99	67	7,18	cyclictest	24048	-21	apt-get	23:15:15	1
565	99	67	6,20	cyclictest	24026	-21	apt-get	21:45:13	1
560	99	67	21,19	cyclictest	0	-21	swapper/0	22:40:21	0
565	99	66	6,19	cyclictest	18783	-21	apt-get	23:00:15	1
565	99	65	7,17	cyclictest	15284	-21	apt-get	22:50:15	1
565	99	65	6,20	cyclictest	2161	-21	apt-get	23:45:15	1
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17