OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack3slot6.osadl.org (updated Sun Sep 20, 2026 00:45:00)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
11488	2	99	0,5	chrt	10960	-21	/usr/sbin/munin	22:55:26	0
27489	2	98	15,38	sleep0	0	-21	swapper/0	19:08:00	0
27863	99	94	14,51	cyclictest	0	-21	swapper/1	22:20:14	1
27863	99	93	17,67	cyclictest	0	-21	swapper/1	20:30:15	1
27531	2	93	23,16	sleep1	0	-21	swapper/1	19:08:28	1
27856	99	92	38,48	cyclictest	10271	-21	irqstats	21:20:27	0
27863	99	91	19,60	cyclictest	0	-21	swapper/1	22:00:10	1
27863	99	89	14,62	cyclictest	0	-21	swapper/1	19:20:12	1
27863	99	88	18,54	cyclictest	0	-21	swapper/1	23:10:14	1
27863	99	88	15,60	cyclictest	0	-21	swapper/1	23:05:23	1
27863	99	88	14,67	cyclictest	0	-21	swapper/1	21:10:14	1
27863	99	88	13,69	cyclictest	0	-21	swapper/1	23:55:24	1
27856	99	88	2,62	cyclictest	3872	-21	cat	22:35:01	0
27856	99	88	2,62	cyclictest	3872	-21	cat	22:35:01	0
27863	99	87	30,50	cyclictest	29200	-21	cron	22:15:00	1
27863	99	87	13,66	cyclictest	0	-21	swapper/1	21:00:16	1
27863	99	87	13,66	cyclictest	0	-21	swapper/1	21:00:16	1
27863	99	85	8,18	cyclictest	14844	-21	apt-get	00:35:17	1
27863	99	85	7,27	cyclictest	8038	-21	apt-get	21:15:13	1
27863	99	85	7,22	cyclictest	12091	-21	apt-get	19:55:17	1
27863	99	85	6,18	cyclictest	25850	-21	apt-get	22:05:14	1
27863	99	85	15,60	cyclictest	0	-21	swapper/1	00:15:14	1
5468	2	84	0,9	chrt	0	-21	swapper/0	19:35:29	0
27863	99	84	7,21	cyclictest	31690	-21	apt-get	20:50:25	1
27863	99	84	12,60	cyclictest	0	-21	swapper/1	21:20:14	1
27863	99	83	8,18	cyclictest	29655	-21	apt-get	20:45:14	1
27863	99	83	6,22	cyclictest	29381	-21	apt-get	22:15:13	1
27863	99	83	6,19	cyclictest	4333	-21	apt-get	00:05:14	1
27863	99	83	3,42	cyclictest	15588	-21	/usr/sbin/munin	20:05:31	1
27863	99	83	25,20	cyclictest	20675	-21	cron	20:20:00	1
27863	99	83	24,22	cyclictest	1862	-21	sshd	22:28:24	1
27856	99	83	5,48	cyclictest	0	-21	swapper/0	23:30:27	0
27856	99	83	2,61	cyclictest	22525	-21	cut	21:55:28	0
27863	99	82	8,18	cyclictest	12720	-21	apt-get	23:00:15	1
27863	99	82	7,22	cyclictest	26101	-21	apt-get	20:35:14	1
27863	99	82	7,22	cyclictest	23839	-21	apt-get	23:30:14	1
27863	99	82	7,19	cyclictest	22617	-21	apt-get	20:25:11	1
27863	99	82	7,18	cyclictest	3018	-21	apt-get	19:30:13	1
27863	99	82	7,18	cyclictest	27915	-21	apt-get	20:40:13	1
27863	99	82	7,18	cyclictest	27915	-21	apt-get	20:40:12	1
27863	99	82	6,19	cyclictest	9225	-21	apt-get	22:50:15	1
27863	99	82	6,18	cyclictest	20356	-21	apt-get	23:20:25	1
27863	99	82	17,59	cyclictest	0	-21	swapper/1	23:35:22	1
27856	99	82	5,49	cyclictest	0	-21	swapper/0	19:15:26	0
27856	99	82	3,53	cyclictest	4153	-21	cron	00:05:00	0
27863	99	81	8,19	cyclictest	18566	-21	apt-get	21:45:11	1
27863	99	81	7,25	cyclictest	1007	-21	apt-get	20:55:13	1
27863	99	81	7,18	cyclictest	5745	-21	apt-get	22:40:12	1
27856	99	81	6,46	cyclictest	0	-21	swapper/0	00:35:24	0
27856	99	81	5,47	cyclictest	0	-21	swapper/0	20:25:17	0
27863	99	80	7,18	cyclictest	13280	-21	apt-get	21:30:15	1
27863	99	80	19,21	cyclictest	15409	-21	cron	20:05:00	1
27856	99	80	5,47	cyclictest	0	-21	swapper/0	23:15:22	0
27863	99	79	6,21	cyclictest	15020	-21	apt-get	21:35:15	1
27863	99	79	6,17	cyclictest	10963	-21	apt-get	22:55:14	1
27863	99	79	5,17	cyclictest	22349	-21	apt-get	21:55:25	1
27863	99	78	6,18	cyclictest	1279	-21	apt-get	19:25:15	1
27863	99	78	6,17	cyclictest	13102	-21	apt-get	00:30:14	1
27863	99	78	5,18	cyclictest	2193	-21	apt-get	22:30:13	1
27863	99	78	5,18	cyclictest	2193	-21	apt-get	22:30:13	1
27863	99	78	20,19	cyclictest	21042	-21	head	20:20:20	1
27863	99	78	19,18	cyclictest	10320	-21	tr	00:20:28	1
27863	99	78	17,18	cyclictest	457	-21	dbus-daemon	19:50:05	1
27863	99	78	10,24	cyclictest	29440	-21	apt-get	23:45:14	1
27856	99	78	5,43	cyclictest	0	-21	swapper/0	23:35:34	0
27863	99	77	7,17	cyclictest	6123	-21	apt-get	00:10:15	1
27863	99	77	6,23	cyclictest	7485	-21	apt-get	22:45:14	1
27863	99	77	6,19	cyclictest	20304	-21	apt-get	21:50:14	1
27863	99	77	3,23	cyclictest	885	-21	apache2	00:00:04	1
27856	99	77	28,44	cyclictest	243	-21	systemd-journal	00:10:24	0
27863	99	76	8,17	cyclictest	0	-21	swapper/1	23:25:12	1
27863	99	76	6,19	cyclictest	3934	-21	apt-get	22:35:13	1
27863	99	76	6,19	cyclictest	18291	-21	apt-get	23:15:12	1
27863	99	76	6,18	cyclictest	8310	-21	apt-get	19:45:16	1
27863	99	76	5,16	cyclictest	6570	-21	apt-get	19:40:13	1
27863	99	76	3,26	cyclictest	11360	-21	/usr/sbin/munin	00:25:27	1
27863	99	76	22,18	cyclictest	31235	-21	grep	19:15:43	1
27856	99	76	5,43	cyclictest	0	-21	swapper/0	21:45:11	0
27856	99	76	26,43	cyclictest	21516	-21	users	23:20:49	0
27863	99	75	7,19	cyclictest	17374	-21	apt-get	20:10:13	1
27863	99	75	7,18	cyclictest	16831	-21	apt-get	21:40:16	1
27863	99	75	6,21	cyclictest	4506	-21	apt-get	21:05:14	1
27863	99	75	6,19	cyclictest	27948	-21	apt-get	19:10:17	1
27863	99	75	3,26	cyclictest	27824	-21	telnet	23:40:17	1
27863	99	75	3,25	cyclictest	5238	-21	uname	19:35:25	1
27863	99	75	3,23	cyclictest	12180	-21	cut	21:25:28	1
27856	99	75	9,59	cyclictest	0	-21	swapper/0	22:35:13	0
27863	99	73	7,21	cyclictest	31175	-21	apt-get	23:50:14	1
27856	99	73	6,49	cyclictest	2769	-21	apt-get	21:00:16	0
27856	99	73	6,49	cyclictest	2769	-21	apt-get	21:00:16	0
27856	99	73	5,53	cyclictest	31145	-21	apt-get	22:20:14	0
27856	99	73	5,52	cyclictest	0	-21	swapper/0	23:06:18	0
27856	99	73	26,41	cyclictest	20000	-21	sed	20:15:36	0
27856	99	72	7,55	cyclictest	783	-21	apt-get	23:55:25	0
27856	99	72	7,51	cyclictest	24361	-21	apt-get	20:30:15	0
27856	99	71	7,57	cyclictest	0	-21	swapper/0	21:50:14	0
27856	99	71	6,57	cyclictest	0	-21	swapper/0	20:45:15	0
27856	99	71	6,49	cyclictest	7863	-21	apt-get	00:15:14	0
27856	99	71	25,39	cyclictest	27952	-21	dpkg	19:10:15	0
27856	99	70	8,54	cyclictest	0	-21	swapper/0	21:40:14	0
27856	99	70	8,53	cyclictest	0	-21	swapper/0	21:25:15	0
27856	99	70	7,56	cyclictest	0	-21	swapper/0	19:30:13	0
27856	99	70	7,55	cyclictest	0	-21	swapper/0	22:50:14	0
27856	99	70	7,55	cyclictest	0	-21	swapper/0	19:45:16	0
27856	99	70	7,13	cyclictest	0	-21	swapper/0	00:25:16	0
27856	99	70	6,36	cyclictest	0	-21	swapper/0	22:10:15	0
27856	99	69	9,52	cyclictest	0	-21	swapper/0	22:40:14	0
27856	99	69	8,53	cyclictest	0	-21	swapper/0	20:10:13	0
27856	99	69	7,54	cyclictest	0	-21	swapper/0	22:45:12	0
27856	99	69	7,54	cyclictest	0	-21	swapper/0	20:00:14	0
27856	99	69	6,55	cyclictest	0	-21	swapper/0	19:40:15	0
27856	99	69	6,52	cyclictest	16553	-21	apt-get	23:10:14	0
27856	99	69	6,50	cyclictest	24107	-21	apt-get	22:00:10	0
27856	99	69	5,59	cyclictest	0	-21	swapper/0	20:20:40	0
27856	99	69	5,52	cyclictest	31965	-21	apt-get	19:20:12	0
27856	99	69	3,46	cyclictest	812	-21	/usr/sbin/munin	22:25:22	0
27856	99	69	2,48	cyclictest	31585	-21	if_enp3s0	23:50:23	0
27856	99	69	23,40	cyclictest	29728	-21	diskstats	22:15:22	0
27856	99	69	22,41	cyclictest	13719	-21	cat	23:00:37	0
27856	99	69	17,35	cyclictest	9675	-21	aten2.4_r3power	00:20:16	0
27856	99	68	7,54	cyclictest	0	-21	swapper/0	21:15:31	0
27856	99	68	25,16	cyclictest	29567	-21	aten2.4-expect	23:45:18	0
27856	99	68	21,41	cyclictest	5388	-21	sed	21:05:35	0
27856	99	68	11,48	cyclictest	0	-21	swapper/0	00:05:14	0
27856	99	67	9,51	cyclictest	0	-21	swapper/0	00:30:16	0
27856	99	67	8,52	cyclictest	0	-21	swapper/0	19:55:16	0
27856	99	67	7,53	cyclictest	0	-21	swapper/0	20:55:13	0
27856	99	67	7,53	cyclictest	0	-21	swapper/0	20:50:23	0
27856	99	67	6,47	cyclictest	6300	-21	apt-get	21:10:14	0
27856	99	67	22,39	cyclictest	11373	-21	unixbench-2d	19:50:46	0
27856	99	66	9,51	cyclictest	0	-21	swapper/0	21:35:14	0
27856	99	66	3,43	cyclictest	15674	-21	/usr/sbin/munin	20:05:19	0
26879	2	66	0,5	sleep0	17	1	rcuc/0	22:05:37	0
27856	99	65	7,52	cyclictest	0	-21	swapper/0	23:25:14	0
27856	99	65	4,52	cyclictest	562	-21	runrttasks	21:30:15	0
27856	99	65	2,57	cyclictest	28096	-21	cpu	20:40:17	0
27856	99	65	2,57	cyclictest	28096	-21	cpu	20:40:17	0
27856	99	64	11,45	cyclictest	0	-21	swapper/0	19:25:17	0
27856	99	63	7,49	cyclictest	0	-21	swapper/0	23:40:13	0
27856	99	62	7,48	cyclictest	0	-21	swapper/0	20:35:13	0
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17