OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #b, slot #1

Click here to display the system's profile data.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Note that at least one system runs a non-RT kernel (x scale expanded to 2000 µs).
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs

Characteristics of the 100 highest latencies:

System (updated Sun Jan 04, 2026 12:44:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1776490	2	217	193,14	sleep1	0	-21	swapper/1	07:07:48	1
1776579	2	205	181,14	sleep2	0	-21	swapper/2	07:08:58	2
1776550	2	201	177,16	sleep3	0	-21	swapper/3	07:08:35	3
1776632	2	196	172,14	sleep0	0	-21	swapper/0	07:09:36	0
1849318	2	126	0,2	sleep1	3125725	-21	ThreadPoolForeg	08:15:40	1
1776875	99	90	1,2	cyclictest	1813877	-21	snmp_wanswitch.	07:45:02	2
1776866	99	88	31,12	cyclictest	1937771	-21	snmp_wlanpowerc	09:45:02	0
1776872	99	82	24,56	cyclictest	1937801	-21	chrome	09:45:02	1
1776879	99	64	61,2	cyclictest	1756106	-21	kworker/u8:0+flush-179:0	07:30:11	3
1776879	99	64	60,3	cyclictest	1968738	-21	kworker/u8:0+flush-179:0	11:00:00	3
1776875	99	60	36,21	cyclictest	2061360	-21	ThreadPoolForeg	11:45:22	2
1776879	99	58	54,3	cyclictest	1923716	-21	kworker/u8:2+events_unbound	09:48:00	3
1776866	99	58	50,7	cyclictest	247560	-21	ThreadPoolForeg	11:03:29	0
1776866	99	58	50,7	cyclictest	247560	-21	ThreadPoolForeg	11:03:29	0
1776879	99	56	52,3	cyclictest	1813169	-21	kworker/u8:2+events_unbound	09:15:30	3
1776879	99	55	50,4	cyclictest	1942563	-21	kworker/u8:0+flush-179:0	10:05:29	3
1776879	99	54	50,3	cyclictest	1823640	-21	kworker/u8:1+events_unbound	08:40:27	3
1776879	99	53	50,2	cyclictest	2030354	-21	kworker/u8:2+events_unbound	11:29:43	3
1776879	99	53	50,2	cyclictest	1982817	-21	kworker/u8:1+events_unbound	10:45:00	3
1776879	99	53	45,6	cyclictest	247560	-21	ThreadPoolForeg	12:35:26	3
1776872	99	53	47,4	cyclictest	2066594	-21	/usr/share/muni	11:50:18	1
1776879	99	52	50,2	cyclictest	1968738	-21	kworker/u8:0+events_unbound	10:16:51	3
1776879	99	52	50,2	cyclictest	1777155	-21	kworker/u8:3+flush-179:0	07:10:20	3
1776879	99	52	48,3	cyclictest	1823640	-21	kworker/u8:1+events_unbound	08:50:37	3
1776879	99	51	47,3	cyclictest	2007339	-21	kworker/u8:4+events_unbound	11:45:29	3
1776879	99	51	47,3	cyclictest	1968738	-21	kworker/u8:0+flush-179:0	10:51:54	3
1776879	99	51	46,4	cyclictest	1864768	-21	ntpq	08:30:28	3
1776879	99	51	46,4	cyclictest	1864768	-21	ntpq	08:30:28	3
1776866	99	51	45,4	cyclictest	1860312	-21	cp	08:30:01	0
1776879	99	50	45,4	cyclictest	2007339	-21	kworker/u8:4+events_unbound	11:15:01	3
1776875	99	50	45,4	cyclictest	1797922	-21	kworker/u8:4+flush-179:0	08:45:06	2
1776866	99	50	45,4	cyclictest	1813169	-21	kworker/u8:2+flush-179:0	09:00:30	0
1776866	99	50	41,7	cyclictest	1819134	-21	/usr/share/muni	07:50:14	0
1776879	99	49	44,4	cyclictest	1823640	-21	kworker/u8:1+events_unbound	08:15:15	3
1776879	99	49	43,4	cyclictest	1947812	-21	kworker/u8:1+events_unbound	10:10:07	3
1776879	99	49	43,4	cyclictest	1886015	-21	kworker/u8:3+events_unbound	08:56:21	3
1776875	99	49	42,5	cyclictest	1808708	-21	/usr/share/muni	07:40:16	2
1776866	99	49	45,3	cyclictest	2086750	-21	apt-get	12:09:59	0
1776879	99	48	44,3	cyclictest	2004450	-21	kworker/u8:3+flush-179:0	11:05:31	3
1776879	99	48	44,3	cyclictest	1813169	-21	kworker/u8:2+flush-179:0	07:50:12	3
1776879	99	48	43,4	cyclictest	2036349	-21	kworker/u8:0+flush-179:0	11:36:52	3
1776879	99	48	43,3	cyclictest	2075516	-21	kworker/u8:3+events_unbound	12:30:01	3
1776879	99	48	42,4	cyclictest	1756106	-21	kworker/u8:0+events_unbound	07:55:13	3
1776879	99	47	44,2	cyclictest	1968738	-21	kworker/u8:0+flush-179:0	10:46:29	3
1776879	99	47	44,2	cyclictest	1968738	-21	kworker/u8:0+flush-179:0	10:46:28	3
1776879	99	47	43,3	cyclictest	2071060	-21	kworker/u8:1+flush-179:0	11:52:51	3
1776879	99	47	43,3	cyclictest	1813169	-21	kworker/u8:2+events_unbound	07:45:17	3
1776879	99	47	43,3	cyclictest	1756106	-21	kworker/u8:0+events_unbound	07:15:20	3
1776879	99	47	42,3	cyclictest	1823640	-21	kworker/u8:1+events_unbound	08:13:34	3
1776879	99	47	34,12	cyclictest	1394450	-1	kworker/3:0H+kblockd	09:30:33	3
1776866	99	47	40,6	cyclictest	1949412	-21	apt-get	09:55:15	0
1776879	99	46	43,2	cyclictest	1917506	-21	kworker/u8:4+events_unbound	10:30:21	3
1776879	99	46	43,2	cyclictest	1911580	-21	kworker/u8:1+events_unbound	09:26:17	3
1776879	99	46	41,4	cyclictest	2071060	-21	kworker/u8:1+events_unbound	12:09:23	3
1776879	99	46	41,4	cyclictest	2039855	-31	munin-html	11:20:37	3
1776875	99	46	43,2	cyclictest	1893466	-21	kworker/u8:4+events_unbound	09:05:00	2
1776875	99	46	42,3	cyclictest	1813169	-21	kworker/u8:2+flush-179:0	08:15:16	2
1776866	99	46	41,4	cyclictest	1872589	-21	ntpq	08:40:16	0
1776866	99	46	33,1	cyclictest	1581588	-1	kworker/0:0H+kblockd	09:34:21	0
1820634	2	45	0,1	sleep2	0	-21	swapper/2	07:50:15	2
1803574	2	45	0,1	chrt	1803568	-21	snmp_wlanswitch	07:35:01	0
1776879	99	45	41,3	cyclictest	2030354	-21	kworker/u8:2+events_unbound	11:30:20	3
1776879	99	45	41,3	cyclictest	1813169	-21	kworker/u8:2+events_unbound	09:05:27	3
1776879	99	45	41,3	cyclictest	1813169	-21	kworker/u8:2+events_unbound	08:00:50	3
1776879	99	45	41,2	cyclictest	2071060	-21	kworker/u8:1+events_unbound	12:10:15	3
1776879	99	45	40,4	cyclictest	1968738	-21	kworker/u8:0+events_unbound	11:00:13	3
1776879	99	45	40,4	cyclictest	1968738	-21	kworker/u8:0+events_unbound	11:00:13	3
1776879	99	45	40,4	cyclictest	1848942	-21	kworker/u8:0+events_unbound	08:25:12	3
1776872	99	45	39,3	cyclictest	1823989	-21	latency_hist	07:55:01	1
1776872	99	45	36,8	cyclictest	1994594	-21	/usr/share/muni	10:40:20	1
1776866	99	45	43,1	cyclictest	1876282	-21	chrome	08:45:03	0
1776879	99	44	41,2	cyclictest	1886015	-21	kworker/u8:3+events_unbound	09:35:22	3
1776879	99	44	41,2	cyclictest	1813169	-21	kworker/u8:2+flush-179:0	08:20:24	3
1776879	99	44	31,12	cyclictest	1797922	-21	kworker/u8:4+events_unbound	08:09:38	3
1776879	99	44	17,26	cyclictest	3125724	-21	ThreadPoolForeg	12:33:01	3
1776875	99	44	7,36	cyclictest	3125728	-13	Compositor	11:01:40	2
1776875	99	44	7,36	cyclictest	3125728	-13	Compositor	11:01:40	2
1776875	99	44	42,1	cyclictest	2062005	-21	kworker/2:0+pm	12:26:19	2
1776875	99	44	40,3	cyclictest	1893466	-21	kworker/u8:4+flush-179:0	09:05:29	2
1776872	99	44	30,11	cyclictest	2045932	-21	/usr/share/muni	11:30:03	1
1776866	99	44	39,4	cyclictest	1989356	-21	/usr/share/muni	10:35:06	0
1776866	99	44	37,5	cyclictest	247563	-13	Compositor	07:20:12	0
1776879	99	43	39,3	cyclictest	1886015	-21	kworker/u8:3+flush-179:0	09:24:35	3
1776879	99	43	36,5	cyclictest	1917506	-21	kworker/u8:4+flush-179:0	10:23:25	3
1776875	99	43	38,4	cyclictest	2030354	-21	kworker/u8:2+flush-179:0	11:35:12	2
1776875	99	43	38,3	cyclictest	2030354	-21	kworker/u8:2+flush-179:0	12:15:18	2
1776875	99	43	29,13	cyclictest	1831976	-21	kworker/2:1+pm	10:56:46	2
1776875	99	43	20,4	cyclictest	2075516	-21	kworker/u8:3+flush-179:0	12:30:05	2
1776866	99	43	40,2	cyclictest	1886015	-21	kworker/u8:3+flush-179:0	09:21:41	0
1776866	99	43	40,2	cyclictest	1581588	-1	kworker/0:0H+kblockd	08:38:16	0
1776866	99	43	37,4	cyclictest	2050786	-21	latency_hist	11:35:00	0
1776866	99	43	32,2	cyclictest	1581588	-1	kworker/0:0H+kblockd	12:31:24	0
1776879	99	42	39,2	cyclictest	1813169	-21	kworker/u8:2+flush-179:0	08:45:37	3
1776879	99	42	38,3	cyclictest	1756106	-21	kworker/u8:0+events_unbound	07:40:08	3
1776879	99	42	31,10	cyclictest	1756106	-21	kworker/u8:0+flush-179:0	07:37:25	3
1776875	99	42	38,3	cyclictest	1803543	-21	/usr/share/muni	07:35:08	2
1776875	99	42	38,2	cyclictest	1831976	-21	kworker/2:1+pm	09:42:23	2
1776875	99	42	35,5	cyclictest	1334497	-21	apache2	11:55:15	2
1776875	99	42	19,22	cyclictest	2007339	-21	kworker/u8:4+flush-179:0	11:50:43	2
1776872	99	42	38,2	cyclictest	2030354	-21	kworker/u8:2+flush-179:0	11:35:24	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17