OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Thu Feb 26, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1856109	2	107	0,0	sleep4	0	-21	swapper/4	11:35:04	4
1794184	2	99	0,1	sleep7	1794185	-21	sh	11:07:50	7
1317071	2	93	0,0	sleep5	0	-21	swapper/5	07:30:14	5
1260197	2	60	50,7	sleep5	0	-21	swapper/5	07:00:42	5
1260783	99	59	56,2	cyclictest	1251103	-21	kworker/u16:0+events_unbound	07:09:57	4
1260792	99	53	51,2	cyclictest	1795330	-21	kworker/u16:1+events_unbound	11:30:23	6
1260792	99	53	51,2	cyclictest	1614742	-21	kworker/u16:1+flush-8:0	09:55:23	6
1260792	99	53	51,2	cyclictest	1426814	-21	kworker/u16:4+flush-8:0	10:05:23	6
1260792	99	52	50,2	cyclictest	1341265	-21	kworker/u16:1+events_unbound	07:45:25	6
1260794	99	49	45,2	cyclictest	1602796	-21	munin-run	09:49:54	7
1260792	99	49	46,2	cyclictest	1426814	-21	kworker/u16:4+flush-8:0	10:00:09	6
1260792	99	47	45,2	cyclictest	1781157	-21	kworker/u16:0+events_unbound	11:10:23	6
6265	89	46	0,39	rtkit-daemon	0	-21	swapper/4	07:00:54	4
1260794	99	46	40,5	cyclictest	1887372	-21	gtk-update-icon	11:45:28	7
1260792	99	46	44,2	cyclictest	1341265	-21	kworker/u16:1+events_unbound	08:30:24	6
1260792	99	45	42,2	cyclictest	1341265	-21	kworker/u16:1+flush-8:0	08:10:22	6
1260792	99	44	42,2	cyclictest	4006696	-21	kworker/u16:3+events_unbound	07:35:25	6
1260792	99	44	41,2	cyclictest	1968679	-21	kworker/u16:0+events_unbound	12:30:23	6
1260792	99	43	41,2	cyclictest	1795330	-21	kworker/u16:1+events_unbound	11:27:33	6
1260792	99	43	41,2	cyclictest	1517762	-21	kworker/u16:0+events_unbound	09:24:24	6
1260794	99	42	37,4	cyclictest	1542446	-21	latency_hist	09:24:54	7
1260792	99	42	40,2	cyclictest	1426814	-21	kworker/u16:4+events_unbound	10:48:24	6
1260792	99	41	39,2	cyclictest	1940312	-21	kworker/u16:4+events_unbound	12:16:09	6
1260792	99	41	39,2	cyclictest	1940312	-21	kworker/u16:4+events_unbound	12:16:09	6
1260792	99	41	39,2	cyclictest	1859241	-21	kworker/u16:5+events_unbound	11:35:40	6
1260792	99	41	39,2	cyclictest	1675393	-21	kworker/u16:0+events_unbound	10:24:24	6
1260792	99	41	39,2	cyclictest	1566734	-21	kworker/u16:3+events_unbound	10:52:44	6
1260792	99	41	39,2	cyclictest	1426814	-21	kworker/u16:4+events_unbound	10:28:05	6
1260792	99	41	39,2	cyclictest	1312324	-21	kworker/u16:0+flush-8:0	08:20:20	6
1260792	99	40	38,2	cyclictest	1517762	-21	kworker/u16:0+events_unbound	09:53:29	6
1260792	99	40	38,2	cyclictest	1517762	-21	kworker/u16:0+events_unbound	09:15:33	6
1260792	99	40	37,2	cyclictest	1810627	-21	kworker/u16:2+flush-8:0	11:45:04	6
1251379	2	40	3,31	sleep7	0	-21	swapper/7	07:00:04	7
1260794	99	39	0,37	cyclictest	124	-21	kswapd0	11:34:58	7
1260792	99	39	37,2	cyclictest	1795330	-21	kworker/u16:1+events_unbound	11:21:02	6
1260792	99	39	37,2	cyclictest	1566734	-21	kworker/u16:3+events_unbound	09:37:28	6
1260792	99	39	37,2	cyclictest	1426814	-21	kworker/u16:4+flush-8:0	11:05:29	6
1260792	99	39	37,2	cyclictest	1417742	-21	kworker/u16:3+events_unbound	09:06:08	6
1260792	99	38	36,2	cyclictest	1889105	-21	kworker/u16:6+events_unbound	12:00:06	6
1260792	99	38	36,2	cyclictest	1566734	-21	kworker/u16:3+events_unbound	10:42:21	6
1260792	99	38	36,2	cyclictest	1566734	-21	kworker/u16:3+events_unbound	10:16:29	6
1260792	99	38	36,2	cyclictest	1517762	-21	kworker/u16:0+events_unbound	09:48:40	6
1260792	99	38	36,2	cyclictest	1341265	-21	kworker/u16:1+events_unbound	08:05:25	6
1260783	99	38	35,3	cyclictest	1426814	-21	kworker/u16:4+flush-8:0	10:45:21	4
1260783	99	38	35,2	cyclictest	1929232	-21	kworker/u16:3+flush-8:0	12:20:08	4
1260792	99	37	35,2	cyclictest	1940312	-21	kworker/u16:4+events_unbound	12:14:37	6
1260792	99	37	35,2	cyclictest	1517762	-21	kworker/u16:0+events_unbound	09:33:08	6
1260792	99	37	35,2	cyclictest	1517762	-21	kworker/u16:0+events_unbound	09:27:49	6
1260792	99	37	35,2	cyclictest	1331307	-21	kworker/u16:2+flush-8:0	07:40:17	6
1260783	99	37	35,2	cyclictest	1781157	-21	kworker/u16:0+flush-8:0	11:19:54	4
1260794	99	36	31,4	cyclictest	1708947	-21	dnf	10:30:44	7
1260792	99	36	34,2	cyclictest	1929232	-21	kworker/u16:3+events_unbound	12:23:09	6
1260792	99	36	34,2	cyclictest	1915656	-21	kworker/u16:1+events_unbound	12:25:53	6
1260792	99	36	34,2	cyclictest	1810627	-21	kworker/u16:2+events_unbound	11:18:10	6
1260792	99	36	34,2	cyclictest	1781157	-21	kworker/u16:0+events_unbound	11:51:16	6
1260792	99	36	34,2	cyclictest	1659633	-21	kworker/u16:5+events_unbound	10:12:29	6
1260792	99	36	34,2	cyclictest	1426814	-21	kworker/u16:4+events_unbound	10:38:09	6
1260783	99	36	33,2	cyclictest	1178511	-21	kworker/u16:1+flush-8:0	07:20:21	4
1260792	99	35	33,2	cyclictest	1915656	-21	kworker/u16:1+events_unbound	11:58:40	6
1260792	99	35	33,2	cyclictest	1810627	-21	kworker/u16:2+events_unbound	11:43:09	6
1260792	99	35	33,2	cyclictest	1696772	-21	kworker/u16:1+flush-8:0	10:31:21	6
1260792	99	35	33,2	cyclictest	1178511	-21	kworker/u16:1+flush-8:0	07:20:19	6
1260794	99	34	0,33	cyclictest	1650895	-21	sh	10:09:49	7
1260792	99	34	32,2	cyclictest	1915656	-21	kworker/u16:1+events_unbound	12:09:34	6
1260792	99	34	31,2	cyclictest	1426814	-21	kworker/u16:4+events_unbound	10:54:59	6
1260788	99	33	31,2	cyclictest	1696772	-21	kworker/u16:1+flush-8:0	10:30:39	5
1260783	99	33	31,2	cyclictest	1566734	-21	kworker/u16:3+flush-8:0	10:35:09	4
1260783	99	33	30,2	cyclictest	1566734	-21	kworker/u16:3+flush-8:0	11:04:48	4
1260794	99	32	0,30	cyclictest	1331267	-21	latency_hist	07:39:54	7
1260792	99	32	30,2	cyclictest	1566734	-21	kworker/u16:3+events_unbound	11:02:21	6
1260792	99	32	30,2	cyclictest	1517762	-21	kworker/u16:0+flush-8:0	09:40:34	6
1260788	99	32	28,3	cyclictest	1855704	-21	kworker/u16:3+flush-8:0	11:45:31	5
1260792	99	31	29,2	cyclictest	1426814	-21	kworker/u16:4+events_unbound	09:10:59	6
1260792	99	31	29,2	cyclictest	1361186	-21	kworker/u16:3+flush-8:0	07:55:15	6
1260792	99	31	29,2	cyclictest	1312324	-21	kworker/u16:0+events_unbound	08:15:23	6
1260792	99	31	29,2	cyclictest	1178511	-21	kworker/u16:1+events_unbound	07:25:19	6
1260226	2	31	21,7	sleep6	0	-21	swapper/6	07:01:07	6
1260794	99	30	30,0	cyclictest	0	-21	swapper/7	11:43:53	7
1260792	99	30	28,2	cyclictest	1426814	-21	kworker/u16:4+events_unbound	08:36:33	6
1260783	99	30	27,3	cyclictest	1312324	-21	kworker/u16:0+flush-8:0	08:25:25	4
1260783	99	30	27,3	cyclictest	1312324	-21	kworker/u16:0+flush-8:0	08:25:25	4
1372202	2	29	0,1	chrt	0	-21	swapper/6	08:00:12	6
1260783	99	29	27,2	cyclictest	1426851	-21	kworker/u16:5+flush-8:0	08:30:12	4
1260792	99	28	26,2	cyclictest	1417742	-21	kworker/u16:3+flush-8:0	08:45:15	6
1260792	99	28	26,2	cyclictest	1331307	-21	kworker/u16:2+events_unbound	07:53:53	6
1260792	99	28	26,2	cyclictest	1312324	-21	kworker/u16:0+events_unbound	08:28:38	6
1260792	99	28	26,2	cyclictest	1312324	-21	kworker/u16:0+events_unbound	08:28:38	6
1260783	99	28	26,2	cyclictest	1426814	-21	kworker/u16:4+flush-8:0	11:05:16	4
1260794	99	27	22,4	cyclictest	1426702	-21	pool-gnome-soft	08:26:54	7
1260794	99	27	22,4	cyclictest	1426702	-21	pool-gnome-soft	08:26:54	7
1260792	99	27	25,2	cyclictest	1417742	-21	kworker/u16:3+events_unbound	08:51:05	6
1260788	99	27	25,2	cyclictest	1331307	-21	kworker/u16:2+flush-8:0	07:40:20	5
1260783	99	26	22,3	cyclictest	1331307	-21	kworker/u16:2+flush-8:0	07:50:20	4
1260792	99	25	22,2	cyclictest	1474718	-21	kworker/u16:0+flush-8:0	08:55:10	6
1260792	99	24	22,2	cyclictest	4006696	-21	kworker/u16:3+events_unbound	07:34:38	6
1260792	99	24	22,2	cyclictest	1417742	-21	kworker/u16:3+events_unbound	09:00:24	6
1260783	99	24	22,2	cyclictest	1517762	-21	kworker/u16:0+flush-8:0	09:39:49	4
1460576	2	23	0,1	chrt	0	-21	swapper/5	08:45:14	5
1260794	99	23	18,4	cyclictest	650	-21	systemd-journal	08:09:54	7
1260788	99	23	20,2	cyclictest	1855704	-21	kworker/u16:3+flush-8:0	11:55:50	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17