OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sat Jun 27, 2026 00:43:35)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1758611	2	93	0,0	sleep4	0	-21	swapper/4	23:49:40	4
1123062	2	58	47,7	sleep4	0	-21	swapper/4	19:06:22	4
1123237	2	57	23,29	sleep7	0	-21	swapper/7	19:08:50	7
1123224	2	40	29,8	sleep5	0	-21	swapper/5	19:08:40	5
1123578	99	38	32,5	cyclictest	650	-21	systemd-journal	19:59:21	7
1123578	99	37	0,36	cyclictest	1385304	-21	latency_hist	21:19:21	7
1123576	99	37	35,2	cyclictest	1432098	-21	kworker/u16:0+events_unbound	22:24:21	6
1123576	99	37	35,2	cyclictest	1397738	-21	kworker/u16:1+events_unbound	22:08:06	6
1123576	99	36	34,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	23:38:23	6
1123576	99	36	34,2	cyclictest	1482898	-21	kworker/u16:2+events_unbound	22:12:12	6
1123576	99	35	33,2	cyclictest	1641595	-21	kworker/u16:3+events_unbound	23:13:55	6
1123576	99	35	33,2	cyclictest	1432098	-21	kworker/u16:0+events_unbound	22:26:47	6
1123576	99	35	33,2	cyclictest	1397738	-21	kworker/u16:1+events_unbound	21:27:03	6
1123576	99	35	33,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	21:57:48	6
1123576	99	35	33,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	21:54:07	6
1123576	99	34	32,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	22:57:52	6
1123576	99	33	31,2	cyclictest	1678330	-21	kworker/u16:0+events_unbound	23:56:10	6
1123576	99	33	31,2	cyclictest	1641595	-21	kworker/u16:3+events_unbound	23:05:56	6
1123578	99	32	27,4	cyclictest	1568368	-21	latency_hist	22:34:21	7
1123578	99	32	26,5	cyclictest	650	-21	systemd-journal	19:14:22	7
1123576	99	32	30,2	cyclictest	1664040	-21	kworker/u16:2+events_unbound	23:45:07	6
1123576	99	32	30,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	22:31:23	6
1123576	99	32	30,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	00:37:04	6
1123576	99	32	30,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	21:39:51	6
1123576	99	32	30,2	cyclictest	1146940	-21	kworker/u16:4+events_unbound	19:46:44	6
1123576	99	31	29,2	cyclictest	1432098	-21	kworker/u16:0+events_unbound	21:38:39	6
1123576	99	31	29,2	cyclictest	1397738	-21	kworker/u16:1+events_unbound	21:30:11	6
1123576	99	31	29,2	cyclictest	1224768	-21	kworker/u16:1+events_unbound	20:36:44	6
1123207	2	31	21,7	sleep6	0	-21	swapper/6	19:08:25	6
1123578	99	30	29,0	cyclictest	0	-21	swapper/7	00:26:35	7
1123578	99	30	28,1	cyclictest	0	-21	swapper/7	23:24:54	7
1123576	99	30	28,2	cyclictest	1664040	-21	kworker/u16:2+events_unbound	23:29:10	6
1123576	99	30	28,2	cyclictest	1617328	-21	kworker/u16:4+events_unbound	23:00:40	6
1123576	99	30	28,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	21:19:59	6
1123576	99	30	28,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	21:18:23	6
1123576	99	29	27,2	cyclictest	1785688	-21	kworker/u16:6+events_unbound	00:01:01	6
1123576	99	29	27,2	cyclictest	1785688	-21	kworker/u16:6+events_unbound	00:01:01	6
1123576	99	29	27,2	cyclictest	1590492	-21	kworker/u16:3+events_unbound	22:47:52	6
1123576	99	29	27,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	22:17:58	6
1123576	99	28	26,2	cyclictest	1664040	-21	kworker/u16:2+flush-8:0	23:43:11	6
1123576	99	28	26,2	cyclictest	1664040	-21	kworker/u16:2+flush-8:0	23:18:06	6
1123576	99	28	26,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	23:51:46	6
1123576	99	28	26,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	23:23:26	6
1123576	99	28	26,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	23:23:26	6
1123576	99	28	26,2	cyclictest	1288885	-21	kworker/u16:2+events_unbound	21:08:43	6
1123576	99	28	26,2	cyclictest	1205417	-21	kworker/u16:2+events_unbound	20:22:45	6
1123576	99	28	26,2	cyclictest	1132338	-21	kworker/u16:3+flush-8:0	20:03:43	6
1123576	99	28	22,6	cyclictest	1785688	-21	kworker/u16:6+events_unbound	00:12:09	6
1123567	99	28	25,2	cyclictest	1373471	-21	kworker/u16:3+events_unbound	21:34:33	4
1773943	2	27	0,1	chrt	1773941	-21	ssh	23:55:44	7
1123576	99	27	25,2	cyclictest	1814336	-21	kworker/u16:0+events_unbound	00:16:47	6
1123576	99	27	25,2	cyclictest	1785688	-21	kworker/u16:6+events_unbound	00:06:35	6
1123576	99	27	25,2	cyclictest	1533045	-21	kworker/u16:1+flush-8:0	23:29:57	6
1123576	99	27	25,2	cyclictest	1519764	-21	kworker/u16:4+events_unbound	22:38:50	6
1123576	99	27	25,2	cyclictest	1397738	-21	kworker/u16:1+flush-8:0	21:49:01	6
1123576	99	27	25,2	cyclictest	1397738	-21	kworker/u16:1+events_unbound	22:02:02	6
1123576	99	27	25,2	cyclictest	1049610	-21	kworker/u16:0+flush-8:0	19:56:43	6
1123578	99	26	25,0	cyclictest	0	-21	swapper/7	22:04:47	7
1123578	99	26	25,0	cyclictest	0	-21	swapper/7	20:24:51	7
1123576	99	26	24,2	cyclictest	1850880	-21	kworker/u16:0+flush-8:0	00:30:38	6
1123576	99	26	24,2	cyclictest	1775909	-21	kworker/u16:5+flush-8:0	00:20:11	6
1123576	99	26	24,2	cyclictest	1274411	-21	kworker/u16:3+events_unbound	20:27:31	6
1123576	99	26	24,2	cyclictest	1224768	-21	kworker/u16:1+events_unbound	20:57:44	6
1123576	99	26	24,2	cyclictest	1205417	-21	kworker/u16:2+events_unbound	20:07:27	6
1123576	99	26	24,2	cyclictest	1132338	-21	kworker/u16:3+events_unbound	19:23:19	6
1123567	99	26	24,2	cyclictest	1288885	-21	kworker/u16:2+flush-8:0	20:59:44	4
1123567	99	26	23,2	cyclictest	1224768	-21	kworker/u16:1+events_unbound	20:29:36	4
1123578	99	25	24,0	cyclictest	0	-21	swapper/7	22:20:03	7
1123578	99	25	0,23	cyclictest	1726111	-21	sshd-session	23:37:42	7
1123576	99	25	23,2	cyclictest	1590492	-21	kworker/u16:3+events_unbound	22:53:34	6
1123576	99	25	23,2	cyclictest	1274411	-21	kworker/u16:3+events_unbound	20:31:00	6
1123576	99	25	23,2	cyclictest	1132338	-21	kworker/u16:3+events_unbound	19:51:25	6
1123576	99	25	23,2	cyclictest	1132338	-21	kworker/u16:3+events_unbound	19:32:28	6
1123576	99	25	22,2	cyclictest	1049610	-21	kworker/u16:0+events_unbound	20:09:24	6
1123572	99	25	22,2	cyclictest	1094623	-21	kworker/u16:1+flush-8:0	19:09:27	5
1123567	99	25	23,2	cyclictest	1533045	-21	kworker/u16:1+flush-8:0	00:24:21	4
1123567	99	25	23,2	cyclictest	1397738	-21	kworker/u16:1+events_unbound	22:09:16	4
1123578	99	24	22,1	cyclictest	1873226	-21	ssh	00:38:36	7
1123576	99	24	22,2	cyclictest	1748996	-21	kworker/u16:3+flush-8:0	00:25:02	6
1123576	99	24	22,2	cyclictest	1533045	-21	kworker/u16:1+events_unbound	22:42:49	6
1123576	99	24	22,2	cyclictest	1301509	-21	kworker/u16:6+flush-8:0	21:11:38	6
1123576	99	24	22,2	cyclictest	1301509	-21	kworker/u16:6+events_unbound	20:44:05	6
1123576	99	24	21,2	cyclictest	1331933	-21	kworker/u16:0+events_unbound	20:59:24	6
1123572	99	24	22,2	cyclictest	1664040	-21	kworker/u16:2+flush-8:0	23:44:43	5
1123572	99	24	22,2	cyclictest	1331933	-21	kworker/u16:0+events_unbound	21:04:35	5
1123572	99	24	20,3	cyclictest	1301509	-21	kworker/u16:6+events_unbound	20:49:36	5
1123567	99	24	22,2	cyclictest	1432098	-21	kworker/u16:0+events_unbound	22:19:26	4
1123576	99	23	21,2	cyclictest	1205417	-21	kworker/u16:2+events_unbound	20:16:20	6
1123576	99	23	21,2	cyclictest	1049610	-21	kworker/u16:0+events_unbound	19:11:45	6
1123572	99	23	20,2	cyclictest	1785688	-21	kworker/u16:6+flush-8:0	00:09:26	5
1123578	99	22	19,2	cyclictest	1369940	-21	ssh	21:09:44	7
1123578	99	22	18,2	cyclictest	1321935	-21	munin-run	20:49:21	7
1123578	99	22	17,4	cyclictest	1397527	-21	latency_hist	21:24:21	7
1123576	99	22	20,2	cyclictest	1224768	-21	kworker/u16:1+events_unbound	20:50:27	6
1123572	99	22	20,2	cyclictest	1397738	-21	kworker/u16:1+events_unbound	21:49:36	5
1123572	99	22	20,2	cyclictest	1146940	-21	kworker/u16:4+flush-8:0	19:24:22	5
1123567	99	22	20,2	cyclictest	1224768	-21	kworker/u16:1+flush-8:0	21:09:39	4
1123578	99	21	20,0	cyclictest	0	-21	swapper/7	23:44:38	7
1123578	99	21	20,0	cyclictest	0	-21	swapper/7	23:19:38	7
1123578	99	21	20,0	cyclictest	0	-21	swapper/7	23:19:38	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17