OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Feb 10, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1877415	2	96	0,0	sleep6	0	-21	swapper/6	12:31:15	6
1143619	99	57	50,6	cyclictest	1633631	-21	latency_hist	10:50:57	7
1143107	2	57	47,7	sleep5	0	-21	swapper/5	07:02:50	5
1143100	2	53	20,29	sleep7	0	-21	swapper/7	07:02:44	7
1143079	2	53	43,7	sleep4	0	-21	swapper/4	07:02:24	4
1143052	2	49	39,7	sleep6	0	-21	swapper/6	07:02:02	6
1402908	2	45	0,0	sleep7	1402909	-21	rm	09:14:56	7
1143619	99	43	37,5	cyclictest	650	-21	systemd-journal	12:00:58	7
1143619	99	40	0,38	cyclictest	1887642	-21	latency_hist	12:35:58	7
1143617	99	38	35,3	cyclictest	1656486	-21	kworker/u16:0+flush-8:0	11:06:28	6
1143617	99	37	35,2	cyclictest	1766681	-21	kworker/u16:0+flush-8:0	12:21:29	6
1143617	99	37	35,2	cyclictest	1634016	-21	kworker/u16:6+flush-8:0	10:56:34	6
1143617	99	37	35,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	09:26:25	6
1143617	99	37	34,2	cyclictest	1848182	-21	kworker/u16:1+flush-8:0	12:26:29	6
1143617	99	36	34,2	cyclictest	1706659	-21	kworker/u16:4+events_unbound	11:21:29	6
1143617	99	35	33,2	cyclictest	1427815	-21	kworker/u16:4+events_unbound	09:56:23	6
1143617	99	34	32,2	cyclictest	1292541	-21	kworker/u16:3+events_unbound	09:01:30	6
1143617	99	33	31,2	cyclictest	1694149	-21	kworker/u16:1+events_unbound	11:57:19	6
1143617	99	33	31,2	cyclictest	1292541	-21	kworker/u16:3+flush-8:0	09:06:39	6
1143617	99	33	30,2	cyclictest	1533770	-21	kworker/u16:2+events_unbound	10:36:13	6
1143617	99	32	30,2	cyclictest	1359952	-21	kworker/u16:1+events_unbound	09:46:38	6
1143608	99	32	30,2	cyclictest	1292541	-21	kworker/u16:3+flush-8:0	09:10:53	4
1143608	99	32	29,2	cyclictest	1291530	-21	kworker/u16:0+flush-8:0	09:45:58	4
1143619	99	31	25,5	cyclictest	1273046	-21	latency_hist	08:10:58	7
1143617	99	31	29,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	09:31:26	6
1143617	99	30	28,2	cyclictest	1427815	-21	kworker/u16:4+events_unbound	09:36:38	6
1143617	99	30	27,2	cyclictest	1694149	-21	kworker/u16:1+events_unbound	11:41:15	6
1143608	99	29	27,2	cyclictest	1359952	-21	kworker/u16:1+flush-8:0	10:01:32	4
1143619	99	28	23,4	cyclictest	1645681	-21	latency_hist	10:55:58	7
1143619	99	28	23,4	cyclictest	1645681	-21	latency_hist	10:55:58	7
1143617	99	28	26,2	cyclictest	1533770	-21	kworker/u16:2+flush-8:0	10:26:46	6
1143617	99	28	25,2	cyclictest	1597698	-21	kworker/u16:3+events_unbound	11:50:58	6
1143608	99	28	26,2	cyclictest	1291530	-21	kworker/u16:0+flush-8:0	09:51:40	4
1143617	99	27	25,2	cyclictest	1427815	-21	kworker/u16:4+events_unbound	09:51:51	6
1143617	99	27	25,2	cyclictest	1322523	-21	kworker/u16:2+events_unbound	08:40:21	6
1143617	99	27	25,2	cyclictest	1224934	-21	kworker/u16:1+events_unbound	07:50:04	6
1143617	99	27	25,2	cyclictest	1124216	-21	kworker/u16:1+events_unbound	07:07:49	6
1143613	99	27	25,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	10:28:00	5
1143613	99	27	25,2	cyclictest	1070424	-21	kworker/u16:3+events_unbound	07:06:12	5
1143608	99	27	25,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	10:28:08	4
1143608	99	27	25,2	cyclictest	1224934	-21	kworker/u16:1+flush-8:0	07:51:25	4
1143617	99	26	24,2	cyclictest	1814569	-21	kworker/u16:3+flush-8:0	12:06:33	6
1143617	99	26	24,2	cyclictest	1533770	-21	kworker/u16:2+flush-8:0	10:15:52	6
1143617	99	26	24,2	cyclictest	1533770	-21	kworker/u16:2+flush-8:0	10:15:52	6
1143617	99	26	24,2	cyclictest	1292541	-21	kworker/u16:3+events_unbound	08:21:22	6
1143617	99	26	24,2	cyclictest	1224934	-21	kworker/u16:1+events_unbound	08:27:20	6
1143608	99	26	23,2	cyclictest	1681195	-21	kworker/u16:2+flush-8:0	11:11:28	4
1143617	99	25	23,2	cyclictest	1322523	-21	kworker/u16:2+events_unbound	08:47:33	6
1143617	99	25	23,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	08:44:12	6
1143617	99	25	23,2	cyclictest	1166418	-21	kworker/u16:5+events_unbound	07:25:52	6
1143617	99	24	22,2	cyclictest	1244066	-21	kworker/u16:2+events_unbound	07:59:19	6
1143617	99	24	22,2	cyclictest	1166418	-21	kworker/u16:5+events_unbound	07:42:42	6
1143617	99	24	22,2	cyclictest	1070424	-21	kworker/u16:3+events_unbound	07:30:33	6
1143613	99	24	22,2	cyclictest	1166418	-21	kworker/u16:5+events_unbound	07:31:16	5
1143608	99	24	21,2	cyclictest	1583451	-21	kworker/u16:1+flush-8:0	10:56:14	4
1143619	99	23	19,3	cyclictest	124	-21	kswapd0	12:15:57	7
1143617	99	23	21,2	cyclictest	1359952	-21	kworker/u16:1+events_unbound	08:56:29	6
1143617	99	23	21,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	08:18:28	6
1143617	99	23	21,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	08:18:28	6
1143617	99	23	21,2	cyclictest	1224934	-21	kworker/u16:1+events_unbound	08:35:07	6
1143617	99	23	21,2	cyclictest	1214627	-21	kworker/u16:0+events_unbound	07:54:20	6
1143617	99	23	21,2	cyclictest	1166418	-21	kworker/u16:5+events_unbound	07:37:14	6
1143613	99	23	21,2	cyclictest	1681195	-21	kworker/u16:2+events_unbound	11:11:12	5
1143608	99	23	19,3	cyclictest	1224934	-21	kworker/u16:1+flush-8:0	07:56:15	4
1143617	99	22	20,2	cyclictest	1583451	-21	kworker/u16:1+events_unbound	11:01:18	6
1143617	99	22	20,2	cyclictest	1322523	-21	kworker/u16:2+events_unbound	08:55:20	6
1143617	99	22	20,2	cyclictest	1143826	-21	kworker/u16:4+events_unbound	07:14:20	6
1143613	99	22	20,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	10:00:19	5
1143617	99	21	19,2	cyclictest	1694149	-21	kworker/u16:1+flush-8:0	11:46:35	6
1143617	99	21	19,2	cyclictest	1224934	-21	kworker/u16:1+events_unbound	08:12:11	6
1143617	99	21	19,2	cyclictest	1166418	-21	kworker/u16:5+events_unbound	08:02:36	6
1143617	99	20	18,2	cyclictest	1359952	-21	kworker/u16:1+events_unbound	09:21:47	6
1143613	99	20	18,2	cyclictest	1533770	-21	kworker/u16:2+flush-8:0	10:20:52	5
1143613	99	20	18,2	cyclictest	1533770	-21	kworker/u16:2+events_unbound	10:40:58	5
1143613	99	20	18,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	08:57:18	5
1143619	99	19	9,9	cyclictest	0	-21	swapper/7	10:28:00	7
1143619	99	19	15,3	cyclictest	1267	-21	snmpd	11:42:48	7
1143617	99	19	17,2	cyclictest	1070424	-21	kworker/u16:3+events_unbound	07:33:31	6
1143613	99	19	17,2	cyclictest	1706659	-21	kworker/u16:4+events_unbound	11:33:48	5
1143613	99	19	17,2	cyclictest	1706659	-21	kworker/u16:4+events_unbound	11:33:48	5
1143613	99	19	17,2	cyclictest	1070424	-21	kworker/u16:3+events_unbound	07:26:25	5
1143619	99	18	11,6	cyclictest	0	-21	swapper/7	11:26:12	7
1143617	99	18	16,2	cyclictest	1694149	-21	kworker/u16:1+flush-8:0	11:31:19	6
1143617	99	18	16,2	cyclictest	1694149	-21	kworker/u16:1+flush-8:0	11:31:19	6
1143617	99	18	16,2	cyclictest	1166418	-21	kworker/u16:5+events_unbound	08:07:51	6
1143613	99	18	16,2	cyclictest	1814569	-21	kworker/u16:3+flush-8:0	12:21:16	5
1143613	99	18	16,2	cyclictest	1583451	-21	kworker/u16:1+events_unbound	11:01:33	5
1143613	99	18	16,2	cyclictest	1359952	-21	kworker/u16:1+events_unbound	10:01:20	5
1143613	99	18	16,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	09:45:36	5
1143613	99	18	15,2	cyclictest	1291530	-21	kworker/u16:0+events_unbound	08:36:13	5
1143619	99	17	13,3	cyclictest	1267	-21	snmpd	07:53:33	7
1143619	99	17	11,5	cyclictest	1282696	-21	latency_hist	08:15:58	7
1143619	99	17	10,6	cyclictest	0	-21	swapper/7	09:26:26	7
1143617	99	17	15,2	cyclictest	1583451	-21	kworker/u16:1+flush-8:0	10:46:22	6
1143617	99	17	15,2	cyclictest	1583451	-21	kworker/u16:1+flush-8:0	10:31:28	6
1143617	99	17	15,2	cyclictest	1359952	-21	kworker/u16:1+events_unbound	09:42:01	6
1143613	99	17	15,2	cyclictest	1634016	-21	kworker/u16:6+events_unbound	10:55:57	5
1143613	99	17	15,2	cyclictest	1634016	-21	kworker/u16:6+events_unbound	10:55:57	5
1143613	99	17	15,2	cyclictest	1597698	-21	kworker/u16:3+events_unbound	11:46:03	5
1143613	99	17	15,2	cyclictest	1583451	-21	kworker/u16:1+events_unbound	10:56:05	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17