OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Feb 04, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2296320	2	99	0,1	sleep7	2296321	-21	rm	12:32:54	7
1585385	2	99	0,1	sleep6	0	-21	swapper/6	07:16:22	6
1562358	99	56	54,2	cyclictest	1962291	-21	kworker/u16:4+events_unbound	10:41:51	6
1562073	2	56	46,7	sleep4	0	-21	swapper/4	07:06:20	4
2029615	2	52	0,0	sleep4	0	-21	swapper/4	10:41:35	4
1561956	2	52	42,7	sleep5	0	-21	swapper/5	07:04:44	5
1561861	2	50	40,7	sleep6	0	-21	swapper/6	07:03:25	6
1553874	2	46	2,31	sleep7	0	-21	swapper/7	07:01:41	7
1562358	99	45	43,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	11:39:20	6
1562358	99	41	39,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	09:19:52	6
1562358	99	40	38,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	12:28:00	6
1562358	99	40	38,2	cyclictest	1962291	-21	kworker/u16:4+events_unbound	10:18:00	6
1562358	99	40	38,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	09:30:56	6
1562358	99	39	37,2	cyclictest	2111441	-21	kworker/u16:0+events_unbound	12:10:32	6
1562358	99	39	37,2	cyclictest	2111441	-21	kworker/u16:0+events_unbound	11:20:56	6
1562358	99	39	37,2	cyclictest	1788148	-21	kworker/u16:1+flush-8:0	09:22:13	6
1562354	99	39	37,2	cyclictest	4149426	-21	kworker/u16:2+events_unbound	07:34:02	5
1562358	99	38	36,2	cyclictest	2053078	-21	kworker/u16:3+events_unbound	11:00:40	6
1562358	99	38	36,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	11:09:53	6
1562358	99	38	36,2	cyclictest	1962291	-21	kworker/u16:4+events_unbound	10:50:32	6
1562358	99	38	36,2	cyclictest	1882043	-21	kworker/u16:2+events_unbound	10:04:25	6
1562358	99	38	36,2	cyclictest	1600239	-21	kworker/u16:3+events_unbound	09:08:08	6
1562358	99	37	35,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	12:02:53	6
1562358	99	37	35,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	11:53:32	6
1562358	99	37	35,2	cyclictest	1962291	-21	kworker/u16:4+events_unbound	10:35:57	6
1562358	99	37	35,2	cyclictest	1822418	-21	kworker/u16:0+events_unbound	10:30:52	6
1562358	99	37	35,2	cyclictest	1711578	-21	kworker/u16:0+flush-8:0	08:21:47	6
1562358	99	36	34,2	cyclictest	2162770	-21	kworker/u16:2+events_unbound	11:50:56	6
1562358	99	36	34,2	cyclictest	2053078	-21	kworker/u16:3+flush-8:0	11:04:56	6
1562358	99	36	34,2	cyclictest	2053078	-21	kworker/u16:3+flush-8:0	11:04:56	6
1562358	99	36	34,2	cyclictest	1992383	-21	kworker/u16:2+events_unbound	10:54:17	6
1562358	99	36	34,2	cyclictest	1882043	-21	kworker/u16:2+events_unbound	09:49:32	6
1562358	99	36	34,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	09:44:41	6
1562358	99	35	33,2	cyclictest	2261982	-21	kworker/u16:2+events_unbound	12:24:40	6
1562358	99	35	33,2	cyclictest	2111441	-21	kworker/u16:0+events_unbound	12:32:16	6
1562358	99	35	33,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	11:42:41	6
1562358	99	35	33,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	11:28:25	6
1562358	99	35	33,2	cyclictest	1992383	-21	kworker/u16:2+events_unbound	10:40:52	6
1562358	99	35	33,2	cyclictest	1882043	-21	kworker/u16:2+events_unbound	10:09:20	6
1562360	99	34	31,2	cyclictest	650	-21	systemd-journal	07:41:22	7
1562360	99	34	31,2	cyclictest	650	-21	systemd-journal	07:41:22	7
1562358	99	34	32,2	cyclictest	2077673	-21	kworker/u16:4+flush-8:0	11:32:04	6
1562358	99	34	32,2	cyclictest	2077673	-21	kworker/u16:4+events_unbound	11:26:08	6
1562358	99	34	32,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	12:19:12	6
1562358	99	34	31,2	cyclictest	2111441	-21	kworker/u16:0+events_unbound	12:11:31	6
1562360	99	33	32,0	cyclictest	0	-21	swapper/7	11:56:45	7
1562360	99	33	32,0	cyclictest	0	-21	swapper/7	09:16:40	7
1562358	99	33	31,2	cyclictest	2111441	-21	kworker/u16:0+events_unbound	11:58:13	6
1562358	99	33	31,2	cyclictest	2077673	-21	kworker/u16:4+events_unbound	11:14:25	6
1562358	99	33	31,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	10:22:40	6
1562358	99	33	31,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	10:12:20	6
1562360	99	32	29,2	cyclictest	6471	-21	gnome-software	07:28:36	7
1562358	99	32	30,2	cyclictest	1882043	-21	kworker/u16:2+flush-8:0	09:57:22	6
1562358	99	32	30,2	cyclictest	1882043	-21	kworker/u16:2+events_unbound	09:52:57	6
1562358	99	32	30,2	cyclictest	1822418	-21	kworker/u16:0+events_unbound	09:38:57	6
1562358	99	32	30,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	09:33:25	6
1562354	99	32	29,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	11:56:40	5
1562360	99	31	30,0	cyclictest	0	-21	swapper/7	12:21:35	7
1562358	99	30	28,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	09:14:41	6
1562358	99	30	27,2	cyclictest	1600239	-21	kworker/u16:3+events_unbound	07:51:24	6
1562358	99	30	27,2	cyclictest	1562123	-21	kworker/u16:1+events_unbound	07:06:25	6
1562354	99	30	28,2	cyclictest	2053078	-21	kworker/u16:3+flush-8:0	10:51:48	5
1562358	99	29	26,2	cyclictest	4149426	-21	kworker/u16:2+flush-8:0	07:16:32	6
1562354	99	29	26,2	cyclictest	2016560	-21	kworker/u16:1+flush-8:0	11:41:37	5
1562354	99	29	26,2	cyclictest	1788148	-21	kworker/u16:1+flush-8:0	09:31:28	5
1562354	99	29	25,3	cyclictest	1822418	-21	kworker/u16:0+flush-8:0	10:11:28	5
2237881	2	28	0,1	chrt	2237776	-21	ssh	12:11:22	7
1932100	2	28	0,1	sleep4	0	-21	swapper/4	10:01:27	4
1562358	99	28	26,2	cyclictest	4149426	-21	kworker/u16:2+events_unbound	08:16:56	6
1562354	99	28	26,2	cyclictest	2016560	-21	kworker/u16:1+flush-8:0	11:21:40	5
1562354	99	28	26,2	cyclictest	1882043	-21	kworker/u16:2+flush-8:0	10:06:17	5
1562354	99	28	26,2	cyclictest	1788148	-21	kworker/u16:1+events_unbound	09:16:36	5
2136839	2	27	0,1	chrt	2136837	-21	ssh	11:24:25	7
1562354	99	26	23,2	cyclictest	2111441	-21	kworker/u16:0+flush-8:0	12:26:36	5
1562360	99	25	0,23	cyclictest	1643609	-21	latency_hist	07:46:22	7
1562354	99	25	23,2	cyclictest	1711578	-21	kworker/u16:0+flush-8:0	08:21:48	5
1562354	99	25	22,2	cyclictest	1600239	-21	kworker/u16:3+flush-8:0	08:51:40	5
1562360	99	23	9,2	cyclictest	2174124	-21	ssh	11:41:41	7
1562358	99	23	21,2	cyclictest	4149426	-21	kworker/u16:2+flush-8:0	08:11:53	6
1562354	99	23	20,2	cyclictest	1798536	-21	kworker/u16:4+flush-8:0	09:21:28	5
1562360	99	22	22,0	cyclictest	0	-21	swapper/7	10:57:58	7
1562360	99	22	17,4	cyclictest	2004373	-21	latency_hist	10:31:22	7
1562358	99	22	20,2	cyclictest	1614250	-21	kworker/u16:4+flush-8:0	07:28:41	6
1562358	99	22	19,2	cyclictest	4149426	-21	kworker/u16:2+flush-8:0	07:41:32	6
1562354	99	22	20,2	cyclictest	4149426	-21	kworker/u16:2+events_unbound	07:28:30	5
1562354	99	22	19,2	cyclictest	2016560	-21	kworker/u16:1+flush-8:0	11:46:27	5
1562358	99	21	19,2	cyclictest	1711578	-21	kworker/u16:0+flush-8:0	08:26:46	6
1562358	99	21	19,2	cyclictest	1600239	-21	kworker/u16:3+flush-8:0	09:01:56	6
1562358	99	21	19,2	cyclictest	1600239	-21	kworker/u16:3+events_unbound	07:34:28	6
1562354	99	21	19,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	12:01:46	5
1562354	99	21	19,2	cyclictest	2016560	-21	kworker/u16:1+events_unbound	10:56:44	5
1562354	99	21	18,3	cyclictest	2162770	-21	kworker/u16:2+flush-8:0	11:51:37	5
1562350	99	21	0,16	cyclictest	1596228	-21	iostat	07:21:39	4
1562354	99	20	18,2	cyclictest	4149426	-21	kworker/u16:2+events_unbound	08:56:46	5
1562354	99	20	18,2	cyclictest	1788148	-21	kworker/u16:1+flush-8:0	09:56:47	5
1562360	99	19	17,1	cyclictest	0	-21	swapper/7	07:56:35	7
1562360	99	19	14,4	cyclictest	1931671	-21	latency_hist	10:01:22	7
1562358	99	19	17,2	cyclictest	1711578	-21	kworker/u16:0+events_unbound	08:32:32	6
1562358	99	19	17,2	cyclictest	1600239	-21	kworker/u16:3+events_unbound	08:03:29	6
1562354	99	19	16,2	cyclictest	1788148	-21	kworker/u16:1+flush-8:0	09:26:41	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17