OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Thu Jan 29, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2224809	2	104	0,1	sleep7	2224810	-21	ssh	09:29:40	7
2456540	2	101	0,1	sleep7	2456541	-21	sh	11:05:01	7
2504690	2	98	0,1	sleep7	2504691	-21	sh	11:24:00	7
2603049	2	55	0,0	sleep7	2603050	-21	sh	12:05:44	7
1928988	2	54	20,30	sleep7	0	-21	swapper/7	07:06:30	7
1928977	2	49	39,7	sleep5	0	-21	swapper/5	07:06:20	5
1928994	2	48	37,8	sleep4	0	-21	swapper/4	07:06:35	4
1928940	2	45	35,7	sleep6	0	-21	swapper/6	07:05:49	6
1929301	99	44	42,2	cyclictest	2237053	-21	kworker/u16:1+flush-8:0	11:12:10	4
1929307	99	41	39,2	cyclictest	2186164	-21	kworker/u16:3+flush-8:0	09:27:12	6
1929301	99	39	37,2	cyclictest	2491533	-21	kworker/u16:2+events_unbound	11:17:56	4
1929301	99	39	37,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:41:08	4
1929301	99	39	37,2	cyclictest	1930332	-21	kworker/u16:0+flush-8:0	08:11:45	4
1929307	99	37	35,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:43:11	6
1929301	99	36	34,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	12:24:56	4
1929301	99	36	34,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	12:24:56	4
1929301	99	35	33,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:29:56	4
1929301	99	35	33,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:43:52	4
1929301	99	35	33,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	09:48:57	4
1929301	99	34	32,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:17:56	4
1929301	99	34	32,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:22:16	4
1929307	99	33	31,2	cyclictest	2554960	-21	kworker/u16:5+events_unbound	12:14:19	6
1929307	99	33	31,2	cyclictest	2530587	-21	kworker/u16:3+events_unbound	12:09:51	6
1929307	99	33	31,2	cyclictest	2491533	-21	kworker/u16:2+events_unbound	11:20:15	6
1929307	99	33	31,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:29:03	6
1929301	99	33	31,2	cyclictest	2554960	-21	kworker/u16:5+events_unbound	11:53:24	4
1929301	99	33	31,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	12:00:56	4
1929301	99	33	31,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:37:15	4
1929301	99	33	31,2	cyclictest	2260449	-21	kworker/u16:3+events_unbound	09:58:41	4
1929301	99	33	31,2	cyclictest	2186164	-21	kworker/u16:3+events_unbound	09:25:05	4
1929301	99	33	31,2	cyclictest	2186164	-21	kworker/u16:3+events_unbound	09:14:45	4
1929301	99	33	30,3	cyclictest	2443173	-21	kworker/u16:4+flush-8:0	11:11:45	4
1929301	99	33	30,2	cyclictest	1930332	-21	kworker/u16:0+flush-8:0	07:11:47	4
1929307	99	32	30,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:14:51	6
1929307	99	32	30,2	cyclictest	2260449	-21	kworker/u16:3+events_unbound	10:03:19	6
1929307	99	32	30,2	cyclictest	2189307	-21	kworker/u16:4+events_unbound	09:19:19	6
1929301	99	32	30,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	12:32:55	4
1929301	99	32	30,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:59:57	4
1929301	99	32	30,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:02:57	4
1929301	99	32	29,3	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:28:24	4
1929311	99	31	0,30	cyclictest	650	-21	systemd-journal	11:56:45	7
1929307	99	31	29,2	cyclictest	2491533	-21	kworker/u16:2+events_unbound	11:45:55	6
1929307	99	31	29,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	11:59:59	6
1929307	99	31	29,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	11:25:55	6
1929307	99	31	29,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:51:39	6
1929307	99	31	29,2	cyclictest	2260449	-21	kworker/u16:3+events_unbound	09:53:07	6
1929307	99	31	29,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:00:07	6
1929301	99	31	29,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	09:52:57	4
1929301	99	31	29,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	09:09:05	4
1929307	99	30	28,2	cyclictest	2554960	-21	kworker/u16:5+events_unbound	12:35:02	6
1929307	99	30	28,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	12:31:23	6
1929307	99	30	28,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:28:51	6
1929307	99	30	28,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:24:14	6
1929307	99	30	28,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:24:14	6
1929301	99	30	28,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	12:01:47	4
1929311	99	29	24,4	cyclictest	2235135	-21	ssh	09:32:09	7
1929307	99	29	27,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	09:47:43	6
1929307	99	29	27,2	cyclictest	2186164	-21	kworker/u16:3+events_unbound	09:13:11	6
1929307	99	29	27,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:38:30	6
1929307	99	29	27,2	cyclictest	1930332	-21	kworker/u16:0+events_unbound	09:05:16	6
1929307	99	29	27,2	cyclictest	1930332	-21	kworker/u16:0+events_unbound	09:05:16	6
1929301	99	29	27,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:46:05	4
1929301	99	29	27,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:49:11	4
1929301	99	29	27,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:24:12	4
1929301	99	29	27,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	10:24:12	4
1929301	99	29	27,2	cyclictest	2213788	-21	kworker/u16:0+flush-8:0	09:29:27	4
1929301	99	29	27,2	cyclictest	2189307	-21	kworker/u16:4+events_unbound	09:19:56	4
1929301	99	29	27,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:36:36	4
1929301	99	29	27,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:16:17	4
1929301	99	29	27,2	cyclictest	1971629	-21	kworker/u16:1+events_unbound	07:37:16	4
1929301	99	29	26,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	11:46:48	4
1929311	99	28	22,5	cyclictest	2087657	-21	latency_hist	08:26:46	7
1929307	99	28	26,2	cyclictest	2334490	-21	kworker/u16:0+events_unbound	10:54:15	6
1929307	99	28	26,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:05:34	6
1929307	99	28	26,2	cyclictest	1908979	-21	kworker/u16:2+events_unbound	07:10:16	6
1929301	99	28	26,2	cyclictest	2530587	-21	kworker/u16:3+flush-8:0	12:27:19	4
1929301	99	28	26,2	cyclictest	2530587	-21	kworker/u16:3+events_unbound	12:17:48	4
1929301	99	28	26,2	cyclictest	2530587	-21	kworker/u16:3+events_unbound	12:17:48	4
1929301	99	28	26,2	cyclictest	2213788	-21	kworker/u16:0+events_unbound	09:41:04	4
1929301	99	28	26,2	cyclictest	1908979	-21	kworker/u16:2+flush-8:0	07:26:41	4
1929307	99	27	25,2	cyclictest	2491533	-21	kworker/u16:2+events_unbound	11:51:06	6
1929307	99	27	25,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:00:15	6
1929307	99	27	25,2	cyclictest	2020173	-21	kworker/u16:3+events_unbound	08:04:49	6
1929307	99	27	25,2	cyclictest	1971629	-21	kworker/u16:1+events_unbound	08:57:08	6
1929307	99	27	25,2	cyclictest	1930332	-21	kworker/u16:0+events_unbound	08:50:00	6
1929307	99	27	24,2	cyclictest	1930332	-21	kworker/u16:0+events_unbound	08:21:47	6
1929307	99	27	24,2	cyclictest	1908979	-21	kworker/u16:2+events_unbound	07:46:47	6
1929301	99	27	25,2	cyclictest	2554960	-21	kworker/u16:5+events_unbound	12:12:53	4
1929301	99	27	24,2	cyclictest	2155409	-21	kworker/u16:2+flush-8:0	10:11:46	4
1929307	99	26	24,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:35:38	6
1929301	99	26	24,2	cyclictest	2260449	-21	kworker/u16:3+events_unbound	09:45:01	4
1929301	99	26	24,2	cyclictest	2260449	-21	kworker/u16:3+events_unbound	09:45:01	4
1929301	99	26	24,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:06:44	4
1929311	99	25	23,1	cyclictest	2286950	-21	ssh	09:56:45	7
1929307	99	25	23,2	cyclictest	2554960	-21	kworker/u16:5+events_unbound	11:52:28	6
1929307	99	25	23,2	cyclictest	2491533	-21	kworker/u16:2+events_unbound	11:32:26	6
1929307	99	25	23,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	10:18:59	6
1929301	99	25	23,2	cyclictest	2554960	-21	kworker/u16:5+events_unbound	12:09:32	4
1929301	99	25	22,2	cyclictest	2237053	-21	kworker/u16:1+events_unbound	11:31:47	4
1929307	99	24	22,2	cyclictest	2155409	-21	kworker/u16:2+events_unbound	09:23:15	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17