OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Mon Dec 08, 2025 12:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2291601	2	61	51,7	sleep4	0	-21	swapper/4	15:22:12	4
2291576	2	57	47,7	sleep6	0	-21	swapper/6	15:21:50	6
2291930	99	55	53,2	cyclictest	2749386	-21	kworker/u16:5+events_unbound	19:22:53	5
2291584	2	55	45,7	sleep5	0	-21	swapper/5	15:21:57	5
2291930	99	52	50,2	cyclictest	2788357	-21	kworker/u16:0+events_unbound	20:22:55	5
2291430	2	48	15,29	sleep7	0	-21	swapper/7	15:19:51	7
2291937	99	45	39,5	cyclictest	2608396	-21	latency_hist	17:57:27	7
2291937	99	41	40,0	cyclictest	0	-21	swapper/7	18:24:37	7
2291937	99	40	39,0	cyclictest	0	-21	swapper/7	16:12:30	7
2291935	99	40	37,2	cyclictest	2657801	-21	kworker/u16:1+flush-8:0	18:32:29	6
2291930	99	38	36,2	cyclictest	2739716	-21	kworker/u16:1+flush-8:0	19:08:38	5
2291937	99	37	34,2	cyclictest	2315963	-21	munin-run	15:32:28	7
2291935	99	37	35,2	cyclictest	2291457	-21	kworker/u16:1+flush-8:0	15:42:51	6
2291930	99	37	35,2	cyclictest	2597357	-21	kworker/u16:3+flush-8:0	17:48:37	5
2291937	99	36	29,5	cyclictest	589235	-21	systemd-journal	20:12:26	7
2291935	99	36	34,2	cyclictest	2577830	-21	kworker/u16:0+events_unbound	18:44:39	6
2291930	99	36	34,2	cyclictest	2788357	-21	kworker/u16:0+events_unbound	20:09:24	5
2291937	99	35	34,0	cyclictest	0	-21	swapper/7	20:41:39	7
2291937	99	35	34,0	cyclictest	0	-21	swapper/7	18:51:37	7
2291937	99	35	34,0	cyclictest	0	-21	swapper/7	17:35:38	7
2291937	99	35	34,0	cyclictest	0	-21	swapper/7	17:30:37	7
2291935	99	35	33,2	cyclictest	2788357	-21	kworker/u16:0+flush-8:0	20:22:51	6
2291935	99	35	33,2	cyclictest	2788357	-21	kworker/u16:0+events_unbound	19:59:14	6
2291935	99	35	33,2	cyclictest	2291457	-21	kworker/u16:1+events_unbound	15:24:26	6
2291930	99	35	33,2	cyclictest	2873240	-21	kworker/u16:3+flush-8:0	20:14:45	5
2291930	99	35	33,2	cyclictest	2395908	-21	kworker/u16:4+flush-8:0	16:40:06	5
2291926	99	35	32,2	cyclictest	2860717	-21	kworker/u16:1+events_unbound	20:32:45	4
2291937	99	34	33,0	cyclictest	0	-21	swapper/7	19:54:39	7
2291935	99	34	32,2	cyclictest	2476894	-21	kworker/u16:2+events_unbound	17:39:42	6
2291935	99	34	32,2	cyclictest	2261613	-21	kworker/u16:2+flush-8:0	15:57:38	6
2291930	99	34	32,2	cyclictest	2749386	-21	kworker/u16:5+flush-8:0	20:03:39	5
2291930	99	34	32,2	cyclictest	2749386	-21	kworker/u16:5+events_unbound	19:22:22	5
2291930	99	34	32,2	cyclictest	2395908	-21	kworker/u16:4+flush-8:0	16:57:38	5
2291935	99	33	31,2	cyclictest	2688166	-21	kworker/u16:3+events_unbound	19:01:37	6
2291930	99	33	31,2	cyclictest	2788357	-21	kworker/u16:0+flush-8:0	19:42:57	5
2291930	99	33	31,2	cyclictest	2788357	-21	kworker/u16:0+events_unbound	19:36:40	5
2291930	99	33	31,2	cyclictest	2577830	-21	kworker/u16:0+events_unbound	18:24:45	5
2291930	99	33	31,1	cyclictest	2588225	-21	kworker/u16:1+flush-8:0	17:57:42	5
2291937	99	32	31,0	cyclictest	0	-21	swapper/7	15:32:38	7
2291937	99	32	30,1	cyclictest	0	-21	swapper/7	18:00:53	7
2291935	99	32	30,2	cyclictest	2749386	-21	kworker/u16:5+events_unbound	20:33:42	6
2291930	99	32	30,2	cyclictest	2476917	-21	kworker/u16:3+events_unbound	17:36:40	5
2291930	99	32	30,2	cyclictest	2476894	-21	kworker/u16:2+flush-8:0	17:08:06	5
2291930	99	32	30,2	cyclictest	2281397	-21	kworker/u16:0+flush-8:0	15:49:46	5
2291930	99	32	30,2	cyclictest	2261613	-21	kworker/u16:2+events_unbound	15:37:26	5
2291937	99	31	31,0	cyclictest	0	-21	swapper/7	18:54:37	7
2291937	99	31	30,0	cyclictest	0	-21	swapper/7	19:33:40	7
2291935	99	31	29,2	cyclictest	2920801	-21	kworker/u16:2+flush-8:0	20:31:36	6
2291935	99	31	29,2	cyclictest	2739716	-21	kworker/u16:1+flush-8:0	19:06:22	6
2291935	99	31	29,2	cyclictest	2688166	-21	kworker/u16:3+events_unbound	19:14:45	6
2291935	99	31	29,2	cyclictest	2577830	-21	kworker/u16:0+events_unbound	18:49:22	6
2291935	99	31	29,2	cyclictest	2546953	-21	kworker/u16:5+events_unbound	17:54:45	6
2291935	99	31	29,2	cyclictest	2291457	-21	kworker/u16:1+events_unbound	16:31:43	6
2291930	99	31	29,2	cyclictest	2577830	-21	kworker/u16:0+events_unbound	18:30:33	5
2291930	99	31	29,2	cyclictest	2546953	-21	kworker/u16:5+events_unbound	18:17:41	5
2291930	99	31	29,2	cyclictest	2546953	-21	kworker/u16:5+events_unbound	17:44:29	5
2291930	99	31	29,2	cyclictest	2476917	-21	kworker/u16:3+events_unbound	16:55:42	5
2291937	99	30	29,0	cyclictest	0	-21	swapper/7	19:46:40	7
2291937	99	30	29,0	cyclictest	0	-21	swapper/7	19:16:40	7
2291935	99	30	28,2	cyclictest	2657801	-21	kworker/u16:1+flush-8:0	18:31:45	6
2291935	99	30	28,2	cyclictest	2546953	-21	kworker/u16:5+flush-8:0	18:02:38	6
2291935	99	30	28,2	cyclictest	2546953	-21	kworker/u16:5+events_unbound	18:14:41	6
2291935	99	30	28,2	cyclictest	2291457	-21	kworker/u16:1+events_unbound	15:56:58	6
2291935	99	30	28,2	cyclictest	2291457	-21	kworker/u16:1+events_unbound	15:56:58	6
2291935	99	30	27,2	cyclictest	2788357	-21	kworker/u16:0+events_unbound	19:27:27	6
2291930	99	30	29,1	cyclictest	2395908	-21	kworker/u16:4+flush-8:0	16:28:50	5
2291930	99	30	28,2	cyclictest	2920801	-21	kworker/u16:2+flush-8:0	20:37:09	5
2291930	99	30	28,2	cyclictest	2873240	-21	kworker/u16:3+flush-8:0	20:20:25	5
2291930	99	30	28,2	cyclictest	2800067	-21	kworker/u16:2+events_unbound	19:55:02	5
2291930	99	30	28,2	cyclictest	2800067	-21	kworker/u16:2+events_unbound	19:51:29	5
2291930	99	30	28,2	cyclictest	2749386	-21	kworker/u16:5+events_unbound	20:41:13	5
2291930	99	30	28,2	cyclictest	2710169	-21	kworker/u16:4+events_unbound	18:58:37	5
2291930	99	30	28,2	cyclictest	2688166	-21	kworker/u16:3+events_unbound	19:16:30	5
2291930	99	30	28,2	cyclictest	2395908	-21	kworker/u16:4+flush-8:0	16:14:34	5
2291930	99	30	28,2	cyclictest	2291457	-21	kworker/u16:1+flush-8:0	15:46:05	5
2291930	99	30	28,2	cyclictest	2261613	-21	kworker/u16:2+flush-8:0	15:58:33	5
2291937	99	29	28,0	cyclictest	0	-21	swapper/7	19:29:36	7
2291937	99	29	28,0	cyclictest	0	-21	swapper/7	17:07:18	7
2291937	99	29	23,5	cyclictest	589235	-21	systemd-journal	15:49:25	7
2291937	99	29	23,5	cyclictest	2598275	-21	latency_hist	17:52:28	7
2291935	99	29	27,2	cyclictest	2739716	-21	kworker/u16:1+events_unbound	19:25:26	6
2291935	99	29	27,2	cyclictest	2597357	-21	kworker/u16:3+flush-8:0	17:49:42	6
2291935	99	29	27,2	cyclictest	2476894	-21	kworker/u16:2+events_unbound	17:03:37	6
2291935	99	29	27,2	cyclictest	2395908	-21	kworker/u16:4+flush-8:0	17:07:54	6
2291935	99	29	27,2	cyclictest	2395908	-21	kworker/u16:4+events_unbound	16:54:45	6
2291935	99	29	27,2	cyclictest	2346318	-21	kworker/u16:4+flush-8:0	15:48:34	6
2291935	99	29	26,2	cyclictest	2546953	-21	kworker/u16:5+flush-8:0	17:26:42	6
2291935	99	29	26,2	cyclictest	2546953	-21	kworker/u16:5+flush-8:0	17:26:42	6
2291930	99	29	27,2	cyclictest	2788357	-21	kworker/u16:0+flush-8:0	20:01:08	5
2291930	99	29	27,2	cyclictest	2710169	-21	kworker/u16:4+events_unbound	18:52:06	5
2291930	99	29	27,2	cyclictest	2688166	-21	kworker/u16:3+events_unbound	19:30:34	5
2291930	99	29	27,2	cyclictest	2688166	-21	kworker/u16:3+events_unbound	18:35:41	5
2291930	99	29	27,2	cyclictest	2657801	-21	kworker/u16:1+events_unbound	18:46:05	5
2291930	99	29	27,2	cyclictest	2577830	-21	kworker/u16:0+events_unbound	17:55:34	5
2291930	99	29	27,2	cyclictest	2291457	-21	kworker/u16:1+events_unbound	16:44:30	5
2291930	99	29	27,2	cyclictest	2291457	-21	kworker/u16:1+events_unbound	15:26:11	5
2291937	99	28	28,0	cyclictest	0	-21	swapper/7	16:30:37	7
2291935	99	28	26,2	cyclictest	2800067	-21	kworker/u16:2+flush-8:0	19:56:30	6
2291935	99	28	26,2	cyclictest	2788357	-21	kworker/u16:0+events_unbound	19:42:55	6
2291935	99	28	26,2	cyclictest	2749386	-21	kworker/u16:5+events_unbound	19:11:33	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17