OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sat Nov 08, 2025 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2425786	99	58	55,2	cyclictest	2530865	-21	kworker/u16:1+flush-8:0	04:34:53	4
2425362	2	58	23,31	sleep7	0	-21	swapper/7	03:23:05	7
2425162	2	56	46,7	sleep5	0	-21	swapper/5	03:20:23	5
2425786	99	53	51,2	cyclictest	2833078	-21	kworker/u16:6+events_unbound	06:59:57	4
2425144	2	49	37,8	sleep6	0	-21	swapper/6	03:20:07	6
2425794	99	48	0,46	cyclictest	2863747	-21	latency_hist	06:59:29	7
2425453	2	48	39,6	sleep4	0	-21	swapper/4	03:24:23	4
2425792	99	45	43,2	cyclictest	2833078	-21	kworker/u16:6+events_unbound	06:45:59	6
2425786	99	43	41,2	cyclictest	3085381	-21	kworker/u16:2+events_unbound	08:52:55	4
2425792	99	41	39,2	cyclictest	2560765	-21	kworker/u16:2+flush-8:0	04:31:56	6
2425788	99	41	39,2	cyclictest	2924650	-21	kworker/u16:0+events_unbound	07:47:07	5
2425786	99	40	38,2	cyclictest	2671724	-21	kworker/u16:4+events_unbound	05:52:57	4
2425792	99	39	37,2	cyclictest	2974500	-21	kworker/u16:4+events_unbound	08:48:23	6
2425792	99	39	37,2	cyclictest	2651972	-21	kworker/u16:2+events_unbound	05:23:56	6
2425786	99	39	37,2	cyclictest	2974500	-21	kworker/u16:4+events_unbound	08:15:55	4
2425788	99	38	36,2	cyclictest	2833078	-21	kworker/u16:6+flush-8:0	06:59:57	5
2425788	99	38	36,2	cyclictest	2762667	-21	kworker/u16:1+flush-8:0	06:40:50	5
2425786	99	38	36,2	cyclictest	2762667	-21	kworker/u16:1+events_unbound	06:35:57	4
2425786	99	38	36,2	cyclictest	2395615	-21	kworker/u16:3+events_unbound	03:59:54	4
2425794	99	37	32,4	cyclictest	2813216	-21	latency_hist	06:34:28	7
2425792	99	37	35,2	cyclictest	2954453	-21	kworker/u16:3+flush-8:0	07:45:03	6
2425792	99	37	35,2	cyclictest	2873847	-21	kworker/u16:2+flush-8:0	07:39:44	6
2425788	99	37	36,1	cyclictest	2772600	-21	kworker/u16:4+events_unbound	06:24:55	5
2425788	99	37	35,2	cyclictest	2994723	-21	kworker/u16:0+events_unbound	08:24:54	5
2425786	99	37	35,2	cyclictest	2752558	-21	kworker/u16:0+events_unbound	06:10:27	4
2425792	99	36	34,2	cyclictest	2762667	-21	kworker/u16:1+events_unbound	06:25:51	6
2425792	99	36	34,2	cyclictest	2395615	-21	kworker/u16:3+flush-8:0	03:59:54	6
2425786	99	36	34,2	cyclictest	2530865	-21	kworker/u16:1+flush-8:0	04:21:20	4
2425794	99	35	30,4	cyclictest	2469935	-21	latency_hist	03:44:29	7
2425794	99	35	30,4	cyclictest	2469935	-21	latency_hist	03:44:29	7
2425792	99	35	33,2	cyclictest	2994723	-21	kworker/u16:0+events_unbound	08:43:57	6
2425792	99	35	33,2	cyclictest	2793376	-21	kworker/u16:0+flush-8:0	06:39:44	6
2425792	99	35	33,2	cyclictest	2571270	-21	kworker/u16:4+events_unbound	04:41:20	6
2425792	99	35	33,2	cyclictest	2510055	-21	kworker/u16:0+events_unbound	04:10:59	6
2425786	99	35	33,2	cyclictest	2974500	-21	kworker/u16:4+events_unbound	08:46:57	4
2425786	99	35	33,2	cyclictest	2823432	-21	kworker/u16:3+events_unbound	06:45:57	4
2425786	99	35	33,2	cyclictest	2651972	-21	kworker/u16:2+events_unbound	05:27:21	4
2425786	99	35	32,2	cyclictest	2349273	-21	kworker/u16:0+events_unbound	03:44:30	4
2425792	99	34	32,2	cyclictest	2863454	-21	kworker/u16:0+events_unbound	07:08:00	6
2425788	99	34	32,2	cyclictest	2651972	-21	kworker/u16:2+events_unbound	05:29:56	5
2425786	99	34	32,2	cyclictest	2994723	-21	kworker/u16:0+events_unbound	08:35:55	4
2425786	99	34	32,2	cyclictest	2772600	-21	kworker/u16:4+events_unbound	06:19:20	4
2425786	99	34	32,2	cyclictest	2395615	-21	kworker/u16:3+flush-8:0	03:26:24	4
2425794	99	33	32,0	cyclictest	0	-21	swapper/7	03:24:57	7
2425792	99	33	31,2	cyclictest	2954453	-21	kworker/u16:3+events_unbound	08:06:28	6
2425792	99	33	31,2	cyclictest	2691915	-21	kworker/u16:3+events_unbound	05:55:20	6
2425792	99	33	31,2	cyclictest	2671693	-21	kworker/u16:1+events_unbound	05:31:00	6
2425792	99	33	31,2	cyclictest	2651972	-21	kworker/u16:2+events_unbound	05:25:57	6
2425792	99	33	31,2	cyclictest	2601167	-21	kworker/u16:0+events_unbound	05:15:20	6
2425792	99	33	30,2	cyclictest	2823432	-21	kworker/u16:3+events_unbound	06:54:31	6
2425792	99	33	30,2	cyclictest	2752558	-21	kworker/u16:0+events_unbound	06:09:30	6
2425792	99	33	30,2	cyclictest	2395615	-21	kworker/u16:3+events_unbound	04:34:32	6
2425792	99	33	29,3	cyclictest	2631169	-21	kworker/u16:4+flush-8:0	05:01:44	6
2425788	99	33	31,2	cyclictest	2510055	-21	kworker/u16:0+events_unbound	04:28:44	5
2425788	99	33	31,2	cyclictest	2510055	-21	kworker/u16:0+events_unbound	04:28:44	5
2425788	99	33	31,2	cyclictest	2349273	-21	kworker/u16:0+events_unbound	03:41:20	5
2425788	99	33	31,2	cyclictest	2349273	-21	kworker/u16:0+events_unbound	03:41:20	5
2425786	99	33	31,2	cyclictest	2873847	-21	kworker/u16:2+events_unbound	07:10:40	4
2425786	99	33	31,2	cyclictest	2671724	-21	kworker/u16:4+events_unbound	05:41:57	4
2425786	99	33	31,2	cyclictest	2671693	-21	kworker/u16:1+flush-8:0	05:30:57	4
2425792	99	32	30,2	cyclictest	2873847	-21	kworker/u16:2+flush-8:0	07:33:39	6
2425792	99	32	30,2	cyclictest	2722309	-21	kworker/u16:2+events_unbound	06:15:59	6
2425792	99	32	29,3	cyclictest	2772600	-21	kworker/u16:4+flush-8:0	06:24:24	6
2425788	99	32	30,2	cyclictest	2873847	-21	kworker/u16:2+events_unbound	07:11:27	5
2425788	99	32	30,2	cyclictest	2651972	-21	kworker/u16:2+events_unbound	05:20:21	5
2425788	99	32	30,2	cyclictest	2550511	-21	kworker/u16:2+events_unbound	04:22:36	5
2425788	99	32	30,2	cyclictest	2530865	-21	kworker/u16:1+events_unbound	05:03:31	5
2425786	99	32	30,2	cyclictest	2833078	-21	kworker/u16:6+events_unbound	06:51:55	4
2425786	99	32	30,2	cyclictest	2722309	-21	kworker/u16:2+events_unbound	05:59:19	4
2425786	99	32	30,2	cyclictest	2691915	-21	kworker/u16:3+events_unbound	06:02:57	4
2425786	99	32	29,2	cyclictest	2395615	-21	kworker/u16:3+flush-8:0	03:54:32	4
2425794	99	31	30,0	cyclictest	0	-21	swapper/7	04:27:00	7
2425794	99	31	30,0	cyclictest	0	-21	swapper/7	04:27:00	7
2425794	99	31	0,29	cyclictest	589235	-21	systemd-journal	07:24:55	7
2425792	99	31	29,2	cyclictest	2994723	-21	kworker/u16:0+flush-8:0	08:50:55	6
2425792	99	31	29,2	cyclictest	2863710	-21	kworker/u16:1+events_unbound	08:00:32	6
2425792	99	31	29,2	cyclictest	2793376	-21	kworker/u16:0+events_unbound	06:36:56	6
2425792	99	31	29,2	cyclictest	2702239	-21	kworker/u16:0+events_unbound	05:47:29	6
2425792	99	31	29,2	cyclictest	2601167	-21	kworker/u16:0+events_unbound	05:13:21	6
2425792	99	31	29,2	cyclictest	2395615	-21	kworker/u16:3+events_unbound	04:06:58	6
2425788	99	31	29,2	cyclictest	2873847	-21	kworker/u16:2+events_unbound	07:38:35	5
2425788	99	31	29,2	cyclictest	2772600	-21	kworker/u16:4+events_unbound	06:48:55	5
2425788	99	31	29,2	cyclictest	2395615	-21	kworker/u16:3+events_unbound	04:17:37	5
2425786	99	31	29,2	cyclictest	2954453	-21	kworker/u16:3+events_unbound	07:48:39	4
2425786	99	31	29,2	cyclictest	2873847	-21	kworker/u16:2+events_unbound	07:33:32	4
2425786	99	31	29,2	cyclictest	2601167	-21	kworker/u16:0+events_unbound	05:19:19	4
2425786	99	31	29,2	cyclictest	2510055	-21	kworker/u16:0+events_unbound	04:15:20	4
2425786	99	31	28,2	cyclictest	2762667	-21	kworker/u16:1+flush-8:0	06:29:40	4
2425792	99	30	28,2	cyclictest	3045165	-21	kworker/u16:1+flush-8:0	08:27:35	6
2425792	99	30	28,2	cyclictest	2994723	-21	kworker/u16:0+events_unbound	08:32:00	6
2425792	99	30	28,2	cyclictest	2974500	-21	kworker/u16:4+events_unbound	08:23:24	6
2425792	99	30	28,2	cyclictest	2954453	-21	kworker/u16:3+events_unbound	07:50:48	6
2425792	99	30	28,2	cyclictest	2722309	-21	kworker/u16:2+flush-8:0	06:04:04	6
2425792	99	30	28,2	cyclictest	2691915	-21	kworker/u16:3+flush-8:0	05:40:51	6
2425788	99	30	28,2	cyclictest	2924650	-21	kworker/u16:0+events_unbound	07:33:44	5
2425788	99	30	28,2	cyclictest	2833078	-21	kworker/u16:6+flush-8:0	07:21:47	5
2425788	99	30	28,2	cyclictest	2823432	-21	kworker/u16:3+events_unbound	06:59:16	5
2425788	99	30	28,2	cyclictest	2722309	-21	kworker/u16:2+events_unbound	06:04:16	5
2425788	99	30	28,2	cyclictest	2702239	-21	kworker/u16:0+events_unbound	05:45:36	5
2425788	99	30	28,2	cyclictest	2651972	-21	kworker/u16:2+events_unbound	05:27:47	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17