OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Dec 09, 2025 12:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3852886	2	100	0,1	sleep7	3852887	-21	sed	18:32:40	7
3464611	2	73	62,7	sleep5	0	-21	swapper/5	15:22:09	5
3464949	99	60	52,7	cyclictest	589235	-21	systemd-journal	17:02:23	7
3464949	99	58	53,4	cyclictest	3710692	-21	latency_hist	17:22:23	7
3464949	99	54	49,4	cyclictest	4103713	-21	latency_hist	20:37:23	7
3464949	99	54	48,5	cyclictest	3841582	-21	latency_hist	18:27:23	7
3464949	99	54	47,5	cyclictest	589235	-21	systemd-journal	18:57:23	7
3464578	2	47	37,7	sleep6	0	-21	swapper/6	15:21:46	6
3464940	99	43	41,2	cyclictest	3760857	-21	kworker/u16:2+flush-8:0	18:02:51	5
3464939	99	43	41,2	cyclictest	3528975	-21	kworker/u16:0+events_unbound	15:55:54	4
3464939	99	43	41,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	16:26:54	4
3464949	99	42	0,40	cyclictest	589235	-21	systemd-journal	19:37:22	7
3464619	2	41	31,7	sleep4	0	-21	swapper/4	15:22:16	4
3464613	2	41	5,32	sleep7	0	-21	swapper/7	15:22:11	7
3464940	99	40	38,2	cyclictest	3463840	-21	kworker/u16:3+flush-8:0	16:17:50	5
3464939	99	40	38,2	cyclictest	4043451	-21	kworker/u16:2+events_unbound	20:10:34	4
3464939	99	40	38,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	16:17:53	4
3464949	99	39	0,37	cyclictest	3519204	-21	latency_hist	15:47:23	7
3464949	99	38	33,4	cyclictest	3781225	-21	latency_hist	17:57:24	7
3464949	99	38	33,4	cyclictest	3700612	-21	latency_hist	17:17:23	7
3464939	99	38	36,2	cyclictest	3882259	-21	kworker/u16:0+events_unbound	19:06:54	4
3464939	99	38	36,2	cyclictest	3489362	-21	kworker/u16:5+events_unbound	15:46:56	4
3464949	99	37	0,35	cyclictest	3640181	-21	latency_hist	16:47:23	7
3464939	99	37	35,2	cyclictest	4043451	-21	kworker/u16:2+events_unbound	20:33:34	4
3464939	99	37	35,2	cyclictest	3972431	-21	kworker/u16:5+events_unbound	20:06:18	4
3464939	99	37	35,2	cyclictest	3740015	-21	kworker/u16:4+events_unbound	19:10:33	4
3464940	99	36	34,2	cyclictest	3740015	-21	kworker/u16:4+flush-8:0	19:12:49	5
3464939	99	36	34,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	17:09:58	4
3464949	99	35	31,3	cyclictest	3992921	-21	latency_hist	19:42:23	7
3464940	99	35	33,2	cyclictest	4063860	-21	kworker/u16:0+events_unbound	20:18:33	5
3464940	99	35	33,2	cyclictest	3489362	-21	kworker/u16:5+events_unbound	15:43:27	5
3464939	99	35	33,2	cyclictest	4024354	-21	kworker/u16:0+events_unbound	20:01:33	4
3464939	99	35	33,2	cyclictest	3952809	-21	kworker/u16:3+events_unbound	20:19:55	4
3464939	99	35	33,2	cyclictest	3892365	-21	kworker/u16:1+events_unbound	18:53:34	4
3464939	99	35	33,2	cyclictest	3760857	-21	kworker/u16:2+flush-8:0	18:05:54	4
3464939	99	35	33,2	cyclictest	3740015	-21	kworker/u16:4+events_unbound	18:50:54	4
3464939	99	35	33,2	cyclictest	3689511	-21	kworker/u16:0+events_unbound	17:28:34	4
3464939	99	35	33,2	cyclictest	3689511	-21	kworker/u16:0+events_unbound	17:28:34	4
3464939	99	35	33,2	cyclictest	3475100	-21	kworker/u16:0+events_unbound	15:32:52	4
3464939	99	35	33,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	17:04:53	4
3464940	99	34	32,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	15:54:26	5
3464940	99	34	31,3	cyclictest	3489362	-21	kworker/u16:5+flush-8:0	15:32:55	5
3464939	99	34	32,2	cyclictest	3882259	-21	kworker/u16:0+events_unbound	19:21:55	4
3464939	99	34	32,2	cyclictest	3760857	-21	kworker/u16:2+events_unbound	17:45:34	4
3464939	99	34	32,2	cyclictest	3590046	-21	kworker/u16:1+events_unbound	16:54:53	4
3464939	99	34	32,2	cyclictest	3463840	-21	kworker/u16:3+flush-8:0	16:27:47	4
3464939	99	34	31,2	cyclictest	3730565	-21	kworker/u16:1+events_unbound	18:17:33	4
3464949	99	33	27,5	cyclictest	589235	-21	systemd-journal	18:47:53	7
3464940	99	33	31,2	cyclictest	3760857	-21	kworker/u16:2+flush-8:0	18:32:17	5
3464940	99	33	31,2	cyclictest	3730565	-21	kworker/u16:1+events_unbound	17:42:18	5
3464940	99	33	30,2	cyclictest	3590046	-21	kworker/u16:1+flush-8:0	16:29:49	5
3464939	99	33	31,2	cyclictest	3952809	-21	kworker/u16:3+events_unbound	19:49:55	4
3464939	99	33	31,2	cyclictest	3942754	-21	kworker/u16:2+events_unbound	19:56:34	4
3464949	99	32	30,1	cyclictest	0	-21	swapper/7	17:22:54	7
3464949	99	32	27,4	cyclictest	589235	-21	systemd-journal	19:43:53	7
3464949	99	32	27,4	cyclictest	589235	-21	systemd-journal	19:43:53	7
3464940	99	32	30,2	cyclictest	3463840	-21	kworker/u16:3+flush-8:0	17:10:57	5
3464940	99	32	29,2	cyclictest	3882259	-21	kworker/u16:0+events_unbound	18:47:29	5
3464939	99	32	30,2	cyclictest	3569498	-21	kworker/u16:2+events_unbound	17:34:14	4
3464939	99	32	29,2	cyclictest	3740015	-21	kworker/u16:4+events_unbound	18:27:33	4
3464939	99	32	28,3	cyclictest	3730565	-21	kworker/u16:1+flush-8:0	18:22:30	4
3464940	99	31	29,2	cyclictest	4063860	-21	kworker/u16:0+flush-8:0	20:24:49	5
3464940	99	31	29,2	cyclictest	3882259	-21	kworker/u16:0+events_unbound	19:26:01	5
3464940	99	31	29,2	cyclictest	3590046	-21	kworker/u16:1+events_unbound	16:49:29	5
3464939	99	31	29,2	cyclictest	4043451	-21	kworker/u16:2+flush-8:0	20:15:55	4
3464939	99	31	29,2	cyclictest	3942754	-21	kworker/u16:2+events_unbound	19:29:57	4
3464939	99	31	29,2	cyclictest	3740015	-21	kworker/u16:4+flush-8:0	17:58:13	4
3464939	99	31	29,2	cyclictest	3590046	-21	kworker/u16:1+flush-8:0	17:02:08	4
3464939	99	31	29,2	cyclictest	3569498	-21	kworker/u16:2+flush-8:0	16:42:04	4
3464949	99	30	25,4	cyclictest	3589792	-21	latency_hist	16:22:24	7
3464940	99	30	28,2	cyclictest	3972431	-21	kworker/u16:5+flush-8:0	19:36:09	5
3464940	99	30	28,2	cyclictest	3911773	-21	kworker/u16:2+events_unbound	19:02:22	5
3464940	99	30	28,2	cyclictest	3911773	-21	kworker/u16:2+events_unbound	19:02:22	5
3464940	99	30	28,2	cyclictest	3866686	-21	kworker/u16:3+flush-8:0	18:39:01	5
3464940	99	30	28,2	cyclictest	3740015	-21	kworker/u16:4+events_unbound	18:24:37	5
3464940	99	30	28,2	cyclictest	3730565	-21	kworker/u16:1+flush-8:0	18:15:58	5
3464940	99	30	28,2	cyclictest	3730565	-21	kworker/u16:1+flush-8:0	18:15:58	5
3464940	99	30	28,2	cyclictest	3569498	-21	kworker/u16:2+events_unbound	17:30:17	5
3464940	99	30	28,2	cyclictest	3569498	-21	kworker/u16:2+events_unbound	17:30:17	5
3464940	99	30	28,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	15:49:17	5
3464939	99	30	28,2	cyclictest	3760857	-21	kworker/u16:2+events_unbound	17:52:38	4
3464939	99	30	28,2	cyclictest	3730565	-21	kworker/u16:1+flush-8:0	17:38:30	4
3464949	99	29	27,1	cyclictest	0	-21	swapper/7	16:12:49	7
3464949	99	29	24,4	cyclictest	4073537	-21	latency_hist	20:22:23	7
3464949	99	29	0,28	cyclictest	3972672	-21	latency_hist	19:32:23	7
3464940	99	29	27,2	cyclictest	3952809	-21	kworker/u16:3+events_unbound	20:41:41	5
3464940	99	29	27,2	cyclictest	3590046	-21	kworker/u16:1+events_unbound	16:55:38	5
3464940	99	29	27,2	cyclictest	3569498	-21	kworker/u16:2+events_unbound	16:34:39	5
3464940	99	29	27,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	17:15:42	5
3464939	99	29	27,2	cyclictest	3660102	-21	kworker/u16:4+events_unbound	17:20:54	4
3464939	99	29	27,2	cyclictest	3569498	-21	kworker/u16:2+events_unbound	16:12:34	4
3464939	99	29	27,2	cyclictest	3463840	-21	kworker/u16:3+flush-8:0	15:50:34	4
3464939	99	29	27,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	16:00:06	4
3464939	99	29	27,2	cyclictest	3463840	-21	kworker/u16:3+events_unbound	16:00:06	4
3464949	99	28	27,0	cyclictest	0	-21	swapper/7	18:37:39	7
3464949	99	28	24,3	cyclictest	589235	-21	systemd-journal	17:36:53	7
3464940	99	28	26,2	cyclictest	4043451	-21	kworker/u16:2+flush-8:0	20:15:24	5
3464940	99	28	26,2	cyclictest	4033002	-21	kworker/u16:1+flush-8:0	20:08:57	5
3464940	99	28	26,2	cyclictest	3952809	-21	kworker/u16:3+events_unbound	19:55:02	5
3464940	99	28	26,2	cyclictest	3882259	-21	kworker/u16:0+events_unbound	19:09:26	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17