OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Dec 03, 2025 12:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
756347	99	53	49,3	cyclictest	1297922	-21	fwupd	17:29:13	7
755905	2	53	40,10	sleep4	0	-21	swapper/4	15:21:15	4
756339	99	51	48,2	cyclictest	961034	-21	kworker/u16:4+flush-8:0	17:22:47	5
756347	99	50	44,5	cyclictest	589235	-21	systemd-journal	20:07:46	7
756347	99	49	44,4	cyclictest	951863	-21	latency_hist	16:57:48	7
756347	99	47	44,2	cyclictest	982293	-21	cat	17:12:47	7
756335	99	47	45,2	cyclictest	840706	-21	kworker/u16:2+events_unbound	16:11:58	4
747191	2	47	12,30	sleep7	0	-21	swapper/7	15:18:05	7
756347	99	46	40,5	cyclictest	1053104	-21	latency_hist	17:47:47	7
756339	99	46	44,2	cyclictest	911666	-21	kworker/u16:3+flush-8:0	16:38:14	5
756335	99	45	42,2	cyclictest	1120108	-21	kworker/u16:0+events_unbound	18:19:19	4
756339	99	43	41,2	cyclictest	1264638	-21	kworker/u16:3+flush-8:0	20:28:15	5
756335	99	42	39,2	cyclictest	1120108	-21	kworker/u16:0+flush-8:0	18:38:13	4
756335	99	41	39,2	cyclictest	1124892	-21	kworker/u16:2+events_unbound	18:37:14	4
756347	99	40	35,4	cyclictest	851008	-21	latency_hist	16:07:48	7
756335	99	40	38,2	cyclictest	1406305	-21	kworker/u16:4+events_unbound	20:52:22	4
756345	99	39	37,2	cyclictest	1264638	-21	kworker/u16:3+flush-8:0	20:28:11	6
756345	99	38	36,2	cyclictest	1012188	-21	kworker/u16:3+events_unbound	17:46:13	6
756339	99	38	36,2	cyclictest	811089	-21	kworker/u16:1+events_unbound	16:01:58	5
755925	2	38	28,7	sleep6	0	-21	swapper/6	15:21:33	6
756347	99	37	35,1	cyclictest	1325405	-21	latency_hist	20:02:46	7
756345	99	37	35,2	cyclictest	799675	-21	kworker/u16:4+events_unbound	15:44:09	6
756345	99	37	35,2	cyclictest	799675	-21	kworker/u16:4+events_unbound	15:44:09	6
756335	99	37	35,2	cyclictest	961034	-21	kworker/u16:4+events_unbound	17:09:15	4
756335	99	37	35,2	cyclictest	891138	-21	kworker/u16:1+events_unbound	16:45:11	4
755933	2	37	27,7	sleep5	0	-21	swapper/5	15:21:40	5
756347	99	36	35,0	cyclictest	0	-21	swapper/7	18:43:07	7
756347	99	36	32,3	cyclictest	4184977	-17	auditd	17:57:06	7
756345	99	36	34,2	cyclictest	953043	-21	kworker/u16:0+events_unbound	16:59:50	6
756345	99	36	34,2	cyclictest	911666	-21	kworker/u16:3+events_unbound	16:40:12	6
756339	99	36	34,2	cyclictest	1345777	-21	kworker/u16:0+events_unbound	20:20:21	5
756339	99	36	34,2	cyclictest	1022411	-21	kworker/u16:6+events_unbound	17:53:58	5
756335	99	36	34,2	cyclictest	1022411	-21	kworker/u16:6+events_unbound	17:45:11	4
756339	99	35	33,2	cyclictest	1143667	-21	kworker/u16:3+events_unbound	18:30:12	5
756335	99	35	33,2	cyclictest	941813	-21	kworker/u16:5+events_unbound	17:19:15	4
756335	99	35	33,2	cyclictest	1345777	-21	kworker/u16:0+flush-8:0	20:41:57	4
756335	99	35	33,2	cyclictest	1012188	-21	kworker/u16:3+events_unbound	17:57:49	4
756345	99	34	32,2	cyclictest	860169	-21	kworker/u16:0+events_unbound	16:15:58	6
756339	99	34	32,2	cyclictest	1355865	-21	kworker/u16:1+flush-8:0	20:38:16	5
756335	99	34	32,2	cyclictest	1002773	-21	kworker/u16:0+events_unbound	17:29:42	4
756345	99	33	31,2	cyclictest	1244455	-21	kworker/u16:1+events_unbound	19:20:58	6
756345	99	33	31,2	cyclictest	1164161	-21	kworker/u16:2+events_unbound	19:14:27	6
756345	99	33	31,2	cyclictest	1083798	-21	kworker/u16:0+flush-8:0	18:04:14	6
756339	99	33	31,2	cyclictest	840706	-21	kworker/u16:2+events_unbound	17:29:29	5
756339	99	33	31,2	cyclictest	790405	-21	kworker/u16:3+events_unbound	16:06:47	5
756339	99	33	31,2	cyclictest	1244455	-21	kworker/u16:1+events_unbound	19:34:14	5
756339	99	33	31,2	cyclictest	1164161	-21	kworker/u16:2+events_unbound	18:59:13	5
756335	99	33	30,2	cyclictest	953043	-21	kworker/u16:0+events_unbound	17:12:50	4
756347	99	32	30,1	cyclictest	0	-21	swapper/7	15:27:49	7
756345	99	32	30,2	cyclictest	961034	-21	kworker/u16:4+flush-8:0	17:12:03	6
756345	99	32	30,2	cyclictest	861187	-21	kworker/u16:4+events_unbound	16:21:59	6
756345	99	32	30,2	cyclictest	790405	-21	kworker/u16:3+flush-8:0	15:38:50	6
756345	99	32	30,2	cyclictest	1264638	-21	kworker/u16:3+flush-8:0	19:59:46	6
756345	99	32	30,2	cyclictest	1002773	-21	kworker/u16:0+events_unbound	17:25:13	6
756339	99	32	30,2	cyclictest	790405	-21	kworker/u16:3+flush-8:0	15:39:12	5
756339	99	32	30,2	cyclictest	746076	-21	kworker/u16:2+flush-8:0	15:29:14	5
756339	99	32	30,2	cyclictest	1406305	-21	kworker/u16:4+flush-8:0	20:52:30	5
756339	99	32	30,2	cyclictest	1264638	-21	kworker/u16:3+events_unbound	20:24:58	5
756339	99	32	30,2	cyclictest	1164161	-21	kworker/u16:2+events_unbound	19:30:11	5
756339	99	32	29,2	cyclictest	911666	-21	kworker/u16:3+flush-8:0	16:57:58	5
756335	99	32	30,2	cyclictest	1012188	-21	kworker/u16:3+flush-8:0	18:09:11	4
756347	99	31	30,0	cyclictest	0	-21	swapper/7	15:58:07	7
756347	99	31	26,4	cyclictest	992298	-21	latency_hist	17:17:47	7
756345	99	31	29,2	cyclictest	1406305	-21	kworker/u16:4+events_unbound	20:52:18	6
756345	99	31	29,2	cyclictest	1355865	-21	kworker/u16:1+events_unbound	20:19:26	6
756345	99	31	29,2	cyclictest	1264638	-21	kworker/u16:3+events_unbound	20:09:01	6
756345	99	31	29,2	cyclictest	1244455	-21	kworker/u16:1+events_unbound	19:32:30	6
756345	99	31	28,2	cyclictest	1143667	-21	kworker/u16:3+events_unbound	18:52:54	6
756339	99	31	29,2	cyclictest	941813	-21	kworker/u16:5+events_unbound	16:55:46	5
756339	99	31	29,2	cyclictest	1264638	-21	kworker/u16:3+flush-8:0	19:38:38	5
756339	99	31	29,2	cyclictest	1120108	-21	kworker/u16:0+flush-8:0	19:03:01	5
756339	99	31	29,2	cyclictest	1012188	-21	kworker/u16:3+events_unbound	17:39:03	5
756345	99	30	28,2	cyclictest	961034	-21	kworker/u16:4+events_unbound	17:19:54	6
756345	99	30	28,2	cyclictest	911666	-21	kworker/u16:3+events_unbound	16:57:12	6
756345	99	30	28,2	cyclictest	891138	-21	kworker/u16:1+events_unbound	16:50:57	6
756345	99	30	28,2	cyclictest	1395658	-21	kworker/u16:2+events_unbound	20:44:11	6
756345	99	30	28,2	cyclictest	1164161	-21	kworker/u16:2+flush-8:0	19:02:25	6
756345	99	30	28,2	cyclictest	1153877	-21	kworker/u16:1+events_unbound	18:39:11	6
756345	99	30	28,2	cyclictest	1022411	-21	kworker/u16:6+events_unbound	18:19:11	6
756339	99	30	28,2	cyclictest	860169	-21	kworker/u16:0+flush-8:0	16:16:58	5
756339	99	30	28,2	cyclictest	811089	-21	kworker/u16:1+events_unbound	16:11:54	5
756339	99	30	28,2	cyclictest	1395658	-21	kworker/u16:2+events_unbound	20:36:09	5
756339	99	30	28,2	cyclictest	1264638	-21	kworker/u16:3+events_unbound	20:05:38	5
756339	99	30	28,2	cyclictest	1264638	-21	kworker/u16:3+events_unbound	20:00:41	5
756339	99	30	28,2	cyclictest	1164161	-21	kworker/u16:2+flush-8:0	19:45:14	5
756339	99	30	28,2	cyclictest	1153877	-21	kworker/u16:1+events_unbound	18:38:20	5
756335	99	30	28,2	cyclictest	1244455	-21	kworker/u16:1+events_unbound	19:22:46	4
756347	99	29	24,4	cyclictest	830893	-21	awk	15:57:47	7
756345	99	29	27,2	cyclictest	840706	-21	kworker/u16:2+events_unbound	16:29:19	6
756345	99	29	27,2	cyclictest	735816	-21	kworker/u16:0+events_unbound	15:50:51	6
756345	99	29	27,2	cyclictest	1164161	-21	kworker/u16:2+events_unbound	20:06:09	6
756345	99	29	27,2	cyclictest	1120108	-21	kworker/u16:0+flush-8:0	19:07:47	6
756345	99	29	27,2	cyclictest	1012188	-21	kworker/u16:3+events_unbound	18:02:03	6
756345	99	29	27,2	cyclictest	1012188	-21	kworker/u16:3+events_unbound	17:56:15	6
756339	99	29	27,2	cyclictest	941813	-21	kworker/u16:5+events_unbound	17:14:02	5
756339	99	29	27,2	cyclictest	840706	-21	kworker/u16:2+flush-8:0	16:19:06	5
756339	99	29	27,2	cyclictest	840706	-21	kworker/u16:2+events_unbound	16:44:26	5
756339	99	29	27,2	cyclictest	735816	-21	kworker/u16:0+events_unbound	15:24:03	5
756339	99	29	27,2	cyclictest	1264638	-21	kworker/u16:3+flush-8:0	20:09:09	5
756339	99	29	27,2	cyclictest	1164161	-21	kworker/u16:2+events_unbound	19:49:13	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17