OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Mon Nov 24, 2025 12:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2997590	2	52	41,7	sleep5	0	-21	swapper/5	15:22:49	5
2998006	99	51	0,49	cyclictest	3163670	-21	latency_hist	16:43:23	7
2998005	99	51	49,2	cyclictest	3486559	-21	kworker/u16:3+events_unbound	19:21:44	6
2998000	99	51	48,2	cyclictest	3446959	-21	kworker/u16:2+events_unbound	19:20:45	4
2998000	99	50	48,2	cyclictest	3446959	-21	kworker/u16:2+events_unbound	19:27:45	4
2998000	99	50	48,2	cyclictest	3183888	-21	kworker/u16:5+flush-8:0	18:50:45	4
2998000	99	48	46,2	cyclictest	1574040	-21	kworker/u16:0+events_unbound	15:47:46	4
2998000	99	47	45,2	cyclictest	3405816	-21	kworker/u16:0+flush-8:0	19:09:45	4
2998000	99	47	45,2	cyclictest	3405816	-21	kworker/u16:0+flush-8:0	19:09:45	4
2997569	2	47	36,8	sleep4	0	-21	swapper/4	15:22:32	4
2998006	99	46	44,1	cyclictest	0	-21	swapper/7	19:11:44	7
2998006	99	46	44,1	cyclictest	0	-21	swapper/7	19:11:44	7
2998000	99	46	44,2	cyclictest	3477121	-21	kworker/u16:1+events_unbound	20:14:24	4
2998006	99	45	40,4	cyclictest	3588694	-21	latency_hist	20:13:23	7
2997429	2	45	11,30	sleep7	0	-21	swapper/7	15:20:34	7
2998000	99	42	40,2	cyclictest	3092555	-21	kworker/u16:3+flush-8:0	16:08:48	4
2998000	99	42	40,2	cyclictest	3092555	-21	kworker/u16:3+flush-8:0	16:08:48	4
2998006	99	41	40,0	cyclictest	0	-21	swapper/7	18:35:46	7
2998006	99	41	40,0	cyclictest	0	-21	swapper/7	16:10:45	7
2998006	99	41	40,0	cyclictest	0	-21	swapper/7	16:10:45	7
2998006	99	41	36,4	cyclictest	3477448	-21	latency_hist	19:18:23	7
2998000	99	41	39,2	cyclictest	3183888	-21	kworker/u16:5+events_unbound	17:50:44	4
2998000	99	41	39,2	cyclictest	3182899	-21	kworker/u16:3+flush-8:0	17:39:36	4
2998000	99	41	39,2	cyclictest	3182899	-21	kworker/u16:3+flush-8:0	17:39:36	4
2998000	99	41	39,2	cyclictest	1574040	-21	kworker/u16:0+flush-8:0	16:23:48	4
2997543	2	41	29,8	sleep6	0	-21	swapper/6	15:22:10	6
2998006	99	40	38,1	cyclictest	0	-21	swapper/7	17:45:45	7
2998005	99	40	38,2	cyclictest	3042304	-21	kworker/u16:1+events_unbound	15:53:38	6
2998000	99	40	38,2	cyclictest	3657516	-21	kworker/u16:3+events_unbound	20:46:45	4
2998000	99	40	38,2	cyclictest	3557425	-21	kworker/u16:5+flush-8:0	20:04:28	4
2998005	99	39	37,2	cyclictest	3365336	-21	kworker/u16:3+events_unbound	18:31:44	6
2998006	99	38	36,1	cyclictest	0	-21	swapper/7	19:04:44	7
2998006	99	38	36,1	cyclictest	0	-21	swapper/7	16:37:46	7
2998006	99	38	35,2	cyclictest	3026792	-21	kthreadcore	15:33:43	7
2998006	99	38	33,4	cyclictest	589235	-21	systemd-journal	17:52:44	7
2998000	99	38	36,2	cyclictest	3557425	-21	kworker/u16:5+events_unbound	20:32:44	4
2998000	99	38	36,2	cyclictest	1574040	-21	kworker/u16:0+events_unbound	15:56:44	4
2998005	99	37	34,2	cyclictest	3446959	-21	kworker/u16:2+flush-8:0	20:03:49	6
2998005	99	37	34,2	cyclictest	3182899	-21	kworker/u16:3+flush-8:0	17:48:32	6
2998004	99	37	35,2	cyclictest	3557425	-21	kworker/u16:5+flush-8:0	20:03:51	5
2998004	99	37	35,2	cyclictest	3092555	-21	kworker/u16:3+flush-8:0	16:08:49	5
2998004	99	37	35,2	cyclictest	3092555	-21	kworker/u16:3+flush-8:0	16:08:49	5
2998006	99	36	35,0	cyclictest	0	-21	swapper/7	17:22:33	7
2998005	99	36	34,2	cyclictest	3477121	-21	kworker/u16:1+events_unbound	19:47:32	6
2998004	99	36	33,2	cyclictest	3334983	-21	kworker/u16:4+events_unbound	18:23:32	5
2998004	99	36	33,2	cyclictest	3334983	-21	kworker/u16:4+events_unbound	18:23:32	5
2998000	99	36	34,2	cyclictest	3356308	-21	kworker/u16:0+events_unbound	18:20:45	4
2998000	99	36	34,2	cyclictest	3133052	-21	kworker/u16:1+events_unbound	16:30:26	4
2998000	99	36	34,2	cyclictest	3092555	-21	kworker/u16:3+flush-8:0	16:20:45	4
2998000	99	36	33,2	cyclictest	3477121	-21	kworker/u16:1+events_unbound	19:48:24	4
2998004	99	35	33,2	cyclictest	3446959	-21	kworker/u16:2+flush-8:0	19:28:25	5
2998004	99	35	33,2	cyclictest	3133052	-21	kworker/u16:1+flush-8:0	16:28:49	5
2998000	99	35	33,2	cyclictest	3587743	-21	kworker/u16:0+events_unbound	20:26:33	4
2998000	99	35	33,2	cyclictest	3477121	-21	kworker/u16:1+flush-8:0	19:34:36	4
2998000	99	35	33,2	cyclictest	3183888	-21	kworker/u16:5+flush-8:0	17:59:33	4
2998004	99	34	32,2	cyclictest	1574040	-21	kworker/u16:0+flush-8:0	15:25:57	5
2998004	99	34	32,2	cyclictest	1574040	-21	kworker/u16:0+flush-8:0	15:25:57	5
2998000	99	34	32,2	cyclictest	3477121	-21	kworker/u16:1+events_unbound	20:12:44	4
2998000	99	34	31,2	cyclictest	3587743	-21	kworker/u16:0+events_unbound	20:18:32	4
2998006	99	33	32,0	cyclictest	0	-21	swapper/7	18:42:33	7
2998006	99	33	0,31	cyclictest	3447143	-21	latency_hist	19:03:23	7
2998005	99	33	31,2	cyclictest	3537960	-21	kworker/u16:0+flush-8:0	19:55:25	6
2998005	99	33	31,2	cyclictest	3537960	-21	kworker/u16:0+flush-8:0	19:55:25	6
2998005	99	33	31,2	cyclictest	3222035	-21	kworker/u16:2+events_unbound	17:45:37	6
2998005	99	33	31,2	cyclictest	3092555	-21	kworker/u16:3+events_unbound	16:08:49	6
2998005	99	33	31,2	cyclictest	3092555	-21	kworker/u16:3+events_unbound	16:08:49	6
2998004	99	33	31,2	cyclictest	3183888	-21	kworker/u16:5+events_unbound	17:35:41	5
2998004	99	33	31,2	cyclictest	3042304	-21	kworker/u16:1+events_unbound	15:44:46	5
2998000	99	33	31,2	cyclictest	3405816	-21	kworker/u16:0+events_unbound	19:07:44	4
2998000	99	33	31,2	cyclictest	3376843	-21	kworker/u16:1+events_unbound	18:28:52	4
2998000	99	33	31,2	cyclictest	1574040	-21	kworker/u16:0+flush-8:0	15:32:06	4
2998006	99	32	31,0	cyclictest	0	-21	swapper/7	16:26:33	7
2998006	99	32	27,4	cyclictest	3527982	-21	latency_hist	19:43:22	7
2998005	99	32	30,2	cyclictest	3417221	-21	kworker/u16:1+events_unbound	19:00:36	6
2998005	99	32	30,2	cyclictest	3203897	-21	kworker/u16:1+flush-8:0	17:00:09	6
2998005	99	32	30,2	cyclictest	3183888	-21	kworker/u16:5+events_unbound	17:56:37	6
2998005	99	32	30,2	cyclictest	3143107	-21	kworker/u16:2+events_unbound	16:34:38	6
2998005	99	32	30,2	cyclictest	3042304	-21	kworker/u16:1+flush-8:0	16:14:25	6
2998005	99	32	30,2	cyclictest	2941111	-21	kworker/u16:4+flush-8:0	15:34:44	6
2998004	99	32	30,2	cyclictest	3657516	-21	kworker/u16:3+flush-8:0	20:45:56	5
2998004	99	32	30,2	cyclictest	3537960	-21	kworker/u16:0+flush-8:0	19:56:17	5
2998004	99	32	30,2	cyclictest	3537960	-21	kworker/u16:0+flush-8:0	19:56:17	5
2998004	99	32	30,2	cyclictest	3477121	-21	kworker/u16:1+flush-8:0	20:27:48	5
2998004	99	32	30,2	cyclictest	3477121	-21	kworker/u16:1+events_unbound	19:34:28	5
2998004	99	32	30,2	cyclictest	3446959	-21	kworker/u16:2+events_unbound	19:24:28	5
2998004	99	32	30,2	cyclictest	3365336	-21	kworker/u16:3+flush-8:0	18:56:04	5
2998004	99	32	30,2	cyclictest	3042304	-21	kworker/u16:1+events_unbound	16:05:37	5
2998000	99	32	30,2	cyclictest	3477121	-21	kworker/u16:1+events_unbound	19:44:32	4
2998000	99	32	30,2	cyclictest	3183888	-21	kworker/u16:5+events_unbound	17:25:33	4
2998000	99	32	30,2	cyclictest	3153485	-21	kworker/u16:0+flush-8:0	16:59:02	4
2998005	99	31	29,2	cyclictest	3376843	-21	kworker/u16:1+events_unbound	18:39:37	6
2998005	99	31	29,2	cyclictest	3222035	-21	kworker/u16:2+flush-8:0	17:36:41	6
2998005	99	31	29,2	cyclictest	3183888	-21	kworker/u16:5+events_unbound	17:20:30	6
2998004	99	31	29,2	cyclictest	3587743	-21	kworker/u16:0+flush-8:0	20:33:08	5
2998004	99	31	29,2	cyclictest	3182899	-21	kworker/u16:3+events_unbound	17:39:33	5
2998004	99	31	29,2	cyclictest	3182899	-21	kworker/u16:3+events_unbound	17:39:33	5
2998000	99	31	28,2	cyclictest	3183888	-21	kworker/u16:5+flush-8:0	17:28:33	4
2998006	99	30	28,1	cyclictest	0	-21	swapper/7	15:26:22	7
2998006	99	30	28,1	cyclictest	0	-21	swapper/7	15:26:22	7
2998006	99	30	25,3	cyclictest	3426932	-21	idleruntime-cro	18:53:23	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17