OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sat Aug 15, 2026 12:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3538496	2	295	0,13	sleep6	0	-21	swapper/6	07:03:15	6
3975070	2	101	0,1	sleep7	3975071	-21	kthreadcore	10:23:17	7
4177538	2	99	0,1	sleep7	4177539	-21	cut	11:48:13	7
3543969	2	54	19,30	sleep7	0	-21	swapper/7	07:04:05	7
3544094	2	47	22,7	sleep5	0	-21	swapper/5	07:05:48	5
3544537	99	46	44,2	cyclictest	3657003	-21	kworker/u16:0+flush-8:0	08:28:24	5
3544537	99	46	43,2	cyclictest	3597051	-21	kworker/u16:3+flush-8:0	08:43:11	5
3544537	99	46	43,2	cyclictest	3597051	-21	kworker/u16:3+flush-8:0	08:43:11	5
3544544	99	45	42,2	cyclictest	3832799	-21	iostat	09:28:11	7
3544544	99	42	41,0	cyclictest	0	-21	swapper/7	09:58:28	7
3544544	99	42	0,40	cyclictest	648	-21	systemd-journal	10:57:53	7
3544544	99	41	39,1	cyclictest	0	-21	swapper/7	11:37:56	7
3544537	99	41	39,2	cyclictest	3714476	-21	kworker/u16:4+flush-8:0	08:33:11	5
3544537	99	41	39,2	cyclictest	3597051	-21	kworker/u16:3+flush-8:0	10:08:24	5
3544537	99	41	38,3	cyclictest	4151278	-21	kworker/u16:3+flush-8:0	11:53:15	5
3544544	99	40	37,2	cyclictest	3959348	-21	kthreadcore	10:18:16	7
3544537	99	40	38,2	cyclictest	3657003	-21	kworker/u16:0+flush-8:0	08:08:24	5
3544529	99	39	37,2	cyclictest	3644864	-21	kworker/u16:1+flush-8:0	08:28:20	4
3544544	99	38	37,0	cyclictest	0	-21	swapper/7	07:23:23	7
3544537	99	38	36,2	cyclictest	4089878	-21	kworker/u16:0+flush-8:0	11:48:09	5
3544537	99	37	35,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	10:03:09	5
3544537	99	36	34,2	cyclictest	4065383	-21	kworker/u16:4+flush-8:0	11:03:25	5
3544529	99	36	34,2	cyclictest	3490308	-21	kworker/u16:5+events_unbound	08:03:10	4
3544111	2	36	25,8	sleep4	0	-21	swapper/4	07:06:03	4
3544542	99	35	33,2	cyclictest	4027150	-21	kworker/u16:0+events_unbound	10:58:08	6
3544542	99	35	33,2	cyclictest	3277196	-21	kworker/u16:4+flush-8:0	07:48:09	6
3544542	99	35	32,2	cyclictest	3714476	-21	kworker/u16:4+flush-8:0	08:33:09	6
3544544	99	34	30,3	cyclictest	125	-21	kswapd0	08:16:52	7
3544537	99	34	32,2	cyclictest	4089878	-21	kworker/u16:0+flush-8:0	11:13:23	5
3544537	99	34	32,2	cyclictest	4050185	-21	kworker/u16:3+flush-8:0	11:18:28	5
3544537	99	34	32,2	cyclictest	4027150	-21	kworker/u16:0+flush-8:0	11:02:53	5
3544537	99	34	31,2	cyclictest	66655	-21	kworker/u16:1+events_unbound	12:32:55	5
3544537	99	34	31,2	cyclictest	66655	-21	kworker/u16:1+events_unbound	12:32:55	5
3544529	99	34	32,2	cyclictest	3644864	-21	kworker/u16:1+events_unbound	08:13:27	4
3544544	99	33	30,2	cyclictest	4172226	-21	ntpq	11:43:21	7
3544537	99	33	31,2	cyclictest	3856061	-21	kworker/u16:4+events_unbound	09:43:27	5
3544537	99	32	30,2	cyclictest	3277196	-21	kworker/u16:4+flush-8:0	07:13:09	5
3544537	99	32	29,2	cyclictest	3774575	-21	kworker/u16:0+flush-8:0	09:08:11	5
3544537	99	32	29,2	cyclictest	3490308	-21	kworker/u16:5+flush-8:0	07:33:11	5
3544544	99	31	30,0	cyclictest	0	-21	swapper/7	10:49:19	7
3544544	99	31	28,2	cyclictest	3818817	-21	ssh	09:22:15	7
3544544	99	31	27,2	cyclictest	4061768	-21	sh	10:58:19	7
3544542	99	31	29,2	cyclictest	4089878	-21	kworker/u16:0+flush-8:0	11:32:48	6
3544542	99	31	29,2	cyclictest	3760339	-21	kworker/u16:2+events_unbound	09:08:24	6
3544542	99	31	29,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	08:51:28	6
3544537	99	31	29,2	cyclictest	3760339	-21	kworker/u16:2+flush-8:0	10:38:20	5
3544544	99	30	28,1	cyclictest	3903730	-21	sh	09:55:19	7
3544544	99	30	24,5	cyclictest	4077222	-21	latency_hist	11:07:54	7
3544544	99	30	0,28	cyclictest	4026942	-21	sh	10:45:49	7
3544542	99	30	28,2	cyclictest	3684625	-21	kworker/u16:2+events_unbound	08:25:28	6
3544537	99	30	28,2	cyclictest	4028527	-21	kworker/u16:1+events_unbound	10:48:16	5
3544537	99	30	28,2	cyclictest	3844762	-21	kworker/u16:1+events_unbound	09:58:12	5
3544537	99	30	28,2	cyclictest	3760339	-21	kworker/u16:2+flush-8:0	11:23:10	5
3544537	99	30	28,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	09:33:27	5
3544537	99	30	27,2	cyclictest	3597051	-21	kworker/u16:3+flush-8:0	07:43:11	5
3544544	99	29	27,1	cyclictest	90832	-21	ssh	12:30:03	7
3544542	99	29	27,2	cyclictest	3644864	-21	kworker/u16:1+events_unbound	08:46:27	6
3544542	99	29	27,2	cyclictest	3644864	-21	kworker/u16:1+events_unbound	08:46:27	6
3544542	99	29	27,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	10:12:30	6
3544542	99	29	27,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	09:05:07	6
3544542	99	29	27,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	08:31:08	6
3544542	99	29	26,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	07:42:56	6
3544537	99	29	26,2	cyclictest	3856061	-21	kworker/u16:4+flush-8:0	09:52:56	5
3544537	99	29	26,2	cyclictest	3760339	-21	kworker/u16:2+events_unbound	09:28:28	5
3544529	99	29	27,2	cyclictest	4028527	-21	kworker/u16:1+flush-8:0	11:13:35	4
3544544	99	28	26,1	cyclictest	4136510	-21	ssh	11:29:11	7
3544544	99	28	25,2	cyclictest	3937814	-21	ssh	10:08:17	7
3544542	99	28	26,2	cyclictest	3760339	-21	kworker/u16:2+events_unbound	08:55:07	6
3544542	99	28	26,2	cyclictest	3684625	-21	kworker/u16:2+events_unbound	08:20:33	6
3544537	99	28	26,2	cyclictest	4175488	-21	kworker/u16:4+events_unbound	12:07:48	5
3544537	99	28	26,2	cyclictest	4175488	-21	kworker/u16:4+events_unbound	12:07:48	5
3544537	99	28	26,2	cyclictest	4151278	-21	kworker/u16:3+flush-8:0	12:08:16	5
3544537	99	28	26,2	cyclictest	3657003	-21	kworker/u16:0+flush-8:0	08:38:23	5
3544529	99	28	26,2	cyclictest	3714476	-21	kworker/u16:4+events_unbound	08:44:08	4
3544529	99	28	26,2	cyclictest	3714476	-21	kworker/u16:4+events_unbound	08:44:08	4
3544529	99	28	26,2	cyclictest	3684625	-21	kworker/u16:2+events_unbound	08:19:44	4
3544529	99	28	26,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	07:34:28	4
3544529	99	28	26,2	cyclictest	3482452	-21	kworker/u16:1+flush-8:0	07:23:20	4
3544544	99	27	26,0	cyclictest	0	-21	swapper/7	12:11:04	7
3544544	99	27	26,0	cyclictest	0	-21	swapper/7	08:28:09	7
3544544	99	27	23,3	cyclictest	3826140	-21	ssh	09:23:19	7
3544544	99	27	0,25	cyclictest	30955	-21	ssh	12:07:44	7
3544544	99	27	0,25	cyclictest	30955	-21	ssh	12:07:44	7
3544542	99	27	25,2	cyclictest	4065383	-21	kworker/u16:4+events_unbound	11:04:45	6
3544542	99	27	25,2	cyclictest	3760339	-21	kworker/u16:2+events_unbound	10:15:55	6
3544542	99	27	25,2	cyclictest	3657003	-21	kworker/u16:0+events_unbound	08:39:27	6
3544542	99	27	25,2	cyclictest	3597051	-21	kworker/u16:3+events_unbound	10:37:06	6
3544542	99	27	25,2	cyclictest	3490308	-21	kworker/u16:5+events_unbound	07:59:39	6
3544542	99	27	25,2	cyclictest	3490308	-21	kworker/u16:5+events_unbound	07:42:52	6
3544542	99	27	25,2	cyclictest	3490308	-21	kworker/u16:5+events_unbound	07:33:27	6
3544542	99	27	25,2	cyclictest	3277196	-21	kworker/u16:4+events_unbound	07:10:53	6
3544537	99	27	25,2	cyclictest	3893310	-21	kworker/u16:0+events_unbound	10:13:27	5
3544537	99	27	25,2	cyclictest	3760339	-21	kworker/u16:2+events_unbound	08:58:12	5
3544529	99	27	25,2	cyclictest	79927	-21	kworker/u16:3+events_unbound	12:30:54	4
3544529	99	27	25,2	cyclictest	3644864	-21	kworker/u16:1+events_unbound	08:33:47	4
3544529	99	27	25,2	cyclictest	3625843	-21	kworker/u16:0+events_unbound	07:45:32	4
102898	2	27	0,1	chrt	102896	-21	ssh	12:34:45	7
102898	2	27	0,1	chrt	102896	-21	ssh	12:34:45	7
3559121	2	26	0,1	chrt	3559122	-21	if_err_p4p1	07:13:10	7
3544544	99	26	25,0	cyclictest	0	-21	swapper/7	10:38:09	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17