OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Feb 18, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
4049569	2	99	0,1	sleep7	4049572	-21	sort	12:00:50	7
3761251	2	65	0,1	sleep7	3761252	-21	cat	10:01:00	7
3385797	99	60	58,2	cyclictest	3513685	-21	kworker/u16:2+events_unbound	08:37:32	6
3385398	2	59	49,7	sleep5	0	-21	swapper/5	07:03:42	5
3385797	99	54	52,2	cyclictest	3980661	-21	kworker/u16:2+events_unbound	12:00:39	6
3385797	99	52	50,2	cyclictest	4000037	-21	kworker/u16:5+events_unbound	12:10:58	6
3385797	99	51	49,2	cyclictest	3726780	-21	kworker/u16:2+events_unbound	10:15:52	6
3385346	2	51	40,7	sleep4	0	-21	swapper/4	07:03:00	4
3385797	99	48	45,2	cyclictest	3872691	-21	kworker/u16:2+events_unbound	11:10:41	6
3385464	2	47	36,8	sleep6	0	-21	swapper/6	07:04:39	6
3385797	99	46	44,2	cyclictest	3980661	-21	kworker/u16:2+events_unbound	11:57:47	6
3385797	99	44	42,2	cyclictest	3968574	-21	kworker/u16:0+events_unbound	12:05:51	6
3385797	99	44	41,2	cyclictest	3763789	-21	kworker/u16:4+events_unbound	10:10:43	6
3385801	99	42	37,4	cyclictest	3872273	-21	latency_hist	10:50:26	7
3385797	99	42	40,2	cyclictest	3475235	-21	kworker/u16:0+flush-8:0	07:55:26	6
3385797	99	41	39,2	cyclictest	3894105	-21	kworker/u16:3+events_unbound	11:15:59	6
3385797	99	41	39,2	cyclictest	3894105	-21	kworker/u16:3+events_unbound	11:15:59	6
3385797	99	41	39,2	cyclictest	3872691	-21	kworker/u16:2+events_unbound	11:09:52	6
3385797	99	41	39,2	cyclictest	3872691	-21	kworker/u16:2+events_unbound	10:51:32	6
3385797	99	41	39,2	cyclictest	3821701	-21	kworker/u16:4+events_unbound	10:31:36	6
3385797	99	41	39,2	cyclictest	3691521	-21	kworker/u16:0+flush-8:0	09:45:42	6
3385795	99	41	38,2	cyclictest	1925032	-21	kworker/u16:5+events_unbound	07:10:28	5
3385797	99	40	38,2	cyclictest	4000037	-21	kworker/u16:5+events_unbound	12:30:23	6
3385797	99	40	38,2	cyclictest	3968574	-21	kworker/u16:0+events_unbound	11:44:39	6
3385797	99	40	38,2	cyclictest	3714189	-21	kworker/u16:3+events_unbound	10:00:53	6
3376456	2	40	3,31	sleep7	0	-21	swapper/7	07:00:39	7
3385797	99	39	37,2	cyclictest	3955653	-21	kworker/u16:1+events_unbound	11:24:28	6
3385797	99	39	37,2	cyclictest	3872691	-21	kworker/u16:2+flush-8:0	10:55:36	6
3385797	99	39	37,2	cyclictest	3714189	-21	kworker/u16:3+events_unbound	10:42:36	6
3385797	99	39	37,2	cyclictest	3631571	-21	kworker/u16:1+events_unbound	09:50:51	6
3385797	99	38	37,1	cyclictest	3955653	-21	kworker/u16:1+events_unbound	11:29:07	6
3385797	99	37	35,2	cyclictest	4074476	-21	kworker/u16:0+events_unbound	12:33:11	6
3385797	99	37	35,2	cyclictest	4074476	-21	kworker/u16:0+events_unbound	12:33:11	6
3385797	99	37	35,2	cyclictest	4062047	-21	kworker/u16:1+events_unbound	12:20:39	6
3385797	99	37	35,2	cyclictest	3821661	-21	kworker/u16:1+events_unbound	10:35:44	6
3385797	99	36	34,2	cyclictest	4000037	-21	kworker/u16:5+events_unbound	11:49:36	6
3385797	99	36	34,2	cyclictest	3821661	-21	kworker/u16:1+events_unbound	11:00:55	6
3385797	99	36	34,2	cyclictest	3714189	-21	kworker/u16:3+events_unbound	10:06:35	6
3385797	99	36	34,2	cyclictest	3691521	-21	kworker/u16:0+flush-8:0	09:55:59	6
3385797	99	36	34,2	cyclictest	3691521	-21	kworker/u16:0+flush-8:0	09:36:00	6
3385797	99	36	34,2	cyclictest	3691521	-21	kworker/u16:0+events_unbound	10:26:00	6
3385797	99	36	34,2	cyclictest	3631571	-21	kworker/u16:1+flush-8:0	09:26:08	6
3385797	99	36	34,2	cyclictest	3551852	-21	kworker/u16:6+events_unbound	09:15:23	6
3385797	99	36	34,2	cyclictest	3551852	-21	kworker/u16:6+events_unbound	09:15:23	6
3385797	99	36	34,2	cyclictest	3513685	-21	kworker/u16:2+events_unbound	09:42:35	6
3385797	99	36	34,2	cyclictest	3513685	-21	kworker/u16:2+events_unbound	09:24:36	6
3385797	99	35	33,2	cyclictest	3590590	-21	kworker/u16:3+events_unbound	09:19:47	6
3385797	99	35	33,2	cyclictest	3513685	-21	kworker/u16:2+events_unbound	09:31:07	6
3385797	99	34	32,2	cyclictest	4062047	-21	kworker/u16:1+events_unbound	12:18:43	6
3385797	99	34	32,2	cyclictest	3590590	-21	kworker/u16:3+events_unbound	09:10:24	6
3385797	99	33	31,2	cyclictest	3980661	-21	kworker/u16:2+flush-8:0	11:50:59	6
3385797	99	33	31,2	cyclictest	3980661	-21	kworker/u16:2+flush-8:0	11:50:59	6
3385797	99	33	31,2	cyclictest	3714189	-21	kworker/u16:3+events_unbound	10:45:54	6
3385797	99	33	31,2	cyclictest	3691521	-21	kworker/u16:0+events_unbound	10:20:59	6
3385795	99	33	31,2	cyclictest	3580560	-21	kworker/u16:0+events_unbound	09:05:52	5
3385797	99	32	30,2	cyclictest	3968574	-21	kworker/u16:0+events_unbound	11:37:04	6
3385795	99	32	30,2	cyclictest	3417381	-21	kworker/u16:1+events_unbound	07:55:52	5
3385797	99	30	28,2	cyclictest	3955653	-21	kworker/u16:1+events_unbound	11:33:00	6
3385793	99	30	27,2	cyclictest	3523659	-21	kworker/u16:0+events_unbound	08:25:39	4
3385801	99	29	25,2	cyclictest	3837416	-21	iostat	10:35:42	7
3385793	99	29	27,2	cyclictest	3551852	-21	kworker/u16:6+flush-8:0	09:30:40	4
3385793	99	29	27,2	cyclictest	3347226	-21	kworker/u16:0+events_unbound	07:15:52	4
3800960	2	27	0,0	chrt	3800959	-21	df	10:20:39	7
3385801	99	27	24,2	cyclictest	124	-21	kswapd0	11:05:39	7
3385797	99	27	25,2	cyclictest	3551852	-21	kworker/u16:6+events_unbound	08:45:56	6
3385797	99	27	25,2	cyclictest	3523659	-21	kworker/u16:0+events_unbound	08:28:36	6
3385797	99	27	25,2	cyclictest	3347226	-21	kworker/u16:0+events_unbound	07:36:37	6
3385795	99	26	23,2	cyclictest	3631571	-21	kworker/u16:1+flush-8:0	09:35:41	5
3385801	99	25	19,5	cyclictest	3418165	-21	latency_hist	07:20:27	7
3385797	99	25	23,2	cyclictest	3417381	-21	kworker/u16:1+events_unbound	07:41:01	6
3385793	99	25	23,2	cyclictest	3580560	-21	kworker/u16:0+events_unbound	09:05:56	4
3385801	99	24	22,1	cyclictest	0	-21	swapper/7	07:50:44	7
3385797	99	24	22,2	cyclictest	3590590	-21	kworker/u16:3+events_unbound	08:51:36	6
3385797	99	24	22,2	cyclictest	3551852	-21	kworker/u16:6+events_unbound	08:31:52	6
3385797	99	24	22,2	cyclictest	3386021	-21	kworker/u16:2+events_unbound	07:33:36	6
3385795	99	24	22,2	cyclictest	3347226	-21	kworker/u16:0+events_unbound	07:05:51	5
3385801	99	23	23,0	cyclictest	0	-21	swapper/7	09:04:38	7
3385801	99	23	20,2	cyclictest	3857879	-21	ssh	10:41:10	7
3385797	99	23	21,2	cyclictest	3417381	-21	kworker/u16:1+events_unbound	07:20:57	6
3385793	99	23	21,2	cyclictest	3551852	-21	kworker/u16:6+flush-8:0	09:15:49	4
3385801	99	22	21,0	cyclictest	0	-21	swapper/7	08:42:31	7
3385801	99	22	17,4	cyclictest	3775858	-21	latency_hist	10:10:26	7
3385801	99	21	20,0	cyclictest	0	-21	swapper/7	07:35:43	7
3385797	99	21	19,2	cyclictest	3523659	-21	kworker/u16:0+events_unbound	08:24:36	6
3385801	99	20	20,0	cyclictest	0	-21	swapper/7	09:10:22	7
3385801	99	20	19,0	cyclictest	0	-21	swapper/7	08:35:07	7
3385801	99	20	19,0	cyclictest	0	-21	swapper/7	08:05:45	7
3385801	99	20	19,0	cyclictest	0	-21	swapper/7	07:06:51	7
3385801	99	20	18,1	cyclictest	3823575	-21	ssh	10:29:37	7
3385801	99	20	18,1	cyclictest	0	-21	swapper/7	07:55:44	7
3385801	99	20	15,4	cyclictest	3981882	-21	cat	11:35:40	7
3385801	99	20	14,5	cyclictest	3787939	-21	latency_hist	10:15:26	7
3385801	99	20	0,18	cyclictest	5656	-21	ssh-agent	08:40:06	7
3385801	99	19	9,8	cyclictest	0	-21	swapper/7	09:35:38	7
3385801	99	19	19,0	cyclictest	0	-21	swapper/7	08:04:34	7
3385801	99	19	18,0	cyclictest	0	-21	swapper/7	07:27:59	7
3385801	99	19	15,3	cyclictest	1267	-21	snmpd	12:06:02	7
3385801	99	19	13,5	cyclictest	3465936	-21	latency_hist	07:45:26	7
3385801	99	19	12,6	cyclictest	0	-21	swapper/7	10:50:46	7
3385797	99	19	17,2	cyclictest	3513685	-21	kworker/u16:2+events_unbound	08:17:36	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17