OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sun Jun 28, 2026 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2547013	2	56	0,1	sleep7	2547012	-21	cat	19:49:17	7
2465021	2	44	34,7	sleep4	0	-21	swapper/4	19:07:17	4
2465078	2	42	32,7	sleep5	0	-21	swapper/5	19:08:06	5
2465454	99	40	35,4	cyclictest	3118896	-21	latency_hist	23:59:16	7
2464871	2	40	4,31	sleep7	0	-21	swapper/7	19:05:11	7
2465444	99	37	35,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	00:11:03	4
2465444	99	37	35,2	cyclictest	2748309	-21	kworker/u16:0+events_unbound	22:03:37	4
2465444	99	37	35,2	cyclictest	2748309	-21	kworker/u16:0+events_unbound	22:03:37	4
2465444	99	36	34,2	cyclictest	3167018	-21	kworker/u16:0+events_unbound	00:21:20	4
2465444	99	36	34,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	00:03:21	4
2465444	99	36	34,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	23:25:19	4
2465444	99	36	34,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	23:25:19	4
2465444	99	36	34,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	23:05:36	4
2465444	99	36	34,2	cyclictest	2849551	-21	kworker/u16:4+events_unbound	22:54:12	4
2464880	2	36	25,8	sleep6	0	-21	swapper/6	19:05:18	6
2465453	99	35	33,2	cyclictest	2751615	-21	kworker/u16:2+events_unbound	21:34:32	6
2465444	99	35	34,1	cyclictest	2859344	-21	kworker/u16:5+events_unbound	22:56:55	4
2465444	99	35	33,2	cyclictest	2935676	-21	kworker/u16:1+events_unbound	00:31:36	4
2465444	99	35	33,2	cyclictest	2886356	-21	kworker/u16:0+events_unbound	22:30:36	4
2465444	99	35	33,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	23:02:05	4
2958058	2	34	0,0	chrt	2958057	-21	ssh	22:50:59	7
2465453	99	34	32,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	23:39:31	6
2465444	99	34	32,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	00:09:05	4
2465444	99	34	32,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	00:09:05	4
2465444	99	34	32,2	cyclictest	2993451	-21	kworker/u16:0+events_unbound	23:54:12	4
2465444	99	34	32,2	cyclictest	2935676	-21	kworker/u16:1+events_unbound	23:17:20	4
2465444	99	34	32,2	cyclictest	2935676	-21	kworker/u16:1+events_unbound	00:27:41	4
2465444	99	34	32,2	cyclictest	2849551	-21	kworker/u16:4+events_unbound	22:36:12	4
2465444	99	34	32,2	cyclictest	2751615	-21	kworker/u16:2+events_unbound	21:40:53	4
2465444	99	33	31,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	23:47:37	4
2465444	99	33	31,2	cyclictest	2751615	-21	kworker/u16:2+events_unbound	21:50:20	4
2465444	99	33	31,2	cyclictest	2751615	-21	kworker/u16:2+events_unbound	21:32:37	4
2465444	99	32	30,2	cyclictest	2993451	-21	kworker/u16:0+events_unbound	23:41:52	4
2465444	99	32	30,2	cyclictest	2993451	-21	kworker/u16:0+events_unbound	23:21:28	4
2465444	99	32	30,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	23:30:12	4
2465444	99	32	30,2	cyclictest	2849551	-21	kworker/u16:4+events_unbound	22:43:37	4
2465444	99	32	30,2	cyclictest	2825085	-21	kworker/u16:3+events_unbound	22:23:00	4
2465444	99	32	30,2	cyclictest	2825085	-21	kworker/u16:3+events_unbound	22:17:36	4
2465444	99	32	30,2	cyclictest	2701504	-21	kworker/u16:4+events_unbound	21:26:28	4
2465444	99	32	30,2	cyclictest	2692848	-21	kworker/u16:3+flush-8:0	21:09:40	4
2465444	99	32	30,2	cyclictest	2692848	-21	kworker/u16:3+events_unbound	21:14:56	4
2465444	99	31	29,2	cyclictest	3167019	-21	kworker/u16:3+events_unbound	00:35:11	4
2465444	99	31	29,2	cyclictest	2993451	-21	kworker/u16:0+events_unbound	23:37:41	4
2465444	99	31	29,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	22:27:57	4
2465444	99	31	29,2	cyclictest	2825085	-21	kworker/u16:3+events_unbound	22:05:05	4
2465444	99	31	29,2	cyclictest	2751615	-21	kworker/u16:2+flush-8:0	21:37:09	4
2465444	99	31	29,2	cyclictest	2614989	-21	kworker/u16:1+events_unbound	21:56:41	4
2465453	99	30	28,2	cyclictest	3045666	-21	kworker/u16:4+flush-8:0	23:49:17	6
2465444	99	30	28,2	cyclictest	2859344	-21	kworker/u16:5+events_unbound	22:48:56	4
2465453	99	29	27,2	cyclictest	2849551	-21	kworker/u16:4+flush-8:0	22:54:17	6
2465444	99	29	27,2	cyclictest	3167018	-21	kworker/u16:0+events_unbound	00:18:56	4
2465444	99	29	27,2	cyclictest	2935676	-21	kworker/u16:1+events_unbound	23:10:24	4
2465444	99	29	27,2	cyclictest	2614989	-21	kworker/u16:1+events_unbound	22:11:01	4
2465444	99	29	26,3	cyclictest	2614989	-21	kworker/u16:1+flush-8:0	21:04:43	4
2465454	99	28	0,26	cyclictest	2910837	-21	latency_hist	22:34:17	7
2465453	99	27	25,2	cyclictest	2993451	-21	kworker/u16:0+events_unbound	23:19:31	6
2465444	99	27	25,2	cyclictest	2751615	-21	kworker/u16:2+events_unbound	21:46:20	4
2465444	99	27	25,2	cyclictest	2692848	-21	kworker/u16:3+events_unbound	21:22:04	4
3068712	2	25	0,1	chrt	3068710	-21	ssh	23:36:43	7
2465454	99	24	23,0	cyclictest	0	-21	swapper/7	19:39:32	7
2465444	99	24	22,2	cyclictest	2614989	-21	kworker/u16:1+flush-8:0	20:32:28	4
2465444	99	24	22,2	cyclictest	2498365	-21	kworker/u16:0+events_unbound	20:06:28	4
2465454	99	23	22,0	cyclictest	0	-21	swapper/7	20:24:31	7
2465444	99	23	21,2	cyclictest	2614989	-21	kworker/u16:1+events_unbound	20:55:33	4
2465454	99	22	21,0	cyclictest	0	-21	swapper/7	00:24:32	7
2465453	99	22	20,2	cyclictest	3119333	-21	kworker/u16:3+events_unbound	00:04:47	6
2465453	99	22	20,2	cyclictest	3119333	-21	kworker/u16:3+events_unbound	00:04:47	6
2465453	99	22	19,3	cyclictest	2380824	-21	kworker/u16:3+flush-8:0	20:39:32	6
2465444	99	21	19,2	cyclictest	2380824	-21	kworker/u16:3+events_unbound	20:24:40	4
2465454	99	20	19,0	cyclictest	0	-21	swapper/7	21:34:48	7
2465454	99	20	12,7	cyclictest	0	-21	swapper/7	20:04:33	7
2465453	99	20	18,2	cyclictest	2498365	-21	kworker/u16:0+flush-8:0	19:24:45	6
2465453	99	20	15,4	cyclictest	2875392	-21	cat	22:19:32	6
2465444	99	20	18,2	cyclictest	2469403	-21	kworker/u16:2+events_unbound	19:18:29	4
2465454	99	19	0,17	cyclictest	2874110	-21	latency_hist	22:19:17	7
2465446	99	19	17,2	cyclictest	2859344	-21	kworker/u16:5+flush-8:0	22:20:15	5
2465444	99	19	17,2	cyclictest	2498365	-21	kworker/u16:0+events_unbound	19:24:48	4
2465444	99	19	17,2	cyclictest	2380824	-21	kworker/u16:3+events_unbound	20:12:29	4
2465454	99	18	9,8	cyclictest	0	-21	swapper/7	19:24:33	7
2465454	99	18	12,5	cyclictest	2595500	-21	latency_hist	20:14:18	7
2465453	99	18	16,2	cyclictest	2701504	-21	kworker/u16:4+flush-8:0	21:14:37	6
2465444	99	18	16,2	cyclictest	2380824	-21	kworker/u16:3+flush-8:0	20:39:38	4
2465444	99	18	16,2	cyclictest	2380824	-21	kworker/u16:3+flush-8:0	19:42:28	4
2465444	99	18	15,2	cyclictest	2498365	-21	kworker/u16:0+flush-8:0	19:29:43	4
2465454	99	17	13,3	cyclictest	3155564	-21	latency_hist	00:14:17	7
2465454	99	17	13,3	cyclictest	1278	-21	snmpd	20:24:06	7
2465454	99	17	0,16	cyclictest	0	-21	swapper/7	20:54:40	7
2465444	99	17	15,2	cyclictest	2614989	-21	kworker/u16:1+flush-8:0	20:44:51	4
2465444	99	17	15,2	cyclictest	2380824	-21	kworker/u16:3+flush-8:0	19:59:32	4
2465444	99	17	14,2	cyclictest	2653569	-21	kworker/u16:0+events_unbound	20:59:24	4
2465454	99	16	13,2	cyclictest	1102	-21	ntpd	23:19:42	7
2465454	99	16	13,1	cyclictest	3167858	-21	awk	00:19:17	7
2465453	99	16	14,2	cyclictest	3045666	-21	kworker/u16:4+events_unbound	00:09:33	6
2465453	99	16	14,2	cyclictest	2653569	-21	kworker/u16:0+events_unbound	20:54:32	6
2465453	99	16	13,2	cyclictest	2469403	-21	kworker/u16:2+events_unbound	19:14:19	6
2465453	99	16	13,1	cyclictest	0	-21	swapper/6	23:38:56	6
2465453	99	16	12,3	cyclictest	2547068	-21	kworker/u16:1+flush-8:0	19:59:48	6
2465444	99	16	14,2	cyclictest	2469403	-21	kworker/u16:2+events_unbound	19:49:12	4
2465444	99	16	14,2	cyclictest	2380824	-21	kworker/u16:3+events_unbound	19:55:29	4
2465454	99	15	9,5	cyclictest	650	-21	systemd-journal	20:44:17	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17