OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri Aug 14, 2026 00:43:36)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3947214	2	100	0,1	sleep7	3947215	-21	cut	21:28:24	7
3895358	2	97	0,1	sleep7	3895357	-21	kthreadcore	21:08:21	7
3909919	2	95	0,0	sleep5	0	-21	swapper/5	21:13:24	5
3702386	2	55	0,1	sleep6	0	-21	swapper/6	19:28:20	6
3654238	99	53	50,2	cyclictest	95917	-21	kworker/u16:1+flush-8:0	23:57:59	6
3654237	99	52	50,2	cyclictest	4066197	-21	kworker/u16:4+events_unbound	22:28:26	5
3654237	99	50	48,2	cyclictest	4082868	-21	kworker/u16:0+flush-8:0	22:53:26	5
3653801	2	50	40,7	sleep4	0	-21	swapper/4	19:05:43	4
3649360	2	43	1,31	sleep7	0	-21	swapper/7	19:03:22	7
3654237	99	42	40,2	cyclictest	3784775	-21	kworker/u16:1+flush-8:0	22:08:15	5
3654237	99	42	39,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	22:43:17	5
3654237	99	41	39,2	cyclictest	3961908	-21	kworker/u16:3+events_unbound	21:38:28	5
3654243	99	40	36,3	cyclictest	3842883	-21	cat	20:43:00	7
3654237	99	40	38,2	cyclictest	4066197	-21	kworker/u16:4+events_unbound	22:23:31	5
3654237	99	40	38,2	cyclictest	3927118	-21	kworker/u16:2+events_unbound	21:58:32	5
3654237	99	40	38,2	cyclictest	3813463	-21	kworker/u16:3+events_unbound	20:28:30	5
3654237	99	40	38,2	cyclictest	3735800	-21	kworker/u16:2+events_unbound	20:23:25	5
3654237	99	40	38,2	cyclictest	3687052	-21	kworker/u16:0+flush-8:0	19:53:24	5
3654237	99	40	38,2	cyclictest	3687052	-21	kworker/u16:0+events_unbound	20:33:25	5
3654237	99	40	37,2	cyclictest	4154351	-21	kworker/u16:1+flush-8:0	22:58:17	5
3654237	99	39	37,2	cyclictest	4189803	-21	kworker/u16:1+events_unbound	23:31:22	5
3654237	99	39	37,2	cyclictest	3882043	-21	kworker/u16:0+flush-8:0	21:03:25	5
3654237	99	39	37,2	cyclictest	3784775	-21	kworker/u16:1+events_unbound	20:48:13	5
3653903	2	39	29,7	sleep5	0	-21	swapper/5	19:07:13	5
3654237	99	38	36,2	cyclictest	3813463	-21	kworker/u16:3+events_unbound	21:23:29	5
3654243	99	37	35,1	cyclictest	3987796	-21	ssh	21:47:59	7
3654237	99	37	35,2	cyclictest	3597852	-21	kworker/u16:4+flush-8:0	19:18:25	5
3654237	99	37	34,2	cyclictest	95917	-21	kworker/u16:1+events_unbound	00:30:11	5
3654237	99	36	34,2	cyclictest	4189803	-21	kworker/u16:1+events_unbound	23:35:38	5
3654237	99	36	34,2	cyclictest	4095189	-21	kworker/u16:1+events_unbound	22:42:20	5
3654237	99	36	34,2	cyclictest	4025246	-21	kworker/u16:0+events_unbound	22:04:26	5
3654237	99	35	33,2	cyclictest	82634	-21	kworker/u16:2+events_unbound	00:05:14	5
3654237	99	35	33,2	cyclictest	4120832	-21	kworker/u16:2+flush-8:0	23:03:29	5
3654237	99	35	33,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	23:45:42	5
3654237	99	35	33,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	23:25:18	5
3654237	99	35	33,2	cyclictest	3986500	-21	kworker/u16:3+flush-8:0	22:18:27	5
3654237	99	35	33,2	cyclictest	3927118	-21	kworker/u16:2+events_unbound	21:55:34	5
3654237	99	35	33,2	cyclictest	3784775	-21	kworker/u16:1+events_unbound	21:44:14	5
3654237	99	35	33,2	cyclictest	3687052	-21	kworker/u16:0+events_unbound	19:43:31	5
3654237	99	35	32,2	cyclictest	3735800	-21	kworker/u16:2+events_unbound	20:13:14	5
3654243	99	34	33,0	cyclictest	0	-21	swapper/7	00:22:54	7
3654237	99	34	32,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	22:49:07	5
3654237	99	34	32,2	cyclictest	4025246	-21	kworker/u16:0+events_unbound	22:13:31	5
3654237	99	34	31,2	cyclictest	3813463	-21	kworker/u16:3+events_unbound	20:58:13	5
3654237	99	33	31,2	cyclictest	83842	-21	kworker/u16:4+events_unbound	23:50:19	5
3654237	99	33	31,2	cyclictest	4193048	-21	kworker/u16:3+events_unbound	00:11:23	5
3654237	99	33	31,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	23:17:02	5
3654237	99	33	31,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	00:24:53	5
3654237	99	33	31,2	cyclictest	3986500	-21	kworker/u16:3+events_unbound	22:37:33	5
3654237	99	33	31,2	cyclictest	3927118	-21	kworker/u16:2+events_unbound	21:22:32	5
3654237	99	33	31,2	cyclictest	3735800	-21	kworker/u16:2+events_unbound	20:22:03	5
3654237	99	33	30,3	cyclictest	4082868	-21	kworker/u16:0+flush-8:0	23:59:29	5
3654237	99	33	30,2	cyclictest	4193048	-21	kworker/u16:3+events_unbound	00:13:15	5
46746	2	32	0,1	chrt	46747	-21	sh	23:31:51	7
3654243	99	32	0,30	cyclictest	203881	-21	latency_hist	00:37:59	7
3654237	99	32	30,2	cyclictest	95917	-21	kworker/u16:1+events_unbound	00:37:31	5
3654237	99	32	30,2	cyclictest	4189803	-21	kworker/u16:1+events_unbound	23:19:34	5
3654237	99	32	30,2	cyclictest	4189803	-21	kworker/u16:1+events_unbound	23:12:50	5
3654237	99	32	30,2	cyclictest	3813463	-21	kworker/u16:3+flush-8:0	21:28:22	5
3653734	2	32	8,7	sleep6	0	-21	swapper/6	19:04:49	6
3654243	99	31	0,29	cyclictest	4120344	-21	latency_hist	22:42:59	7
3654237	99	31	29,2	cyclictest	4193048	-21	kworker/u16:3+flush-8:0	23:38:27	5
3654237	99	30	28,2	cyclictest	3711611	-21	kworker/u16:1+flush-8:0	19:33:28	5
3654237	99	30	28,2	cyclictest	3654460	-21	kworker/u16:3+flush-8:0	19:38:25	5
3654243	99	29	23,5	cyclictest	648	-21	systemd-journal	19:47:59	7
3654237	99	29	27,2	cyclictest	3927118	-21	kworker/u16:2+events_unbound	21:35:15	5
3654237	99	29	27,2	cyclictest	3813463	-21	kworker/u16:3+flush-8:0	20:43:14	5
3654237	99	29	27,2	cyclictest	3774594	-21	kworker/u16:3+flush-8:0	20:08:20	5
3654238	99	28	26,2	cyclictest	3986500	-21	kworker/u16:3+flush-8:0	21:52:35	6
3654238	99	28	26,2	cyclictest	3986500	-21	kworker/u16:3+flush-8:0	21:52:35	6
3654237	99	28	26,2	cyclictest	4082868	-21	kworker/u16:0+events_unbound	00:21:06	5
3654237	99	28	26,2	cyclictest	3813463	-21	kworker/u16:3+events_unbound	21:11:51	5
3996793	2	27	0,1	chrt	3996792	-21	missed_timers	21:48:28	7
3996793	2	27	0,1	chrt	3996792	-21	missed_timers	21:48:28	7
3654243	99	27	22,4	cyclictest	83423	-21	latency_hist	23:47:59	7
3654237	99	27	25,2	cyclictest	95917	-21	kworker/u16:1+events_unbound	23:55:20	5
3654243	99	26	25,0	cyclictest	0	-21	swapper/7	20:08:31	7
3654243	99	26	21,4	cyclictest	143682	-21	latency_hist	00:12:59	7
3654237	99	26	24,2	cyclictest	3597852	-21	kworker/u16:4+flush-8:0	19:23:19	5
3654237	99	25	22,2	cyclictest	3687052	-21	kworker/u16:0+events_unbound	19:58:03	5
3654237	99	24	22,2	cyclictest	3852841	-21	kworker/u16:4+events_unbound	20:54:04	5
3654237	99	24	22,2	cyclictest	3784775	-21	kworker/u16:1+events_unbound	20:42:11	5
3654237	99	24	22,2	cyclictest	3711611	-21	kworker/u16:1+flush-8:0	20:03:20	5
3654233	99	24	22,2	cyclictest	4189803	-21	kworker/u16:1+flush-8:0	23:38:25	4
3654233	99	23	21,2	cyclictest	3986500	-21	kworker/u16:3+flush-8:0	22:29:11	4
3654237	99	22	20,2	cyclictest	3654460	-21	kworker/u16:3+events_unbound	19:09:14	5
3654243	99	21	18,2	cyclictest	155738	-21	latency_hist	00:17:59	7
3654243	99	21	16,4	cyclictest	3927514	-21	latency_hist	21:23:00	7
3654243	99	21	0,19	cyclictest	180989	-21	ssh	00:28:14	7
3654243	99	20	19,0	cyclictest	0	-21	swapper/7	00:07:39	7
3654243	99	20	16,3	cyclictest	4025111	-21	/usr/sbin/munin	22:03:14	7
3654243	99	20	0,18	cyclictest	12425	-21	cut	23:18:14	7
3654237	99	20	18,2	cyclictest	3927118	-21	kworker/u16:2+events_unbound	21:48:14	5
3654237	99	20	18,2	cyclictest	3927118	-21	kworker/u16:2+events_unbound	21:48:14	5
3654237	99	20	18,2	cyclictest	3735800	-21	kworker/u16:2+flush-8:0	19:48:33	5
3654237	99	20	18,2	cyclictest	3654460	-21	kworker/u16:3+events_unbound	19:28:19	5
3654237	99	19	17,2	cyclictest	3654460	-21	kworker/u16:3+events_unbound	19:17:12	5
3979954	2	17	0,0	sleep6	0	-21	swapper/6	21:43:22	6
3654243	99	17	12,4	cyclictest	4023923	-21	latency_hist	22:03:00	7
3654243	99	17	11,5	cyclictest	3963736	-21	latency_hist	21:38:00	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17