OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri Nov 28, 2025 12:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3445394	99	61	56,4	cyclictest	3812395	-21	latency_hist	18:23:07	7
3444992	2	61	51,7	sleep4	0	-21	swapper/4	15:21:49	4
3444869	2	56	23,29	sleep7	0	-21	swapper/7	15:20:07	7
3445388	99	52	49,2	cyclictest	3751682	-21	kworker/u16:0+flush-8:0	18:48:07	5
3445024	2	52	42,7	sleep6	0	-21	swapper/6	15:22:15	6
3444963	2	51	41,7	sleep5	0	-21	swapper/5	15:21:26	5
3445383	99	50	46,3	cyclictest	3691715	-21	kworker/u16:3+flush-8:0	19:48:22	4
3445394	99	41	36,4	cyclictest	3539897	-21	latency_hist	16:08:07	7
3445388	99	41	39,2	cyclictest	3981081	-21	kworker/u16:0+flush-8:0	20:08:07	5
3445388	99	41	39,2	cyclictest	3981081	-21	kworker/u16:0+flush-8:0	20:08:07	5
3445394	99	40	35,4	cyclictest	3862754	-21	latency_hist	18:48:07	7
3445388	99	37	35,2	cyclictest	2778414	-21	kworker/u16:7+flush-8:0	15:53:35	5
3445388	99	36	34,2	cyclictest	3953781	-21	kworker/u16:5+events_unbound	19:43:36	5
3445389	99	34	32,2	cyclictest	2778414	-21	kworker/u16:7+flush-8:0	15:53:39	6
3445394	99	32	27,4	cyclictest	3792234	-21	latency_hist	18:13:07	7
3445388	99	32	30,2	cyclictest	3981081	-21	kworker/u16:0+flush-8:0	20:13:29	5
3445383	99	31	27,3	cyclictest	3539649	-21	kworker/u16:2+flush-8:0	16:08:18	4
3445394	99	30	25,4	cyclictest	3923152	-21	latency_hist	19:18:07	7
3445388	99	30	27,2	cyclictest	3943279	-21	kworker/u16:2+events_unbound	19:28:25	5
3445389	99	29	27,2	cyclictest	3640651	-21	kworker/u16:5+events_unbound	17:03:35	6
3445388	99	29	27,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	17:38:16	5
3445388	99	29	27,2	cyclictest	3640651	-21	kworker/u16:5+events_unbound	17:13:52	5
3445388	99	29	26,2	cyclictest	3415314	-21	kworker/u16:0+flush-8:0	15:25:19	5
3445394	99	28	23,4	cyclictest	3661008	-21	latency_hist	17:08:07	7
3445383	99	28	26,2	cyclictest	3691715	-21	kworker/u16:3+flush-8:0	18:33:31	4
3445383	99	28	26,2	cyclictest	3691715	-21	kworker/u16:3+flush-8:0	18:33:31	4
3445394	99	27	25,1	cyclictest	0	-21	swapper/7	17:53:35	7
3445388	99	27	25,2	cyclictest	3981081	-21	kworker/u16:0+events_unbound	20:28:35	5
3445388	99	27	25,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	19:23:19	5
3445388	99	27	25,2	cyclictest	3404956	-21	kworker/u16:1+flush-8:0	15:28:12	5
3445388	99	27	24,2	cyclictest	3404956	-21	kworker/u16:1+events_unbound	16:03:21	5
3445394	99	26	21,4	cyclictest	3549996	-21	latency_hist	16:13:07	7
3445394	99	26	0,24	cyclictest	3741804	-21	latency_hist	17:48:07	7
3445388	99	26	23,2	cyclictest	3893276	-21	kworker/u16:4+flush-8:0	19:08:24	5
3445389	99	25	22,2	cyclictest	3691715	-21	kworker/u16:3+flush-8:0	17:48:32	6
3445388	99	25	23,2	cyclictest	3751682	-21	kworker/u16:0+events_unbound	18:30:48	5
3445388	99	25	22,2	cyclictest	3751682	-21	kworker/u16:0+flush-8:0	18:08:51	5
3445394	99	24	23,0	cyclictest	0	-21	swapper/7	16:18:22	7
3445389	99	24	22,2	cyclictest	3751682	-21	kworker/u16:0+events_unbound	18:43:32	6
3445389	99	24	21,3	cyclictest	3893276	-21	kworker/u16:4+flush-8:0	19:08:35	6
3445388	99	24	22,2	cyclictest	4004107	-21	kworker/u16:1+events_unbound	19:58:31	5
3445388	99	24	22,2	cyclictest	3802692	-21	kworker/u16:2+events_unbound	18:26:48	5
3445388	99	24	22,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	18:20:48	5
3445388	99	24	22,2	cyclictest	3640639	-21	kworker/u16:3+flush-8:0	17:08:25	5
3445388	99	24	22,2	cyclictest	3562072	-21	kworker/u16:1+events_unbound	16:35:48	5
3445388	99	24	22,2	cyclictest	3469028	-21	kworker/u16:3+flush-8:0	15:48:24	5
3445394	99	23	17,5	cyclictest	3469370	-21	latency_hist	15:33:07	7
3445389	99	23	21,2	cyclictest	3832408	-21	kworker/u16:1+flush-8:0	18:38:32	6
3445389	99	23	21,2	cyclictest	3404956	-21	kworker/u16:1+events_unbound	15:58:36	6
3445388	99	23	20,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	19:33:22	5
3445388	99	22	20,2	cyclictest	3943279	-21	kworker/u16:2+events_unbound	19:53:28	5
3445388	99	22	20,2	cyclictest	3600753	-21	kworker/u16:4+events_unbound	16:38:30	5
3445388	99	22	20,2	cyclictest	3580134	-21	kworker/u16:2+flush-8:0	16:51:17	5
3445389	99	21	18,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	19:38:25	6
3445388	99	21	19,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	18:34:47	5
3445388	99	21	19,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	18:34:47	5
3445388	99	21	19,2	cyclictest	3469028	-21	kworker/u16:3+flush-8:0	16:08:34	5
3445388	99	21	18,2	cyclictest	3539649	-21	kworker/u16:2+flush-8:0	16:13:35	5
3445388	99	21	18,2	cyclictest	2778414	-21	kworker/u16:7+flush-8:0	16:03:08	5
3445394	99	20	15,4	cyclictest	4114683	-21	latency_hist	20:53:07	7
3445394	99	20	15,4	cyclictest	4114683	-21	latency_hist	20:53:07	7
3445394	99	20	0,18	cyclictest	4064307	-21	latency_hist	20:28:07	7
3445389	99	20	17,2	cyclictest	3469028	-21	kworker/u16:3+flush-8:0	15:40:07	6
3445388	99	20	18,2	cyclictest	4075579	-21	kworker/u16:2+flush-8:0	20:48:02	5
3445388	99	20	18,2	cyclictest	3832408	-21	kworker/u16:1+events_unbound	18:55:47	5
3445388	99	20	18,2	cyclictest	3691715	-21	kworker/u16:3+events_unbound	20:18:31	5
3445388	99	20	18,2	cyclictest	3549756	-21	kworker/u16:0+events_unbound	16:25:48	5
3445389	99	19	16,2	cyclictest	3981081	-21	kworker/u16:0+flush-8:0	20:33:31	6
3445389	99	19	15,3	cyclictest	3549756	-21	kworker/u16:0+flush-8:0	16:13:31	6
3445388	99	19	17,2	cyclictest	4054190	-21	kworker/u16:1+flush-8:0	20:23:33	5
3445388	99	19	17,2	cyclictest	2778414	-21	kworker/u16:7+events_unbound	15:33:27	5
3445389	99	18	16,2	cyclictest	3562072	-21	kworker/u16:1+flush-8:0	16:48:07	6
3445389	99	18	15,2	cyclictest	3691715	-21	kworker/u16:3+flush-8:0	17:33:31	6
3445389	99	18	14,3	cyclictest	3882748	-21	kworker/u16:2+flush-8:0	18:58:32	6
3445388	99	18	16,2	cyclictest	3882748	-21	kworker/u16:2+flush-8:0	18:58:26	5
3445388	99	18	16,2	cyclictest	3832408	-21	kworker/u16:1+events_unbound	19:18:31	5
3445388	99	18	16,2	cyclictest	3832408	-21	kworker/u16:1+events_unbound	19:18:31	5
3445388	99	18	16,2	cyclictest	3681020	-21	kworker/u16:1+flush-8:0	17:18:33	5
3445388	99	18	16,2	cyclictest	3640651	-21	kworker/u16:5+events_unbound	17:00:47	5
3445388	99	18	16,2	cyclictest	3600753	-21	kworker/u16:4+flush-8:0	16:53:33	5
3445388	99	18	16,2	cyclictest	3562072	-21	kworker/u16:1+events_unbound	16:43:48	5
3445388	99	18	15,2	cyclictest	4075579	-21	kworker/u16:2+flush-8:0	20:38:19	5
3445394	99	17	14,2	cyclictest	0	-21	swapper/7	19:53:07	7
3445394	99	17	14,2	cyclictest	0	-21	swapper/7	18:23:07	7
3445394	99	17	12,4	cyclictest	3913064	-21	latency_hist	19:13:07	7
3445394	99	17	11,5	cyclictest	3590305	-21	latency_hist	16:33:08	7
3445394	99	17	0,15	cyclictest	3973656	-21	latency_hist	19:43:07	7
3445389	99	17	15,2	cyclictest	3580134	-21	kworker/u16:2+events_unbound	17:53:31	6
3445389	99	17	15,2	cyclictest	3469028	-21	kworker/u16:3+events_unbound	15:48:22	6
3445388	99	17	15,2	cyclictest	3792063	-21	kworker/u16:1+flush-8:0	18:13:31	5
3445388	99	17	15,2	cyclictest	3701118	-21	kworker/u16:4+events_unbound	17:50:48	5
3445388	99	17	15,2	cyclictest	3681020	-21	kworker/u16:1+flush-8:0	17:23:33	5
3445394	99	16	8,7	cyclictest	27924	-21	gmain	15:43:31	7
3445389	99	16	0,1	cyclictest	0	-21	swapper/6	19:58:33	6
3445388	99	16	14,2	cyclictest	3832408	-21	kworker/u16:1+events_unbound	18:48:33	5
3445388	99	16	14,2	cyclictest	3580134	-21	kworker/u16:2+flush-8:0	17:03:38	5
3445388	99	16	13,2	cyclictest	3691715	-21	kworker/u16:3+flush-8:0	17:28:23	5
3445388	99	16	13,2	cyclictest	3469028	-21	kworker/u16:3+flush-8:0	15:38:23	5
3445383	99	16	0,1	cyclictest	4093801	-21	switchtime	20:38:32	4
3445394	99	15	13,1	cyclictest	0	-21	swapper/7	17:28:25	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17