OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed May 06, 2026 12:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3150088	2	60	0,0	sleep7	3150090	-21	cut	10:23:55	7
2723132	99	56	54,2	cyclictest	2794174	-21	kworker/u16:0+events_unbound	08:18:48	5
2723132	99	52	50,2	cyclictest	2994526	-21	kworker/u16:0+events_unbound	09:24:00	5
2723132	99	51	49,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	12:19:01	5
2723132	99	51	48,2	cyclictest	2669832	-21	kworker/u16:2+flush-8:0	07:14:04	5
2723138	99	49	48,0	cyclictest	0	-21	swapper/7	11:13:48	7
2723132	99	49	47,2	cyclictest	3350093	-21	kworker/u16:2+events_unbound	12:28:59	5
2723132	99	49	46,2	cyclictest	2757202	-21	kworker/u16:1+events_unbound	07:38:48	5
2723138	99	47	43,2	cyclictest	2880719	-21	munin-run	08:28:34	7
2722722	2	47	37,7	sleep5	0	-21	swapper/5	07:06:49	5
2723138	99	45	39,4	cyclictest	3362298	-21	latency_hist	11:53:33	7
2723132	99	45	43,2	cyclictest	3033032	-21	kworker/u16:1+events_unbound	10:03:47	5
2723138	99	44	0,43	cyclictest	1708	-21	dockerd	11:18:48	7
2723132	99	44	42,2	cyclictest	3350093	-21	kworker/u16:2+events_unbound	11:48:47	5
2723132	99	44	42,2	cyclictest	3264310	-21	kworker/u16:1+flush-8:0	12:34:03	5
2723132	99	43	41,2	cyclictest	2919302	-21	kworker/u16:4+flush-8:0	10:44:03	5
2723129	99	43	41,2	cyclictest	3166589	-21	kworker/u16:0+flush-8:0	10:43:49	4
2723129	99	43	40,2	cyclictest	2994526	-21	kworker/u16:0+flush-8:0	09:33:49	4
2722474	2	43	7,30	sleep7	0	-21	swapper/7	07:04:11	7
2722778	2	42	32,7	sleep4	0	-21	swapper/4	07:07:36	4
2723138	99	41	38,2	cyclictest	5997	-21	gnome-software	07:59:36	7
2723132	99	41	39,2	cyclictest	3264310	-21	kworker/u16:1+events_unbound	11:46:19	5
2723132	99	41	39,2	cyclictest	3202908	-21	kworker/u16:1+events_unbound	10:51:20	5
2723132	99	41	39,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	11:32:20	5
2723132	99	41	39,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	11:10:04	5
2723132	99	41	38,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	11:43:05	5
2723138	99	40	37,2	cyclictest	3108279	-21	grep	10:08:52	7
2723132	99	40	38,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	10:54:29	5
2723132	99	40	38,2	cyclictest	3114904	-21	kworker/u16:2+events_unbound	10:23:33	5
2723132	99	40	37,2	cyclictest	3033032	-21	kworker/u16:1+flush-8:0	10:09:03	5
2723132	99	39	37,2	cyclictest	3264310	-21	kworker/u16:1+events_unbound	11:57:04	5
2723132	99	39	37,2	cyclictest	3249565	-21	kworker/u16:3+events_unbound	11:06:09	5
2723132	99	39	37,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	10:37:33	5
2723132	99	39	37,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	10:37:33	5
2723132	99	39	37,2	cyclictest	3117349	-21	kworker/u16:3+flush-8:0	10:29:08	5
2723132	99	39	37,2	cyclictest	3033032	-21	kworker/u16:1+events_unbound	09:47:33	5
2723132	99	39	36,2	cyclictest	2794174	-21	kworker/u16:0+flush-8:0	08:48:49	5
2723132	99	39	36,2	cyclictest	2669832	-21	kworker/u16:2+events_unbound	07:19:04	5
2723129	99	39	36,2	cyclictest	2794174	-21	kworker/u16:0+flush-8:0	08:59:02	4
2723134	99	38	35,2	cyclictest	1160040	-21	kworker/u16:3+events_unbound	07:13:39	6
2723132	99	38	36,2	cyclictest	3264310	-21	kworker/u16:1+events_unbound	11:21:12	5
2723132	99	38	36,2	cyclictest	3264310	-21	kworker/u16:1+events_unbound	11:15:53	5
2723132	99	38	36,2	cyclictest	3114904	-21	kworker/u16:2+flush-8:0	11:23:59	5
2723132	99	38	36,2	cyclictest	2957394	-21	kworker/u16:1+events_unbound	09:17:08	5
2723132	99	38	36,2	cyclictest	2841985	-21	kworker/u16:2+flush-8:0	08:24:00	5
2723132	99	37	35,2	cyclictest	3350093	-21	kworker/u16:2+events_unbound	12:09:03	5
2723132	99	37	35,2	cyclictest	3166589	-21	kworker/u16:0+flush-8:0	11:59:03	5
2723132	99	37	35,2	cyclictest	3117349	-21	kworker/u16:3+events_unbound	10:26:13	5
2723132	99	37	35,2	cyclictest	2919302	-21	kworker/u16:4+flush-8:0	09:59:02	5
2723132	99	37	35,2	cyclictest	2919302	-21	kworker/u16:4+events_unbound	09:36:00	5
2723132	99	37	35,2	cyclictest	2692980	-21	kworker/u16:4+events_unbound	08:04:01	5
2723138	99	36	35,0	cyclictest	0	-21	swapper/7	11:28:28	7
2723132	99	36	34,2	cyclictest	3350093	-21	kworker/u16:2+flush-8:0	12:23:54	5
2723132	99	36	34,2	cyclictest	3350093	-21	kworker/u16:2+events_unbound	12:03:56	5
2723132	99	36	34,2	cyclictest	3166589	-21	kworker/u16:0+events_unbound	11:36:05	5
2723132	99	36	34,2	cyclictest	3114904	-21	kworker/u16:2+events_unbound	10:39:40	5
2723132	99	36	34,2	cyclictest	2957394	-21	kworker/u16:1+flush-8:0	09:09:28	5
2723132	99	36	34,2	cyclictest	2957394	-21	kworker/u16:1+flush-8:0	09:09:28	5
2723132	99	35	33,2	cyclictest	3055903	-21	kworker/u16:3+events_unbound	09:52:01	5
2723132	99	35	33,2	cyclictest	3055903	-21	kworker/u16:3+events_unbound	09:52:01	5
2723132	99	35	33,2	cyclictest	3033032	-21	kworker/u16:1+flush-8:0	09:54:03	5
2723132	99	35	33,2	cyclictest	3033032	-21	kworker/u16:1+flush-8:0	09:39:29	5
2723132	99	35	33,2	cyclictest	2938404	-21	kworker/u16:3+events_unbound	09:19:52	5
2723138	99	34	30,2	cyclictest	3162908	-21	ntpq	10:29:00	7
2723138	99	34	0,32	cyclictest	3019035	-21	latency_hist	09:33:34	7
2723132	99	34	32,2	cyclictest	3264310	-21	kworker/u16:1+events_unbound	12:14:33	5
2723132	99	34	32,2	cyclictest	2692980	-21	kworker/u16:4+events_unbound	07:28:48	5
2723129	99	34	32,2	cyclictest	3350093	-21	kworker/u16:2+flush-8:0	11:53:59	4
2723129	99	34	31,2	cyclictest	3117349	-21	kworker/u16:3+events_unbound	10:28:48	4
2723129	99	34	31,2	cyclictest	3055902	-21	kworker/u16:0+events_unbound	10:08:48	4
2722632	2	34	23,8	sleep6	0	-21	swapper/6	07:05:38	6
2723132	99	33	31,2	cyclictest	3114904	-21	kworker/u16:2+flush-8:0	10:14:00	5
2723132	99	33	31,2	cyclictest	3114904	-21	kworker/u16:2+events_unbound	11:00:51	5
2723132	99	33	31,2	cyclictest	2919302	-21	kworker/u16:4+flush-8:0	08:53:52	5
2723132	99	33	31,2	cyclictest	2919302	-21	kworker/u16:4+events_unbound	09:30:53	5
2723132	99	33	31,2	cyclictest	2794174	-21	kworker/u16:0+events_unbound	07:53:53	5
2723129	99	33	30,2	cyclictest	2919302	-21	kworker/u16:4+events_unbound	09:58:48	4
2889830	2	32	0,1	chrt	2889831	-21	unixbench-2d	08:29:03	7
2889830	2	32	0,1	chrt	2889831	-21	unixbench-2d	08:29:03	7
2723138	99	32	31,0	cyclictest	0	-21	swapper/7	12:03:36	7
2723138	99	32	0,30	cyclictest	3327674	-21	tr	11:38:51	7
2723132	99	32	30,2	cyclictest	2794174	-21	kworker/u16:0+flush-8:0	09:04:01	5
2723132	99	32	30,2	cyclictest	2794174	-21	kworker/u16:0+events_unbound	08:00:45	5
2723129	99	32	29,2	cyclictest	3350093	-21	kworker/u16:2+events_unbound	12:03:48	4
3065955	2	31	0,0	chrt	3065954	-21	ssh	09:49:49	7
3065955	2	31	0,0	chrt	3065954	-21	ssh	09:49:49	7
2723129	99	31	28,2	cyclictest	2757202	-21	kworker/u16:1+events_unbound	07:58:48	4
2723129	99	31	27,3	cyclictest	3202908	-21	kworker/u16:1+flush-8:0	11:02:00	4
2723138	99	30	0,28	cyclictest	3043539	-21	perf	09:43:33	7
2723132	99	30	27,2	cyclictest	2794174	-21	kworker/u16:0+flush-8:0	07:43:54	5
2723129	99	30	29,1	cyclictest	3033032	-21	kworker/u16:1+events_unbound	09:43:29	4
2723129	99	30	27,2	cyclictest	2757202	-21	kworker/u16:1+events_unbound	07:53:49	4
2723138	99	29	0,27	cyclictest	3250921	-21	sh	11:06:44	7
2723132	99	29	28,1	cyclictest	2692980	-21	kworker/u16:4+events_unbound	07:48:58	5
2723132	99	29	27,2	cyclictest	2794174	-21	kworker/u16:0+events_unbound	08:44:44	5
2723129	99	29	27,2	cyclictest	3114904	-21	kworker/u16:2+flush-8:0	10:23:48	4
2723129	99	29	26,2	cyclictest	2994526	-21	kworker/u16:0+flush-8:0	09:24:05	4
2723129	99	29	26,2	cyclictest	2692980	-21	kworker/u16:4+flush-8:0	07:29:00	4
2723138	99	28	27,0	cyclictest	0	-21	swapper/7	12:13:59	7
2723138	99	28	27,0	cyclictest	0	-21	swapper/7	11:38:28	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17