OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #4

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance

Characteristics of the 100 highest latencies:

System rack3slot4.osadl.org (updated Tue May 26, 2026 12:46:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2986947	2	930	912,10	sleep5	0	-21	swapper/5	07:05:38	5
2987871	99	162	125,29	cyclictest	3125014	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:55:04	0
2987871	99	154	128,19	cyclictest	3212583	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:30:14	0
2987871	99	152	126,19	cyclictest	3125014	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:24:55	0
2987871	99	141	61,54	cyclictest	2993781	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:40:37	0
2987871	99	139	69,50	cyclictest	3156241	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:45:38	0
2987871	99	139	61,53	cyclictest	3015818	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	08:40:30	0
2987871	99	134	61,28	cyclictest	3146824	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:39:00	0
2987871	99	134	61,28	cyclictest	3146824	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:39:00	0
2987871	99	133	95,30	cyclictest	3225147	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:50:20	0
2987871	99	132	69,55	cyclictest	3285009	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:25:19	0
2987871	99	131	74,49	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:25:28	0
2987871	99	131	74,49	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:25:28	0
2987871	99	129	97,23	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:05:22	0
2987871	99	127	59,7	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:18:21	0
2987871	99	125	66,52	cyclictest	2984192	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	07:20:34	0
2987871	99	125	56,44	cyclictest	3285009	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:35:33	0
2987871	99	125	55,45	cyclictest	3271062	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	12:11:06	0
2987871	99	122	91,23	cyclictest	3030266	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:54:35	0
2987871	99	120	24,31	cyclictest	0	-21	swapper/0	11:19:13	0
2987871	99	120	24,31	cyclictest	0	-21	swapper/0	11:19:13	0
2987871	99	117	68,26	cyclictest	3036269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:11:38	0
2987871	99	117	63,27	cyclictest	3092732	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:59:35	0
2987871	99	116	58,54	cyclictest	3092732	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:13:30	0
2987871	99	116	2,5	cyclictest	3229637	-21	latency_hist	11:40:00	0
2987871	99	114	82,24	cyclictest	3156241	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:43:37	0
2987871	99	114	82,24	cyclictest	3156241	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:43:37	0
2987871	99	114	2,49	cyclictest	0	-21	swapper/0	07:50:40	0
2987871	99	112	53,50	cyclictest	3136866	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:30:12	0
2987871	99	112	53,50	cyclictest	3136866	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:30:12	0
2987871	99	109	1,63	cyclictest	0	-21	swapper/0	07:20:00	0
2987871	99	108	1,5	cyclictest	3011093	-21	ntpq	08:05:32	0
2987871	99	107	55,30	cyclictest	690	50	irq/528-eth0-Tx	11:20:32	0
2987871	99	107	55,30	cyclictest	690	50	irq/528-eth0-Tx	11:20:32	0
2987871	99	106	63,27	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:34:35	0
2987871	99	106	39,24	cyclictest	142	50	irq/98-imx_mu_c	11:00:40	0
2987871	99	104	52,28	cyclictest	0	-21	swapper/0	10:25:29	0
2987871	99	104	52,28	cyclictest	0	-21	swapper/0	10:25:29	0
2987871	99	104	28,13	cyclictest	0	-21	swapper/0	11:40:14	0
2987871	99	104	28,13	cyclictest	0	-21	swapper/0	11:40:14	0
2987871	99	103	66,28	cyclictest	3022814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:48:00	0
2987871	99	102	77,16	cyclictest	3085854	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:49:33	0
2987871	99	102	77,16	cyclictest	3085854	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:49:32	0
2987871	99	102	56,37	cyclictest	3225147	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:56:34	0
2987871	99	102	1,4	cyclictest	3257562	-21	ssh	12:00:44	0
2987884	99	101	87,7	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:18:35	3
2987871	99	100	74,22	cyclictest	3015818	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	08:38:35	0
2987871	99	100	74,17	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:52:04	0
2987871	99	100	68,25	cyclictest	3013728	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:30:06	0
2987884	99	99	64,16	cyclictest	3111024	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:27:57	3
2987884	99	99	64,16	cyclictest	3111024	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:27:57	3
2987871	99	99	57,34	cyclictest	3036455	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:19:44	0
2987871	99	99	28,22	cyclictest	0	-21	swapper/0	11:10:21	0
2987871	99	99	28,22	cyclictest	0	-21	swapper/0	11:10:21	0
3058778	2	98	0,5	sleep2	0	-21	swapper/2	09:28:07	2
2987875	99	98	85,6	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:24:41	1
2987871	99	98	65,25	cyclictest	3006282	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:55:40	0
2987871	99	98	59,17	cyclictest	3003952	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	08:15:36	0
2987875	99	96	82,7	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:15:22	1
2987871	99	96	61,16	cyclictest	2984192	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	07:31:25	0
2987871	99	96	58,17	cyclictest	3015818	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	08:25:36	0
2987871	99	95	69,18	cyclictest	3291610	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:30:42	0
2987871	99	95	58,16	cyclictest	2993781	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:38:18	0
2987871	99	95	58,16	cyclictest	2993781	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:38:17	0
2987871	99	93	65,25	cyclictest	3013728	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:23:26	0
2987871	99	93	60,16	cyclictest	3146824	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:50:29	0
2987871	99	93	55,17	cyclictest	2993781	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:25:06	0
2987884	99	92	78,7	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:15:10	3
2987884	99	92	78,7	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:15:10	3
2987871	99	92	64,25	cyclictest	3092732	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:09:46	0
2987884	99	90	69,8	cyclictest	3006282	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:00:06	3
2987880	99	89	77,6	cyclictest	3118423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:33:00	2
2987880	99	89	77,6	cyclictest	3118423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:33:00	2
2987880	99	88	75,6	cyclictest	3146824	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:53:11	2
2987871	99	88	68,16	cyclictest	15	-21	rcuc/0	09:25:36	0
2987871	99	88	61,24	cyclictest	3022814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:55:29	0
2987871	99	88	56,17	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:36:36	0
2987871	99	88	56,16	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:07:23	0
2987871	99	88	44,17	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:20:32	0
2987893	99	87	81,3	cyclictest	0	-21	swapper/5	09:44:46	5
2987871	99	87	64,7	cyclictest	3041694	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:24:14	0
2987871	99	87	50,4	cyclictest	0	-21	swapper/0	09:00:45	0
2987871	99	86	53,17	cyclictest	3032151	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:05:15	0
2987871	99	86	53,17	cyclictest	3006282	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:10:33	0
2987871	99	86	49,16	cyclictest	2993781	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:45:10	0
2987871	99	86	48,3	cyclictest	0	-21	swapper/0	10:00:33	0
2987871	99	86	25,25	cyclictest	3276306	-21	/usr/sbin/munin	12:15:28	0
2987871	99	85	54,3	cyclictest	0	-21	swapper/0	07:10:35	0
2987871	99	85	54,16	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:45:37	0
2987871	99	85	54,16	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:45:37	0
2987875	99	84	69,8	cyclictest	3118423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:15:27	1
3077715	2	83	0,5	sleep1	0	-21	swapper/1	09:42:32	1
2987875	99	82	60,7	cyclictest	3163226	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:01:40	1
2987889	99	81	76,2	cyclictest	0	-21	swapper/4	11:37:14	4
2987880	99	81	67,7	cyclictest	3125014	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:35:03	2
2987880	99	81	67,7	cyclictest	3125014	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:35:03	2
2987880	99	81	61,8	cyclictest	3156241	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:46:09	2
2987875	99	81	67,7	cyclictest	3118423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:25:31	1
2987875	99	81	67,7	cyclictest	3118423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:25:31	1
2987893	99	80	73,3	cyclictest	0	-21	swapper/5	09:35:37	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17