OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #4

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance

Characteristics of the 100 highest latencies:

System rack3slot4.osadl.org (updated Sun Feb 01, 2026 12:46:25)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
4038510	2	2087	2060,17	sleep5	0	-21	swapper/5	07:05:19	5
4040323	99	124	109,7	cyclictest	4123594	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:54:24	3
4040323	99	124	109,7	cyclictest	4123594	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:54:24	3
4040313	99	123	109,7	cyclictest	72904	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:39:40	1
4040323	99	119	107,6	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:39:20	3
4040323	99	114	102,6	cyclictest	4058147	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	07:59:02	3
4040323	99	112	100,6	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:09:45	3
4040323	99	112	100,6	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:09:45	3
4141198	2	109	0,3	sleep0	0	-21	swapper/0	09:53:04	0
4141198	2	109	0,3	sleep0	0	-21	swapper/0	09:53:03	0
4040323	99	109	97,6	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:39:42	3
4040323	99	109	96,6	cyclictest	4060307	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:24:05	3
115189	2	108	0,5	sleep4	0	-21	swapper/4	12:08:27	4
115189	2	108	0,5	sleep4	0	-21	swapper/4	12:08:27	4
4040323	99	102	90,6	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:04:15	3
4040308	99	101	73,16	cyclictest	690	50	irq/528-eth0-Tx	10:20:01	0
4040308	99	100	64,17	cyclictest	0	-21	swapper/0	07:50:35	0
4040323	99	97	84,7	cyclictest	63425	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:29:25	3
4040323	99	97	84,7	cyclictest	63425	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:29:25	3
4040308	99	97	69,5	cyclictest	15	-21	rcuc/0	11:20:00	0
4040323	99	96	66,7	cyclictest	106909	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	12:27:02	3
4040308	99	96	61,13	cyclictest	4042056	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	07:37:52	0
4169358	2	95	0,6	sleep3	4169355	-21	ssh	10:15:27	3
4040308	99	95	2,77	cyclictest	4131809	-21	ntpq	09:45:28	0
4184516	2	94	0,4	sleep0	0	-21	swapper/0	10:27:37	0
4040323	99	94	82,6	cyclictest	4072382	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	08:49:13	3
4040323	99	94	81,7	cyclictest	4166589	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:07:33	3
4040313	99	94	80,7	cyclictest	4173236	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:19:35	1
4040313	99	94	80,7	cyclictest	4072382	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	08:39:14	1
4040308	99	94	83,5	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:09:03	0
4040308	99	94	83,5	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:09:03	0
4040323	99	92	80,6	cyclictest	4088620	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:14:19	3
4040323	99	92	80,6	cyclictest	4088620	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:14:18	3
4040313	99	92	64,9	cyclictest	4096573	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:20:32	1
4040313	99	92	61,7	cyclictest	72904	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:40:51	1
4040323	99	91	77,7	cyclictest	4103323	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:05:13	3
4040323	99	91	61,7	cyclictest	106909	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	12:19:57	3
4040308	99	91	60,16	cyclictest	0	-21	swapper/0	12:00:05	0
4040308	99	89	81,5	cyclictest	87223	-21	ssh	11:45:32	0
4040308	99	89	63,16	cyclictest	691	50	irq/529-eth0-Tx	09:15:43	0
4040308	99	89	56,17	cyclictest	15	-21	rcuc/0	10:10:14	0
4040308	99	89	2,74	cyclictest	0	-21	swapper/0	12:15:23	0
4040308	99	89	2,70	cyclictest	0	-21	swapper/0	11:05:37	0
4040308	99	89	1,76	cyclictest	0	-21	swapper/0	08:42:18	0
4040323	99	88	75,7	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:15:27	3
4040313	99	88	73,7	cyclictest	55289	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:22:07	1
4040308	99	88	64,5	cyclictest	91376	-21	kworker/u12:2+rpciod@ rpc_async_schedule	11:53:52	0
4040308	99	88	63,18	cyclictest	99704	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:59:32	0
4040308	99	88	62,16	cyclictest	690	50	irq/528-eth0-Tx	11:40:35	0
4040308	99	88	48,24	cyclictest	4089802	-21	/usr/sbin/munin	09:10:24	0
4040308	99	88	48,24	cyclictest	4089802	-21	/usr/sbin/munin	09:10:23	0
4040308	99	88	47,6	cyclictest	15	-21	rcuc/0	11:24:59	0
4040323	99	87	74,7	cyclictest	4139837	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:55:02	3
4040323	99	87	74,7	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:57:27	3
4040313	99	87	75,5	cyclictest	91376	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:54:41	1
4040308	99	87	62,14	cyclictest	690	50	irq/528-eth0-Tx	12:39:15	0
4040308	99	87	60,15	cyclictest	691	50	irq/529-eth0-Tx	10:35:30	0
4040323	99	86	60,6	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:08:36	3
4040308	99	86	75,5	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:04:18	0
4040308	99	86	65,12	cyclictest	691	50	irq/529-eth0-Tx	10:30:23	0
4040308	99	86	57,16	cyclictest	0	-21	swapper/0	09:25:27	0
4040308	99	86	17,47	cyclictest	4091917	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:21:34	0
4040323	99	85	54,8	cyclictest	118419	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:20:13	3
4040313	99	85	72,6	cyclictest	4159488	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:11:35	1
4040313	99	85	55,7	cyclictest	4166589	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:48:54	1
4040308	99	85	73,6	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:34:13	0
4040308	99	85	47,15	cyclictest	14092	-21	ssh	10:46:00	0
4040323	99	84	69,7	cyclictest	137316	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	12:30:16	3
4040308	99	84	57,14	cyclictest	0	-21	swapper/0	08:15:36	0
4040308	99	84	2,69	cyclictest	0	-21	swapper/0	12:05:19	0
4040308	99	84	2,69	cyclictest	0	-21	swapper/0	12:05:19	0
4040323	99	83	71,6	cyclictest	4166589	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:43:56	3
4040323	99	83	70,6	cyclictest	91376	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:50:31	3
4040323	99	83	69,7	cyclictest	4166589	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:23:59	3
4040313	99	83	70,6	cyclictest	4103323	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:25:34	1
4040308	99	83	60,16	cyclictest	4052322	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	07:44:07	0
4040308	99	83	60,16	cyclictest	4052322	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	07:44:07	0
4040308	99	83	50,4	cyclictest	4157747	-21	grep	10:05:44	0
4040308	99	83	49,18	cyclictest	4088751	-21	diskmemload	10:57:45	0
4040308	99	83	43,18	cyclictest	0	-21	swapper/0	07:45:30	0
4040308	99	83	40,17	cyclictest	73892	-21	head	11:35:17	0
4040327	99	82	69,8	cyclictest	4151580	-21	ssh	10:00:47	4
4040323	99	82	70,6	cyclictest	4052322	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	07:44:10	3
4040323	99	82	70,6	cyclictest	4052322	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	07:44:09	3
4040323	99	82	69,7	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:00:38	3
4040323	99	82	68,7	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:14:29	3
4040313	99	82	66,8	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:35:35	1
4040308	99	82	67,4	cyclictest	691	50	irq/529-eth0-Tx	11:05:01	0
4040323	99	81	69,6	cyclictest	43099	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:22:19	3
4040323	99	81	69,6	cyclictest	4062386	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:37:14	3
4040313	99	81	66,8	cyclictest	4039867	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:10:35	1
4040308	99	81	62,4	cyclictest	691	50	irq/529-eth0-Tx	12:10:33	0
4040308	99	81	49,7	cyclictest	4150566	-21	cat	10:00:27	0
4040308	99	81	46,17	cyclictest	15	-21	rcuc/0	12:20:43	0
4040327	99	80	74,3	cyclictest	0	-21	swapper/4	11:35:28	4
4040327	99	80	74,3	cyclictest	0	-21	swapper/4	09:10:37	4
4040327	99	80	74,3	cyclictest	0	-21	swapper/4	09:10:37	4
4040327	99	80	67,7	cyclictest	135418	-21	ssh	12:25:15	4
4040317	99	80	59,9	cyclictest	83431	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:53:22	2
4040313	99	80	66,7	cyclictest	34458	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:58:29	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17