OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #4

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance

Characteristics of the 100 highest latencies:

System rack3slot4.osadl.org (updated Mon May 18, 2026 12:46:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
465031	2	124	0,3	sleep0	0	-21	swapper/0	10:15:43	0
334552	99	117	42,38	cyclictest	399468	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:28:31	0
334565	99	116	102,7	cyclictest	569470	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:15:49	3
334565	99	114	100,7	cyclictest	560776	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:50:47	3
334565	99	113	101,6	cyclictest	464203	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:17:33	3
334565	99	110	96,7	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:35:21	3
334565	99	106	92,7	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:20:20	3
334565	99	106	92,7	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:20:20	3
334552	99	106	0,63	cyclictest	0	-21	swapper/0	09:52:01	0
404705	2	105	0,5	sleep3	0	-21	swapper/3	09:28:22	3
334556	99	104	90,7	cyclictest	626011	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	12:25:52	1
334552	99	104	31,33	cyclictest	690	50	irq/528-eth0-Tx	10:25:36	0
334552	99	104	31,33	cyclictest	690	50	irq/528-eth0-Tx	10:25:36	0
334565	99	102	89,7	cyclictest	609277	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:30:52	3
334565	99	99	84,8	cyclictest	618083	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	12:20:35	3
334565	99	96	81,7	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:10:26	3
334561	99	96	81,8	cyclictest	515260	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:15:42	2
334565	99	94	81,7	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:00:38	3
334552	99	94	67,18	cyclictest	373359	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:50:14	0
334574	99	93	86,3	cyclictest	0	-21	swapper/5	10:25:39	5
334574	99	93	86,3	cyclictest	0	-21	swapper/5	10:25:39	5
463606	2	92	0,6	sleep4	386322	-21	diskmemload-4	10:15:17	4
334565	99	92	78,7	cyclictest	378814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:10:20	3
334565	99	91	76,8	cyclictest	418207	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:40:28	3
334565	99	91	61,8	cyclictest	472314	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:44:34	3
334565	99	89	75,7	cyclictest	560776	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	12:00:37	3
334552	99	89	64,17	cyclictest	351104	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	08:15:01	0
334565	99	88	74,7	cyclictest	410461	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:54:35	3
334565	99	88	56,9	cyclictest	609277	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:39:53	3
334556	99	87	72,7	cyclictest	378814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:05:17	1
334574	99	86	79,3	cyclictest	0	-21	swapper/5	07:55:33	5
334574	99	86	79,3	cyclictest	0	-21	swapper/5	07:55:33	5
334574	99	86	77,5	cyclictest	0	-21	swapper/5	10:00:25	5
334565	99	86	51,19	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:51:43	3
334561	99	86	71,7	cyclictest	560776	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:42:24	2
334570	99	85	78,3	cyclictest	0	-21	swapper/4	08:50:13	4
334570	99	85	76,5	cyclictest	469732	-21	ssh	10:20:06	4
334570	99	85	76,5	cyclictest	469732	-21	ssh	10:20:06	4
334570	99	85	72,8	cyclictest	445217	-21	ssh	10:00:24	4
334570	99	85	66,13	cyclictest	597088	-21	sh	12:00:29	4
334565	99	85	71,7	cyclictest	569470	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:55:47	3
334565	99	85	59,9	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:16:19	3
334561	99	85	42,9	cyclictest	446244	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	10:05:22	2
334556	99	85	71,7	cyclictest	376750	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	09:00:16	1
334552	99	85	61,16	cyclictest	439673	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:00:24	0
334565	99	84	54,9	cyclictest	548435	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:25:12	3
334565	99	84	54,9	cyclictest	548435	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:25:11	3
334556	99	84	72,6	cyclictest	569470	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:52:34	1
334552	99	84	58,7	cyclictest	334849	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	07:16:09	0
334561	99	83	68,8	cyclictest	618083	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	12:22:32	2
334556	99	83	55,8	cyclictest	609277	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:30:25	1
334552	99	83	61,15	cyclictest	538431	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:29:34	0
334552	99	83	61,15	cyclictest	538431	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	11:29:34	0
334561	99	82	66,9	cyclictest	378814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:20:28	2
334561	99	82	66,9	cyclictest	378814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:20:28	2
334556	99	82	67,8	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:48:03	1
334552	99	82	30,22	cyclictest	13	-21	ksoftirqd/0	11:10:18	0
334565	99	81	67,7	cyclictest	569470	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:05:26	3
334565	99	81	67,7	cyclictest	560776	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:48:33	3
334565	99	81	66,8	cyclictest	464203	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:20:35	3
334565	99	81	66,8	cyclictest	464203	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	10:20:35	3
334556	99	81	70,7	cyclictest	210	-21	kswapd0	11:05:17	1
334556	99	81	67,7	cyclictest	515260	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:10:44	1
334556	99	81	66,8	cyclictest	569470	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:55:42	1
334574	99	80	75,2	cyclictest	0	-21	swapper/5	10:20:06	5
334574	99	80	75,2	cyclictest	0	-21	swapper/5	10:20:06	5
334570	99	80	75,3	cyclictest	0	-21	swapper/4	08:15:02	4
334570	99	80	68,6	cyclictest	519758	-21	rm	10:59:50	4
334565	99	80	67,7	cyclictest	364546	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	08:32:59	3
334565	99	80	67,6	cyclictest	609277	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:25:21	3
334565	99	80	66,7	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:10:14	3
334556	99	80	66,7	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	11:04:48	1
334556	99	80	66,7	cyclictest	366802	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	08:30:41	1
334556	99	80	62,6	cyclictest	378814	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	09:17:06	1
334574	99	79	73,3	cyclictest	0	-21	swapper/5	08:13:07	5
334574	99	79	73,3	cyclictest	0	-21	swapper/5	08:13:06	5
334570	99	79	72,3	cyclictest	0	-21	swapper/4	11:29:34	4
334570	99	79	72,3	cyclictest	0	-21	swapper/4	11:29:34	4
334565	99	79	69,7	cyclictest	210	-21	kswapd0	11:15:11	3
334565	99	79	66,7	cyclictest	560776	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	11:35:01	3
334565	99	79	64,8	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:45:34	3
334561	99	79	64,7	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:11:56	2
334552	99	79	51,8	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:20:22	0
334552	99	79	51,8	cyclictest	385305	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	09:20:22	0
334552	99	79	51,6	cyclictest	569470	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	12:15:19	0
334552	99	79	3,73	cyclictest	0	-21	swapper/0	11:05:32	0
334574	99	78	72,3	cyclictest	0	-21	swapper/5	07:10:19	5
334561	99	78	62,8	cyclictest	399468	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	09:25:19	2
334556	99	78	64,7	cyclictest	626011	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	12:35:44	1
334552	99	78	46,6	cyclictest	472314	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:36:03	0
484362	2	77	0,4	sleep3	0	-21	swapper/3	10:31:03	3
484362	2	77	0,4	sleep3	0	-21	swapper/3	10:31:02	3
334565	99	77	64,7	cyclictest	609277	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	12:10:35	3
334565	99	77	63,7	cyclictest	472314	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:28:34	3
334565	99	77	63,7	cyclictest	472314	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:28:34	3
334565	99	77	63,7	cyclictest	454061	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	10:48:59	3
334565	99	77	63,7	cyclictest	340869	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:55:56	3
334565	99	77	63,7	cyclictest	340869	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	07:55:55	3
334565	99	77	56,9	cyclictest	472314	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	10:35:04	3
334561	99	77	64,6	cyclictest	548435	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	11:47:06	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17