OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #4

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance

Characteristics of the 100 highest latencies:

System rack3slot4.osadl.org (updated Thu Feb 26, 2026 00:46:21)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
278873	2	1005	981,14	sleep4	0	-21	swapper/4	19:05:55	4
278873	2	1005	981,14	sleep4	0	-21	swapper/4	19:05:54	4
279339	99	151	30,81	cyclictest	523798	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	23:50:20	0
279339	99	147	112,28	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	21:51:45	0
279339	99	137	63,39	cyclictest	311417	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	20:40:53	0
279339	99	137	63,39	cyclictest	311417	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	20:40:52	0
279339	99	135	76,51	cyclictest	435561	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	22:35:05	0
279339	99	133	98,15	cyclictest	297381	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	20:06:25	0
279339	99	133	98,15	cyclictest	297381	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	20:06:24	0
279339	99	132	96,15	cyclictest	303804	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	20:16:27	0
279339	99	129	95,14	cyclictest	427977	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:02:03	0
279339	99	129	95,14	cyclictest	427977	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:02:03	0
279339	99	128	70,38	cyclictest	330892	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	21:21:18	0
279339	99	128	69,40	cyclictest	541539	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:55:40	0
279339	99	127	67,36	cyclictest	483317	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:39:24	0
279339	99	127	56,39	cyclictest	427977	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	22:49:50	0
279344	99	122	108,7	cyclictest	461112	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	23:12:04	1
279339	99	122	66,36	cyclictest	553323	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	00:12:13	0
279339	99	119	51,51	cyclictest	396457	-21	runrttasks	22:03:19	0
279339	99	118	58,37	cyclictest	390498	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	22:14:39	0
279339	99	117	52,43	cyclictest	282944	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	19:18:33	0
279339	99	116	57,37	cyclictest	279676	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	19:29:29	0
279339	99	112	71,38	cyclictest	354092	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	21:32:55	0
279339	99	111	1,90	cyclictest	0	-21	swapper/0	22:55:37	0
279339	99	110	50,41	cyclictest	0	-21	swapper/0	22:06:53	0
279339	99	110	49,37	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:32:46	0
417664	2	107	0,8	sleep3	37	-21	ksoftirqd/3	22:19:44	3
279339	99	107	67,37	cyclictest	361907	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:41:43	0
279339	99	107	58,42	cyclictest	282944	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	19:55:34	0
279358	99	105	99,3	cyclictest	0	-21	swapper/4	00:03:05	4
279358	99	105	99,3	cyclictest	0	-21	swapper/4	00:03:05	4
279339	99	105	83,15	cyclictest	327633	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	21:11:36	0
279339	99	105	60,37	cyclictest	279676	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	19:35:36	0
279339	99	104	65,24	cyclictest	461112	-21	kworker/u12:0+flush-179:96@ wb_workfn	23:10:41	0
279339	99	104	61,36	cyclictest	344985	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:26:39	0
279339	99	104	61,36	cyclictest	344985	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:26:38	0
279339	99	103	58,38	cyclictest	449889	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:50:20	0
279353	99	102	88,7	cyclictest	390498	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	22:11:51	3
279339	99	102	82,16	cyclictest	533423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	00:07:35	0
279348	99	101	87,7	cyclictest	344985	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:26:43	2
279348	99	101	87,7	cyclictest	344985	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:26:43	2
279348	99	101	62,8	cyclictest	378087	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	21:57:27	2
279339	99	101	79,14	cyclictest	461112	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	23:05:48	0
279339	99	101	48,31	cyclictest	541539	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	00:01:14	0
279339	99	101	48,31	cyclictest	541539	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	00:01:14	0
279363	99	100	94,3	cyclictest	0	-21	swapper/5	22:46:58	5
279339	99	100	80,15	cyclictest	327633	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	21:18:53	0
279339	99	100	77,14	cyclictest	303804	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	20:30:41	0
279339	99	100	76,15	cyclictest	561287	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	00:26:23	0
279339	99	100	53,28	cyclictest	0	-21	swapper/0	20:10:24	0
279348	99	99	69,9	cyclictest	435561	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:00:39	2
279348	99	99	69,9	cyclictest	435561	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:00:38	2
279344	99	99	86,6	cyclictest	378087	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:06:50	1
279339	99	99	1,78	cyclictest	0	-21	swapper/0	23:27:05	0
279339	99	98	76,14	cyclictest	282944	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	19:23:38	0
279363	99	97	92,2	cyclictest	0	-21	swapper/5	19:15:34	5
279339	99	97	76,14	cyclictest	561287	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	00:22:33	0
279339	99	97	68,20	cyclictest	690	50	irq/528-eth0-Tx	22:29:11	0
279339	99	97	66,21	cyclictest	689	50	irq/527-eth0-Tx	22:18:44	0
279339	99	97	56,31	cyclictest	690	50	irq/528-eth0-Tx	00:15:36	0
279339	99	97	43,31	cyclictest	0	-21	swapper/0	00:33:47	0
279339	99	96	73,15	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:24:16	0
279339	99	96	72,15	cyclictest	279676	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	19:10:25	0
279339	99	96	59,25	cyclictest	361907	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:37:03	0
279339	99	96	59,25	cyclictest	361907	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:37:02	0
279339	99	96	56,37	cyclictest	483317	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:18:15	0
279339	99	96	55,38	cyclictest	502298	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	23:24:25	0
279339	99	96	1,75	cyclictest	0	-21	swapper/0	23:40:43	0
279339	99	95	74,14	cyclictest	513718	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:45:20	0
279339	99	95	71,16	cyclictest	442436	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:40:26	0
279339	99	95	50,28	cyclictest	0	-21	swapper/0	21:45:41	0
279344	99	94	70,6	cyclictest	513718	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:45:34	1
279339	99	94	70,16	cyclictest	325502	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	21:04:55	0
279339	99	94	59,21	cyclictest	0	-21	swapper/0	20:50:33	0
279348	99	93	80,7	cyclictest	502298	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	23:37:08	2
279344	99	93	57,8	cyclictest	442436	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:42:04	1
279339	99	93	71,15	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	21:55:14	0
279339	99	93	61,22	cyclictest	689	50	irq/527-eth0-Tx	19:50:27	0
279339	99	93	61,22	cyclictest	689	50	irq/527-eth0-Tx	19:50:26	0
279348	99	92	78,7	cyclictest	449889	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:45:11	2
279348	99	92	78,7	cyclictest	321423	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	20:56:35	2
279348	99	92	62,7	cyclictest	327633	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	21:19:34	2
279339	99	92	51,27	cyclictest	0	-21	swapper/0	20:35:27	0
279348	99	90	77,7	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:31:01	2
279348	99	90	76,7	cyclictest	561287	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	00:38:43	2
279339	99	90	73,14	cyclictest	306500	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	20:22:20	0
279339	99	90	66,16	cyclictest	502298	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	23:30:12	0
279339	99	90	66,12	cyclictest	0	-21	swapper/0	20:45:23	0
279339	99	90	49,27	cyclictest	0	-21	swapper/0	19:35:00	0
279344	99	89	75,7	cyclictest	279676	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	19:39:42	1
279339	99	89	66,15	cyclictest	533423	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	00:36:23	0
279348	99	88	74,7	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	21:55:02	2
279348	99	88	63,9	cyclictest	583999	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	00:28:17	2
279344	99	88	73,8	cyclictest	378087	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:30:32	1
279344	99	88	73,8	cyclictest	378087	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:19:54	1
279339	99	88	67,16	cyclictest	303804	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	20:25:43	0
279348	99	87	73,7	cyclictest	483317	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:25:35	2
279348	99	87	73,7	cyclictest	461112	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:56:42	2
279348	99	87	54,8	cyclictest	370269	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	22:20:54	2
279344	99	87	74,6	cyclictest	483317	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:25:59	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17