OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Jan 14, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
61292	2	93	0,0	sleep5	0	-21	swapper/5	21:57:21	5
3897179	99	66	64,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:57:01	4
3896887	2	58	33,7	sleep5	0	-21	swapper/5	19:07:43	5
3896774	2	57	22,30	sleep7	0	-21	swapper/7	19:06:08	7
3896707	2	49	39,7	sleep6	0	-21	swapper/6	19:05:11	6
3896798	2	45	35,7	sleep4	0	-21	swapper/4	19:06:29	4
3897193	99	44	39,4	cyclictest	4007408	-21	latency_hist	20:02:47	7
3897184	99	40	38,2	cyclictest	86366	-21	kworker/u16:2+events_unbound	22:14:01	5
3897184	99	40	38,2	cyclictest	86366	-21	kworker/u16:2+events_unbound	22:14:01	5
3897184	99	40	38,2	cyclictest	59276	-21	kworker/u16:4+events_unbound	22:07:25	5
3897193	99	39	35,2	cyclictest	4055854	-21	munin-run	20:27:48	7
3897189	99	39	37,2	cyclictest	4055541	-21	kworker/u16:0+flush-8:0	21:48:25	6
3897184	99	39	37,2	cyclictest	109041	-21	kworker/u16:1+events_unbound	22:26:57	5
3897193	99	38	32,5	cyclictest	268731	-21	latency_hist	23:22:47	7
3897184	99	38	36,2	cyclictest	71674	-21	kworker/u16:5+events_unbound	22:19:37	5
3897184	99	38	36,2	cyclictest	109041	-21	kworker/u16:1+events_unbound	22:44:05	5
3897184	99	38	36,2	cyclictest	109041	-21	kworker/u16:1+events_unbound	22:36:10	5
3897189	99	37	34,2	cyclictest	317397	-21	kworker/u16:5+flush-8:0	23:57:52	6
3897184	99	37	35,2	cyclictest	305230	-21	kworker/u16:0+events_unbound	23:54:05	5
3897184	99	37	35,2	cyclictest	147332	-21	kworker/u16:0+events_unbound	22:50:30	5
3897184	99	36	34,2	cyclictest	653	-21	kworker/u16:3+events_unbound	21:34:13	5
3897184	99	36	34,2	cyclictest	653	-21	kworker/u16:3+events_unbound	21:34:13	5
3897184	99	36	34,2	cyclictest	4155287	-21	kworker/u16:5+events_unbound	21:21:17	5
3897184	99	36	34,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:39:14	5
3897184	99	36	34,2	cyclictest	278681	-21	kworker/u16:3+events_unbound	23:32:25	5
3897184	99	36	34,2	cyclictest	110302	-21	kworker/u16:3+events_unbound	22:54:13	5
3897193	99	35	34,0	cyclictest	0	-21	swapper/7	23:33:12	7
3897189	99	35	33,2	cyclictest	71674	-21	kworker/u16:5+events_unbound	22:28:04	6
3897184	99	35	33,2	cyclictest	86366	-21	kworker/u16:2+events_unbound	22:12:31	5
3897184	99	35	33,2	cyclictest	378291	-21	kworker/u16:3+events_unbound	00:14:52	5
3897193	99	34	33,0	cyclictest	0	-21	swapper/7	23:08:05	7
3897189	99	34	32,2	cyclictest	3907124	-21	kworker/u16:0+flush-8:0	19:47:48	6
3897184	99	34	32,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:13:52	5
3897184	99	34	32,2	cyclictest	290591	-21	kworker/u16:4+events_unbound	23:39:33	5
3897184	99	34	32,2	cyclictest	290591	-21	kworker/u16:4+events_unbound	23:39:33	5
3897184	99	34	32,2	cyclictest	244023	-21	kworker/u16:1+events_unbound	23:18:25	5
3897193	99	33	32,0	cyclictest	0	-21	swapper/7	21:47:50	7
3897189	99	33	31,2	cyclictest	653	-21	kworker/u16:3+flush-8:0	21:52:57	6
3897189	99	33	30,2	cyclictest	363616	-21	kworker/u16:2+events_unbound	00:27:48	6
3897184	99	33	31,2	cyclictest	4017089	-21	kworker/u16:2+events_unbound	20:08:16	5
3897189	99	32	30,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:58:04	6
3897184	99	32	30,2	cyclictest	363616	-21	kworker/u16:2+flush-8:0	00:33:20	5
3897193	99	31	31,0	cyclictest	0	-21	swapper/7	22:48:46	7
3897193	99	31	29,1	cyclictest	0	-21	swapper/7	00:03:05	7
3897189	99	31	29,2	cyclictest	4055541	-21	kworker/u16:0+flush-8:0	20:57:47	6
3897189	99	31	29,2	cyclictest	278681	-21	kworker/u16:3+flush-8:0	23:53:07	6
3897184	99	31	29,2	cyclictest	4055541	-21	kworker/u16:0+events_unbound	21:31:02	5
3897184	99	31	29,2	cyclictest	354288	-21	kworker/u16:1+events_unbound	00:06:18	5
3897184	99	31	29,2	cyclictest	305230	-21	kworker/u16:0+flush-8:0	00:31:55	5
3897184	99	31	29,2	cyclictest	305230	-21	kworker/u16:0+events_unbound	00:10:14	5
3897184	99	31	29,2	cyclictest	278681	-21	kworker/u16:3+flush-8:0	23:33:26	5
3897184	99	31	29,2	cyclictest	110302	-21	kworker/u16:3+events_unbound	23:01:35	5
3897193	99	30	25,4	cyclictest	73797	-21	latency_hist	22:02:48	7
3897184	99	30	28,2	cyclictest	71674	-21	kworker/u16:5+events_unbound	22:40:59	5
3897184	99	30	28,2	cyclictest	3907124	-21	kworker/u16:0+events_unbound	19:29:58	5
3897193	99	29	0,27	cyclictest	183070	-21	ssh	22:47:17	7
3897189	99	29	27,2	cyclictest	71674	-21	kworker/u16:5+events_unbound	22:13:05	6
3897189	99	29	27,2	cyclictest	71674	-21	kworker/u16:5+events_unbound	22:13:05	6
3897189	99	29	27,2	cyclictest	147332	-21	kworker/u16:0+flush-8:0	23:08:21	6
3897189	99	29	26,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:17:50	6
3897184	99	29	27,2	cyclictest	4055541	-21	kworker/u16:0+events_unbound	21:43:54	5
3897184	99	29	27,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:06:30	5
3897184	99	29	27,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	20:19:14	5
3897184	99	29	27,2	cyclictest	4017089	-21	kworker/u16:2+events_unbound	21:09:55	5
3897184	99	29	27,2	cyclictest	4017089	-21	kworker/u16:2+events_unbound	20:59:30	5
3897184	99	29	27,2	cyclictest	4017089	-21	kworker/u16:2+events_unbound	20:59:30	5
3897184	99	29	27,2	cyclictest	305230	-21	kworker/u16:0+events_unbound	23:57:55	5
3897184	99	29	27,2	cyclictest	147332	-21	kworker/u16:0+flush-8:0	23:13:18	5
3897184	99	29	27,2	cyclictest	110302	-21	kworker/u16:3+events_unbound	23:10:16	5
3897184	99	29	27,2	cyclictest	109041	-21	kworker/u16:1+events_unbound	22:31:07	5
3897193	99	28	28,0	cyclictest	0	-21	swapper/7	23:47:32	7
3897193	99	28	27,0	cyclictest	0	-21	swapper/7	21:22:02	7
3897193	99	28	0,27	cyclictest	4193548	-21	ssh	21:31:37	7
3897193	99	28	0,27	cyclictest	133995	-21	ssh	22:26:30	7
3897184	99	28	26,2	cyclictest	3940000	-21	kworker/u16:2+events_unbound	19:33:38	5
3897184	99	28	26,2	cyclictest	363616	-21	kworker/u16:2+events_unbound	00:24:54	5
3897184	99	28	26,2	cyclictest	3186022	-21	kworker/u16:5+events_unbound	19:10:37	5
3897184	99	28	26,2	cyclictest	278681	-21	kworker/u16:3+events_unbound	23:45:27	5
3897193	99	27	27,0	cyclictest	0	-21	swapper/7	00:27:38	7
3897193	99	27	25,1	cyclictest	218677	-21	ssh	23:00:14	7
3897189	99	27	25,2	cyclictest	3186022	-21	kworker/u16:5+flush-8:0	19:22:47	6
3897189	99	27	25,2	cyclictest	110302	-21	kworker/u16:3+flush-8:0	22:58:58	6
3897184	99	27	25,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	21:59:10	5
3897184	99	27	25,2	cyclictest	4017089	-21	kworker/u16:2+events_unbound	20:38:49	5
3897184	99	27	25,2	cyclictest	3967623	-21	kworker/u16:4+events_unbound	19:56:26	5
3897184	99	27	25,2	cyclictest	3907124	-21	kworker/u16:0+events_unbound	19:19:50	5
3897184	99	27	25,2	cyclictest	3186022	-21	kworker/u16:5+events_unbound	19:27:02	5
3897184	99	27	25,2	cyclictest	3186022	-21	kworker/u16:5+events_unbound	19:27:02	5
3897193	99	26	26,0	cyclictest	0	-21	swapper/7	22:38:47	7
3897193	99	26	26,0	cyclictest	0	-21	swapper/7	00:21:19	7
3897193	99	26	0,25	cyclictest	5747	-21	gnome-shell	22:19:57	7
3897189	99	26	24,2	cyclictest	3907124	-21	kworker/u16:0+flush-8:0	19:33:09	6
3897189	99	26	24,2	cyclictest	363616	-21	kworker/u16:2+flush-8:0	00:07:42	6
3897189	99	26	23,2	cyclictest	244023	-21	kworker/u16:1+flush-8:0	23:32:57	6
3897184	99	26	24,2	cyclictest	4055541	-21	kworker/u16:0+events_unbound	21:27:19	5
3897184	99	26	24,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	20:54:29	5
3897184	99	26	24,2	cyclictest	4027168	-21	kworker/u16:1+events_unbound	20:50:05	5
3897184	99	26	24,2	cyclictest	4007069	-21	kworker/u16:1+events_unbound	20:05:34	5
3897184	99	26	24,2	cyclictest	3867006	-21	kworker/u16:1+events_unbound	19:50:25	5
3897184	99	26	24,2	cyclictest	378291	-21	kworker/u16:3+flush-8:0	00:18:26	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17