OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Jul 08, 2026 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
6278	89	56	0,32	rtkit-daemon	0	-21	swapper/5	19:08:54	5
3448406	2	55	20,30	sleep7	0	-21	swapper/7	19:07:23	7
3448877	99	40	0,38	cyclictest	4125882	-21	latency_hist	00:09:55	7
3448873	99	40	38,2	cyclictest	3482932	-21	kworker/u16:0+flush-8:0	19:34:55	6
3448871	99	36	34,2	cyclictest	4004086	-21	kworker/u16:3+events_unbound	00:09:02	5
3448871	99	35	33,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	22:21:11	5
3448316	2	35	25,7	sleep4	0	-21	swapper/4	19:06:08	4
3448871	99	34	32,2	cyclictest	3954401	-21	kworker/u16:1+flush-8:0	23:16:26	5
3448871	99	34	32,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	22:13:42	5
3448873	99	33	31,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	22:59:55	6
3448871	99	33	31,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:53:02	5
3448871	99	33	31,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:10:13	5
3448871	99	33	31,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	22:53:22	5
3448871	99	33	30,2	cyclictest	3666860	-21	kworker/u16:2+events_unbound	21:29:59	5
3942129	2	32	0,1	chrt	3942130	-21	ssh	22:54:07	7
3448871	99	32	30,2	cyclictest	4088340	-21	kworker/u16:2+events_unbound	00:25:45	5
3448871	99	32	30,2	cyclictest	4088340	-21	kworker/u16:2+events_unbound	00:17:34	5
3448871	99	32	30,2	cyclictest	4050324	-21	kworker/u16:1+events_unbound	23:36:22	5
3448871	99	32	30,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:46:02	5
3448871	99	32	30,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	22:31:41	5
3448871	99	32	30,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	22:31:41	5
3448871	99	32	30,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	22:07:42	5
3448871	99	32	30,2	cyclictest	3646995	-21	kworker/u16:0+events_unbound	21:16:50	5
3448871	99	32	29,2	cyclictest	3759290	-21	kworker/u16:0+flush-8:0	21:59:58	5
3448405	2	32	22,7	sleep6	0	-21	swapper/6	19:07:22	6
3448871	99	31	29,2	cyclictest	3954401	-21	kworker/u16:1+events_unbound	23:09:14	5
3448871	99	31	29,2	cyclictest	3915820	-21	kworker/u16:2+flush-8:0	22:59:58	5
3448871	99	31	29,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:43:26	5
3448871	99	31	29,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:29:02	5
3448871	99	31	29,2	cyclictest	3735010	-21	kworker/u16:3+flush-8:0	22:16:42	5
3448871	99	31	29,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	21:28:10	5
3448871	99	31	29,2	cyclictest	3666860	-21	kworker/u16:2+flush-8:0	21:11:30	5
3448871	99	31	29,2	cyclictest	2498	-21	kworker/u16:6+events_unbound	00:38:54	5
3448871	99	31	29,2	cyclictest	2498	-21	kworker/u16:6+events_unbound	00:38:54	5
3448871	99	30	28,2	cyclictest	4101836	-21	kworker/u16:1+events_unbound	00:20:42	5
3448871	99	30	28,2	cyclictest	3915820	-21	kworker/u16:2+flush-8:0	22:43:50	5
3448871	99	30	28,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:21:27	5
3448871	99	30	28,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	22:29:34	5
3448871	99	30	28,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	00:12:58	5
3448871	99	30	28,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	00:03:30	5
3448871	99	30	28,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	22:55:51	5
3448871	99	30	28,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	21:40:46	5
3448871	99	29	27,2	cyclictest	4004086	-21	kworker/u16:3+events_unbound	00:32:49	5
3448871	99	29	27,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:32:31	5
3448871	99	29	27,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	21:58:35	5
3448871	99	29	27,2	cyclictest	3684745	-21	kworker/u16:1+flush-8:0	21:21:35	5
3448871	99	29	26,2	cyclictest	3759290	-21	kworker/u16:0+events_unbound	21:49:59	5
3448877	99	28	25,2	cyclictest	3880110	-21	ssh	22:27:29	7
3448873	99	28	26,2	cyclictest	4050324	-21	kworker/u16:1+events_unbound	23:50:18	6
3448873	99	28	26,2	cyclictest	3735010	-21	kworker/u16:3+flush-8:0	22:15:10	6
3448873	99	28	26,2	cyclictest	3684745	-21	kworker/u16:1+flush-8:0	21:47:31	6
3448871	99	28	26,2	cyclictest	3915820	-21	kworker/u16:2+events_unbound	22:47:28	5
3448871	99	28	26,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	22:36:59	5
3448877	99	27	26,0	cyclictest	0	-21	swapper/7	20:50:11	7
3448877	99	27	23,2	cyclictest	3902859	-21	ntpq	22:35:20	7
3448873	99	27	25,2	cyclictest	4004086	-21	kworker/u16:3+events_unbound	23:40:18	6
3448871	99	27	25,2	cyclictest	3759290	-21	kworker/u16:0+flush-8:0	21:46:14	5
3448871	99	27	25,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	21:35:43	5
3448877	99	26	24,1	cyclictest	0	-21	swapper/7	00:30:22	7
3448873	99	26	24,2	cyclictest	4004086	-21	kworker/u16:3+events_unbound	00:07:30	6
3448877	99	25	23,1	cyclictest	4078054	-21	ssh	23:50:10	7
3448871	99	25	23,2	cyclictest	3869585	-21	kworker/u16:0+events_unbound	23:56:37	5
3448871	99	25	23,2	cyclictest	3448648	-21	kworker/u16:2+flush-8:0	19:10:31	5
3448871	99	25	23,2	cyclictest	3448648	-21	kworker/u16:2+events_unbound	19:47:06	5
3448873	99	24	22,2	cyclictest	3482932	-21	kworker/u16:0+events_unbound	19:50:09	6
3448871	99	24	22,2	cyclictest	3448648	-21	kworker/u16:2+events_unbound	20:49:14	5
3448871	99	24	22,2	cyclictest	3448648	-21	kworker/u16:2+events_unbound	20:37:15	5
3448871	99	24	22,2	cyclictest	3414538	-21	kworker/u16:1+flush-8:0	19:15:26	5
3448877	99	23	22,0	cyclictest	0	-21	swapper/7	23:15:19	7
3448873	99	23	21,2	cyclictest	3597416	-21	kworker/u16:1+events_unbound	20:36:15	6
3448873	99	23	21,2	cyclictest	3414538	-21	kworker/u16:1+flush-8:0	19:10:10	6
3448873	99	23	20,2	cyclictest	3684745	-21	kworker/u16:1+flush-8:0	22:40:01	6
3448873	99	22	19,2	cyclictest	3735010	-21	kworker/u16:3+events_unbound	21:40:13	6
3448871	99	22	20,2	cyclictest	3558413	-21	kworker/u16:4+events_unbound	20:13:07	5
3448877	99	21	20,0	cyclictest	0	-21	swapper/7	21:50:17	7
3448877	99	21	15,5	cyclictest	648	-21	systemd-journal	20:49:55	7
3448873	99	21	19,2	cyclictest	3646995	-21	kworker/u16:0+flush-8:0	21:10:09	6
3448871	99	21	19,2	cyclictest	3448648	-21	kworker/u16:2+flush-8:0	19:49:56	5
3448877	99	20	18,1	cyclictest	0	-21	swapper/7	23:30:50	7
3448877	99	20	16,2	cyclictest	4174741	-21	munin-run	00:29:54	7
3448873	99	20	18,2	cyclictest	3448648	-21	kworker/u16:2+flush-8:0	20:40:25	6
3448871	99	20	18,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	21:05:16	5
3448871	99	20	18,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	21:05:16	5
3448871	99	20	17,2	cyclictest	3482932	-21	kworker/u16:0+events_unbound	20:20:07	5
3448877	99	19	17,1	cyclictest	3860735	-21	ssh	22:20:13	7
3448873	99	19	17,2	cyclictest	4004086	-21	kworker/u16:3+flush-8:0	00:00:19	6
3448873	99	19	17,2	cyclictest	3684745	-21	kworker/u16:1+events_unbound	21:30:09	6
3448871	99	19	16,2	cyclictest	3646995	-21	kworker/u16:0+events_unbound	20:49:57	5
3448871	99	19	16,2	cyclictest	3448648	-21	kworker/u16:2+events_unbound	19:25:05	5
3448877	99	18	17,0	cyclictest	0	-21	swapper/7	20:20:34	7
3448877	99	18	17,0	cyclictest	0	-21	swapper/7	20:10:16	7
3448877	99	18	16,1	cyclictest	0	-21	swapper/7	21:55:19	7
3448877	99	18	0,17	cyclictest	0	-21	swapper/7	22:00:18	7
3448873	99	18	15,2	cyclictest	3869585	-21	kworker/u16:0+flush-8:0	00:34:56	6
3448873	99	18	15,2	cyclictest	3869585	-21	kworker/u16:0+flush-8:0	00:34:56	6
3448871	99	18	16,2	cyclictest	3482932	-21	kworker/u16:0+events_unbound	19:59:58	5
3448871	99	18	16,2	cyclictest	3448648	-21	kworker/u16:2+events_unbound	20:43:05	5
3448871	99	18	16,2	cyclictest	2756729	-21	kworker/u16:3+events_unbound	19:31:14	5
3448877	99	17	16,0	cyclictest	0	-21	swapper/7	21:10:11	7
3448877	99	17	15,1	cyclictest	3964532	-21	ssh	23:00:26	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17