OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed May 20, 2026 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1777896	2	95	0,1	sleep7	1777898	-21	cut	20:37:59	7
1595772	2	59	48,8	sleep5	0	-21	swapper/5	19:07:28	5
1594838	2	46	9,31	sleep7	0	-21	swapper/7	19:03:13	7
1596069	99	40	39,0	cyclictest	0	-21	swapper/7	21:23:03	7
1596069	99	40	39,0	cyclictest	0	-21	swapper/7	21:23:03	7
1596069	99	37	32,4	cyclictest	2048134	-21	latency_hist	22:37:40	7
1595771	2	36	26,7	sleep4	0	-21	swapper/4	19:07:27	4
1596069	99	33	30,1	cyclictest	1743883	-21	munin-run	20:22:40	7
1596069	99	33	28,4	cyclictest	2265438	-21	latency_hist	00:07:39	7
1596063	99	33	29,3	cyclictest	2277990	-21	kworker/u16:4+flush-8:0	00:23:02	5
1595705	2	33	9,8	sleep6	0	-21	swapper/6	19:06:33	6
1596063	99	31	29,2	cyclictest	1951407	-21	kworker/u16:2+flush-8:0	22:58:02	5
1596058	99	31	29,2	cyclictest	1609642	-21	kworker/u16:2+events_unbound	19:12:55	4
1596069	99	30	27,2	cyclictest	1976090	-21	ssh	22:07:46	7
1596058	99	30	28,2	cyclictest	2217373	-21	kworker/u16:1+events_unbound	23:53:06	4
1596069	99	28	25,1	cyclictest	1937200	-21	ssh	21:48:55	7
1596058	99	28	26,2	cyclictest	1951408	-21	kworker/u16:4+flush-8:0	23:22:34	4
1596058	99	28	26,2	cyclictest	1696054	-21	kworker/u16:3+flush-8:0	21:18:02	4
1596058	99	28	25,2	cyclictest	1876448	-21	kworker/u16:1+events_unbound	21:57:54	4
1596069	99	27	26,0	cyclictest	0	-21	swapper/7	23:17:19	7
1596069	99	27	26,0	cyclictest	0	-21	swapper/7	21:35:23	7
1596069	99	27	0,25	cyclictest	1924951	-21	sshd-session	21:43:42	7
1596063	99	27	25,2	cyclictest	1890998	-21	kworker/u16:4+events_unbound	21:43:12	5
1596069	99	26	25,0	cyclictest	0	-21	swapper/7	23:28:50	7
1596069	99	26	23,2	cyclictest	124	-21	kswapd0	23:02:52	7
1596069	99	26	23,1	cyclictest	2071817	-21	ssh	22:46:47	7
1596069	99	26	23,1	cyclictest	2071817	-21	ssh	22:46:47	7
1596058	99	26	24,2	cyclictest	2024369	-21	kworker/u16:0+flush-8:0	22:43:02	4
1596058	99	26	24,2	cyclictest	2024369	-21	kworker/u16:0+flush-8:0	22:43:02	4
1596058	99	26	24,2	cyclictest	1951407	-21	kworker/u16:2+flush-8:0	22:08:17	4
1596069	99	25	24,0	cyclictest	0	-21	swapper/7	21:40:07	7
1596069	99	25	23,1	cyclictest	1974118	-21	ssh	22:04:50	7
1596069	99	25	23,1	cyclictest	1974118	-21	ssh	22:04:50	7
1596069	99	25	23,1	cyclictest	1830350	-21	diskmemload	00:13:49	7
1596063	99	25	21,3	cyclictest	1951407	-21	kworker/u16:2+flush-8:0	00:12:44	5
1596058	99	25	23,2	cyclictest	1696054	-21	kworker/u16:3+flush-8:0	21:53:09	4
1596069	99	24	23,0	cyclictest	0	-21	swapper/7	23:52:02	7
1596069	99	24	22,1	cyclictest	2153738	-21	ssh	23:18:06	7
1596069	99	24	0,22	cyclictest	1866653	-21	perf	21:22:40	7
1596058	99	24	23,1	cyclictest	1951408	-21	kworker/u16:4+events_unbound	22:06:36	4
1596058	99	24	23,1	cyclictest	1951408	-21	kworker/u16:4+events_unbound	22:06:36	4
1596069	99	23	21,1	cyclictest	1950114	-21	ssh	21:55:23	7
1596069	99	23	21,1	cyclictest	1841107	-21	ssh	21:10:15	7
1596063	99	23	21,2	cyclictest	1951407	-21	kworker/u16:2+flush-8:0	00:32:35	5
1596063	99	23	20,3	cyclictest	1951408	-21	kworker/u16:4+flush-8:0	22:28:06	5
1596069	99	22	21,0	cyclictest	0	-21	swapper/7	22:57:59	7
1596069	99	22	21,0	cyclictest	0	-21	swapper/7	22:53:51	7
1596069	99	22	20,1	cyclictest	1998111	-21	ssh	22:14:31	7
1596069	99	22	17,4	cyclictest	2277521	-21	updatedb	00:12:38	7
1596063	99	22	20,2	cyclictest	1924757	-21	kworker/u16:0+flush-8:0	21:48:12	5
1596063	99	22	20,2	cyclictest	1782313	-21	kworker/u16:0+events_unbound	21:32:55	5
1596063	99	22	20,2	cyclictest	1696054	-21	kworker/u16:3+flush-8:0	20:33:03	5
1596063	99	22	19,2	cyclictest	1696054	-21	kworker/u16:3+flush-8:0	21:12:55	5
1596058	99	22	18,3	cyclictest	2277565	-21	kworker/u16:3+events_unbound	00:13:02	4
1596069	99	21	20,0	cyclictest	0	-21	swapper/7	22:18:34	7
1596069	99	21	20,0	cyclictest	0	-21	swapper/7	21:31:48	7
1596069	99	21	19,1	cyclictest	2336209	-21	ssh	00:34:10	7
1596069	99	21	15,4	cyclictest	635	-21	systemd-journal	20:17:40	7
1596069	99	21	0,19	cyclictest	2240751	-21	sshd-session	23:56:47	7
1596058	99	21	19,2	cyclictest	1951407	-21	kworker/u16:2+events_unbound	23:03:08	4
1596058	99	21	19,2	cyclictest	1674401	-21	kworker/u16:2+flush-8:0	20:12:59	4
1596058	99	21	18,2	cyclictest	1951408	-21	kworker/u16:4+flush-8:0	23:32:59	4
1596058	99	21	17,3	cyclictest	1657295	-21	kworker/u16:0+events_unbound	19:37:54	4
1596069	99	20	19,0	cyclictest	0	-21	swapper/7	22:49:41	7
1596069	99	20	15,4	cyclictest	2313837	-21	latency_hist	00:27:40	7
1596069	99	20	0,18	cyclictest	2022595	-21	sh	22:25:10	7
1596064	99	20	18,2	cyclictest	2024369	-21	kworker/u16:0+flush-8:0	23:37:53	6
1596058	99	20	18,2	cyclictest	1674401	-21	kworker/u16:2+events_unbound	20:38:08	4
1596058	99	20	17,3	cyclictest	1674401	-21	kworker/u16:2+flush-8:0	19:53:03	4
1596069	99	19	18,0	cyclictest	0	-21	swapper/7	23:41:54	7
1596069	99	19	18,0	cyclictest	0	-21	swapper/7	00:28:13	7
1596069	99	19	17,1	cyclictest	1830350	-21	diskmemload	23:11:40	7
1596069	99	19	17,1	cyclictest	0	-21	swapper/7	23:46:39	7
1596069	99	19	16,2	cyclictest	1961576	-21	ssh	21:59:04	7
1596069	99	19	16,2	cyclictest	1638190	-21	munin-run	19:27:40	7
1596069	99	19	14,4	cyclictest	2168842	-21	latency_hist	23:27:40	7
1596069	99	19	14,4	cyclictest	2168842	-21	latency_hist	23:27:40	7
1596058	99	19	17,2	cyclictest	1951408	-21	kworker/u16:4+flush-8:0	23:23:05	4
1596058	99	19	17,2	cyclictest	1951408	-21	kworker/u16:4+flush-8:0	23:23:05	4
1596058	99	19	17,2	cyclictest	1951407	-21	kworker/u16:2+events_unbound	22:52:53	4
1596069	99	18	17,0	cyclictest	0	-21	swapper/7	19:33:06	7
1596058	99	18	4,13	cyclictest	0	-21	swapper/4	20:42:55	4
1596058	99	18	4,13	cyclictest	0	-21	swapper/4	20:42:55	4
1596058	99	18	16,2	cyclictest	1924757	-21	kworker/u16:0+events_unbound	21:48:02	4
1596058	99	18	16,2	cyclictest	1842842	-21	kworker/u16:4+events_unbound	21:17:40	4
1596058	99	18	15,2	cyclictest	1782313	-21	kworker/u16:0+flush-8:0	20:53:03	4
1596069	99	17	16,0	cyclictest	0	-21	swapper/7	23:36:47	7
1596069	99	17	16,0	cyclictest	0	-21	swapper/7	19:58:02	7
1596064	99	17	15,2	cyclictest	1657295	-21	kworker/u16:0+flush-8:0	19:53:37	6
1596063	99	17	15,2	cyclictest	1951407	-21	kworker/u16:2+flush-8:0	23:53:02	5
1596063	99	17	15,2	cyclictest	1951407	-21	kworker/u16:2+flush-8:0	22:08:08	5
1596063	99	17	15,2	cyclictest	1696054	-21	kworker/u16:3+flush-8:0	21:28:12	5
1596069	99	16	14,1	cyclictest	2058569	-21	ssh	22:39:39	7
1596064	99	16	14,2	cyclictest	1876448	-21	kworker/u16:1+flush-8:0	22:13:07	6
1596063	99	16	14,2	cyclictest	1951408	-21	kworker/u16:4+flush-8:0	22:33:07	5
1596058	99	16	13,2	cyclictest	1337918	-21	kworker/u16:1+events_unbound	19:07:58	4
1596058	99	16	13,2	cyclictest	1337918	-21	kworker/u16:1+events_unbound	19:07:58	4
1596069	99	15	6,7	cyclictest	6101	-21	gmain	21:02:58	7
1596069	99	15	14,0	cyclictest	0	-21	swapper/7	00:21:54	7
1596064	99	15	2,5	cyclictest	0	-21	swapper/6	00:17:42	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17