OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Mar 31, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3981507	2	101	0,1	sleep6	0	-21	swapper/6	22:52:44	6
3881302	2	97	0,1	sleep5	0	-21	swapper/5	22:08:12	5
3657316	2	96	0,0	sleep6	0	-21	swapper/6	20:32:45	6
3511071	2	65	0,1	sleep7	3511073	-21	kthreadcore	19:13:06	7
3481638	99	50	48,2	cyclictest	3471875	-21	kworker/u16:1+events_unbound	19:13:15	6
3481122	2	45	35,7	sleep4	0	-21	swapper/4	18:59:41	4
6265	89	42	0,36	rtkit-daemon	0	-21	swapper/6	19:01:50	6
3481638	99	42	40,2	cyclictest	3978849	-21	kworker/u16:1+events_unbound	23:25:48	6
3481629	99	42	40,2	cyclictest	3848029	-21	kworker/u16:0+flush-8:0	22:02:45	4
3481638	99	39	37,2	cyclictest	3861120	-21	kworker/u16:3+flush-8:0	22:37:45	6
3481638	99	39	37,2	cyclictest	3695455	-21	kworker/u16:3+events_unbound	21:15:59	6
3481114	2	39	3,31	sleep7	0	-21	swapper/7	18:59:36	7
3481639	99	38	34,2	cyclictest	3638223	-21	munin-run	20:22:45	7
3481639	99	38	0,37	cyclictest	650	-21	systemd-journal	23:12:39	7
3481638	99	38	36,2	cyclictest	4162361	-21	kworker/u16:0+events_unbound	00:27:26	6
3481638	99	38	36,2	cyclictest	3956971	-21	kworker/u16:0+events_unbound	23:14:03	6
3481638	99	38	36,2	cyclictest	3763212	-21	kworker/u16:0+events_unbound	21:39:06	6
3481638	99	38	36,2	cyclictest	3763212	-21	kworker/u16:0+events_unbound	21:39:06	6
3481638	99	38	36,2	cyclictest	3695455	-21	kworker/u16:3+events_unbound	21:20:15	6
3481638	99	38	35,2	cyclictest	3861120	-21	kworker/u16:3+events_unbound	22:27:46	6
3481634	99	38	33,5	cyclictest	3978849	-21	kworker/u16:1+flush-8:0	23:02:44	5
3481634	99	38	33,5	cyclictest	3978849	-21	kworker/u16:1+flush-8:0	23:02:44	5
3481132	2	38	26,8	sleep5	0	-21	swapper/5	18:59:50	5
3481639	99	37	32,4	cyclictest	3884307	-21	latency_hist	22:12:44	7
3481638	99	37	35,2	cyclictest	3861120	-21	kworker/u16:3+events_unbound	22:12:00	6
3481638	99	37	35,2	cyclictest	3419176	-21	kworker/u16:0+flush-8:0	19:18:09	6
3481634	99	37	34,2	cyclictest	4126698	-21	kworker/u16:3+flush-8:0	00:03:42	5
3481639	99	36	32,3	cyclictest	650	-21	systemd-journal	22:47:40	7
3481638	99	36	34,2	cyclictest	4078544	-21	kworker/u16:4+events_unbound	00:11:15	6
3481638	99	36	34,2	cyclictest	4078544	-21	kworker/u16:4+events_unbound	00:06:18	6
3481638	99	36	34,2	cyclictest	3956971	-21	kworker/u16:0+events_unbound	23:40:38	6
3481638	99	36	34,2	cyclictest	3695455	-21	kworker/u16:3+events_unbound	21:03:56	6
3481638	99	36	34,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	21:10:52	6
3481638	99	36	34,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	20:52:12	6
3481638	99	35	33,2	cyclictest	4126698	-21	kworker/u16:3+events_unbound	00:32:21	6
3481638	99	35	33,2	cyclictest	4078544	-21	kworker/u16:4+events_unbound	23:56:06	6
3481638	99	35	33,2	cyclictest	3956971	-21	kworker/u16:0+events_unbound	23:46:08	6
3481638	99	35	33,2	cyclictest	3861120	-21	kworker/u16:3+events_unbound	22:58:48	6
3481638	99	35	33,2	cyclictest	3861120	-21	kworker/u16:3+events_unbound	22:58:48	6
3481638	99	35	33,2	cyclictest	3570529	-21	kworker/u16:3+events_unbound	20:02:12	6
3481638	99	35	33,2	cyclictest	3523906	-21	kworker/u16:4+events_unbound	19:54:09	6
3481629	99	35	32,2	cyclictest	3861120	-21	kworker/u16:3+events_unbound	22:22:59	4
3481638	99	34	32,2	cyclictest	3981522	-21	kworker/u16:4+events_unbound	22:57:18	6
3481638	99	34	32,2	cyclictest	3978849	-21	kworker/u16:1+events_unbound	23:20:22	6
3481638	99	34	32,2	cyclictest	3956971	-21	kworker/u16:0+events_unbound	23:36:37	6
3481638	99	34	32,2	cyclictest	3929585	-21	kworker/u16:1+events_unbound	22:39:06	6
3481638	99	34	32,2	cyclictest	3906613	-21	kworker/u16:2+events_unbound	22:45:21	6
3481638	99	34	32,2	cyclictest	3763212	-21	kworker/u16:0+events_unbound	21:33:30	6
3481638	99	34	32,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	22:19:20	6
3481638	99	34	32,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	22:19:20	6
3481638	99	34	32,2	cyclictest	3570529	-21	kworker/u16:3+events_unbound	20:06:56	6
3481638	99	33	31,2	cyclictest	3978849	-21	kworker/u16:1+events_unbound	23:06:06	6
3481638	99	33	31,2	cyclictest	3978849	-21	kworker/u16:1+events_unbound	23:06:06	6
3481638	99	33	31,2	cyclictest	3570529	-21	kworker/u16:3+events_unbound	20:19:59	6
3481638	99	33	30,2	cyclictest	3848029	-21	kworker/u16:0+events_unbound	22:22:53	6
3481629	99	33	30,2	cyclictest	3823843	-21	kworker/u16:3+flush-8:0	21:52:55	4
3481638	99	32	30,2	cyclictest	4017991	-21	kworker/u16:2+events_unbound	23:09:53	6
3481638	99	32	30,2	cyclictest	3848029	-21	kworker/u16:0+events_unbound	21:59:22	6
3481638	99	32	30,2	cyclictest	3749293	-21	kworker/u16:2+events_unbound	21:45:02	6
3481638	99	32	30,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	21:24:07	6
3481638	99	31	29,2	cyclictest	4162361	-21	kworker/u16:0+events_unbound	00:20:24	6
3481638	99	31	29,2	cyclictest	4126698	-21	kworker/u16:3+flush-8:0	23:58:20	6
3481638	99	31	29,2	cyclictest	3848029	-21	kworker/u16:0+events_unbound	22:05:37	6
3481638	99	31	29,2	cyclictest	3763212	-21	kworker/u16:0+flush-8:0	21:28:23	6
3481634	99	31	28,2	cyclictest	3419176	-21	kworker/u16:0+events_unbound	19:07:47	5
3481639	99	30	29,0	cyclictest	0	-21	swapper/7	22:54:54	7
3481639	99	30	29,0	cyclictest	0	-21	swapper/7	21:21:03	7
3481639	99	30	25,4	cyclictest	650	-21	systemd-journal	22:22:44	7
3481639	99	30	25,4	cyclictest	650	-21	systemd-journal	22:22:44	7
3481638	99	30	28,2	cyclictest	4162361	-21	kworker/u16:0+events_unbound	00:14:18	6
3481638	99	30	28,2	cyclictest	3619475	-21	kworker/u16:2+events_unbound	20:14:23	6
3481638	99	30	28,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	21:50:22	6
3481638	99	30	28,2	cyclictest	3419176	-21	kworker/u16:0+events_unbound	19:10:56	6
3481639	99	29	29,0	cyclictest	0	-21	swapper/7	00:00:18	7
3481639	99	29	28,0	cyclictest	0	-21	swapper/7	19:39:28	7
3481638	99	29	27,2	cyclictest	3956971	-21	kworker/u16:0+events_unbound	23:28:58	6
3481638	99	29	27,2	cyclictest	3619475	-21	kworker/u16:2+events_unbound	20:25:00	6
3481638	99	29	27,2	cyclictest	3419176	-21	kworker/u16:0+events_unbound	19:37:23	6
3481639	99	28	27,0	cyclictest	0	-21	swapper/7	23:48:42	7
3481638	99	28	26,2	cyclictest	3590862	-21	kworker/u16:1+events_unbound	20:57:36	6
3481638	99	28	26,2	cyclictest	3523906	-21	kworker/u16:4+flush-8:0	19:42:40	6
3481638	99	28	26,2	cyclictest	3481832	-21	kworker/u16:2+events_unbound	19:04:37	6
3481638	99	28	26,2	cyclictest	3471875	-21	kworker/u16:1+events_unbound	19:30:23	6
3481638	99	28	26,2	cyclictest	3471875	-21	kworker/u16:1+events_unbound	19:30:23	6
3481629	99	28	27,1	cyclictest	3981522	-21	kworker/u16:4+events_unbound	22:55:19	4
3481629	99	28	26,2	cyclictest	3695455	-21	kworker/u16:3+flush-8:0	20:53:17	4
3481629	99	28	25,2	cyclictest	3523906	-21	kworker/u16:4+flush-8:0	19:23:00	4
3481638	99	27	25,2	cyclictest	3848029	-21	kworker/u16:0+events_unbound	22:15:49	6
3481638	99	27	25,2	cyclictest	3823843	-21	kworker/u16:3+flush-8:0	21:53:43	6
3481639	99	26	26,0	cyclictest	0	-21	swapper/7	22:07:34	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	20:59:00	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	20:46:16	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	20:40:36	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	20:13:12	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	20:08:33	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	20:00:11	7
3481639	99	26	25,0	cyclictest	0	-21	swapper/7	19:24:13	7
3481639	99	26	24,1	cyclictest	4136551	-21	ssh	23:53:43	7
3481639	99	26	12,13	cyclictest	0	-21	swapper/7	19:48:10	7
3481638	99	26	24,2	cyclictest	3705401	-21	kworker/u16:0+events_unbound	21:00:40	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17