OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Thu Jun 25, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3004800	2	145	0,1	sleep7	3004798	-21	ssh	22:06:50	7
2869323	2	99	0,1	sleep7	2869324	-21	sh	21:11:05	7
2703088	2	99	0,0	sleep4	0	-21	swapper/4	19:49:29	4
3139021	2	72	0,1	sleep7	3139022	-21	sh	23:00:20	7
2620906	2	54	20,29	sleep7	0	-21	swapper/7	19:06:47	7
2620848	2	53	43,7	sleep5	0	-21	swapper/5	19:05:56	5
2621044	2	45	19,8	sleep4	0	-21	swapper/4	19:08:44	4
2621382	99	41	37,3	cyclictest	3040802	-21	kworker/u16:1+flush-8:0	23:19:44	5
2621385	99	37	35,2	cyclictest	2930289	-21	kworker/u16:5+flush-8:0	22:15:40	6
2621385	99	37	34,2	cyclictest	3202627	-21	kworker/u16:4+events_unbound	23:54:31	6
2621385	99	37	34,2	cyclictest	3202627	-21	kworker/u16:4+events_unbound	23:54:31	6
2621380	99	37	35,2	cyclictest	2772874	-21	kworker/u16:2+events_unbound	22:09:29	4
2621380	99	37	34,2	cyclictest	3239363	-21	kworker/u16:3+events_unbound	00:04:31	4
2621380	99	36	34,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:09:23	4
2621385	99	34	32,2	cyclictest	3276725	-21	kworker/u16:1+events_unbound	00:05:38	6
2621382	99	34	29,4	cyclictest	2930289	-21	kworker/u16:5+flush-8:0	21:49:29	5
2621055	2	34	24,7	sleep6	0	-21	swapper/6	19:08:54	6
2621385	99	33	31,2	cyclictest	3300599	-21	kworker/u16:0+events_unbound	00:28:39	6
2621380	99	33	31,2	cyclictest	2616030	-21	kworker/u16:1+events_unbound	20:04:30	4
2621385	99	32	30,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:13:19	6
2621385	99	32	30,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:13:19	6
2621385	99	32	30,2	cyclictest	3080079	-21	kworker/u16:2+events_unbound	22:45:24	6
2621385	99	32	30,2	cyclictest	3040802	-21	kworker/u16:1+events_unbound	23:31:08	6
2621385	99	32	30,2	cyclictest	2871352	-21	kworker/u16:4+flush-8:0	22:01:40	6
2621385	99	32	29,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:19:32	6
2621382	99	32	30,2	cyclictest	2616030	-21	kworker/u16:1+events_unbound	20:04:44	5
2621380	99	32	29,2	cyclictest	3202627	-21	kworker/u16:4+flush-8:0	23:54:39	4
2621380	99	32	29,2	cyclictest	3202627	-21	kworker/u16:4+flush-8:0	23:54:39	4
2621385	99	31	29,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:01:28	6
2621385	99	30	28,2	cyclictest	3080079	-21	kworker/u16:2+events_unbound	22:43:48	6
2621385	99	30	28,2	cyclictest	3040802	-21	kworker/u16:1+events_unbound	22:28:40	6
2621385	99	30	28,2	cyclictest	2930289	-21	kworker/u16:5+flush-8:0	22:23:51	6
2621385	99	30	28,2	cyclictest	2761596	-21	kworker/u16:3+events_unbound	21:27:37	6
2621380	99	30	28,2	cyclictest	2871352	-21	kworker/u16:4+flush-8:0	21:39:51	4
2621380	99	30	28,2	cyclictest	2761596	-21	kworker/u16:3+events_unbound	21:19:29	4
2621385	99	29	27,2	cyclictest	3362402	-21	kworker/u16:3+events_unbound	00:37:35	6
2621385	99	29	27,2	cyclictest	3362402	-21	kworker/u16:3+events_unbound	00:37:35	6
2621385	99	29	27,2	cyclictest	3314499	-21	kworker/u16:2+events_unbound	00:23:19	6
2621385	99	29	27,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:16:44	6
2621385	99	29	27,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:05:24	6
2621385	99	29	27,2	cyclictest	2772874	-21	kworker/u16:2+events_unbound	21:57:12	6
2621385	99	29	27,2	cyclictest	2772874	-21	kworker/u16:2+events_unbound	21:37:59	6
2621385	99	29	27,2	cyclictest	2761596	-21	kworker/u16:3+events_unbound	21:04:24	6
2621385	99	29	27,2	cyclictest	2616030	-21	kworker/u16:1+events_unbound	21:34:24	6
2621380	99	29	27,2	cyclictest	3314499	-21	kworker/u16:2+flush-8:0	00:19:29	4
2621380	99	29	26,2	cyclictest	2772874	-21	kworker/u16:2+flush-8:0	21:44:30	4
2621385	99	28	26,2	cyclictest	3276725	-21	kworker/u16:1+events_unbound	00:30:40	6
2621385	99	28	26,2	cyclictest	2930289	-21	kworker/u16:5+events_unbound	22:54:08	6
2621385	99	28	26,2	cyclictest	2761596	-21	kworker/u16:3+events_unbound	21:19:25	6
2621382	99	28	24,3	cyclictest	2616030	-21	kworker/u16:1+flush-8:0	19:34:47	5
2621382	99	28	24,3	cyclictest	2616030	-21	kworker/u16:1+flush-8:0	19:34:47	5
2621380	99	28	26,2	cyclictest	3202627	-21	kworker/u16:4+events_unbound	23:42:40	4
2621380	99	28	26,2	cyclictest	2930289	-21	kworker/u16:5+events_unbound	22:34:39	4
2621380	99	28	26,2	cyclictest	1191689	-21	kworker/u16:5+flush-8:0	19:54:30	4
2621380	99	28	25,3	cyclictest	2761596	-21	kworker/u16:3+flush-8:0	20:34:43	4
2621385	99	27	25,2	cyclictest	3080079	-21	kworker/u16:2+flush-8:0	00:03:52	6
2621385	99	27	25,2	cyclictest	2930289	-21	kworker/u16:5+events_unbound	21:42:40	6
2621385	99	27	25,2	cyclictest	2772874	-21	kworker/u16:2+events_unbound	21:22:48	6
2621385	99	27	24,2	cyclictest	2871352	-21	kworker/u16:4+events_unbound	22:04:32	6
2621380	99	27	25,2	cyclictest	2871352	-21	kworker/u16:4+flush-8:0	22:25:07	4
2621380	99	27	25,2	cyclictest	2616030	-21	kworker/u16:1+flush-8:0	20:24:44	4
2621380	99	27	25,2	cyclictest	2616030	-21	kworker/u16:1+flush-8:0	20:24:44	4
2621380	99	27	25,2	cyclictest	1191689	-21	kworker/u16:5+flush-8:0	19:29:24	4
2621380	99	27	24,2	cyclictest	2871352	-21	kworker/u16:4+events_unbound	21:59:31	4
2621380	99	27	24,2	cyclictest	2772874	-21	kworker/u16:2+events_unbound	21:24:32	4
2621385	99	26	24,2	cyclictest	3314499	-21	kworker/u16:2+events_unbound	00:17:23	6
2621385	99	26	24,2	cyclictest	3202627	-21	kworker/u16:4+events_unbound	23:52:03	6
2621385	99	26	24,2	cyclictest	3080079	-21	kworker/u16:2+flush-8:0	23:39:50	6
2621385	99	26	24,2	cyclictest	3040802	-21	kworker/u16:1+events_unbound	23:35:51	6
2621385	99	26	24,2	cyclictest	3040802	-21	kworker/u16:1+events_unbound	22:32:11	6
2621385	99	26	24,2	cyclictest	3040802	-21	kworker/u16:1+events_unbound	22:32:11	6
2621385	99	26	24,2	cyclictest	2772874	-21	kworker/u16:2+events_unbound	22:14:20	6
2621380	99	26	24,2	cyclictest	3300599	-21	kworker/u16:0+events_unbound	00:14:29	4
2621380	99	26	24,2	cyclictest	2616030	-21	kworker/u16:1+flush-8:0	21:29:47	4
2621387	99	25	20,4	cyclictest	650	-21	systemd-journal	19:34:29	7
2621385	99	25	23,2	cyclictest	2930289	-21	kworker/u16:5+flush-8:0	21:45:19	6
2621385	99	25	23,2	cyclictest	2930289	-21	kworker/u16:5+events_unbound	22:39:11	6
2621385	99	25	23,2	cyclictest	2859148	-21	kworker/u16:0+events_unbound	21:11:32	6
2621385	99	25	23,2	cyclictest	2772874	-21	kworker/u16:2+flush-8:0	21:50:11	6
2621385	99	25	22,2	cyclictest	3080079	-21	kworker/u16:2+flush-8:0	23:44:34	6
2621380	99	25	23,2	cyclictest	3202627	-21	kworker/u16:4+events_unbound	23:52:40	4
2621380	99	25	23,2	cyclictest	3202627	-21	kworker/u16:4+events_unbound	23:48:59	4
2621380	99	25	23,2	cyclictest	3040802	-21	kworker/u16:1+events_unbound	22:40:39	4
2621380	99	25	23,2	cyclictest	2616030	-21	kworker/u16:1+events_unbound	19:29:43	4
2621380	99	25	22,2	cyclictest	2589897	-21	kworker/u16:2+events_unbound	19:14:34	4
2621380	99	25	21,3	cyclictest	2930289	-21	kworker/u16:5+flush-8:0	22:49:48	4
2621380	99	24	22,2	cyclictest	3314499	-21	kworker/u16:2+events_unbound	00:37:39	4
2621380	99	24	22,2	cyclictest	3314499	-21	kworker/u16:2+events_unbound	00:37:39	4
2621380	99	24	22,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:10:39	4
2621380	99	24	22,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:10:39	4
2621380	99	24	21,2	cyclictest	2871352	-21	kworker/u16:4+events_unbound	21:49:31	4
2621385	99	23	21,2	cyclictest	3276725	-21	kworker/u16:1+events_unbound	00:13:52	6
2621385	99	23	21,2	cyclictest	3080079	-21	kworker/u16:2+events_unbound	22:55:07	6
2621385	99	23	21,2	cyclictest	1191689	-21	kworker/u16:5+flush-8:0	19:54:40	6
2621382	99	23	20,2	cyclictest	3040802	-21	kworker/u16:1+flush-8:0	22:39:43	5
2621380	99	23	21,2	cyclictest	3300599	-21	kworker/u16:0+flush-8:0	00:29:24	4
2621380	99	22	20,2	cyclictest	3125887	-21	kworker/u16:0+events_unbound	23:23:40	4
2621380	99	22	20,2	cyclictest	3080079	-21	kworker/u16:2+flush-8:0	23:14:34	4
2621380	99	22	20,2	cyclictest	2859148	-21	kworker/u16:0+flush-8:0	21:09:47	4
2621380	99	22	20,2	cyclictest	2673950	-21	kworker/u16:0+events_unbound	20:19:29	4
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17