OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Dec 30, 2025 00:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2707795	2	104	0,1	sleep7	2707796	-21	cat	20:19:09	7
3094949	2	93	0,1	sleep7	3094951	-21	tr	23:39:09	7
3060953	2	67	0,1	sleep7	3060956	-21	tr	23:19:13	7
2956207	2	63	0,1	sleep4	0	-21	swapper/4	22:29:03	4
2564255	2	58	22,31	sleep7	0	-21	swapper/7	19:06:27	7
2564373	2	54	44,7	sleep4	0	-21	swapper/4	19:08:08	4
2564281	2	51	29,7	sleep6	0	-21	swapper/6	19:06:50	6
2564723	99	44	41,2	cyclictest	2674808	-21	kworker/u16:0+flush-8:0	20:09:07	5
2564403	2	43	31,9	sleep5	0	-21	swapper/5	19:08:33	5
2564723	99	41	38,2	cyclictest	2762140	-21	kworker/u16:0+events_unbound	21:28:51	5
2564725	99	38	35,2	cyclictest	2884256	-21	kworker/u16:5+events_unbound	21:54:07	6
2564725	99	37	35,2	cyclictest	2674808	-21	kworker/u16:0+flush-8:0	20:04:15	6
2564729	99	36	29,6	cyclictest	650	-21	systemd-journal	19:43:50	7
2564725	99	36	33,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	22:54:07	6
2564725	99	36	33,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	22:54:07	6
2564723	99	36	34,2	cyclictest	3169220	-21	kworker/u16:3+flush-8:0	00:38:49	5
2564723	99	36	33,2	cyclictest	2636169	-21	kworker/u16:3+events_unbound	20:19:03	5
2564725	99	35	32,2	cyclictest	2950256	-21	kworker/u16:2+flush-8:0	22:28:51	6
2564723	99	35	33,2	cyclictest	2884256	-21	kworker/u16:5+events_unbound	21:59:12	5
2564723	99	34	32,2	cyclictest	2965760	-21	kworker/u16:1+flush-8:0	23:49:14	5
2564723	99	34	32,2	cyclictest	2771766	-21	kworker/u16:2+events_unbound	21:04:07	5
2564725	99	33	31,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	23:06:20	6
2564725	99	33	31,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	21:12:16	6
2564723	99	33	31,2	cyclictest	2975196	-21	kworker/u16:3+flush-8:0	22:39:05	5
2564723	99	33	31,2	cyclictest	2636169	-21	kworker/u16:3+events_unbound	19:59:20	5
2564723	99	33	30,2	cyclictest	2829905	-21	kworker/u16:3+events_unbound	21:24:09	5
2564723	99	33	30,2	cyclictest	2674808	-21	kworker/u16:0+flush-8:0	20:29:07	5
2564729	99	32	31,0	cyclictest	0	-21	swapper/7	21:34:07	7
2564725	99	32	30,2	cyclictest	3101215	-21	kworker/u16:2+events_unbound	23:53:00	6
2564725	99	32	30,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	21:14:19	6
2564725	99	32	29,2	cyclictest	3149399	-21	kworker/u16:0+events_unbound	00:09:06	6
2564725	99	32	29,2	cyclictest	2946019	-21	kworker/u16:0+events_unbound	22:24:03	6
2564723	99	32	30,2	cyclictest	3072414	-21	kworker/u16:0+events_unbound	23:36:59	5
2564723	99	32	30,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	23:29:59	5
2564723	99	32	30,2	cyclictest	2897071	-21	kworker/u16:0+events_unbound	22:04:19	5
2564723	99	32	30,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	20:59:51	5
2564729	99	31	26,4	cyclictest	2994462	-21	latency_hist	22:48:49	7
2564729	99	31	24,6	cyclictest	2752234	-21	latency_hist	20:43:50	7
2564725	99	31	29,2	cyclictest	2876117	-21	kworker/u16:4+events_unbound	23:02:00	6
2564725	99	31	28,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	21:44:03	6
2564723	99	31	29,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	22:59:22	5
2564723	99	31	29,2	cyclictest	2704202	-21	kworker/u16:2+flush-8:0	20:24:03	5
2564723	99	31	29,2	cyclictest	2674808	-21	kworker/u16:0+events_unbound	20:35:00	5
2564719	99	31	29,2	cyclictest	3130452	-21	kworker/u16:5+events_unbound	00:05:11	4
2564725	99	30	28,2	cyclictest	3033433	-21	kworker/u16:3+events_unbound	23:26:00	6
2564725	99	30	28,2	cyclictest	2897760	-21	kworker/u16:3+events_unbound	22:04:13	6
2564725	99	30	28,2	cyclictest	2839747	-21	kworker/u16:2+events_unbound	21:49:18	6
2564723	99	30	28,2	cyclictest	3101215	-21	kworker/u16:2+events_unbound	23:58:52	5
2564723	99	30	28,2	cyclictest	2876117	-21	kworker/u16:4+flush-8:0	21:48:50	5
2564723	99	30	28,2	cyclictest	2876117	-21	kworker/u16:4+events_unbound	00:06:59	5
2564723	99	30	28,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	20:44:19	5
2564723	99	30	27,2	cyclictest	2578420	-21	kworker/u16:1+events_unbound	20:03:52	5
3174356	2	29	0,1	chrt	0	-21	swapper/4	00:19:12	4
2564729	99	29	24,4	cyclictest	650	-21	systemd-journal	20:08:45	7
2564725	99	29	27,2	cyclictest	3149399	-21	kworker/u16:0+events_unbound	00:29:12	6
2564725	99	29	27,2	cyclictest	3149399	-21	kworker/u16:0+events_unbound	00:29:12	6
2564725	99	29	27,2	cyclictest	3130452	-21	kworker/u16:5+events_unbound	23:59:49	6
2564725	99	29	27,2	cyclictest	2771766	-21	kworker/u16:2+events_unbound	20:59:24	6
2564725	99	29	27,2	cyclictest	2578420	-21	kworker/u16:1+flush-8:0	20:09:19	6
2564723	99	29	27,2	cyclictest	3072414	-21	kworker/u16:0+events_unbound	23:43:00	5
2564719	99	29	27,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	23:06:55	4
2564725	99	28	26,2	cyclictest	2897071	-21	kworker/u16:0+events_unbound	22:02:20	6
2564725	99	28	26,2	cyclictest	2762140	-21	kworker/u16:0+events_unbound	21:32:20	6
2564725	99	28	25,2	cyclictest	2762140	-21	kworker/u16:0+events_unbound	21:33:52	6
2564723	99	28	26,2	cyclictest	3159397	-21	kworker/u16:1+events_unbound	00:30:00	5
2564723	99	28	26,2	cyclictest	3159397	-21	kworker/u16:1+events_unbound	00:30:00	5
2564723	99	28	26,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	23:04:11	5
2564723	99	28	26,2	cyclictest	2876117	-21	kworker/u16:4+events_unbound	22:34:07	5
2564719	99	28	26,2	cyclictest	2897760	-21	kworker/u16:3+events_unbound	22:16:08	4
2564719	99	28	26,2	cyclictest	2762140	-21	kworker/u16:0+events_unbound	21:04:43	4
2564719	99	28	26,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	20:45:08	4
2564719	99	28	26,2	cyclictest	2704202	-21	kworker/u16:2+events_unbound	20:20:48	4
2564719	99	28	26,2	cyclictest	2578420	-21	kworker/u16:1+events_unbound	20:07:09	4
2564729	99	27	26,0	cyclictest	0	-21	swapper/7	23:51:19	7
2564729	99	27	26,0	cyclictest	0	-21	swapper/7	22:18:11	7
2564729	99	27	24,2	cyclictest	2888617	-21	df	21:54:05	7
2564725	99	27	25,2	cyclictest	3149399	-21	kworker/u16:0+events_unbound	00:25:31	6
2564725	99	27	25,2	cyclictest	3101215	-21	kworker/u16:2+events_unbound	23:47:28	6
2564725	99	27	25,2	cyclictest	2965760	-21	kworker/u16:1+events_unbound	22:39:20	6
2564725	99	27	25,2	cyclictest	2897760	-21	kworker/u16:3+events_unbound	22:16:00	6
2564725	99	27	24,2	cyclictest	2617605	-21	kworker/u16:2+events_unbound	19:38:51	6
2564723	99	27	25,2	cyclictest	2884256	-21	kworker/u16:5+events_unbound	21:50:32	5
2564723	99	27	25,2	cyclictest	2876117	-21	kworker/u16:4+events_unbound	22:56:04	5
2564723	99	27	25,2	cyclictest	2876117	-21	kworker/u16:4+events_unbound	22:56:04	5
2564723	99	27	25,2	cyclictest	2839747	-21	kworker/u16:2+events_unbound	21:35:00	5
2564723	99	27	24,2	cyclictest	3159397	-21	kworker/u16:1+events_unbound	00:23:51	5
2564723	99	27	24,2	cyclictest	2950256	-21	kworker/u16:2+flush-8:0	22:29:00	5
2564719	99	27	25,2	cyclictest	2771766	-21	kworker/u16:2+events_unbound	21:13:28	4
2564719	99	27	25,2	cyclictest	2746232	-21	kworker/u16:1+events_unbound	21:00:56	4
2564719	99	27	25,2	cyclictest	2674808	-21	kworker/u16:0+events_unbound	20:26:24	4
2564729	99	26	25,0	cyclictest	0	-21	swapper/7	23:32:27	7
2564729	99	26	25,0	cyclictest	0	-21	swapper/7	19:58:12	7
2564729	99	26	25,0	cyclictest	0	-21	swapper/7	19:52:08	7
2564729	99	26	25,0	cyclictest	0	-21	swapper/7	00:06:27	7
2564729	99	26	25,0	cyclictest	0	-21	swapper/7	00:01:21	7
2564729	99	26	20,5	cyclictest	2829698	-21	latency_hist	21:23:50	7
2564729	99	26	20,5	cyclictest	2829698	-21	latency_hist	21:23:50	7
2564725	99	26	24,2	cyclictest	2762140	-21	kworker/u16:0+events_unbound	20:55:00	6
2564725	99	26	23,2	cyclictest	3033433	-21	kworker/u16:3+events_unbound	23:14:03	6
2564725	99	26	23,2	cyclictest	2578420	-21	kworker/u16:1+events_unbound	19:30:01	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17