OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sat Jul 11, 2026 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3420487	2	51	40,8	sleep4	0	-21	swapper/4	19:08:13	4
3420572	2	48	37,8	sleep6	0	-21	swapper/6	19:09:28	6
3420542	2	44	34,7	sleep5	0	-21	swapper/5	19:09:03	5
3420895	99	42	40,2	cyclictest	3927870	-21	kworker/u16:2+events_unbound	23:50:13	5
3419481	2	40	3,31	sleep7	0	-21	swapper/7	19:05:12	7
3420895	99	38	36,2	cyclictest	3670818	-21	kworker/u16:5+events_unbound	21:41:22	5
3420895	99	38	36,2	cyclictest	3670818	-21	kworker/u16:5+events_unbound	21:41:22	5
3420895	99	37	35,2	cyclictest	4013178	-21	kworker/u16:3+events_unbound	00:16:14	5
3420895	99	37	35,2	cyclictest	3878640	-21	kworker/u16:6+events_unbound	22:46:38	5
3420895	99	37	35,2	cyclictest	3670818	-21	kworker/u16:5+events_unbound	22:04:23	5
3420895	99	35	33,2	cyclictest	4013178	-21	kworker/u16:3+events_unbound	00:02:18	5
3420895	99	35	33,2	cyclictest	3878640	-21	kworker/u16:6+events_unbound	22:42:18	5
3420895	99	35	33,2	cyclictest	3829560	-21	kworker/u16:1+flush-8:0	00:34:42	5
3420895	99	35	33,2	cyclictest	3719951	-21	kworker/u16:1+events_unbound	21:51:39	5
3420895	99	35	33,2	cyclictest	3670818	-21	kworker/u16:5+events_unbound	22:07:58	5
3420895	99	34	32,2	cyclictest	4013178	-21	kworker/u16:3+events_unbound	00:26:14	5
3420895	99	34	32,2	cyclictest	3768524	-21	kworker/u16:3+events_unbound	22:11:18	5
3420895	99	34	32,2	cyclictest	3639067	-21	kworker/u16:2+flush-8:0	21:35:18	5
3420895	99	34	31,2	cyclictest	3829560	-21	kworker/u16:1+events_unbound	23:34:49	5
3420895	99	33	31,2	cyclictest	3997799	-21	kworker/u16:0+events_unbound	23:29:05	5
3420895	99	33	31,2	cyclictest	3997799	-21	kworker/u16:0+events_unbound	00:09:45	5
3420895	99	33	31,2	cyclictest	3829560	-21	kworker/u16:1+events_unbound	23:11:33	5
3420895	99	33	31,2	cyclictest	3802078	-21	kworker/u16:0+events_unbound	22:39:05	5
3751091	2	32	0,1	chrt	3751092	-21	kthreadcore	21:45:04	7
3420895	99	32	30,2	cyclictest	3927870	-21	kworker/u16:2+events_unbound	23:40:34	5
3420895	99	32	30,2	cyclictest	3768524	-21	kworker/u16:3+events_unbound	22:34:29	5
3420895	99	32	29,2	cyclictest	3829560	-21	kworker/u16:1+events_unbound	00:19:55	5
3420895	99	31	29,2	cyclictest	3829560	-21	kworker/u16:1+events_unbound	23:34:13	5
3420895	99	31	29,2	cyclictest	3802078	-21	kworker/u16:0+flush-8:0	22:25:45	5
3420895	99	31	29,2	cyclictest	3670818	-21	kworker/u16:5+flush-8:0	21:48:51	5
3420895	99	31	29,2	cyclictest	3639067	-21	kworker/u16:2+flush-8:0	21:10:22	5
3420895	99	31	29,2	cyclictest	3639067	-21	kworker/u16:2+events_unbound	21:27:33	5
3420895	99	31	29,2	cyclictest	3570735	-21	kworker/u16:3+events_unbound	21:31:02	5
3420895	99	30	28,2	cyclictest	3878640	-21	kworker/u16:6+events_unbound	23:02:49	5
3420895	99	30	28,2	cyclictest	3768524	-21	kworker/u16:3+events_unbound	22:23:13	5
3420895	99	30	28,2	cyclictest	3768524	-21	kworker/u16:3+events_unbound	22:23:13	5
3917570	2	29	0,1	chrt	3917571	-21	kthreadcore	22:55:01	7
3420895	99	29	27,2	cyclictest	4144880	-21	kworker/u16:2+flush-8:0	00:35:17	5
3420895	99	29	27,2	cyclictest	3997799	-21	kworker/u16:0+flush-8:0	23:46:29	5
3420895	99	29	27,2	cyclictest	3878640	-21	kworker/u16:6+flush-8:0	23:19:21	5
3420895	99	29	27,2	cyclictest	3878640	-21	kworker/u16:6+flush-8:0	22:58:47	5
3420895	99	29	27,2	cyclictest	3365835	-21	kworker/u16:0+events_unbound	19:34:47	5
3420895	99	29	26,2	cyclictest	3445223	-21	kworker/u16:3+flush-8:0	20:04:45	5
3420895	99	28	26,2	cyclictest	3997799	-21	kworker/u16:0+flush-8:0	00:05:34	5
4061129	2	27	0,0	chrt	4061130	-21	ssh	23:53:35	7
3901840	2	27	0,1	chrt	3901841	-21	ssh	22:47:39	7
3420895	99	27	25,2	cyclictest	3768524	-21	kworker/u16:3+events_unbound	22:17:51	5
3420895	99	27	25,2	cyclictest	3492842	-21	kworker/u16:4+events_unbound	19:56:11	5
3420904	99	26	0,24	cyclictest	3841730	-21	latency_hist	22:24:43	7
3420904	99	26	0,24	cyclictest	3841730	-21	latency_hist	22:24:43	7
3420895	99	26	24,2	cyclictest	3997799	-21	kworker/u16:0+events_unbound	23:55:08	5
3420895	99	26	24,2	cyclictest	3866438	-21	kworker/u16:4+events_unbound	22:54:12	5
3420895	99	26	24,2	cyclictest	3551635	-21	kworker/u16:1+events_unbound	20:21:42	5
3420895	99	26	24,2	cyclictest	3550252	-21	kworker/u16:0+events_unbound	21:07:35	5
3420904	99	25	19,4	cyclictest	3817331	-21	latency_hist	22:14:43	7
3420895	99	25	23,2	cyclictest	3878640	-21	kworker/u16:6+flush-8:0	23:05:22	5
3420895	99	25	23,2	cyclictest	3570735	-21	kworker/u16:3+events_unbound	20:34:14	5
3420895	99	25	23,2	cyclictest	3492842	-21	kworker/u16:4+flush-8:0	19:52:58	5
3420895	99	25	23,2	cyclictest	3419355	-21	kworker/u16:1+events_unbound	19:41:46	5
3420895	99	25	22,2	cyclictest	3878640	-21	kworker/u16:6+flush-8:0	23:19:51	5
3420895	99	24	22,2	cyclictest	3600198	-21	kworker/u16:1+events_unbound	20:41:59	5
3420895	99	24	22,2	cyclictest	3570735	-21	kworker/u16:3+events_unbound	21:22:56	5
3420895	99	24	22,2	cyclictest	3570735	-21	kworker/u16:3+events_unbound	21:04:31	5
3420895	99	24	22,2	cyclictest	3570735	-21	kworker/u16:3+events_unbound	21:04:31	5
3420895	99	24	22,2	cyclictest	3551635	-21	kworker/u16:1+events_unbound	20:16:23	5
3420895	99	24	21,2	cyclictest	3492842	-21	kworker/u16:4+flush-8:0	19:44:46	5
3420892	99	24	21,3	cyclictest	3639067	-21	kworker/u16:2+flush-8:0	21:54:43	4
3420892	99	24	21,2	cyclictest	3829560	-21	kworker/u16:1+flush-8:0	00:24:52	4
3420904	99	23	17,5	cyclictest	648	-21	systemd-journal	23:59:43	7
3420895	99	23	21,2	cyclictest	3719951	-21	kworker/u16:1+events_unbound	21:59:07	5
3420895	99	23	21,2	cyclictest	3550252	-21	kworker/u16:0+flush-8:0	20:27:06	5
3420895	99	23	21,2	cyclictest	3365835	-21	kworker/u16:0+events_unbound	19:34:26	5
3420895	99	22	20,2	cyclictest	3639067	-21	kworker/u16:2+events_unbound	21:15:36	5
3420895	99	22	20,2	cyclictest	3570735	-21	kworker/u16:3+flush-8:0	20:39:03	5
3420895	99	22	20,2	cyclictest	3492842	-21	kworker/u16:4+events_unbound	20:12:09	5
3420895	99	22	20,2	cyclictest	3445223	-21	kworker/u16:3+flush-8:0	20:00:44	5
3420895	99	22	20,2	cyclictest	3365835	-21	kworker/u16:0+events_unbound	19:29:27	5
3420895	99	22	20,2	cyclictest	3365835	-21	kworker/u16:0+events_unbound	19:29:27	5
3420895	99	21	19,2	cyclictest	3600198	-21	kworker/u16:1+events_unbound	20:45:18	5
3420895	99	21	18,2	cyclictest	3419355	-21	kworker/u16:1+flush-8:0	19:09:59	5
3420892	99	21	18,2	cyclictest	4144880	-21	kworker/u16:2+flush-8:0	00:34:53	4
3420904	99	20	14,5	cyclictest	4124133	-21	cat	00:19:58	7
3420895	99	20	18,2	cyclictest	2686190	-21	kworker/u16:2+events_unbound	19:15:58	5
3420904	99	19	14,4	cyclictest	3792949	-21	latency_hist	22:04:43	7
3420904	99	19	0,2	cyclictest	3692033	-21	ssh	21:20:06	7
3420895	99	19	17,2	cyclictest	3570735	-21	kworker/u16:3+flush-8:0	20:51:30	5
3420904	99	18	0,16	cyclictest	3853999	-21	latency_hist	22:29:43	7
3420904	99	17	14,2	cyclictest	1274	-21	snmpd	19:18:17	7
3420904	99	17	14,1	cyclictest	3731810	-21	awk	21:39:42	7
3420904	99	17	0,1	cyclictest	0	-21	swapper/7	19:25:08	7
3420904	99	17	0,1	cyclictest	0	-21	swapper/7	19:25:08	7
3420901	99	17	16,1	cyclictest	0	-21	swapper/6	22:00:17	6
3420895	99	17	15,2	cyclictest	3600198	-21	kworker/u16:1+events_unbound	20:56:29	5
3420895	99	17	15,2	cyclictest	3445223	-21	kworker/u16:3+flush-8:0	19:20:17	5
3420904	99	16	11,4	cyclictest	3805175	-21	latency_hist	22:09:43	7
3420904	99	16	0,14	cyclictest	1081	-21	ntpd	00:11:48	7
3420901	99	16	16,0	cyclictest	0	-21	swapper/6	22:41:59	6
3420904	99	15	9,5	cyclictest	3648751	-21	latency_hist	21:04:43	7
3420904	99	15	9,5	cyclictest	3648751	-21	latency_hist	21:04:43	7
3420904	99	15	9,5	cyclictest	3453761	-21	latency_hist	19:24:43	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17