OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Jul 29, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2867451	2	53	19,29	sleep7	0	-21	swapper/7	07:05:27	7
2867998	99	50	44,5	cyclictest	3066351	-21	latency_hist	08:48:59	7
2867998	99	46	42,3	cyclictest	125	-21	kswapd0	09:17:57	7
2867998	99	46	42,3	cyclictest	125	-21	kswapd0	09:17:57	7
2867988	99	44	40,4	cyclictest	2949735	-21	kworker/u16:1+flush-8:0	07:53:59	5
2867988	99	44	40,4	cyclictest	2949735	-21	kworker/u16:1+flush-8:0	07:53:59	5
2867430	2	42	32,7	sleep4	0	-21	swapper/4	07:05:07	4
2867998	99	41	38,2	cyclictest	3216059	-21	ssh	09:53:59	7
2866316	2	41	28,8	sleep5	0	-21	swapper/5	07:04:28	5
2867988	99	40	37,2	cyclictest	3127827	-21	kworker/u16:1+events_unbound	09:29:14	5
2867432	2	40	27,9	sleep6	0	-21	swapper/6	07:05:10	6
3607228	2	39	0,0	chrt	0	-21	swapper/5	12:31:39	5
2867998	99	35	29,5	cyclictest	2998218	-21	latency_hist	08:14:00	7
2867988	99	35	33,2	cyclictest	3066735	-21	kworker/u16:4+flush-8:0	09:04:00	5
2867998	99	33	32,0	cyclictest	0	-21	swapper/7	08:19:14	7
2867998	99	33	30,2	cyclictest	3316116	-21	ssh	10:34:17	7
2867988	99	33	31,2	cyclictest	3127827	-21	kworker/u16:1+flush-8:0	09:34:12	5
2867988	99	32	30,2	cyclictest	2949735	-21	kworker/u16:1+events_unbound	08:04:21	5
2867998	99	31	25,5	cyclictest	2910610	-21	latency_hist	07:29:00	7
2867985	99	31	29,2	cyclictest	3251442	-21	kworker/u16:0+events_unbound	10:24:26	4
2867998	99	30	0,28	cyclictest	2879577	-21	grep	07:14:03	7
2867988	99	29	27,2	cyclictest	3507859	-21	kworker/u16:2+events_unbound	12:19:01	5
2867988	99	29	27,2	cyclictest	3251442	-21	kworker/u16:0+events_unbound	10:13:10	5
2867988	99	29	27,1	cyclictest	3127827	-21	kworker/u16:1+events_unbound	10:25:42	5
2867985	99	29	25,3	cyclictest	2900871	-21	kworker/u16:3+flush-8:0	08:49:17	4
3253201	2	28	0,1	chrt	3253203	-21	sed	10:08:59	7
2911160	2	28	0,0	sleep6	0	-21	swapper/6	07:29:13	6
2867998	99	28	22,5	cyclictest	2920360	-21	latency_hist	07:34:00	7
2867988	99	28	26,2	cyclictest	3313610	-21	kworker/u16:1+flush-8:0	11:44:25	5
2867988	99	28	26,2	cyclictest	3209088	-21	kworker/u16:2+flush-8:0	09:54:29	5
2867988	99	28	26,2	cyclictest	3209088	-21	kworker/u16:2+flush-8:0	09:54:29	5
2867985	99	28	26,2	cyclictest	3129293	-21	kworker/u16:5+flush-8:0	09:34:17	4
3190722	2	26	0,1	chrt	3190720	-21	ssh	09:42:47	7
2867998	99	26	23,2	cyclictest	3532318	-21	ssh	11:59:31	7
2867998	99	26	23,2	cyclictest	3532318	-21	ssh	11:59:31	7
2867988	99	26	24,2	cyclictest	3127827	-21	kworker/u16:1+flush-8:0	10:03:54	5
2867988	99	26	24,2	cyclictest	3085539	-21	kworker/u16:0+events_unbound	09:14:11	5
2867988	99	26	24,2	cyclictest	3085539	-21	kworker/u16:0+events_unbound	09:14:11	5
2867988	99	26	24,2	cyclictest	2838574	-21	kworker/u16:1+flush-8:0	07:29:25	5
2867998	99	25	24,0	cyclictest	0	-21	swapper/7	12:04:18	7
2867998	99	25	20,4	cyclictest	3412170	-21	latency_hist	11:13:59	7
2867988	99	25	23,2	cyclictest	3338854	-21	kworker/u16:4+events_unbound	10:58:54	5
2867988	99	24	23,1	cyclictest	3323967	-21	kworker/u16:3+events_unbound	10:34:57	5
2867988	99	24	21,2	cyclictest	3300470	-21	kworker/u16:2+flush-8:0	10:44:09	5
2867993	99	23	19,3	cyclictest	2891163	-21	kworker/u16:2+flush-8:0	08:29:17	6
2867988	99	23	21,2	cyclictest	2891163	-21	kworker/u16:2+flush-8:0	07:18:55	5
2867985	99	23	20,2	cyclictest	2891163	-21	kworker/u16:2+flush-8:0	07:44:25	4
2867998	99	22	0,20	cyclictest	3363267	-21	systemd-run	10:53:59	7
2867993	99	22	19,3	cyclictest	3018041	-21	kworker/u16:1+flush-8:0	08:54:25	6
2867988	99	22	20,2	cyclictest	3412557	-21	kworker/u16:4+events_unbound	11:28:30	5
2867988	99	22	20,2	cyclictest	3385554	-21	kworker/u16:2+flush-8:0	11:09:25	5
2867988	99	22	20,2	cyclictest	3018041	-21	kworker/u16:1+flush-8:0	09:04:25	5
2867988	99	22	19,2	cyclictest	3129293	-21	kworker/u16:5+events_unbound	09:24:01	5
2867998	99	21	20,0	cyclictest	0	-21	swapper/7	07:54:28	7
2867988	99	21	19,2	cyclictest	3372964	-21	kworker/u16:0+events_unbound	11:50:41	5
2867988	99	21	19,2	cyclictest	2877283	-21	kworker/u16:0+events_unbound	07:24:25	5
2867988	99	21	19,2	cyclictest	2838574	-21	kworker/u16:1+events_unbound	07:39:28	5
2867988	99	21	18,2	cyclictest	3178347	-21	kworker/u16:0+flush-8:0	09:44:04	5
2867985	99	21	19,2	cyclictest	3385554	-21	kworker/u16:2+events_unbound	11:19:17	4
2867988	99	20	18,2	cyclictest	3412557	-21	kworker/u16:4+events_unbound	11:18:26	5
3264824	2	19	0,0	chrt	3264822	-21	ssh	10:13:24	7
2867998	99	19	0,17	cyclictest	1302	-21	snmpd	09:27:23	7
2867988	99	19	17,2	cyclictest	3583409	-21	kworker/u16:0+events_unbound	12:34:26	5
2867988	99	19	17,2	cyclictest	3372964	-21	kworker/u16:0+events_unbound	12:04:26	5
2867988	99	19	17,2	cyclictest	3129293	-21	kworker/u16:5+events_unbound	09:43:10	5
2867988	99	19	16,2	cyclictest	3129293	-21	kworker/u16:5+flush-8:0	10:19:05	5
2867985	99	19	16,2	cyclictest	3018041	-21	kworker/u16:1+flush-8:0	08:24:14	4
2867988	99	18	15,2	cyclictest	3453928	-21	kthreadcore	11:29:19	5
2867985	99	18	16,2	cyclictest	3251442	-21	kworker/u16:0+events_unbound	10:39:21	4
2867985	99	18	16,2	cyclictest	3251442	-21	kworker/u16:0+events_unbound	10:39:21	4
2867985	99	18	16,2	cyclictest	3085539	-21	kworker/u16:0+events_unbound	09:19:29	4
2867998	99	17	12,4	cyclictest	648	-21	systemd-journal	12:33:59	7
2867998	99	17	12,4	cyclictest	3314151	-21	latency_hist	10:33:59	7
2867993	99	17	14,2	cyclictest	3519482	-21	ntpq	11:54:24	6
2867988	99	17	15,2	cyclictest	3583993	-21	kworker/u16:1+events_unbound	12:25:09	5
2867988	99	17	15,2	cyclictest	3209088	-21	kworker/u16:2+events_unbound	09:50:10	5
2867988	99	17	15,2	cyclictest	3018041	-21	kworker/u16:1+events_unbound	08:54:19	5
2867988	99	17	14,2	cyclictest	3412557	-21	kworker/u16:4+flush-8:0	11:34:01	5
3580464	2	16	0,0	sleep7	3580467	-21	cut	12:19:22	7
2867998	99	16	0,14	cyclictest	0	-21	swapper/7	09:59:25	7
2867993	99	16	13,2	cyclictest	0	-21	swapper/6	10:39:12	6
2867993	99	16	13,2	cyclictest	0	-21	swapper/6	10:39:12	6
2867993	99	16	0,15	cyclictest	3455248	-21	grep	11:29:20	6
2867988	99	16	14,2	cyclictest	3526935	-21	kworker/u16:3+flush-8:0	12:14:24	5
2867988	99	16	14,2	cyclictest	3507859	-21	kworker/u16:2+flush-8:0	11:54:09	5
2867988	99	16	14,2	cyclictest	3228752	-21	kworker/u16:3+events_unbound	10:06:31	5
2867988	99	16	14,2	cyclictest	2939628	-21	kworker/u16:0+flush-8:0	07:44:18	5
2867988	99	16	14,2	cyclictest	2891163	-21	kworker/u16:2+flush-8:0	08:24:27	5
2867988	99	16	14,2	cyclictest	2891163	-21	kworker/u16:2+flush-8:0	08:14:15	5
2867985	99	16	0,15	cyclictest	3506823	-21	kthreadcore	11:49:20	4
2867985	99	16	0,15	cyclictest	3196100	-21	grep	09:44:19	4
2867998	99	15	10,3	cyclictest	3435786	-21	ssh	11:22:13	7
2867998	99	15	0,13	cyclictest	0	-21	swapper/7	11:49:22	7
2867998	99	15	0,13	cyclictest	0	-21	swapper/7	08:49:23	7
2867993	99	15	13,2	cyclictest	2949735	-21	kworker/u16:1+events_unbound	07:59:14	6
2867988	99	15	13,2	cyclictest	3313610	-21	kworker/u16:1+events_unbound	10:59:19	5
2867988	99	15	13,2	cyclictest	2794161	-21	kworker/u16:2+flush-8:0	07:09:17	5
2867988	99	15	12,2	cyclictest	3300470	-21	kworker/u16:2+events_unbound	10:49:00	5
2867998	99	14	9,4	cyclictest	1302	-21	snmpd	11:04:23	7
2867998	99	14	8,5	cyclictest	2877574	-21	latency_hist	07:14:00	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17