OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Thu Jan 08, 2026 12:43:47)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
20088	2	92	67,9	sleep2	0	-21	swapper/2	07:06:36	2
20245	2	84	65,15	sleep0	0	-21	swapper/0	07:08:38	0
20228	2	82	69,9	sleep3	0	-21	swapper/3	07:08:26	3
20077	2	81	64,13	sleep1	0	-21	swapper/1	07:06:26	1
20444	99	28	7,21	cyclictest	0	-21	swapper/2	11:40:49	2
20444	99	23	0,0	cyclictest	0	-21	swapper/2	07:52:01	2
20439	99	22	0,16	cyclictest	0	-21	swapper/0	09:15:12	0
20439	99	22	0,16	cyclictest	0	-21	swapper/0	09:15:12	0
20448	99	20	18,1	cyclictest	28275	-21	munin-run	10:55:01	3
20448	99	19	14,3	cyclictest	39	-21	ksoftirqd/3	09:25:00	3
20444	99	19	1,15	cyclictest	1962	-21	apt-get	08:50:12	2
20444	99	19	0,18	cyclictest	0	-21	swapper/2	12:26:24	2
20448	99	17	13,1	cyclictest	39	-21	ksoftirqd/3	12:05:14	3
20448	99	17	10,6	cyclictest	39	-21	ksoftirqd/3	08:50:13	3
20439	99	17	4,6	cyclictest	14794	-21	apt-get	08:09:59	0
20448	99	16	14,1	cyclictest	29634	-21	apt-get	08:40:12	3
20439	99	16	6,3	cyclictest	273	-21	systemd-udevd	11:07:04	0
20448	99	15	9,2	cyclictest	7594	-21	cron	12:30:00	3
20448	99	15	4,11	cyclictest	39	-21	ksoftirqd/3	09:30:13	3
20448	99	15	13,1	cyclictest	23356	-21	munin-run	11:55:00	3
20444	99	15	5,10	cyclictest	0	-21	swapper/2	11:02:41	2
20441	99	15	0,15	cyclictest	0	-21	swapper/1	10:50:25	1
20448	99	14	11,2	cyclictest	27001	-21	wc	12:00:18	3
20448	99	14	10,3	cyclictest	20373	-21	wc	11:45:25	3
20448	99	14	10,3	cyclictest	14293	-21	tr	09:15:14	3
20448	99	14	10,3	cyclictest	14293	-21	tr	09:15:13	3
20448	99	14	0,13	cyclictest	1083	-21	wc	08:45:25	3
20448	99	14	0,12	cyclictest	0	-21	swapper/3	09:40:14	3
20444	99	14	14,0	cyclictest	0	-21	swapper/2	11:45:13	2
20444	99	14	11,2	cyclictest	1799	-21	sh	10:00:00	2
20441	99	14	9,4	cyclictest	32707	-21	tr	07:35:13	1
20441	99	14	10,3	cyclictest	26253	-21	grep	12:00:13	1
20439	99	14	9,3	cyclictest	28627	-21	cron	09:45:00	0
20439	99	14	5,8	cyclictest	16313	-21	munin-run	10:30:01	0
20439	99	14	5,5	cyclictest	225	-21	systemd-journal	09:00:01	0
20439	99	14	3,6	cyclictest	28147	-21	idleruntime-cro	12:05:00	0
20439	99	14	10,2	cyclictest	29216	-21	grep	09:45:13	0
20448	99	13	9,3	cyclictest	4185	-21	tail	07:40:25	3
20448	99	13	9,3	cyclictest	28438	-21	cut	07:25:16	3
20448	99	13	9,3	cyclictest	26170	-21	cat	09:40:00	3
20448	99	13	9,3	cyclictest	25083	-21	tr	08:30:15	3
20448	99	13	9,3	cyclictest	19988	-21	wc	10:35:19	3
20448	99	13	8,1	cyclictest	25888	-21	ls	10:50:00	3
20448	99	13	2,5	cyclictest	0	-21	swapper/3	08:35:12	3
20448	99	13	1,9	cyclictest	39	-21	ksoftirqd/3	11:30:25	3
20448	99	13	10,2	cyclictest	20940	-21	tr	07:10:13	3
20448	99	13	10,2	cyclictest	15973	-21	grep	08:10:14	3
20448	99	13	10,2	cyclictest	12240	-21	cut	09:10:16	3
20444	99	13	13,0	cyclictest	0	-21	swapper/2	12:10:20	2
20444	99	13	10,2	cyclictest	8827	-21	cron	09:05:00	2
20444	99	13	10,2	cyclictest	32756	-21	sed	08:45:15	2
20444	99	13	10,2	cyclictest	32194	-21	grep	09:50:16	2
20444	99	13	10,2	cyclictest	29087	-21	cat	11:00:00	2
20444	99	13	10,2	cyclictest	27581	-21	cat	09:40:20	2
20444	99	13	10,2	cyclictest	24944	-21	cat	09:35:19	2
20441	99	13	9,3	cyclictest	30752	-21	tr	08:40:20	1
20441	99	13	9,2	cyclictest	10263	-21	grep	07:55:15	1
20441	99	13	1,2	cyclictest	14	1	rcu_preempt	08:50:19	1
20441	99	13	10,2	cyclictest	7650	-21	grep	11:20:16	1
20441	99	13	10,2	cyclictest	7373	-21	grep	09:00:16	1
20441	99	13	10,2	cyclictest	533	-21	ls	11:05:23	1
20441	99	13	10,2	cyclictest	31498	-21	tail	12:10:17	1
20441	99	13	10,2	cyclictest	28874	-21	sort	09:45:00	1
20441	99	13	10,2	cyclictest	24663	-21	grep	07:15:26	1
20441	99	13	10,2	cyclictest	15673	-21	sed	11:35:26	1
20441	99	13	10,2	cyclictest	12267	-21	sed	09:10:16	1
20439	99	13	9,3	cyclictest	7104	-21	grep	12:25:26	0
20439	99	13	9,3	cyclictest	5221	-21	cron	12:25:00	0
20439	99	13	9,3	cyclictest	25165	-21	sleep	07:20:00	0
20439	99	13	9,2	cyclictest	25972	-21	cut	08:30:21	0
20439	99	13	9,2	cyclictest	17191	-21	sed	11:40:16	0
20439	99	13	6,6	cyclictest	7037	-21	munin-run	07:50:00	0
20439	99	13	1,6	cyclictest	1826	-21	tr	09:59:59	0
20439	99	13	10,2	cyclictest	962	-21	tr	12:15:13	0
20439	99	13	10,2	cyclictest	5004	-21	grep	11:15:15	0
20439	99	13	10,2	cyclictest	31140	-21	wc	08:40:26	0
20439	99	13	0,11	cyclictest	2223	-21	tr	08:50:13	0
20448	99	12	9,2	cyclictest	9898	-21	grep	07:55:13	3
20448	99	12	9,2	cyclictest	7369	-21	cut	09:00:16	3
20448	99	12	9,2	cyclictest	6651	-21	rm	11:20:01	3
20448	99	12	9,2	cyclictest	6014	-21	grep	07:45:20	3
20448	99	12	9,2	cyclictest	32115	-21	cut	12:10:22	3
20448	99	12	9,2	cyclictest	2728	-21	sort	10:00:12	3
20448	99	12	9,2	cyclictest	25935	-21	df	07:20:15	3
20448	99	12	9,2	cyclictest	23008	-21	sed	10:40:24	3
20448	99	12	9,2	cyclictest	21802	-21	grep	08:20:25	3
20448	99	12	9,2	cyclictest	18391	-21	grep	08:15:17	3
20448	99	12	9,2	cyclictest	15643	-21	grep	11:35:26	3
20448	99	12	9,2	cyclictest	12106	-21	grep	10:20:13	3
20448	99	12	8,3	cyclictest	10891	-21	uniq	12:35:17	3
20448	99	12	7,4	cyclictest	30672	-21	ls	11:00:21	3
20448	99	12	0,11	cyclictest	16706	-21	tr	11:40:13	3
20444	99	12	9,2	cyclictest	5681	-21	sed	07:45:17	2
20444	99	12	9,2	cyclictest	32302	-21	apt	07:35:00	2
20444	99	12	9,2	cyclictest	2797	-21	grep	07:40:13	2
20444	99	12	9,2	cyclictest	27634	-21	grep	12:00:25	2
20444	99	12	9,2	cyclictest	21130	-21	sleep	10:40:00	2
20444	99	12	9,2	cyclictest	13271	-21	grep	11:30:24	2
20444	99	12	9,2	cyclictest	12342	-21	tr	08:00:17	2
20444	99	12	9,2	cyclictest	10396	-21	ls	10:15:18	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17