OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Sat Aug 01, 2026 00:45:30)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
808267	99	69	6,13	cyclictest	12	-21	ksoftirqd/0	22:50:01	0
808267	99	69	46,8	cyclictest	12	-21	ksoftirqd/0	22:10:09	0
808276	99	68	38,22	cyclictest	3467602	-21	systemd-journal	20:35:00	2
808276	99	68	10,53	cyclictest	1	-21	systemd	19:55:01	2
808267	99	68	3,29	cyclictest	13	1	rcu_preempt	23:21:50	0
808267	99	68	3,29	cyclictest	13	1	rcu_preempt	23:21:50	0
808276	99	67	19,42	cyclictest	275	-21	dbus-daemon	20:55:01	2
808267	99	67	27,22	cyclictest	894776	-21	ssh	00:00:00	0
808276	99	66	27,35	cyclictest	867704	-21	sh	23:02:26	2
808276	99	66	26,35	cyclictest	836813	-21	sh	21:57:11	2
808276	99	65	16,44	cyclictest	3467602	-21	systemd-journal	22:40:02	2
808276	99	65	14,46	cyclictest	885372	-21	systemd-run	23:40:00	2
808276	99	65	1,15	cyclictest	853639	-21	sshd	22:32:43	2
808276	99	64	4,54	cyclictest	1	-21	systemd	23:20:02	2
808276	99	64	2,14	cyclictest	810760	-21	sshd	19:56:24	2
808276	99	64	13,43	cyclictest	0	-21	swapper/2	23:50:00	2
808267	99	64	5,7	cyclictest	12	-21	ksoftirqd/0	21:27:18	0
808267	99	64	2,9	cyclictest	12	1	ksoftirqd/0	00:00:50	0
808267	99	63	12,24	cyclictest	13	1	rcu_preempt	23:27:54	0
808276	99	62	41,16	cyclictest	1	-21	systemd	21:25:01	2
808276	99	62	22,36	cyclictest	871119	-21	latency_hist	23:10:02	2
808276	99	62	18,39	cyclictest	913811	-21	latency_hist	00:40:02	2
808276	99	62	18,39	cyclictest	913811	-21	latency_hist	00:40:01	2
808276	99	62	1,13	cyclictest	364	-21	snmpd	21:05:39	2
808267	99	62	8,7	cyclictest	13	1	rcu_preempt	19:40:01	0
808267	99	62	4,51	cyclictest	3467602	-21	systemd-journal	23:20:00	0
808276	99	61	2,19	cyclictest	824190	-21	ssh	21:30:23	2
808276	99	61	2,13	cyclictest	809798	-21	munin-run	19:40:01	2
808276	99	61	19,37	cyclictest	842192	-21	sh	22:08:47	2
808276	99	61	1,16	cyclictest	811325	-21	munin-run	20:10:01	2
808267	99	61	25,5	cyclictest	0	-21	swapper/0	22:01:33	0
808267	99	61	18,10	cyclictest	12	-21	ksoftirqd/0	00:20:00	0
808267	99	61	18,10	cyclictest	12	-21	ksoftirqd/0	00:19:59	0
808276	99	60	17,9	cyclictest	847787	-21	sh	22:20:06	2
808276	99	60	1,20	cyclictest	864340	-21	cat	22:55:03	2
808276	99	60	1,20	cyclictest	864340	-21	cat	22:55:02	2
808276	99	60	1,15	cyclictest	910480	-21	sshd	00:32:56	2
808276	99	60	1,14	cyclictest	828511	-21	munin-run	21:40:00	2
808276	99	60	11,4	cyclictest	0	-21	swapper/2	21:51:35	2
808276	99	60	1,13	cyclictest	883030	-21	true	23:35:01	2
808276	99	60	1,13	cyclictest	883030	-21	true	23:35:01	2
808267	99	59	9,6	cyclictest	13	1	rcu_preempt	21:49:46	0
808276	99	58	45,8	cyclictest	800493	-21	kworker/2:1	19:20:00	2
808276	99	58	3,10	cyclictest	897186	-21	cron	00:05:00	2
808276	99	58	2,17	cyclictest	829085	-21	ssh	21:40:44	2
808276	99	58	2,15	cyclictest	831546	-21	ssh	21:45:52	2
808276	99	58	2,14	cyclictest	903631	-21	ssh	00:18:40	2
808276	99	58	2,14	cyclictest	903631	-21	ssh	00:18:39	2
808276	99	58	2,14	cyclictest	894774	-21	cron	00:00:00	2
808276	99	58	2,14	cyclictest	859119	-21	ssh	22:44:22	2
808276	99	58	2,14	cyclictest	859119	-21	ssh	22:44:22	2
808276	99	58	1,15	cyclictest	907033	-21	ssh	00:25:25	2
808276	99	58	1,15	cyclictest	907033	-21	ssh	00:25:24	2
808276	99	57	4,9	cyclictest	880666	-21	unin-run	23:30:01	2
808276	99	57	2,4	cyclictest	13	1	rcu_preempt	21:11:01	2
808276	99	57	23,28	cyclictest	906614	-21	latency_hist	00:25:02	2
808276	99	57	20,32	cyclictest	813347	-21	latency_hist	20:45:03	2
808272	99	57	16,18	cyclictest	23	-21	ksoftirqd/1	23:30:18	1
808272	99	57	16,18	cyclictest	23	-21	ksoftirqd/1	23:30:18	1
808267	99	57	27,13	cyclictest	907122	-21	ssh	00:25:35	0
808267	99	57	27,13	cyclictest	907122	-21	ssh	00:25:35	0
808267	99	57	24,9	cyclictest	12	-21	ksoftirqd/0	21:59:02	0
808267	99	57	11,23	cyclictest	0	-21	swapper/0	00:08:09	0
808276	99	56	8,9	cyclictest	890834	-21	sh	23:51:20	2
808276	99	56	46,6	cyclictest	850208	-21	kworker/2:2	22:46:48	2
808276	99	56	2,16	cyclictest	811640	-21	latency_hist	20:15:02	2
808276	99	56	2,14	cyclictest	843568	-21	ssh	22:11:20	2
808276	99	56	21,31	cyclictest	30	-21	ksoftirqd/2	22:01:02	2
808276	99	56	1,16	cyclictest	120	-21	kswapd0	23:25:01	2
808276	99	56	1,16	cyclictest	120	-21	kswapd0	23:25:00	2
808267	99	56	3,5	cyclictest	13	1	rcu_preempt	22:26:11	0
808267	99	56	29,23	cyclictest	12	-21	ksoftirqd/0	22:42:46	0
808267	99	56	29,23	cyclictest	12	-21	ksoftirqd/0	22:42:45	0
808267	99	56	1,7	cyclictest	12	1	ksoftirqd/0	00:23:32	0
808267	99	56	15,5	cyclictest	13	1	rcu_preempt	22:57:15	0
808267	99	56	10,9	cyclictest	13	1	rcu_preempt	21:32:44	0
808280	99	55	40,11	cyclictest	886538	-21	sshd	23:42:13	3
808276	99	55	4,7	cyclictest	847498	-21	modprobe	22:20:00	2
808276	99	55	3,15	cyclictest	899904	-21	ssh	00:10:27	2
808276	99	55	2,17	cyclictest	822765	-21	ssh	21:27:31	2
808276	99	55	2,16	cyclictest	814110	-21	wc	21:00:02	2
808276	99	55	1,17	cyclictest	851766	-21	ssh	22:28:48	2
808267	99	55	8,12	cyclictest	12	-21	ksoftirqd/0	22:55:01	0
808267	99	55	8,12	cyclictest	12	-21	ksoftirqd/0	22:55:00	0
808267	99	55	45,7	cyclictest	12	-21	ksoftirqd/0	23:32:40	0
808267	99	55	45,7	cyclictest	12	-21	ksoftirqd/0	23:32:40	0
808267	99	55	28,12	cyclictest	900365	-21	ssh	00:11:26	0
808267	99	55	27,25	cyclictest	12	-21	ksoftirqd/0	22:15:34	0
808267	99	55	25,14	cyclictest	849848	-21	ssh	22:24:58	0
808267	99	55	23,13	cyclictest	835880	-21	ssh	21:55:02	0
808267	99	55	20,30	cyclictest	814439	-21	diskmemload	23:45:03	0
808267	99	55	18,19	cyclictest	857026	-21	ssh	22:40:02	0
808280	99	54	6,45	cyclictest	814439	-21	diskmemload	00:15:03	3
808280	99	54	6,45	cyclictest	814439	-21	diskmemload	00:15:02	3
808276	99	54	8,8	cyclictest	812089	-21	sshd	20:21:43	2
808276	99	54	4,8	cyclictest	1	-21	systemd	20:30:01	2
808276	99	54	2,16	cyclictest	897442	-21	ssh	00:05:05	2
808276	99	54	2,15	cyclictest	871476	-21	ssh	23:10:18	2
808276	99	54	19,31	cyclictest	865211	-21	sh	22:57:10	2
808276	99	54	1,16	cyclictest	818263	-21	ssh	21:18:04	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33