OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Fri Mar 13, 2026 00:45:28)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2078350	99	108	4,9	cyclictest	13	1	rcu_preempt	19:35:01	1
2078350	99	87	29,19	cyclictest	2147465	-21	sh	23:30:06	1
2078350	99	85	29,21	cyclictest	2105189	-21	sh	21:55:51	1
2078350	99	79	30,16	cyclictest	2097186	-21	ssh	21:38:11	1
2078350	99	79	29,20	cyclictest	2177173	-21	sh	00:36:37	1
2078350	99	78	29,13	cyclictest	2111243	-21	sh	22:09:36	1
2078350	99	77	30,14	cyclictest	2100489	-21	ssh	21:45:08	1
2078347	99	77	27,20	cyclictest	2169656	-21	ssh	00:20:00	0
2078347	99	77	23,37	cyclictest	2100607	-21	ssh	21:45:24	0
2078350	99	73	24,13	cyclictest	2135018	-21	ssh	23:02:40	1
2078350	99	72	26,13	cyclictest	2136691	-21	diskmemload	23:06:16	1
2078350	99	70	26,14	cyclictest	2144395	-21	ssh	23:23:42	1
2078350	99	70	26,14	cyclictest	2144395	-21	ssh	23:23:41	1
2078347	99	70	24,13	cyclictest	2156230	-21	latency_hist	23:50:02	0
2078347	99	68	29,11	cyclictest	1	-21	systemd	21:18:21	0
2078350	99	66	28,6	cyclictest	2140373	-21	ssh	23:14:59	1
2078350	99	66	28,6	cyclictest	2140373	-21	ssh	23:14:58	1
2078347	99	66	27,33	cyclictest	2118314	-21	latency_hist	22:25:03	0
2078347	99	66	16,46	cyclictest	2149478	-21	systemd-run	23:35:01	0
2078347	99	65	54,6	cyclictest	2140388	-21	cron	23:15:01	0
2078347	99	65	54,6	cyclictest	2140388	-21	cron	23:15:01	0
2078350	99	64	19,11	cyclictest	0	-21	swapper/1	22:26:09	1
2078347	99	63	25,34	cyclictest	0	-21	swapper/0	21:25:00	0
2078347	99	63	16,42	cyclictest	2098217	-21	cat	21:40:03	0
2078347	99	62	28,31	cyclictest	2136659	-21	ssh	23:06:11	0
2078347	99	62	19,10	cyclictest	2102928	-21	ssh	21:50:44	0
2078347	99	61	20,38	cyclictest	2160617	-21	modprobe	00:00:00	0
2078347	99	61	19,36	cyclictest	309	-21	in:imuxsock	19:20:00	0
2078347	99	61	19,36	cyclictest	2093682	-21	latency_hist	21:30:03	0
2078350	99	60	46,11	cyclictest	2155762	-21	ssh	23:49:00	1
2078350	99	60	39,17	cyclictest	2129056	-21	sh	22:49:29	1
2078350	99	60	35,21	cyclictest	2130417	-21	ssh	22:52:20	1
2078347	99	60	28,28	cyclictest	168	50	irq/134-8000000	21:40:19	0
2078347	99	60	23,11	cyclictest	2086903	-21	latency_hist	21:15:02	0
2078350	99	59	44,12	cyclictest	2175816	-21	ssh	00:33:49	1
2078347	99	59	42,12	cyclictest	275	-21	dbus-daemon	20:40:01	0
2078347	99	59	42,12	cyclictest	275	-21	dbus-daemon	20:40:00	0
2078347	99	59	20,34	cyclictest	168	50	irq/134-8000000	22:55:00	0
2078347	99	59	18,37	cyclictest	2106943	-21	munin-run	22:00:02	0
2078347	99	59	17,39	cyclictest	2129003	-21	ssh	22:49:23	0
2078356	99	58	14,13	cyclictest	2084530	-21	diskmemload	23:42:44	2
2078350	99	58	37,15	cyclictest	2152510	-21	rm	23:41:36	1
2078350	99	58	32,21	cyclictest	2095164	-21	ssh	21:33:36	1
2078350	99	58	29,24	cyclictest	2141195	-21	ssh	23:16:22	1
2078350	99	58	19,13	cyclictest	150	-21	pr/ttyS2	23:36:14	1
2078350	99	58	19,13	cyclictest	150	-21	pr/ttyS2	23:36:14	1
2078350	99	58	0,15	cyclictest	2173365	-21	sh	00:28:17	1
2078347	99	58	29,25	cyclictest	2170052	-21	ssh	00:20:37	0
2078347	99	58	21,4	cyclictest	2084528	-21	latency_hist	21:10:01	0
2078347	99	58	17,9	cyclictest	2162505	-21	ssh	00:04:05	0
2078347	99	58	17,9	cyclictest	2162505	-21	ssh	00:04:05	0
2078356	99	57	47,6	cyclictest	2098031	-21	kworker/2:1	21:49:59	2
2078350	99	57	44,10	cyclictest	2108942	-21	ssh	22:04:35	1
2078350	99	57	35,18	cyclictest	2082448	-21	modprobe	20:29:59	1
2078347	99	57	17,36	cyclictest	0	-21	swapper/0	23:55:02	0
2078347	99	57	17,13	cyclictest	2122498	-21	unin-run	22:35:02	0
2078350	99	56	43,10	cyclictest	2087176	-21	ssh	21:15:24	1
2078350	99	56	41,12	cyclictest	2121032	-21	ssh	22:31:12	1
2078350	99	56	39,13	cyclictest	1	-21	systemd	19:15:00	1
2078350	99	56	37,15	cyclictest	2159108	-21	ssh	23:56:21	1
2078350	99	56	36,15	cyclictest	2078849	-21	unin-run	19:20:00	1
2078350	99	56	34,17	cyclictest	1	-21	systemd	19:30:01	1
2078350	99	56	32,19	cyclictest	2157488	-21	ssh	23:52:45	1
2078350	99	56	31,20	cyclictest	2099731	-21	ssh	21:43:53	1
2078350	99	56	31,20	cyclictest	2083628	-21	latency_hist	20:50:02	1
2078350	99	56	30,21	cyclictest	2081669	-21	munin-run	20:15:00	1
2078347	99	56	27,25	cyclictest	2084530	-21	diskmemload	23:45:01	0
2078347	99	56	26,14	cyclictest	2150588	-21	ssh	23:37:21	0
2078347	99	56	26,14	cyclictest	2150588	-21	ssh	23:37:21	0
2078347	99	56	2,15	cyclictest	2178561	-21	grep	00:40:01	0
2078347	99	56	18,16	cyclictest	168	50	irq/134-8000000	23:17:53	0
2078350	99	55	42,10	cyclictest	2085520	-21	ssh	21:11:48	1
2078350	99	55	41,12	cyclictest	2112449	-21	ssh	22:12:09	1
2078350	99	55	38,13	cyclictest	2167925	-21	ssh	00:15:55	1
2078350	99	55	37,14	cyclictest	2166518	-21	ssh	00:12:59	1
2078350	99	55	37,14	cyclictest	2103301	-21	ssh	21:51:37	1
2078350	99	55	36,15	cyclictest	2161726	-21	ssh	00:02:05	1
2078350	99	55	36,15	cyclictest	2161726	-21	ssh	00:02:04	1
2078350	99	55	33,17	cyclictest	2126118	-21	ssh	22:42:46	1
2078350	99	55	32,18	cyclictest	2114307	-21	ssh	22:16:08	1
2078350	99	55	30,21	cyclictest	2124941	-21	latency_hist	22:40:02	1
2078347	99	55	29,22	cyclictest	2114720	-21	ssh	22:17:16	0
2078347	99	55	27,3	cyclictest	2123206	-21	ssh	22:36:14	0
2078356	99	54	46,5	cyclictest	2129569	-21	kworker/2:0	23:03:29	2
2078350	99	54	41,10	cyclictest	364	-21	snmpd	19:47:38	1
2078350	99	54	39,12	cyclictest	2089388	-21	ssh	21:20:17	1
2078350	99	54	38,13	cyclictest	2165084	-21	ssh	00:10:00	1
2078350	99	54	38,12	cyclictest	2145629	-21	ssh	23:26:03	1
2078350	99	54	36,16	cyclictest	2116722	-21	ssh	22:21:37	1
2078350	99	54	34,16	cyclictest	2091817	-21	ssh	21:25:51	1
2078350	99	54	31,18	cyclictest	2131762	-21	ssh	22:55:09	1
2078350	99	54	29,22	cyclictest	2170390	-21	ssh	00:21:25	1
2078347	99	54	26,13	cyclictest	2173684	-21	ssh	00:29:03	0
2078347	99	54	26,12	cyclictest	2133245	-21	ssh	22:59:00	0
2078347	99	54	20,30	cyclictest	168	50	irq/134-8000000	00:11:29	0
2078347	99	54	19,31	cyclictest	168	50	irq/134-8000000	23:28:50	0
2078347	99	54	18,32	cyclictest	168	50	irq/134-8000000	22:12:26	0
2078347	99	54	18,31	cyclictest	168	50	irq/134-8000000	22:41:12	0
2078347	99	54	17,8	cyclictest	1	-21	systemd	19:37:20	0
2078347	99	54	17,8	cyclictest	1	-21	systemd	19:37:19	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33