OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Sat Dec 13, 2025 00:43:48)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
25744	2	90	77,9	sleep2	0	-21	swapper/2	19:08:23	2
25577	2	86	70,10	sleep0	0	-21	swapper/0	19:06:11	0
25565	2	84	71,9	sleep3	0	-21	swapper/3	19:06:03	3
25536	2	75	58,13	sleep1	0	-21	swapper/1	19:05:40	1
26842	2	59	0,1	sleep0	26845	-21	sort	21:25:20	0
26842	2	59	0,1	sleep0	26845	-21	sort	21:25:19	0
25957	99	27	17,6	cyclictest	225	-21	systemd-journal	21:25:01	1
25952	99	23	0,20	cyclictest	0	-21	swapper/0	21:20:16	0
25952	99	22	22,0	cyclictest	0	-21	swapper/0	20:16:43	0
25963	99	21	0,1	cyclictest	0	-21	swapper/3	20:32:22	3
25952	99	20	8,6	cyclictest	24175	-21	idleruntime-cro	22:29:59	0
25952	99	20	15,1	cyclictest	0	-21	swapper/0	20:10:14	0
25963	99	19	2,15	cyclictest	0	-21	swapper/3	21:25:15	3
25963	99	19	2,15	cyclictest	0	-21	swapper/3	21:25:15	3
25961	99	19	9,3	cyclictest	20473	-21	cat	00:40:00	2
25961	99	19	9,3	cyclictest	18432	-21	tr	22:15:17	2
25957	99	19	0,18	cyclictest	0	-21	swapper/1	00:20:23	1
25952	99	19	7,6	cyclictest	27952	-21	munin-run	23:45:00	0
25961	99	18	10,2	cyclictest	5523	-21	grep	21:50:13	2
25961	99	17	10,2	cyclictest	7493	-21	uniq	20:45:13	2
25961	99	16	10,2	cyclictest	29493	-21	cron	20:25:00	2
25952	99	16	4,7	cyclictest	18668	-21	awk	21:10:00	0
25963	99	15	11,3	cyclictest	18670	-21	sleep	21:09:59	3
25961	99	15	3,7	cyclictest	225	-21	systemd-journal	19:40:00	2
25957	99	15	11,3	cyclictest	1704	-21	ntpq	23:55:16	1
25952	99	15	1,7	cyclictest	16295	-21	munin-run	21:04:59	0
25952	99	15	0,1	cyclictest	0	-21	swapper/0	19:55:12	0
25963	99	14	10,2	cyclictest	16112	-21	grep	19:55:16	3
25957	99	14	10,3	cyclictest	6701	-21	grep	21:50:21	1
25957	99	14	10,2	cyclictest	26077	-21	tr	23:40:15	1
25952	99	14	9,3	cyclictest	17964	-21	cut	20:00:30	0
25952	99	14	2,6	cyclictest	22187	-21	munin-run	20:10:00	0
25952	99	14	10,3	cyclictest	5280	-21	perf	00:05:01	0
25952	99	14	10,2	cyclictest	9919	-21	sort	23:05:14	0
25963	99	13	9,3	cyclictest	30779	-21	wc	22:40:21	3
25963	99	13	9,3	cyclictest	10378	-21	uname	23:05:18	3
25963	99	13	9,2	cyclictest	30856	-21	tr	23:50:10	3
25963	99	13	11,2	cyclictest	1235	-21	ntpd	23:00:00	3
25963	99	13	10,2	cyclictest	8971	-21	tail	19:40:16	3
25963	99	13	10,2	cyclictest	29491	-21	cron	20:25:00	3
25961	99	13	9,3	cyclictest	9967	-21	basename	22:00:00	2
25961	99	13	9,3	cyclictest	24221	-21	sed	23:35:17	2
25961	99	13	12,1	cyclictest	0	-21	swapper/2	21:34:23	2
25961	99	13	10,2	cyclictest	6123	-21	grep	20:40:25	2
25961	99	13	10,2	cyclictest	31489	-21	grep	21:35:17	2
25961	99	13	10,2	cyclictest	29097	-21	latency_hist	22:39:59	2
25961	99	13	10,2	cyclictest	28120	-21	sh	21:30:01	2
25961	99	13	10,2	cyclictest	28120	-21	sh	21:30:00	2
25961	99	13	10,2	cyclictest	24644	-21	cat	20:15:00	2
25961	99	13	10,2	cyclictest	22203	-21	uname	23:30:17	2
25961	99	13	10,2	cyclictest	21351	-21	grep	22:20:27	2
25961	99	13	0,4	cyclictest	0	-21	swapper/2	19:45:14	2
25957	99	13	9,2	cyclictest	24542	-21	munin-plugin-st	20:15:00	1
25957	99	13	9,2	cyclictest	16397	-21	ls	21:05:01	1
25957	99	13	6,3	cyclictest	225	-21	systemd-journal	20:40:00	1
25957	99	13	10,2	cyclictest	28503	-21	grep	20:20:18	1
25957	99	13	10,2	cyclictest	20449	-21	grep	22:20:20	1
25957	99	13	10,2	cyclictest	14684	-21	cron	22:10:01	1
25952	99	13	9,3	cyclictest	479	-21	cat	21:40:00	0
25952	99	13	9,3	cyclictest	21117	-21	wc	21:15:00	0
25952	99	13	9,2	cyclictest	5128	-21	grep	19:30:26	0
25952	99	13	9,2	cyclictest	23729	-21	tr	23:35:15	0
25952	99	13	8,3	cyclictest	18461	-21	cron	23:25:00	0
25952	99	13	10,2	cyclictest	3739	-21	grep	20:35:24	0
25952	99	13	10,2	cyclictest	32425	-21	wc	20:30:13	0
25952	99	13	10,2	cyclictest	14649	-21	grep	19:50:24	0
25952	99	13	10,2	cyclictest	11337	-21	basename	20:55:00	0
25952	99	13	0,3	cyclictest	225	-21	systemd-journal	00:06:58	0
25952	99	13	0,12	cyclictest	18333	-21	grep	00:30:12	0
25952	99	13	0,12	cyclictest	14984	-21	grep	22:10:09	0
25963	99	12	9,2	cyclictest	950	-21	grep	22:45:21	3
25963	99	12	9,2	cyclictest	746	-21	basename	21:40:13	3
25963	99	12	9,2	cyclictest	6270	-21	grep	00:05:15	3
25963	99	12	9,2	cyclictest	3842	-21	grep	21:45:18	3
25963	99	12	9,2	cyclictest	30690	-21	tr	19:20:00	3
25963	99	12	9,2	cyclictest	29104	-21	grep	21:30:16	3
25963	99	12	9,2	cyclictest	28848	-21	sort	23:45:13	3
25963	99	12	9,2	cyclictest	19691	-21	head	21:10:16	3
25963	99	12	9,2	cyclictest	19229	-21	grep	23:25:15	3
25963	99	12	9,2	cyclictest	19229	-21	grep	23:25:15	3
25963	99	12	9,2	cyclictest	16667	-21	cut	23:20:17	3
25963	99	12	9,2	cyclictest	16011	-21	head	00:25:12	3
25963	99	12	9,2	cyclictest	12081	-21	tr	20:55:14	3
25963	99	12	9,2	cyclictest	10489	-21	grep	22:00:11	3
25963	99	12	8,3	cyclictest	6660	-21	sed	19:35:16	3
25961	99	12	9,2	cyclictest	9947	-21	grep	20:50:15	2
25961	99	12	9,2	cyclictest	5168	-21	sed	19:30:26	2
25961	99	12	9,2	cyclictest	3438	-21	grep	00:00:10	2
25961	99	12	9,2	cyclictest	31114	-21	grep	23:50:11	2
25961	99	12	9,2	cyclictest	31025	-21	tail	22:40:22	2
25961	99	12	9,2	cyclictest	309	-21	sort	20:30:15	2
25961	99	12	9,2	cyclictest	30417	-21	wc	23:50:01	2
25961	99	12	9,2	cyclictest	20347	-21	tr	20:05:27	2
25961	99	12	9,2	cyclictest	1920	-21	head	19:25:16	2
25961	99	12	9,2	cyclictest	15725	-21	grep	22:10:12	2
25961	99	12	9,2	cyclictest	11196	-21	grep	00:15:12	2
25961	99	12	8,3	cyclictest	31855	-21	cron	20:30:00	2
25961	99	12	8,3	cyclictest	17466	-21	tr	20:00:01	2
25957	99	12	9,2	cyclictest	9499	-21	grep	20:50:13	1
25957	99	12	9,2	cyclictest	9402	-21	wc	21:55:21	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17