OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Thu Jan 01, 2026 12:43:48)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
22259	2	94	80,9	sleep0	0	-21	swapper/0	07:06:55	0
22495	2	92	66,8	sleep3	0	-21	swapper/3	07:09:58	3
22188	2	92	79,9	sleep2	0	-21	swapper/2	07:06:00	2
22451	2	82	72,6	sleep1	0	-21	swapper/1	07:09:25	1
2762	2	65	0,0	sleep0	0	-21	swapper/0	08:40:25	0
22591	99	25	24,1	cyclictest	0	-21	swapper/2	07:53:40	2
22588	99	23	2,14	cyclictest	0	-21	swapper/1	10:45:14	1
22588	99	22	4,16	cyclictest	9398	-21	awk	10:05:00	1
22588	99	22	1,3	cyclictest	0	-21	swapper/1	07:25:13	1
22588	99	21	1,17	cyclictest	0	-21	swapper/1	07:35:13	1
22586	99	20	19,1	cyclictest	0	-21	swapper/0	07:51:58	0
22586	99	17	6,6	cyclictest	25143	-21	perf	11:44:59	0
22586	99	17	4,7	cyclictest	9395	-21	munin-run	10:05:00	0
22594	99	16	11,1	cyclictest	39	-21	ksoftirqd/3	10:55:26	3
22588	99	16	9,6	cyclictest	1732	-21	grep	12:00:29	1
22586	99	16	11,3	cyclictest	8583	-21	ls	07:45:26	0
22594	99	15	1,2	cyclictest	14	1	rcu_preempt	11:10:14	3
22594	99	15	11,3	cyclictest	38	1	rcuc/3	11:35:00	3
22594	99	15	0,1	cyclictest	0	-21	swapper/3	11:45:14	3
22591	99	15	12,3	cyclictest	32	-21	ksoftirqd/2	09:50:15	2
22594	99	14	11,2	cyclictest	21358	-21	munin-run	08:15:00	3
22594	99	14	10,3	cyclictest	21201	-21	grep	09:20:16	3
22591	99	14	9,3	cyclictest	12236	-21	cat	12:25:00	2
22591	99	14	4,6	cyclictest	0	-21	swapper/2	09:55:12	2
22591	99	14	11,2	cyclictest	23089	-21	tr	07:10:14	2
22591	99	14	10,3	cyclictest	9391	-21	munin-plugin-st	10:05:00	2
22588	99	14	9,3	cyclictest	15196	-21	grep	08:00:16	1
22586	99	14	7,6	cyclictest	15387	-21	cron	09:10:00	0
22586	99	14	10,2	cyclictest	26499	-21	grep	08:25:16	0
22594	99	13	9,3	cyclictest	18351	-21	grep	12:35:22	3
22594	99	13	9,3	cyclictest	18351	-21	grep	12:35:21	3
22594	99	13	10,2	cyclictest	7172	-21	grep	07:45:13	3
22594	99	13	10,2	cyclictest	6829	-21	sed	08:50:17	3
22594	99	13	10,2	cyclictest	31393	-21	ls	09:40:22	3
22594	99	13	10,2	cyclictest	27117	-21	grep	10:40:19	3
22594	99	13	10,2	cyclictest	27117	-21	grep	10:40:18	3
22594	99	13	10,2	cyclictest	14997	-21	head	09:05:26	3
22594	99	13	10,2	cyclictest	14019	-21	head	11:20:18	3
22591	99	13	9,3	cyclictest	8812	-21	ls	12:15:27	2
22591	99	13	9,3	cyclictest	18358	-21	tr	09:15:12	2
22591	99	13	4,4	cyclictest	225	-21	systemd-journal	07:55:01	2
22591	99	13	10,2	cyclictest	15432	-21	wc	09:10:00	2
22591	99	13	10,2	cyclictest	11886	-21	cat	10:10:01	2
22588	99	13	9,3	cyclictest	7139	-21	wc	08:50:20	1
22588	99	13	9,3	cyclictest	29610	-21	sort	08:30:17	1
22588	99	13	9,3	cyclictest	14908	-21	grep	12:30:14	1
22588	99	13	9,3	cyclictest	10983	-21	cat	07:50:25	1
22588	99	13	9,2	cyclictest	11135	-21	tr	09:00:16	1
22588	99	13	11,1	cyclictest	0	-21	swapper/1	11:35:15	1
22588	99	13	10,2	cyclictest	25	-21	ksoftirqd/1	11:10:23	1
22588	99	13	10,2	cyclictest	17615	-21	tail	08:05:15	1
22588	99	13	10,2	cyclictest	17115	-21	tr	09:10:22	1
22588	99	13	10,2	cyclictest	10202	-21	tr	12:20:15	1
22586	99	13	8,3	cyclictest	9209	-21	cut	11:10:18	0
22586	99	13	10,2	cyclictest	833	2	sleep0	10:55:00	0
22586	99	13	10,2	cyclictest	7496	-21	tail	11:05:27	0
22586	99	13	10,2	cyclictest	30016	-21	wc	08:30:22	0
22586	99	13	10,2	cyclictest	24036	-21	sed	07:10:22	0
22586	99	13	10,2	cyclictest	19394	-21	basename	12:40:00	0
22586	99	13	10,2	cyclictest	19394	-21	basename	12:40:00	0
22586	99	13	0,0	cyclictest	0	-21	swapper/0	09:50:14	0
22594	99	12	9,2	cyclictest	6964	-21	cron	12:15:01	3
22594	99	12	9,2	cyclictest	5663	-21	cut	09:55:16	3
22594	99	12	9,2	cyclictest	4101	-21	tune2fs	11:00:18	3
22594	99	12	9,2	cyclictest	322	-21	grep	09:45:13	3
22594	99	12	9,2	cyclictest	31559	-21	ls	11:55:26	3
22594	99	12	9,2	cyclictest	30706	-21	head	07:25:18	3
22594	99	12	9,2	cyclictest	29284	-21	sed	07:20:26	3
22594	99	12	9,2	cyclictest	2029	-21	cron	07:34:59	3
22594	99	12	9,2	cyclictest	1433	-21	grep	08:40:13	3
22594	99	12	9,2	cyclictest	13301	-21	ls	10:10:21	3
22594	99	12	8,3	cyclictest	26414	-21	ls	07:15:22	3
22594	99	12	8,3	cyclictest	25674	-21	sed	10:35:26	3
22594	99	12	8,3	cyclictest	21078	-21	grep	11:35:17	3
22594	99	12	8,3	cyclictest	17280	-21	grep	11:25:27	3
22594	99	12	8,3	cyclictest	12628	-21	sed	09:00:29	3
22594	99	12	7,3	cyclictest	26316	-21	wc	08:25:01	3
22594	99	12	3,1	cyclictest	14	1	rcu_preempt	12:05:12	3
22591	99	12	9,2	cyclictest	9188	-21	turbostat.cron	07:49:59	2
22591	99	12	9,2	cyclictest	7524	-21	cat	08:50:26	2
22591	99	12	9,2	cyclictest	3914	-21	grep	07:35:27	2
22591	99	12	9,2	cyclictest	32104	-21	grep	08:35:19	2
22591	99	12	9,2	cyclictest	31745	-21	grep	11:55:27	2
22591	99	12	9,2	cyclictest	31375	-21	grep	10:50:15	2
22591	99	12	9,2	cyclictest	27995	-21	grep	10:40:29	2
22591	99	12	9,2	cyclictest	27995	-21	grep	10:40:28	2
22591	99	12	9,2	cyclictest	24057	-21	cut	10:35:13	2
22591	99	12	9,2	cyclictest	23054	-21	grep	11:40:15	2
22591	99	12	9,2	cyclictest	21843	-21	grep	10:30:16	2
22591	99	12	9,2	cyclictest	18654	-21	sort	11:30:16	2
22591	99	12	9,2	cyclictest	17959	-21	cat	08:05:19	2
22591	99	12	3,7	cyclictest	31573	-21	python	09:40:25	2
22591	99	12	11,1	cyclictest	0	-21	swapper/2	11:05:23	2
22588	99	12	9,2	cyclictest	9344	-21	cron	12:20:01	1
22588	99	12	9,2	cyclictest	5605	-21	grep	12:10:19	1
22588	99	12	9,2	cyclictest	353	-21	tr	09:45:13	1
22588	99	12	9,2	cyclictest	28489	-21	grep	09:35:17	1
22588	99	12	9,2	cyclictest	28215	-21	cut	07:20:16	1
22588	99	12	9,2	cyclictest	28057	-21	tr	11:50:17	1
22588	99	12	9,2	cyclictest	2587	-21	wc	08:40:22	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17