OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Fri Dec 19, 2025 12:43:47)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
24970	2	102	89,9	sleep2	0	-21	swapper/2	07:07:51	2
25020	2	89	79,6	sleep0	0	-21	swapper/0	07:08:30	0
25045	2	85	57,10	sleep3	0	-21	swapper/3	07:08:49	3
25006	2	85	67,14	sleep1	0	-21	swapper/1	07:08:19	1
25230	99	22	0,15	cyclictest	0	-21	swapper/1	07:10:14	1
25230	99	21	1,17	cyclictest	8796	-21	/usr/sbin/munin	12:10:27	1
25230	99	20	7,8	cyclictest	381	50	irq/131-snd_hda	11:55:14	1
25225	99	20	6,8	cyclictest	16902	-21	idleruntime-cro	11:20:00	0
25233	99	18	16,1	cyclictest	15967	-21	apache_processe	12:25:11	2
25230	99	17	5,3	cyclictest	14	1	rcu_preempt	07:20:00	1
25230	99	16	14,1	cyclictest	0	-21	swapper/1	10:20:02	1
25230	99	16	0,1	cyclictest	0	-21	swapper/1	08:05:14	1
25236	99	15	7,1	cyclictest	14	1	rcu_preempt	07:20:00	3
25236	99	15	12,2	cyclictest	21805	-21	idleruntime-cro	08:10:00	3
25230	99	15	11,3	cyclictest	2632	-21	cron	10:50:00	1
25230	99	15	11,3	cyclictest	11242	-21	sed	11:05:20	1
25225	99	15	9,4	cyclictest	18738	-21	ls	11:20:28	0
25225	99	15	14,1	cyclictest	0	-21	swapper/0	09:50:12	0
25225	99	15	11,3	cyclictest	32445	-21	ls	09:35:26	0
25225	99	15	11,2	cyclictest	29063	-21	wc	10:35:25	0
25225	99	15	11,2	cyclictest	29063	-21	wc	10:35:25	0
25236	99	14	2,11	cyclictest	19598	-21	apt-get	10:20:02	3
25236	99	14	11,2	cyclictest	6321	-21	cron	08:45:00	3
25236	99	14	11,2	cyclictest	6321	-21	cron	08:45:00	3
25230	99	14	9,3	cyclictest	4281	-21	wc	09:45:20	1
25230	99	14	2,7	cyclictest	381	50	irq/131-snd_hda	12:35:12	1
25230	99	14	0,12	cyclictest	32219	-21	munin-run	10:45:00	1
25225	99	14	13,1	cyclictest	0	-21	swapper/0	10:12:09	0
25225	99	14	11,2	cyclictest	18242	-21	cut	10:15:17	0
25225	99	14	11,2	cyclictest	15214	-21	grep	12:20:26	0
25225	99	14	10,3	cyclictest	31810	-21	grep	10:40:27	0
25225	99	14	10,2	cyclictest	24702	-21	tr	08:15:12	0
25225	99	14	10,2	cyclictest	23110	-21	wc	08:10:21	0
25236	99	13	1,5	cyclictest	0	-21	swapper/3	08:50:14	3
25236	99	13	10,2	cyclictest	30751	-21	cat	09:35:00	3
25236	99	13	10,2	cyclictest	18249	-21	cut	10:15:17	3
25236	99	13	10,2	cyclictest	15419	-21	sed	09:00:26	3
25236	99	13	10,2	cyclictest	1053	-21	grep	08:30:25	3
25233	99	13	13,0	cyclictest	0	-21	swapper/2	12:30:13	2
25233	99	13	12,1	cyclictest	0	-21	swapper/2	10:45:14	2
25233	99	13	10,2	cyclictest	32618	-21	sleep	07:24:59	2
25233	99	13	10,2	cyclictest	27589	-21	uniq	11:40:17	2
25233	99	13	10,2	cyclictest	27231	-21	grep	09:25:19	2
25233	99	13	10,2	cyclictest	25087	-21	grep	09:20:26	2
25233	99	13	10,2	cyclictest	23241	-21	wc	09:20:01	2
25233	99	13	10,2	cyclictest	11033	-21	cat	11:05:18	2
25230	99	13	9,2	cyclictest	10430	-21	grep	10:00:12	1
25230	99	13	6,4	cyclictest	0	-21	swapper/1	08:30:16	1
25230	99	13	12,1	cyclictest	0	-21	swapper/1	07:35:26	1
25230	99	13	11,1	cyclictest	2838	-21	apt-get	07:30:12	1
25230	99	13	10,2	cyclictest	9925	-21	cat	11:05:01	1
25230	99	13	10,2	cyclictest	921	-21	grep	11:50:26	1
25230	99	13	10,2	cyclictest	7322	-21	head	08:45:16	1
25230	99	13	10,2	cyclictest	31097	-21	tr	09:35:15	1
25230	99	13	10,2	cyclictest	28207	-21	tail	10:35:16	1
25230	99	13	10,2	cyclictest	28207	-21	tail	10:35:16	1
25225	99	13	9,3	cyclictest	16224	-21	tr	12:25:13	0
25225	99	13	9,3	cyclictest	12300	-21	cat	11:10:01	0
25225	99	13	6,6	cyclictest	13296	-21	gltestperf	07:50:17	0
25225	99	13	11,1	cyclictest	9586	-21	tune2fs	08:50:17	0
25225	99	13	10,2	cyclictest	24465	-21	wc	09:20:20	0
25225	99	13	10,2	cyclictest	14520	-21	cron	11:15:00	0
25225	99	13	10,2	cyclictest	1380	-21	uname	10:45:20	0
25236	99	12	9,2	cyclictest	7640	-21	tr	07:40:15	3
25236	99	12	9,2	cyclictest	735	-21	grep	07:25:15	3
25236	99	12	9,2	cyclictest	4762	-21	cat	09:45:26	3
25236	99	12	9,2	cyclictest	3453	-21	grep	08:35:27	3
25236	99	12	9,2	cyclictest	32191	-21	grep	09:35:23	3
25236	99	12	9,2	cyclictest	32122	-21	grep	07:20:24	3
25236	99	12	9,2	cyclictest	27847	-21	sleep	07:15:00	3
25236	99	12	9,2	cyclictest	25327	-21	grep	10:30:14	3
25236	99	12	9,2	cyclictest	23334	-21	wc	08:10:22	3
25236	99	12	9,2	cyclictest	22224	-21	grep	11:30:16	3
25236	99	12	9,2	cyclictest	19426	-21	cron	08:05:00	3
25236	99	12	9,2	cyclictest	10244	-21	tr	07:45:14	3
25233	99	12	9,2	cyclictest	4940	-21	latency_hist	07:35:00	2
25233	99	12	9,2	cyclictest	4812	-21	grep	12:00:17	2
25233	99	12	9,2	cyclictest	23018	-21	tr	08:10:20	2
25233	99	12	9,2	cyclictest	21912	-21	ls	12:35:21	2
25233	99	12	8,3	cyclictest	26618	-21	cron	08:20:00	2
25233	99	12	4,3	cyclictest	225	-21	systemd-journal	07:20:00	2
25233	99	12	1,9	cyclictest	0	-21	swapper/2	10:00:00	2
25233	99	12	10,1	cyclictest	30978	-21	cut	09:35:15	2
25233	99	12	10,1	cyclictest	28298	2	sleep2	08:20:23	2
25233	99	12	10,1	cyclictest	17331	-21	grep	09:05:21	2
25233	99	12	10,1	cyclictest	15570	-21	cut	11:15:16	2
25233	99	12	0,11	cyclictest	18418	-21	grep	11:20:23	2
25230	99	12	9,2	cyclictest	31258	-21	sed	07:20:15	1
25230	99	12	9,2	cyclictest	30042	-21	cut	08:25:16	1
25230	99	12	9,2	cyclictest	19430	-21	cron	08:05:00	1
25230	99	12	9,2	cyclictest	18518	-21	wc	11:20:24	1
25230	99	12	9,2	cyclictest	14000	-21	tr	09:00:15	1
25230	99	12	9,2	cyclictest	13433	-21	cut	10:05:16	1
25230	99	12	8,3	cyclictest	10224	-21	wc	08:50:23	1
25230	99	12	8,2	cyclictest	15179	-21	grep	10:10:13	1
25230	99	12	7,2	cyclictest	2248	-21	taskset	07:26:04	1
25230	99	12	4,3	cyclictest	9978	-21	apt-get	07:45:13	1
25230	99	12	1,5	cyclictest	225	-21	systemd-journal	11:29:10	1
25230	99	12	0,11	cyclictest	0	-21	swapper/1	10:25:38	1
25225	99	12	9,2	cyclictest	7863	-21	tr	07:40:16	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17