OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Fri Dec 26, 2025 00:43:47)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
20922	2	93	80,9	sleep2	0	-21	swapper/2	19:05:59	2
21057	2	89	79,6	sleep0	0	-21	swapper/0	19:07:41	0
20985	2	80	66,10	sleep3	0	-21	swapper/3	19:06:48	3
21171	2	79	58,17	sleep1	0	-21	swapper/1	19:09:11	1
21324	99	23	23,0	cyclictest	0	-21	swapper/2	20:35:16	2
21321	99	21	19,1	cyclictest	26616	-21	wget	23:50:18	1
21318	99	19	0,1	cyclictest	0	-21	swapper/0	20:20:13	0
21328	99	18	1,1	cyclictest	0	-21	swapper/3	21:00:14	3
21328	99	16	5,9	cyclictest	0	-21	swapper/3	22:10:13	3
21328	99	16	3,9	cyclictest	27661	-21	apt-get	22:45:15	3
21321	99	16	2,3	cyclictest	0	-21	swapper/1	21:35:13	1
21328	99	15	11,3	cyclictest	3367	-21	wc	23:00:20	3
21324	99	15	0,3	cyclictest	0	-21	swapper/2	00:05:18	2
21318	99	15	5,4	cyclictest	21269	-21	awk	21:25:00	0
21318	99	15	1,7	cyclictest	18863	-21	munin-run	23:35:00	0
21328	99	14	3,6	cyclictest	225	-21	systemd-journal	21:54:55	3
21328	99	14	10,3	cyclictest	23857	-21	cat	23:45:00	3
21324	99	14	13,1	cyclictest	0	-21	swapper/2	19:45:25	2
21324	99	14	13,1	cyclictest	0	-21	swapper/2	19:15:14	2
21321	99	14	9,3	cyclictest	3186	-21	cut	20:45:21	1
21321	99	14	0,13	cyclictest	1913	-21	tail	21:50:18	1
21321	99	14	0,12	cyclictest	29019	-21	grep	19:25:15	1
21318	99	14	7,6	cyclictest	15554	-21	idleruntime-cro	20:05:00	0
21318	99	14	12,1	cyclictest	22657	-21	cron	20:20:00	0
21318	99	14	12,1	cyclictest	22657	-21	cron	20:20:00	0
21318	99	14	0,8	cyclictest	6882	-21	perf	23:09:59	0
21328	99	13	9,3	cyclictest	18842	-21	cron	23:35:00	3
21328	99	13	9,2	cyclictest	638	-21	cut	22:55:19	3
21328	99	13	1,6	cyclictest	0	-21	swapper/3	23:10:15	3
21328	99	13	10,2	cyclictest	5904	-21	grep	00:10:29	3
21328	99	13	10,2	cyclictest	5904	-21	grep	00:10:28	3
21328	99	13	10,2	cyclictest	3615	-21	wc	19:40:00	3
21328	99	13	10,2	cyclictest	12704	-21	head	21:05:18	3
21328	99	13	0,12	cyclictest	30681	-21	grep	23:55:26	3
21328	99	13	0,12	cyclictest	13096	-21	tr	23:20:22	3
21324	99	13	9,3	cyclictest	6966	-21	grep	20:55:00	2
21324	99	13	1,1	cyclictest	0	-21	swapper/2	19:50:12	2
21324	99	13	10,2	cyclictest	8674	-21	cat	20:55:29	2
21324	99	13	10,2	cyclictest	32532	-21	cut	20:40:15	2
21324	99	13	10,2	cyclictest	3179	-21	cut	20:45:21	2
21324	99	13	10,2	cyclictest	1974	-21	sed	21:50:17	2
21324	99	13	10,2	cyclictest	14095	-21	sh	21:09:59	2
21321	99	13	9,3	cyclictest	32165	-21	cron	20:40:00	1
21321	99	13	9,2	cyclictest	8966	-21	cat	00:20:00	1
21321	99	13	10,2	cyclictest	32718	-21	ls	19:30:21	1
21321	99	13	10,2	cyclictest	31339	-21	cat	00:00:00	1
21321	99	13	10,2	cyclictest	29633	-21	cut	21:40:17	1
21321	99	13	10,2	cyclictest	28915	-21	sed	20:30:24	1
21321	99	13	10,2	cyclictest	26811	-21	grep	19:20:14	1
21321	99	13	10,2	cyclictest	25758	-21	grep	23:45:29	1
21321	99	13	10,2	cyclictest	22652	-21	grep	19:10:19	1
21321	99	13	10,2	cyclictest	2259	-21	wc	23:00:00	1
21321	99	13	10,2	cyclictest	12203	-21	tr	23:20:15	1
21321	99	13	10,2	cyclictest	10665	-21	grep	00:20:27	1
21321	99	13	10,2	cyclictest	10528	-21	cat	23:15:20	1
21321	99	13	0,12	cyclictest	0	-21	swapper/1	00:35:23	1
21318	99	13	9,3	cyclictest	31954	-21	grep	21:45:18	0
21318	99	13	9,3	cyclictest	17966	-21	sed	21:15:25	0
21318	99	13	9,3	cyclictest	14595	-21	tr	23:25:16	0
21318	99	13	9,3	cyclictest	10004	-21	grep	22:05:29	0
21318	99	13	10,2	cyclictest	9313	-21	tail	19:50:16	0
21318	99	13	10,2	cyclictest	30573	-21	grep	21:40:26	0
21318	99	13	10,2	cyclictest	26985	-21	tail	20:25:26	0
21318	99	13	10,2	cyclictest	15111	-21	wc	00:30:23	0
21318	99	13	10,2	cyclictest	13673	-21	grep	21:05:26	0
21318	99	13	0,8	cyclictest	32156	-21	basename	22:54:59	0
21328	99	12	9,2	cyclictest	9744	-21	grep	23:15:15	3
21328	99	12	9,2	cyclictest	8612	-21	grep	22:05:16	3
21328	99	12	9,2	cyclictest	6890	-21	cron	20:55:00	3
21328	99	12	9,2	cyclictest	5456	-21	sed	19:40:27	3
21328	99	12	9,2	cyclictest	3800	-21	grep	00:10:00	3
21328	99	12	9,2	cyclictest	30632	-21	sort	22:50:18	3
21328	99	12	9,2	cyclictest	26733	-21	tr	21:35:15	3
21328	99	12	9,2	cyclictest	22229	-21	sort	19:10:16	3
21328	99	12	9,2	cyclictest	16520	-21	head	22:20:18	3
21328	99	12	9,2	cyclictest	13644	-21	tr	20:00:14	3
21328	99	12	9,2	cyclictest	10367	-21	fgrep	00:20:23	3
21328	99	12	8,3	cyclictest	31333	-21	wc	20:35:25	3
21328	99	12	8,2	cyclictest	15964	-21	grep	00:35:00	3
21324	99	12	9,2	cyclictest	774	-21	sed	19:30:26	2
21324	99	12	9,2	cyclictest	7374	-21	grep	23:10:18	2
21324	99	12	9,2	cyclictest	4998	-21	grep	00:10:20	2
21324	99	12	9,2	cyclictest	4998	-21	grep	00:10:19	2
21324	99	12	9,2	cyclictest	32279	-21	cat	22:55:00	2
21324	99	12	9,2	cyclictest	31196	-21	awk	00:00:00	2
21324	99	12	9,2	cyclictest	2941	-21	cat	19:35:29	2
21324	99	12	9,2	cyclictest	28390	-21	tune2fs	20:30:18	2
21324	99	12	9,2	cyclictest	27538	-21	cat	21:35:20	2
21324	99	12	9,2	cyclictest	21239	-21	cron	21:25:00	2
21324	99	12	9,2	cyclictest	17211	-21	cat	20:05:26	2
21324	99	12	9,2	cyclictest	12630	-21	grep	23:20:17	2
21324	99	12	9,2	cyclictest	12450	-21	wc	00:25:21	2
21324	99	12	9,2	cyclictest	11164	-21	dump-pmu-power	00:25:00	2
21324	99	12	9,2	cyclictest	11008	-21	tr	22:10:15	2
21321	99	12	9,2	cyclictest	673	-21	sed	00:00:26	1
21321	99	12	9,2	cyclictest	5488	-21	grep	23:05:17	1
21321	99	12	9,2	cyclictest	5066	-21	grep	20:50:15	1
21321	99	12	9,2	cyclictest	32382	-21	tr	21:45:21	1
21321	99	12	9,2	cyclictest	29347	-21	grep	22:45:28	1
21321	99	12	9,2	cyclictest	25989	-21	head	22:40:15	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17