OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Sun Dec 21, 2025 00:43:47)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
13035	2	90	64,8	sleep0	0	-21	swapper/0	19:09:46	0
12992	2	90	77,9	sleep2	0	-21	swapper/2	19:09:13	2
13016	2	83	66,13	sleep1	0	-21	swapper/1	19:09:32	1
12956	2	79	66,9	sleep3	0	-21	swapper/3	19:08:46	3
13144	99	26	0,1	cyclictest	0	-21	swapper/3	23:40:14	3
13140	99	25	0,20	cyclictest	0	-21	swapper/2	20:35:14	2
13137	99	24	5,10	cyclictest	0	-21	swapper/1	21:10:12	1
13137	99	24	5,10	cyclictest	0	-21	swapper/1	21:10:11	1
13137	99	19	19,0	cyclictest	0	-21	swapper/1	19:10:15	1
13137	99	19	17,1	cyclictest	11962	-21	ls	20:14:59	1
13130	99	19	7,6	cyclictest	30765	-21	munin-run	23:10:01	0
13140	99	18	13,4	cyclictest	14295	-21	systemctl	20:20:00	2
13140	99	17	16,1	cyclictest	0	-21	swapper/2	21:35:14	2
13140	99	17	1,13	cyclictest	0	-21	swapper/2	21:25:18	2
13130	99	16	4,6	cyclictest	22687	-21	perf	21:45:01	0
13144	99	15	2,3	cyclictest	225	-21	systemd-journal	00:00:00	3
13144	99	15	11,2	cyclictest	11650	-21	cron	22:29:59	3
13144	99	15	10,3	cyclictest	6882	-21	cron	22:19:59	3
13137	99	15	2,7	cyclictest	225	-21	systemd-journal	00:19:59	1
13137	99	15	11,3	cyclictest	23382	-21	tune2fs	00:00:18	1
13130	99	15	4,6	cyclictest	9341	-21	perf	00:40:00	0
13144	99	14	4,1	cyclictest	14	1	rcu_preempt	21:20:14	3
13144	99	14	1,7	cyclictest	0	-21	swapper/3	23:20:15	3
13137	99	14	14,0	cyclictest	0	-21	swapper/1	20:05:15	1
13137	99	14	11,2	cyclictest	18639	-21	tail	23:50:15	1
13137	99	14	10,3	cyclictest	3861	-21	chrt	23:20:17	1
13137	99	14	10,3	cyclictest	24649	-21	grep	22:55:18	1
13137	99	14	10,2	cyclictest	25560	-21	tr	21:50:15	1
13137	99	14	10,2	cyclictest	22067	-21	wc	19:25:30	1
13130	99	14	9,3	cyclictest	30717	-21	sort	00:15:15	0
13130	99	14	6,7	cyclictest	27451	-21	munin-run	19:40:01	0
13130	99	14	11,2	cyclictest	2008	-21	wc	23:15:20	0
13130	99	14	11,2	cyclictest	18861	-21	sh	22:45:00	0
13130	99	14	10,2	cyclictest	4503	-21	cron	22:15:00	0
13130	99	14	10,2	cyclictest	25504	-21	grep	22:55:27	0
13130	99	14	10,2	cyclictest	24785	-21	cron	00:05:00	0
13130	99	14	10,2	cyclictest	23621	-21	sleep	22:55:00	0
13130	99	14	10,2	cyclictest	1703	-21	grep	22:05:29	0
13130	99	14	0,3	cyclictest	0	-21	swapper/0	19:20:14	0
13130	99	14	0,13	cyclictest	8617	-21	tr	21:15:17	0
13144	99	13	9,3	cyclictest	24392	-21	sed	00:00:28	3
13144	99	13	9,3	cyclictest	17002	-21	wc	23:45:29	3
13144	99	13	1,11	cyclictest	5266	-21	grep	20:00:18	3
13144	99	13	10,2	cyclictest	6236	-21	grep	21:10:14	3
13144	99	13	10,2	cyclictest	6236	-21	grep	21:10:13	3
13144	99	13	10,2	cyclictest	30549	-21	tr	19:45:16	3
13144	99	13	10,2	cyclictest	2996	-21	grep	00:25:17	3
13144	99	13	10,2	cyclictest	28018	-21	cat	20:45:25	3
13144	99	13	10,2	cyclictest	23798	-21	cat	22:55:01	3
13144	99	13	10,2	cyclictest	16354	-21	sort	19:15:18	3
13144	99	13	10,2	cyclictest	12154	-21	grep	22:30:16	3
13144	99	13	10,2	cyclictest	10237	-21	turbostat.cron	23:35:00	3
13144	99	13	0,12	cyclictest	22655	-21	cron	21:45:01	3
13140	99	13	9,3	cyclictest	18570	-21	wc	20:25:31	2
13140	99	13	10,2	cyclictest	9858	-21	grep	20:10:17	2
13140	99	13	10,2	cyclictest	32525	-21	wc	19:50:01	2
13140	99	13	10,2	cyclictest	25059	-21	sh	19:35:01	2
13140	99	13	10,2	cyclictest	22107	-21	grep	22:50:19	2
13140	99	13	10,2	cyclictest	21908	-21	fgrep	19:25:25	2
13140	99	13	0,12	cyclictest	17984	-21	grep	22:40:21	2
13137	99	13	9,3	cyclictest	25267	-21	tr	20:40:21	1
13137	99	13	9,3	cyclictest	21469	-21	sh	20:35:00	1
13137	99	13	9,2	cyclictest	26949	-21	grep	19:35:28	1
13137	99	13	10,2	cyclictest	6609	-21	wc	20:00:29	1
13137	99	13	10,2	cyclictest	410	-21	grep	20:55:32	1
13137	99	13	10,2	cyclictest	4077	-21	grep	22:10:27	1
13137	99	13	10,2	cyclictest	29987	-21	grep	20:50:19	1
13137	99	13	10,2	cyclictest	27759	-21	cut	19:40:14	1
13137	99	13	10,2	cyclictest	2168	-21	cron	00:25:00	1
13137	99	13	10,2	cyclictest	11216	-21	tail	22:25:27	1
13137	99	13	0,12	cyclictest	4347	-21	taskset	00:27:32	1
13137	99	13	0,11	cyclictest	16007	-21	tr	19:15:15	1
13130	99	13	9,3	cyclictest	16838	-21	grep	21:30:22	0
13130	99	13	9,2	cyclictest	1679	-21	grep	00:20:26	0
13130	99	13	9,2	cyclictest	14131	-21	sed	19:10:19	0
13130	99	13	6,4	cyclictest	455	-21	dbus-daemon	00:00:00	0
13130	99	13	3,2	cyclictest	14	1	rcu_preempt	23:45:01	0
13130	99	13	1,11	cyclictest	0	-21	swapper/0	20:20:17	0
13130	99	13	10,2	cyclictest	9226	-21	grep	23:30:20	0
13130	99	13	10,2	cyclictest	29418	-21	grep	19:40:28	0
13130	99	13	0,12	cyclictest	30839	-21	grep	22:00:17	0
13130	99	13	0,12	cyclictest	23358	-21	sed	20:35:27	0
13144	99	12	9,2	cyclictest	8633	-21	grep	21:15:17	3
13144	99	12	9,2	cyclictest	5542	-21	head	00:30:18	3
13144	99	12	9,2	cyclictest	32696	-21	sed	23:10:27	3
13144	99	12	9,2	cyclictest	27430	-21	cron	00:10:00	3
13144	99	12	9,2	cyclictest	2670	-21	grep	22:10:16	3
13144	99	12	9,2	cyclictest	25981	-21	grep	21:50:18	3
13144	99	12	9,2	cyclictest	25976	-21	cron	22:59:59	3
13144	99	12	9,2	cyclictest	24021	-21	grep	21:45:20	3
13144	99	12	9,2	cyclictest	22842	-21	sed	20:35:21	3
13144	99	12	9,2	cyclictest	20577	-21	wc	22:45:27	3
13144	99	12	8,3	cyclictest	29361	-21	sed	21:55:27	3
13144	99	12	8,3	cyclictest	27426	-21	cron	19:40:01	3
13144	99	12	8,3	cyclictest	24267	-21	basename	20:40:15	3
13144	99	12	8,3	cyclictest	1482	-21	grep	21:00:15	3
13140	99	12	9,2	cyclictest	9072	-21	grep	21:15:19	2
13140	99	12	9,2	cyclictest	30718	-21	uniq	00:15:15	2
13140	99	12	9,2	cyclictest	2900	-21	sed	21:00:26	2
13140	99	12	9,2	cyclictest	28398	-21	head	21:55:18	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17