OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rack7slot7s.osadl.org (updated Wed Apr 08, 2026 12:43:51)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
499	2	90	77,9	sleep0	0	-21	swapper/0	07:05:57	0
603	2	83	63,16	sleep1	0	-21	swapper/1	07:07:06	1
789	2	82	70,8	sleep2	0	-21	swapper/2	07:09:28	2
728	2	79	67,8	sleep3	0	-21	swapper/3	07:08:41	3
19302	2	65	0,0	sleep1	0	-21	swapper/1	08:55:00	1
11427	2	65	0,0	sleep1	0	-21	swapper/1	12:12:04	1
11687	2	64	0,0	sleep2	0	-21	swapper/2	09:39:29	2
922	99	27	0,27	cyclictest	0	-21	swapper/2	11:01:35	2
925	99	26	0,23	cyclictest	0	-21	swapper/3	12:25:09	3
922	99	23	21,1	cyclictest	0	-21	swapper/2	10:09:54	2
922	99	23	0,23	cyclictest	0	-21	swapper/2	10:37:58	2
919	99	23	21,1	cyclictest	0	-21	swapper/1	12:32:43	1
919	99	23	0,3	cyclictest	0	-21	swapper/1	09:41:13	1
922	99	22	1,20	cyclictest	0	-21	swapper/2	09:59:37	2
922	99	22	0,21	cyclictest	0	-21	swapper/2	10:43:06	2
922	99	21	0,21	cyclictest	0	-21	swapper/2	11:22:02	2
919	99	21	18,2	cyclictest	28210	-21	munin-node	12:00:10	1
919	99	21	0,21	cyclictest	0	-21	swapper/1	11:36:38	1
919	99	21	0,21	cyclictest	0	-21	swapper/1	10:40:11	1
922	99	20	0,19	cyclictest	0	-21	swapper/2	12:38:11	2
919	99	20	19,1	cyclictest	0	-21	swapper/1	10:59:53	1
17982	2	20	0,0	sleep3	0	-21	swapper/3	11:56:43	3
922	99	19	11,4	cyclictest	225	-21	systemd-journal	08:40:27	2
919	99	19	0,16	cyclictest	0	-21	swapper/1	11:05:11	1
925	99	18	0,1	cyclictest	0	-21	swapper/3	11:35:52	3
922	99	18	0,18	cyclictest	0	-21	swapper/2	11:08:10	2
919	99	18	17,1	cyclictest	0	-21	swapper/1	10:22:28	1
919	99	18	16,1	cyclictest	0	-21	swapper/1	11:51:49	1
925	99	17	2,15	cyclictest	0	-21	swapper/3	12:17:57	3
925	99	17	15,2	cyclictest	39	-21	ksoftirqd/3	09:35:44	3
925	99	17	1,13	cyclictest	0	-21	swapper/3	10:50:11	3
925	99	17	0,1	cyclictest	0	-21	swapper/3	12:10:24	3
925	99	17	0,0	cyclictest	0	-21	swapper/3	11:46:01	3
922	99	17	15,1	cyclictest	0	-21	swapper/2	10:17:09	2
922	99	17	0,4	cyclictest	0	-21	swapper/2	11:56:56	2
29212	2	17	0,1	chrt	0	-21	swapper/2	09:30:16	2
925	99	16	4,1	cyclictest	16952	-21	apache_processe	12:35:11	3
925	99	16	14,2	cyclictest	39	-21	ksoftirqd/3	11:54:25	3
925	99	16	14,1	cyclictest	0	-21	swapper/3	11:04:59	3
925	99	16	11,3	cyclictest	910	-21	cyclictest	09:34:51	3
925	99	16	0,11	cyclictest	0	-21	swapper/3	10:30:12	3
922	99	16	3,1	cyclictest	0	-21	swapper/2	12:02:44	2
922	99	16	2,0	cyclictest	14	1	rcu_preempt	09:25:00	2
922	99	16	1,1	cyclictest	14	1	rcu_preempt	12:24:37	2
919	99	16	14,2	cyclictest	25	-21	ksoftirqd/1	12:36:53	1
917	99	16	1,15	cyclictest	0	-21	swapper/0	10:01:58	0
917	99	16	11,4	cyclictest	0	-21	swapper/0	11:50:12	0
925	99	15	12,2	cyclictest	39	-21	ksoftirqd/3	09:21:42	3
925	99	15	11,2	cyclictest	26363	-21	awk	09:10:00	3
925	99	15	0,1	cyclictest	0	-21	swapper/3	10:29:32	3
922	99	15	13,1	cyclictest	0	-21	swapper/2	12:29:01	2
922	99	15	0,15	cyclictest	0	-21	swapper/2	12:18:49	2
922	99	15	0,14	cyclictest	0	-21	swapper/2	11:16:52	2
922	99	15	0,14	cyclictest	0	-21	swapper/2	09:14:34	2
919	99	15	15,0	cyclictest	0	-21	swapper/1	12:22:51	1
919	99	15	1,11	cyclictest	0	-21	swapper/1	09:15:50	1
919	99	15	0,1	cyclictest	0	-21	swapper/1	10:02:41	1
925	99	14	5,3	cyclictest	225	-21	systemd-journal	12:04:19	3
925	99	14	13,1	cyclictest	26357	-21	diskmemload	11:27:31	3
925	99	14	10,3	cyclictest	27379	-21	ssh	10:45:26	3
922	99	14	8,5	cyclictest	19165	-21	apache2	07:40:15	2
922	99	14	11,1	cyclictest	32	-21	ksoftirqd/2	10:33:02	2
922	99	14	10,2	cyclictest	11662	-21	sed	07:30:16	2
919	99	14	12,1	cyclictest	6827	-21	fschecks_time	07:20:15	1
919	99	14	10,3	cyclictest	2577	-21	ls	07:10:24	1
919	99	14	0,14	cyclictest	0	-21	swapper/1	09:45:18	1
917	99	14	1,13	cyclictest	26357	-21	diskmemload	11:38:16	0
917	99	14	10,3	cyclictest	2264	-21	cron	08:20:00	0
32311	2	14	0,0	sleep1	0	-21	swapper/1	10:49:13	1
925	99	13	9,3	cyclictest	21220	-21	ssh	11:40:12	3
925	99	13	9,3	cyclictest	20723	-21	wc	08:55:20	3
925	99	13	9,3	cyclictest	20521	-21	grep	11:20:15	3
925	99	13	2,3	cyclictest	225	-21	systemd-journal	11:35:01	3
925	99	13	10,2	cyclictest	7763	-21	tr	08:30:15	3
925	99	13	10,2	cyclictest	7260	-21	turbostat.cron	08:30:00	3
925	99	13	10,2	cyclictest	31004	-21	grep	09:50:20	3
925	99	13	10,2	cyclictest	28672	-21	grep	11:05:24	3
925	99	13	10,2	cyclictest	11632	-21	head	07:30:16	3
925	99	13	0,9	cyclictest	0	-21	swapper/3	09:00:13	3
925	99	13	0,12	cyclictest	5044	-21	sed	09:15:25	3
922	99	13	11,1	cyclictest	7256	-21	ls	08:30:00	2
922	99	13	10,2	cyclictest	769	-21	tail	12:05:25	2
922	99	13	10,2	cyclictest	32	-21	ksoftirqd/2	11:50:49	2
922	99	13	10,2	cyclictest	17641	-21	grep	10:20:26	2
922	99	13	10,2	cyclictest	16166	-21	tail	10:00:25	2
922	99	13	10,2	cyclictest	15370	-21	cron	07:40:01	2
922	99	13	10,2	cyclictest	10519	-21	grep	12:30:26	2
922	99	13	0,3	cyclictest	0	-21	swapper/2	11:34:45	2
922	99	13	0,13	cyclictest	20688	-21	kworker/u8:3	09:54:09	2
922	99	13	0,13	cyclictest	0	-21	swapper/2	11:26:07	2
922	99	13	0,12	cyclictest	0	-21	swapper/2	10:11:04	2
922	99	13	0,12	cyclictest	0	-21	swapper/2	08:35:13	2
919	99	13	9,3	cyclictest	13504	-21	head	08:40:21	1
919	99	13	8,5	cyclictest	19166	-21	apache2	10:30:44	1
919	99	13	8,3	cyclictest	3611	-21	sed	11:10:16	1
919	99	13	10,2	cyclictest	9288	-21	grep	07:25:16	1
919	99	13	10,2	cyclictest	2579	-21	ls	09:14:59	1
919	99	13	10,2	cyclictest	20826	-21	sed	09:25:16	1
919	99	13	10,2	cyclictest	19811	-21	grep	07:45:25	1
919	99	13	0,12	cyclictest	17383	-21	ssh	11:18:28	1
919	99	13	0,12	cyclictest	0	-21	swapper/1	10:07:28	1
919	99	13	0,11	cyclictest	28040	-21	wc	11:25:00	1
917	99	13	6,6	cyclictest	26170	-21	ssh	11:04:57	0
917	99	13	6,1	cyclictest	26357	-21	diskmemload	12:28:22	0
917	99	13	13,0	cyclictest	0	-21	swapper/0	10:27:18	0
917	99	13	12,1	cyclictest	0	-21	swapper/0	12:09:56	0
917	99	13	10,2	cyclictest	9131	-21	grep	08:30:25	0
917	99	13	10,2	cyclictest	2465	-21	gzip	07:10:21	0
917	99	13	10,2	cyclictest	19166	-21	apache2	11:20:41	0
917	99	13	0,12	cyclictest	2214	-21	tail	10:50:15	0
925	99	12	9,2	cyclictest	9142	-21	grep	07:25:16	3
925	99	12	9,2	cyclictest	7818	-21	sed	07:20:25	3
925	99	12	9,2	cyclictest	729	-21	sed	10:10:25	3
925	99	12	9,2	cyclictest	30817	-21	cut	08:10:15	3
925	99	12	9,2	cyclictest	2118	-21	wc	07:10:19	3
925	99	12	9,2	cyclictest	18019	-21	wc	08:50:18	3
925	99	12	9,2	cyclictest	1783	-21	grep	08:15:25	3
925	99	12	9,2	cyclictest	16511	-21	head	10:20:18	3
925	99	12	9,2	cyclictest	16431	-21	grep	07:40:17	3
925	99	12	9,2	cyclictest	13751	-21	head	09:40:16	3
925	99	12	5,6	cyclictest	1	-21	systemd	09:57:27	3
925	99	12	11,1	cyclictest	0	-21	swapper/3	08:43:36	3
922	99	12	9,2	cyclictest	7366	-21	head	07:20:20	2
922	99	12	9,2	cyclictest	6427	-21	sh	10:52:44	2
922	99	12	9,2	cyclictest	31834	-21	grep	08:10:25	2
922	99	12	9,2	cyclictest	25351	-21	cut	10:25:19	2
922	99	12	9,2	cyclictest	18936	-21	chrt	09:43:34	2
922	99	12	9,2	cyclictest	1871	-21	grep	08:15:25	2
922	99	12	9,2	cyclictest	15887	-21	head	08:45:20	2
922	99	12	8,3	cyclictest	4850	-21	cut	08:25:00	2
922	99	12	10,2	cyclictest	19166	-21	apache2	11:42:32	2
922	99	12	0,1	cyclictest	0	-21	swapper/2	11:13:36	2
919	99	12	9,2	cyclictest	8098	-21	sh	12:29:59	1
919	99	12	9,2	cyclictest	6680	-21	grep	09:55:16	1
919	99	12	9,2	cyclictest	5873	-21	head	08:25:18	1
919	99	12	9,2	cyclictest	5819	-21	ssh	09:35:23	1
919	99	12	9,2	cyclictest	4985	-21	grep	07:15:21	1
919	99	12	9,2	cyclictest	29191	-21	grep	09:30:16	1
919	99	12	9,2	cyclictest	28692	-21	tr	11:25:15	1
919	99	12	9,2	cyclictest	16761	-21	sh	08:50:01	1
919	99	12	9,2	cyclictest	14218	-21	chrt	09:20:57	1
919	99	12	8,3	cyclictest	20129	-21	grep	08:55:16	1
919	99	12	8,3	cyclictest	15418	-21	basename	07:40:00	1
919	99	12	8,2	cyclictest	4173	-21	grep	08:20:25	1
919	99	12	7,2	cyclictest	5889	-21	ssh	10:52:19	1
919	99	12	3,6	cyclictest	31232	-21	wget	10:10:13	1
919	99	12	1,6	cyclictest	0	-21	swapper/1	09:05:14	1
919	99	12	10,2	cyclictest	0	-21	swapper/1	12:15:50	1
917	99	12	9,2	cyclictest	9276	-21	grep	12:30:15	0
917	99	12	9,2	cyclictest	5268	-21	head	09:35:18	0
917	99	12	9,2	cyclictest	4446	-21	sed	09:15:20	0
917	99	12	9,2	cyclictest	30784	-21	cron	12:05:00	0
917	99	12	9,2	cyclictest	18336	-21	pluginstate	10:40:20	0
917	99	12	9,1	cyclictest	26357	-21	diskmemload	12:17:53	0
917	99	12	3,7	cyclictest	20688	-21	kworker/u8:3	09:59:26	0
917	99	12	2,5	cyclictest	10712	-21	dpkg	11:15:00	0
917	99	12	10,1	cyclictest	25579	-21	cut	10:45:13	0
917	99	12	0,11	cyclictest	0	-21	swapper/0	10:13:25	0
917	99	12	0,11	cyclictest	0	-21	swapper/0	10:05:12	0
925	99	11	9,1	cyclictest	28473	-21	cut	08:05:15	3
925	99	11	9,1	cyclictest	19355	-21	wc	07:45:21	3
925	99	11	8,2	cyclictest	8420	-21	ssh	10:35:08	3
925	99	11	8,2	cyclictest	26858	-21	ssh	09:48:09	3
925	99	11	8,2	cyclictest	24399	-21	tr	12:20:13	3
925	99	11	8,2	cyclictest	23278	-21	grep	10:05:14	3
925	99	11	8,2	cyclictest	22714	-21	sh	09:25:59	3
925	99	11	8,2	cyclictest	1910	-21	ssh	12:06:14	3
925	99	11	8,2	cyclictest	18592	-21	ssh	10:02:13	3
925	99	11	8,2	cyclictest	17702	-21	grep	10:40:15	3
925	99	11	8,2	cyclictest	11753	-21	awk	10:55:24	3
925	99	11	8,2	cyclictest	10422	-21	wc	12:30:25	3
925	99	11	8,2	cyclictest	10274	-21	rm	11:14:43	3
925	99	11	6,3	cyclictest	556	50	irq/127-enp2s0-	10:15:12	3
925	99	11	2,6	cyclictest	11421	-21	ssh	11:15:12	3
925	99	11	1,8	cyclictest	28244	-21	ssh	09:10:19	3
925	99	11	0,10	cyclictest	5004	-21	grep	07:15:21	3
925	99	11	0,10	cyclictest	27115	-21	chrt	08:02:13	3
922	99	11	9,1	cyclictest	21571	-21	sh	09:00:00	2
922	99	11	8,2	cyclictest	3820	-21	grep	07:15:13	2
922	99	11	8,2	cyclictest	3720	-21	sh	09:15:15	2
922	99	11	8,2	cyclictest	3267	-21	turbostat.cron	07:15:00	2
922	99	11	8,2	cyclictest	31559	-21	grep	11:45:25	2
922	99	11	8,2	cyclictest	23905	-21	ssh	09:26:54	2
922	99	11	8,2	cyclictest	23113	-21	tr	07:55:13	2
922	99	11	8,2	cyclictest	22459	-21	sort	09:00:16	2
922	99	11	8,2	cyclictest	13568	-21	head	11:35:16	2
922	99	11	8,2	cyclictest	11792	-21	sh	10:55:24	2
922	99	11	2,5	cyclictest	32	-21	ksoftirqd/2	09:45:25	2
922	99	11	0,6	cyclictest	0	-21	swapper/2	08:51:32	2
922	99	11	0,10	cyclictest	25929	-21	expr	10:45:14	2
919	99	11	9,1	cyclictest	24263	2	sleep1	11:41:14	1
919	99	11	8,2	cyclictest	7960	-21	grep	10:15:15	1
919	99	11	8,2	cyclictest	6645	-21	sh	12:09:53	1
919	99	11	8,2	cyclictest	364	-21	grep	08:15:14	1
919	99	11	8,2	cyclictest	30530	-21	expr	08:10:14	1
919	99	11	8,2	cyclictest	30505	-21	cut	09:50:16	1
919	99	11	8,2	cyclictest	30481	-21	head	11:45:17	1
919	99	11	8,2	cyclictest	25464	-21	tr	08:00:13	1
919	99	11	8,2	cyclictest	23064	-21	ssh	11:02:34	1
919	99	11	8,2	cyclictest	22998	-21	ntpq	09:00:20	1
919	99	11	8,2	cyclictest	21683	-21	ls	07:50:20	1
919	99	11	8,2	cyclictest	11601	-21	tail	07:30:16	1
919	99	11	7,3	cyclictest	0	-21	swapper/1	10:37:07	1
919	99	11	7,1	cyclictest	27648	-21	ssh	10:26:36	1
919	99	11	10,1	cyclictest	25	-21	ksoftirqd/1	11:55:59	1
917	99	11	9,1	cyclictest	9390	-21	sed	10:35:17	0
917	99	11	9,1	cyclictest	4781	-21	chrt	11:49:44	0
917	99	11	9,1	cyclictest	27025	-21	grep	11:05:13	0
917	99	11	9,1	cyclictest	17737	-21	grep	08:50:15	0
917	99	11	9,1	cyclictest	16386	-21	grep	10:20:17	0
917	99	11	9,1	cyclictest	13039	-21	grep	08:40:16	0
917	99	11	8,2	cyclictest	9265	-21	grep	12:10:26	0
917	99	11	8,2	cyclictest	9222	-21	grep	07:25:17	0
917	99	11	8,2	cyclictest	8873	-21	cat	10:15:24	0
917	99	11	8,2	cyclictest	591	-21	chrt	09:32:26	0
917	99	11	8,2	cyclictest	3987	-21	tr	07:15:14	0
917	99	11	8,2	cyclictest	31392	-21	ls	08:10:20	0
917	99	11	8,2	cyclictest	31320	-21	rm	09:50:23	0
917	99	11	8,2	cyclictest	29229	-21	grep	11:25:18	0
917	99	11	8,2	cyclictest	28448	-21	grep	08:05:15	0
917	99	11	8,2	cyclictest	20690	-21	cut	09:25:15	0
917	99	11	8,2	cyclictest	15699	-21	sed	08:45:19	0
917	99	11	8,2	cyclictest	12035	-21	ssh	11:15:15	0
917	99	11	8,2	cyclictest	11677	-21	grep	10:55:24	0
917	99	11	7,3	cyclictest	4161	-21	grep	08:20:25	0
917	99	11	10,1	cyclictest	0	-21	swapper/0	11:30:13	0
917	99	11	0,10	cyclictest	2526	-21	ssh	09:14:58	0
917	99	11	0,10	cyclictest	21206	-21	head	07:50:15	0
917	99	11	0,10	cyclictest	18970	-21	uname	07:45:17	0
925	99	10	8,1	cyclictest	3317	-21	expr	08:20:16	3
925	99	10	8,1	cyclictest	22214	2	sleep3	07:50:27	3
925	99	10	7,2	cyclictest	23596	-21	grep	07:55:15	3
925	99	10	7,2	cyclictest	14927	-21	grep	08:45:13	3
925	99	10	2,2	cyclictest	225	-21	systemd-journal	07:38:25	3
922	99	10	8,1	cyclictest	8902	-21	cat	07:25:14	2
922	99	10	8,1	cyclictest	25476	-21	tr	08:00:13	2
922	99	10	8,1	cyclictest	10687	-21	ssh	12:11:29	2
922	99	10	7,2	cyclictest	8631	-21	awk	08:30:20	2
922	99	10	7,2	cyclictest	20704	-21	cut	07:50:13	2
922	99	10	7,2	cyclictest	18105	-21	grep	07:45:13	2
919	99	10	8,1	cyclictest	7488	-21	cat	08:30:10	1
919	99	10	8,1	cyclictest	3994	-21	ssh	11:30:01	1
919	99	10	8,1	cyclictest	28190	-21	expr	08:05:13	1
919	99	10	8,1	cyclictest	11739	-21	chrt	08:36:49	1
919	99	10	7,2	cyclictest	23206	-21	expr	07:55:14	1
917	99	10	8,2	cyclictest	13262	-21	expr	09:40:14	0
917	99	10	8,1	cyclictest	5339	-21	ssh	10:32:54	0
917	99	10	8,1	cyclictest	24309	-21	rm	12:20:14	0
917	99	10	8,1	cyclictest	21358	-21	ssh	09:45:12	0
917	99	10	8,1	cyclictest	21191	-21	ssh	11:40:11	0
917	99	10	8,1	cyclictest	15991	-21	cat	11:55:24	0
917	99	10	8,1	cyclictest	12111	-21	grep	07:30:20	0
917	99	10	8,1	cyclictest	10645	-21	grep	08:35:16	0
917	99	10	7,2	cyclictest	13421	-21	grep	09:20:25	0
917	99	10	3,6	cyclictest	24240	-21	apt-get	09:05:13	0
917	99	10	3,3	cyclictest	225	-21	systemd-journal	07:45:00	0
917	99	10	1,1	cyclictest	0	-21	swapper/0	07:55:13	0
917	99	10	0,9	cyclictest	15369	-21	cron	07:40:00	0
925	99	9	6,2	cyclictest	10508	-21	cut	08:35:16	3
922	99	9	7,1	cyclictest	25412	-21	open_inodes	09:05:20	2
922	99	9	6,2	cyclictest	28528	-21	grep	08:05:16	2
919	99	9	7,1	cyclictest	16483	-21	basename	07:40:17	1
917	99	9	8,1	cyclictest	6575	-21	expr	07:20:13	0
917	99	9	7,1	cyclictest	5346	-21	grep	08:25:15	0
917	99	9	7,1	cyclictest	25226	-21	cat	08:00:10	0
917	99	9	7,1	cyclictest	16729	-21	ssh	12:35:02	0
917	99	9	1,6	cyclictest	0	-21	swapper/0	09:00:17	0
917	99	9	0,9	cyclictest	19588	-21	aten2_r7power_v	08:55:13	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17