OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackcslot7s.osadl.org (updated Wed Dec 31, 2025 12:44:51)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2413415	2	271	0,6	sleep0	0	-21	swapper/0	07:05:29	0
2413415	2	271	0,6	sleep0	0	-21	swapper/0	07:05:29	0
1973	99	90	0,72	rtkit-daemon	0	-21	swapper/3	07:05:28	3
1973	99	90	0,72	rtkit-daemon	0	-21	swapper/3	07:05:28	3
2443566	2	83	0,4	sleep3	4497	-21	PK-Backend	08:30:31	3
2462091	2	76	0,5	sleep2	2462093	-21	tr	09:25:15	2
2432829	2	68	0,5	sleep1	25	1	rcuc/1	08:00:27	1
2412686	2	68	42,18	sleep2	0	-21	swapper/2	07:05:16	2
2412686	2	68	42,18	sleep2	0	-21	swapper/2	07:05:16	2
2413545	2	63	44,14	sleep1	0	-21	swapper/1	07:07:12	1
2413545	2	63	44,14	sleep1	0	-21	swapper/1	07:07:12	1
1973	99	63	0,8	rtkit-daemon	1972	-21	rtkit-daemon	07:44:22	2
1973	99	63	0,18	rtkit-daemon	1972	-21	rtkit-daemon	07:51:27	2
2418812	2	60	0,21	sleep3	42	-21	ksoftirqd/3	07:20:25	3
2515770	2	58	0,6	chrt	4497	-21	PK-Backend	11:55:30	2
1973	99	58	0,9	rtkit-daemon	1972	-21	rtkit-daemon	07:56:55	2
2427762	2	55	0,6	sleep1	0	-21	swapper/1	07:45:26	1
2427762	2	55	0,6	sleep1	0	-21	swapper/1	07:45:26	1
2413932	99	54	5,44	cyclictest	2506974	-21	grep	11:30:28	0
2413942	99	53	17,31	cyclictest	2470148	-21	grep	09:45:29	3
2413939	99	53	42,8	cyclictest	34	-21	ksoftirqd/2	08:10:39	2
2413932	99	53	18,30	cyclictest	2493651	-21	cut	10:55:02	0
2413932	99	53	16,25	cyclictest	13	-21	ksoftirqd/0	12:23:16	0
2413932	99	53	15,32	cyclictest	2423414	-21	wc	07:35:14	0
2413932	99	53	1,47	cyclictest	2428964	-21	df	07:50:16	0
2413932	99	52	20,28	cyclictest	1033	-21	gdbus	09:12:49	0
2413932	99	52	20,27	cyclictest	2069	-21	gnome-shell	10:22:59	0
2413932	99	52	16,32	cyclictest	2435999	-21	kernelversion	08:10:19	0
2413932	99	52	15,32	cyclictest	2504548	-21	cat	11:25:15	0
2413932	99	52	14,33	cyclictest	2751	-21	gdbus	07:23:31	0
2413932	99	52	14,33	cyclictest	2450766	-21	sshd	08:54:00	0
2413932	99	52	14,32	cyclictest	2442663	-21	grep	08:30:14	0
2413932	99	52	12,34	cyclictest	2420127	-21	expr	07:25:15	0
1973	99	52	0,8	rtkit-daemon	1972	-21	rtkit-daemon	08:36:01	2
2413932	99	51	15,31	cyclictest	2414255	-21	grep	07:10:14	0
2413932	99	51	13,33	cyclictest	2464441	-21	latency	09:30:21	0
2413942	99	50	11,12	cyclictest	1074	-21	lldpd	12:33:05	3
2413942	99	50	11,12	cyclictest	1074	-21	lldpd	12:33:05	3
2413932	99	50	24,21	cyclictest	2522214	-21	hddtemp_smartct	12:15:17	0
2413932	99	50	20,26	cyclictest	2426829	-21	/usr/sbin/munin	07:45:08	0
2413932	99	50	20,26	cyclictest	2426829	-21	/usr/sbin/munin	07:45:08	0
2413932	99	50	18,27	cyclictest	2497591	-21	diskstats	11:05:17	0
2413932	99	50	17,29	cyclictest	2495659	-21	tr	11:00:15	0
2413932	99	50	16,30	cyclictest	2473649	-21	ls	09:55:31	0
2413932	99	50	15,30	cyclictest	438	-20	systemd-journal	07:41:29	0
2413932	99	50	13,32	cyclictest	2515693	-21	which	11:55:28	0
2413942	99	49	11,33	cyclictest	2524081	-21	expr	12:20:13	3
2413932	99	49	18,26	cyclictest	2485722	-21	missed_timers	10:30:23	0
2413932	99	49	17,27	cyclictest	2458834	-21	hddtemp_smartct	09:15:19	0
2413932	99	49	16,28	cyclictest	2466711	-21	timerwakeupswit	09:35:30	0
2413932	99	49	13,31	cyclictest	2511320	-21	cut	11:45:02	0
2413932	99	49	12,32	cyclictest	2476197	-21	cpuspeed	10:05:13	0
2413932	99	49	10,11	cyclictest	2468041	-21	cat	09:40:22	0
2413932	99	48	16,28	cyclictest	2504171	-21	latency_hist	11:25:01	0
2413932	99	48	16,28	cyclictest	1137	-21	runrttasks	11:14:11	0
2413932	99	48	16,27	cyclictest	2434499	-21	ntpq	08:05:24	0
2413932	99	48	15,28	cyclictest	2490962	-21	needreboot	10:45:21	0
2413932	99	48	14,29	cyclictest	2483550	-21	tr	10:25:14	0
2413932	99	48	13,31	cyclictest	2527927	-21	missed_timers	12:30:22	0
2413932	99	48	13,31	cyclictest	2527927	-21	missed_timers	12:30:22	0
2413932	99	48	13,31	cyclictest	2471188	-21	expr	09:50:14	0
2413932	99	48	12,31	cyclictest	2451522	-21	cpuspeed_turbos	08:55:15	0
2413932	99	48	11,31	cyclictest	2525675	-21	cpuspeed_turbos	12:25:16	0
1973	99	48	0,6	rtkit-daemon	1972	-21	rtkit-daemon	08:52:05	3
2516975	2	47	0,6	sleep3	4497	-21	PK-Backend	12:00:13	3
2449290	2	47	0,17	sleep2	34	-21	ksoftirqd/2	08:50:02	2
2413942	99	47	18,24	cyclictest	2422154	-21	missed_timers	07:30:24	3
2413942	99	47	17,26	cyclictest	4497	-21	PK-Backend	10:31:00	3
2413942	99	47	11,31	cyclictest	2492919	-21	switchtime	10:50:28	3
2413939	99	47	34,9	cyclictest	34	-21	ksoftirqd/2	09:12:47	2
2413939	99	47	31,13	cyclictest	34	-21	ksoftirqd/2	10:50:13	2
2413939	99	47	10,20	cyclictest	34	-21	ksoftirqd/2	10:35:01	2
2413936	99	47	3,16	cyclictest	2486732	-21	sed	10:35:02	1
2413936	99	47	2,19	cyclictest	2478372	-21	kernelversion	10:10:22	1
2413932	99	47	5,37	cyclictest	2440952	-21	cstates	08:25:15	0
2413932	99	47	15,28	cyclictest	2529967	-21	switchtime	12:35:30	0
2413932	99	47	15,27	cyclictest	2488451	-21	latency_hist	10:40:02	0
2413932	99	47	15,27	cyclictest	2453325	-21	cpuspeed_turbos	09:00:15	0
2413932	99	47	15,27	cyclictest	2453325	-21	cpuspeed_turbos	09:00:15	0
2413932	99	47	13,30	cyclictest	2448035	-21	df	08:45:17	0
2413942	99	46	8,12	cyclictest	438	-20	systemd-journal	10:40:00	3
2413942	99	46	1,7	cyclictest	2505313	-21	python3	11:25:29	3
2413942	99	46	12,29	cyclictest	2457467	-21	proc_pri	09:10:29	3
2413939	99	46	0,42	cyclictest	1972	-21	rtkit-daemon	08:32:47	2
2413932	99	46	8,10	cyclictest	2476016	-21	cut	10:05:01	0
2413932	99	46	16,25	cyclictest	2514824	-21	latency_hist	11:55:01	0
2413932	99	46	14,27	cyclictest	2463583	-21	apt-key	09:30:01	0
2413932	99	46	14,27	cyclictest	2455469	-21	latency	09:05:20	0
2413932	99	46	1,39	cyclictest	2069	-21	gnome-shell	11:35:11	0
2413932	99	46	13,29	cyclictest	1137	-21	runrttasks	12:11:01	0
2413932	99	46	13,28	cyclictest	2519879	-21	sed	12:10:01	0
2413932	99	46	13,28	cyclictest	2516812	-21	cpuspeed_turbos	12:00:13	0
2413942	99	45	16,25	cyclictest	2503687	-21	timerwakeupswit	11:20:30	3
2413942	99	45	14,26	cyclictest	2472178	-21	systemd-udevd	09:52:41	3
2413942	99	45	1,39	cyclictest	2476306	-21	cut	10:05:13	3
2413936	99	45	8,14	cyclictest	1046	-21	lldpd	11:20:09	1
2413936	99	45	29,11	cyclictest	2494200	-21	iwlist	10:55:22	1
2413936	99	45	2,15	cyclictest	2499510	-21	sed	11:10:22	1
2413932	99	45	6,36	cyclictest	1137	-21	runrttasks	09:49:40	0
2413932	99	45	16,25	cyclictest	2416535	-21	cstates	07:15:13	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17