OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackaslot8s.osadl.org (updated Fri May 15, 2026 00:45:09)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
364	2	168	100,61	sleep2	0	-21	swapper/2	19:09:43	2
32679	2	146	110,25	sleep1	0	-21	swapper/1	19:07:44	1
32674	2	145	96,24	sleep3	0	-21	swapper/3	19:07:42	3
32666	2	139	102,25	sleep0	0	-21	swapper/0	19:07:35	0
16287	2	131	0,8	sleep2	28	-21	ksoftirqd/2	23:55:28	2
19602	2	99	0,1	sleep3	19603	-21	sshd	19:52:43	3
5522	2	69	0,3	sleep0	0	-21	swapper/0	23:00:00	0
22605	2	66	0,1	sleep0	0	-21	swapper/0	23:12:05	0
19942	2	64	0,1	sleep1	0	-21	swapper/1	21:33:45	1
14307	2	63	0,3	sleep1	22	-21	ksoftirqd/1	19:40:17	1
28714	2	60	0,2	sleep2	0	-21	swapper/2	22:04:48	2
32727	2	58	0,1	sleep3	0	-21	swapper/3	23:44:34	3
26922	2	56	0,3	sleep0	12	-21	ksoftirqd/0	20:10:15	0
25266	2	55	0,2	sleep1	0	-21	swapper/1	00:02:33	1
10741	2	51	0,2	sleep0	0	-21	swapper/0	23:51:38	0
27563	2	49	0,2	sleep3	0	-21	swapper/3	22:51:17	3
6240	2	46	0,4	sleep0	0	-21	swapper/0	23:48:48	0
15804	2	46	0,1	sleep2	0	-21	swapper/2	23:06:57	2
704	99	44	17,17	cyclictest	13824	-21	ssh	00:18:35	2
704	99	44	14,13	cyclictest	11551	-21	ssh	22:15:21	2
704	99	44	12,11	cyclictest	1618	-21	sed	23:20:23	2
704	99	43	15,10	cyclictest	32480	-21	sh	22:55:19	2
704	99	43	14,9	cyclictest	11897	-21	ssh	23:04:44	2
704	99	43	12,9	cyclictest	19818	-21	diskmemload	23:53:43	2
704	99	43	12,12	cyclictest	3818	-21	timerandwakeup	00:10:26	2
704	99	42	14,11	cyclictest	15869	-21	rm	23:31:00	2
704	99	42	13,12	cyclictest	24028	-21	expr	22:25:11	2
704	99	42	13,12	cyclictest	19772	-21	grep	23:10:18	2
704	99	42	12,12	cyclictest	27511	-21	expr	21:15:19	2
704	99	41	14,14	cyclictest	656	-21	ssh	22:31:02	2
704	99	41	13,11	cyclictest	19818	-21	diskmemload	21:25:12	2
28924	2	41	0,1	sleep0	0	-21	swapper/0	23:40:42	0
704	99	40	15,9	cyclictest	10091	-21	rm	23:26:56	2
704	99	40	15,11	cyclictest	1919	-21	taskset	00:33:52	2
704	99	40	15,10	cyclictest	19818	-21	diskmemload	23:48:18	2
704	99	40	15,10	cyclictest	19818	-21	diskmemload	23:44:26	2
704	99	40	13,10	cyclictest	23584	-21	sed	20:00:23	2
704	99	40	12,12	cyclictest	26837	-21	runrttasks	00:04:10	2
704	99	40	12,10	cyclictest	19818	-21	diskmemload	00:08:27	2
3359	2	40	0,1	chrt	0	-21	swapper/3	22:57:50	3
17006	2	40	0,1	sleep1	17008	-21	ssh	21:55:20	1
704	99	39	14,16	cyclictest	22915	-21	ssh	23:36:22	2
704	99	39	13,9	cyclictest	4478	-21	rm	21:21:55	2
704	99	39	13,16	cyclictest	18963	-21	ssh	00:22:01	2
704	99	38	16,9	cyclictest	25615	-21	ssh	00:27:01	2
704	99	38	16,8	cyclictest	16086	-21	ps	21:30:22	2
704	99	38	13,9	cyclictest	8371	-21	ssh	21:49:31	2
704	99	38	12,9	cyclictest	29604	-21	grep	21:40:22	2
704	99	38	12,9	cyclictest	19818	-21	diskmemload	21:54:05	2
704	99	38	12,9	cyclictest	19818	-21	diskmemload	21:54:05	2
704	99	38	12,9	cyclictest	15776	-21	ssh	21:55:03	2
704	99	38	12,10	cyclictest	7972	-21	sh	00:38:15	2
704	99	38	12,10	cyclictest	7586	-21	ssh	22:12:43	2
704	99	38	12,10	cyclictest	28298	-21	rm	22:52:02	2
704	99	37	15,9	cyclictest	19818	-21	diskmemload	22:23:44	2
704	99	37	13,10	cyclictest	29656	-21	ls	22:05:14	2
704	99	37	12,9	cyclictest	19818	-21	diskmemload	23:17:52	2
704	99	37	12,11	cyclictest	11528	-21	diskmemload	22:40:03	2
704	99	37	10,10	cyclictest	0	-21	swapper/2	20:30:22	2
704	99	36	14,9	cyclictest	11844	-21	tune2fs	19:35:14	2
704	99	36	12,8	cyclictest	21630	-21	sed	21:10:24	2
704	99	36	12,8	cyclictest	21368	-21	timerwakeupswit	19:55:24	2
704	99	35	9,9	cyclictest	19818	-21	diskmemload	21:38:26	2
704	99	35	7,17	cyclictest	0	-21	swapper/2	22:35:12	2
704	99	35	7,17	cyclictest	0	-21	swapper/2	22:35:12	2
704	99	35	13,8	cyclictest	16227	-21	grep	21:00:15	2
704	99	35	12,9	cyclictest	20652	-21	sh	22:46:02	2
704	99	35	12,10	cyclictest	887	-21	sh	20:25:12	2
11228	2	35	0,10	sleep1	22	-21	ksoftirqd/1	21:50:40	1
11228	2	35	0,10	sleep1	22	-21	ksoftirqd/1	21:50:39	1
704	99	34	12,8	cyclictest	28668	-21	apt-get	20:15:10	2
704	99	34	12,10	cyclictest	7800	-21	if_enp3s0	19:25:16	2
8120	2	33	0,2	sleep1	0	-21	swapper/1	23:50:13	1
704	99	33	9,15	cyclictest	0	-21	swapper/2	20:05:18	2
704	99	33	9,15	cyclictest	0	-21	swapper/2	19:45:24	2
704	99	33	13,7	cyclictest	12925	-21	taskset	20:52:08	2
17570	2	33	0,8	sleep0	12	-21	ksoftirqd/0	22:44:39	0
704	99	31	7,15	cyclictest	0	-21	swapper/2	19:10:25	2
704	99	31	12,7	cyclictest	27855	-21	sed	20:10:24	2
15842	2	31	0,8	sleep0	0	-21	swapper/0	23:06:58	0
31417	2	30	0,8	sleep1	22	-21	ksoftirqd/1	22:05:47	1
22479	2	30	0,11	sleep0	12	-21	ksoftirqd/0	22:00:01	0
705	99	29	2,11	cyclictest	20279	-21	ssh	23:35:13	3
704	99	29	8,8	cyclictest	0	-21	swapper/2	20:45:11	2
704	99	29	17,9	cyclictest	9334	-21	taskset	19:30:11	2
704	99	29	16,10	cyclictest	18677	-21	/usr/sbin/munin	21:05:17	2
704	99	29	16,10	cyclictest	14209	-21	ls	20:55:18	2
704	99	28	17,8	cyclictest	13688	-21	ls	19:40:12	2
28194	2	28	0,2	sleep3	0	-21	swapper/3	22:04:18	3
704	99	27	7,8	cyclictest	1393	-21	gpm	20:35:20	2
704	99	27	18,6	cyclictest	31806	-21	grep	20:20:21	2
704	99	27	17,8	cyclictest	3868	-21	grep	19:15:22	2
704	99	26	17,7	cyclictest	7902	-21	ntp_offset	20:40:20	2
704	99	26	17,6	cyclictest	18157	-21	ls	19:50:13	2
704	99	26	15,8	cyclictest	5053	-21	apt-get	19:20:10	2
702	99	26	23,2	cyclictest	0	-21	swapper/0	19:12:48	0
1546	2	26	0,2	sleep1	1549	-21	pmu-power	20:25:20	1
705	99	25	23,1	cyclictest	0	-21	swapper/3	19:12:43	3
705	99	25	19,4	cyclictest	20932	-21	sh	00:24:01	3
703	99	25	2,16	cyclictest	21788	-21	chrt	19:58:04	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17