OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Fri Nov 21, 2025 12:44:00)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
23742	2	106	78,9	sleep1	0	-21	swapper/1	07:06:23	1
23989	2	104	91,9	sleep3	0	-21	swapper/3	07:09:34	3
23905	2	94	70,8	sleep0	0	-21	swapper/0	07:08:31	0
17874	2	88	0,1	sleep0	17209	-21	diskmemload	12:35:11	0
21936	2	85	64,15	sleep2	0	-21	swapper/2	07:05:15	2
24109	99	28	0,28	cyclictest	0	-21	swapper/2	11:51:41	2
24105	99	21	3,2	cyclictest	14	1	rcu_preempt	11:53:18	1
24114	99	19	1,14	cyclictest	6915	-21	/usr/sbin/munin	10:20:17	3
24109	99	19	0,17	cyclictest	0	-21	swapper/2	12:34:57	2
24100	99	19	18,1	cyclictest	0	-21	swapper/0	08:25:13	0
24100	99	19	0,2	cyclictest	0	-21	swapper/0	11:50:24	0
24109	99	18	1,2	cyclictest	32	-21	ksoftirqd/2	12:05:27	2
24109	99	18	1,2	cyclictest	32	-21	ksoftirqd/2	12:05:27	2
29351	2	17	8,2	sleep2	29379	-21	sh	09:20:01	2
24114	99	17	9,7	cyclictest	39	-21	ksoftirqd/3	11:54:52	3
24109	99	17	14,2	cyclictest	0	-21	swapper/2	11:45:20	2
24109	99	17	12,3	cyclictest	0	-21	swapper/2	09:25:13	2
24100	99	17	3,8	cyclictest	26468	-21	munin-run	07:15:00	0
24100	99	17	2,8	cyclictest	27079	-21	/usr/sbin/munin	09:45:20	0
24114	99	16	14,2	cyclictest	39	-21	ksoftirqd/3	10:03:33	3
24109	99	16	0,16	cyclictest	0	-21	swapper/2	09:51:40	2
24100	99	16	7,6	cyclictest	3194	-21	apt-get	08:40:15	0
24533	2	15	0,0	chrt	0	-21	swapper/0	12:16:32	0
24114	99	15	1,11	cyclictest	21076	-21	ssh	09:40:15	3
24109	99	15	9,5	cyclictest	22111	-21	cat	09:40:21	2
24109	99	15	13,1	cyclictest	378	50	irq/131-snd_hda	07:45:17	2
24109	99	15	13,1	cyclictest	0	-21	swapper/2	08:25:13	2
24109	99	15	12,1	cyclictest	0	-21	swapper/2	12:38:46	2
24109	99	15	0,15	cyclictest	0	-21	swapper/2	10:43:56	2
24105	99	15	0,14	cyclictest	0	-21	swapper/1	12:03:12	1
24100	99	15	13,1	cyclictest	17209	-21	diskmemload	09:14:02	0
31115	2	14	0,2	sleep2	32	-21	ksoftirqd/2	12:21:03	2
24114	99	14	9,4	cyclictest	19194	-21	grep	08:05:14	3
24114	99	14	4,6	cyclictest	29185	-21	apt-get	07:20:12	3
24114	99	14	12,1	cyclictest	19304	-21	ssh	12:13:19	3
24114	99	14	10,3	cyclictest	25723	-21	grep	10:35:18	3
24109	99	14	10,3	cyclictest	8202	-21	cpuspeed_turbos	11:15:14	2
24105	99	14	3,4	cyclictest	28231	-21	awk	12:20:00	1
24105	99	14	11,2	cyclictest	1578	-21	grep	07:30:12	1
24105	99	14	10,3	cyclictest	27976	-21	cron	08:25:00	1
24105	99	14	10,3	cyclictest	14345	-21	sed	10:25:27	1
24105	99	14	10,2	cyclictest	15505	-21	grep	10:00:25	1
7610	2	13	0,1	sleep1	0	-21	swapper/1	10:46:54	1
24114	99	13	9,3	cyclictest	8829	-21	grep	12:05:22	3
24114	99	13	9,3	cyclictest	8829	-21	grep	12:05:22	3
24114	99	13	8,4	cyclictest	14377	-21	sh	09:35:00	3
24114	99	13	13,0	cyclictest	0	-21	swapper/3	09:56:08	3
24114	99	13	12,1	cyclictest	0	-21	swapper/3	09:26:14	3
24114	99	13	10,2	cyclictest	25127	-21	head	08:15:28	3
24114	99	13	10,2	cyclictest	23404	-21	sed	11:00:26	3
24114	99	13	10,2	cyclictest	18391	-21	wc	10:55:22	3
24109	99	13	9,4	cyclictest	24095	-21	cyclictest	09:55:13	2
24109	99	13	8,4	cyclictest	32222	-21	cut	07:25:14	2
24109	99	13	8,4	cyclictest	28667	-21	tr	09:45:23	2
24109	99	13	7,3	cyclictest	17209	-21	diskmemload	12:14:41	2
24109	99	13	4,8	cyclictest	7744	-21	snmpd	09:33:46	2
24109	99	13	2,2	cyclictest	224	-21	systemd-journal	08:08:30	2
24109	99	13	0,13	cyclictest	0	-21	swapper/2	12:00:24	2
24105	99	13	9,3	cyclictest	5817	-21	grep	08:45:16	1
24105	99	13	9,3	cyclictest	4405	-21	sed	09:25:21	1
24105	99	13	10,3	cyclictest	25	-21	ksoftirqd/1	10:20:31	1
24105	99	13	10,2	cyclictest	9196	-21	tr	07:45:14	1
24105	99	13	10,2	cyclictest	30886	-21	wc	09:20:19	1
24105	99	13	10,2	cyclictest	27130	-21	grep	11:55:20	1
24105	99	13	10,2	cyclictest	2393	-21	grep	11:35:27	1
24100	99	13	6,4	cyclictest	17209	-21	diskmemload	09:41:46	0
24100	99	13	5,1	cyclictest	6053	-21	apt-get	07:40:00	0
24100	99	13	10,2	cyclictest	8674	-21	grep	08:50:17	0
24100	99	13	10,2	cyclictest	8036	-21	cat	11:40:20	0
24100	99	13	0,8	cyclictest	11044	-21	perf	07:50:00	0
24100	99	13	0,13	cyclictest	0	-21	swapper/0	11:58:06	0
28102	2	12	0,1	sleep1	0	-21	swapper/1	10:37:08	1
24114	99	12	9,2	cyclictest	775	-21	cron	09:50:00	3
24114	99	12	9,2	cyclictest	6029	-21	cron	07:40:00	3
24114	99	12	9,2	cyclictest	3333	-21	tr	12:25:14	3
24114	99	12	9,2	cyclictest	31007	-21	cut	09:20:20	3
24114	99	12	9,2	cyclictest	30147	-21	grep	08:25:26	3
24114	99	12	9,2	cyclictest	30036	-21	grep	12:20:24	3
24114	99	12	9,2	cyclictest	23811	-21	tr	09:15:14	3
24114	99	12	9,2	cyclictest	18190	-21	grep	12:35:13	3
24114	99	12	9,2	cyclictest	16320	-21	head	09:05:25	3
24114	99	12	9,2	cyclictest	15796	-21	ls	08:00:00	3
24114	99	12	9,2	cyclictest	14140	-21	ssh	11:20:00	3
24114	99	12	9,2	cyclictest	1258	-21	grep	12:00:17	3
24114	99	12	9,2	cyclictest	10803	-21	df	08:55:17	3
24114	99	12	9,2	cyclictest	1005	-21	tr	08:35:15	3
24114	99	12	8,3	cyclictest	31567	-21	head	08:30:18	3
24114	99	12	8,3	cyclictest	31567	-21	head	08:30:18	3
24114	99	12	8,2	cyclictest	15372	-21	grep	11:20:19	3
24114	99	12	1,5	cyclictest	0	-21	swapper/3	08:40:14	3
24114	99	12	10,1	cyclictest	22394	-21	head	12:15:18	3
24114	99	12	0,11	cyclictest	19620	-21	sort	10:30:19	3
24109	99	12	9,2	cyclictest	8841	-21	grep	11:40:28	2
24109	99	12	9,2	cyclictest	7860	-21	grep	08:50:00	2
24109	99	12	9,2	cyclictest	21752	-21	sed	11:25:17	2
24109	99	12	9,2	cyclictest	16716	-21	sort	08:00:17	2
24109	99	12	9,2	cyclictest	16321	-21	tr	09:05:26	2
24109	99	12	9,2	cyclictest	16291	-21	wc	11:20:27	2
24109	99	12	9,2	cyclictest	10862	-21	grep	08:55:16	2
24109	99	12	8,3	cyclictest	32	-21	ksoftirqd/2	07:30:12	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17