OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Mon Dec 29, 2025 12:43:47)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
30007	2	98	85,9	sleep0	0	-21	swapper/0	07:07:55	0
30087	2	90	66,8	sleep2	0	-21	swapper/2	07:08:57	2
30103	2	81	68,9	sleep3	0	-21	swapper/3	07:09:09	3
29889	2	81	64,13	sleep1	0	-21	swapper/1	07:06:23	1
30265	99	29	17,12	cyclictest	0	-21	swapper/3	09:10:33	3
30265	99	21	2,19	cyclictest	20390	-21	kworker/3:0	09:15:02	3
30265	99	21	2,19	cyclictest	20390	-21	kworker/3:0	09:09:22	3
30260	99	21	0,20	cyclictest	0	-21	swapper/1	09:20:16	1
30260	99	20	4,8	cyclictest	0	-21	swapper/1	09:00:14	1
30260	99	20	19,1	cyclictest	0	-21	swapper/1	10:05:14	1
30265	99	19	10,6	cyclictest	38	1	rcuc/3	08:45:20	3
30265	99	18	16,1	cyclictest	14719	-21	apt-get	10:00:14	3
30265	99	18	14,3	cyclictest	39	-21	ksoftirqd/3	08:15:14	3
30256	99	18	5,6	cyclictest	13674	-21	munin-run	11:05:01	0
30265	99	17	3,3	cyclictest	14	1	rcu_preempt	09:32:55	3
30260	99	17	16,0	cyclictest	0	-21	swapper/1	09:50:26	1
30260	99	17	0,0	cyclictest	0	-21	swapper/1	11:05:06	1
30256	99	17	5,6	cyclictest	14692	-21	munin-run	07:44:59	0
30256	99	16	4,6	cyclictest	7325	-21	awk	09:45:00	0
30265	99	15	0,2	cyclictest	0	-21	swapper/3	10:50:12	3
30260	99	15	11,3	cyclictest	15250	-21	sh	12:15:00	1
30256	99	15	3,6	cyclictest	12060	-21	munin-run	09:54:59	0
30265	99	14	11,2	cyclictest	12973	-21	wc	11:00:28	3
30265	99	14	0,3	cyclictest	0	-21	swapper/3	07:14:57	3
30265	99	14	0,14	cyclictest	0	-21	swapper/3	10:30:17	3
30263	99	14	11,2	cyclictest	28291	-21	grep	10:25:28	2
30260	99	14	12,1	cyclictest	11286	-21	find	11:00:00	1
30260	99	14	10,2	cyclictest	29873	-21	grep	08:15:16	1
30256	99	14	10,3	cyclictest	8103	-21	cron	12:00:00	0
30265	99	13	9,3	cyclictest	29125	-21	tr	11:35:15	3
30265	99	13	4,6	cyclictest	39	-21	ksoftirqd/3	08:40:01	3
30265	99	13	4,6	cyclictest	39	-21	ksoftirqd/3	08:35:00	3
30265	99	13	3,6	cyclictest	39	-21	ksoftirqd/3	11:35:00	3
30265	99	13	1,11	cyclictest	39	-21	ksoftirqd/3	08:20:14	3
30265	99	13	10,2	cyclictest	28844	-21	wc	10:30:00	3
30265	99	13	10,2	cyclictest	24346	-21	grep	12:30:27	3
30265	99	13	10,2	cyclictest	20239	-21	sed	10:10:17	3
30265	99	13	10,2	cyclictest	17185	-21	grep	12:15:28	3
30265	99	13	0,1	cyclictest	0	-21	swapper/3	07:50:20	3
30263	99	13	9,3	cyclictest	19802	-21	grep	07:55:15	2
30263	99	13	5,4	cyclictest	225	-21	systemd-journal	09:44:59	2
30263	99	13	1,8	cyclictest	21571	-21	turbostat.cron	10:15:00	2
30263	99	13	10,2	cyclictest	5247	-21	tr	08:30:24	2
30263	99	13	10,2	cyclictest	24036	-21	cat	10:20:00	2
30263	99	13	10,2	cyclictest	21212	-21	cat	11:20:00	2
30263	99	13	10,2	cyclictest	17143	-21	grep	12:15:28	2
30263	99	13	10,2	cyclictest	13666	-21	cut	08:50:15	2
30263	99	13	10,2	cyclictest	11626	-21	grep	11:00:17	2
30263	99	13	10,2	cyclictest	11007	-21	cut	12:05:15	2
30260	99	13	9,2	cyclictest	3900	-21	tr	11:50:16	1
30260	99	13	10,2	cyclictest	666	-21	grep	07:15:14	1
30260	99	13	10,2	cyclictest	6455	-21	wc	10:50:01	1
30260	99	13	10,2	cyclictest	21677	-21	cat	10:15:00	1
30260	99	13	10,2	cyclictest	15804	-21	tr	12:15:16	1
30260	99	13	10,2	cyclictest	13631	-21	grep	08:50:15	1
30260	99	13	0,13	cyclictest	0	-21	swapper/1	08:25:06	1
30256	99	13	9,3	cyclictest	7853	-21	grep	09:45:14	0
30256	99	13	9,3	cyclictest	19642	-21	head	11:15:17	0
30256	99	13	8,3	cyclictest	1895	-21	cut	10:40:16	0
30256	99	13	10,2	cyclictest	31579	-21	wc	09:25:21	0
30256	99	13	10,2	cyclictest	27196	-21	cron	12:40:00	0
30256	99	13	10,2	cyclictest	26188	-21	cut	09:15:19	0
30256	99	13	10,2	cyclictest	20914	-21	grep	12:25:15	0
30256	99	13	10,2	cyclictest	19022	-21	grep	07:50:25	0
30256	99	13	10,2	cyclictest	1018	-21	grep	09:30:18	0
30256	99	13	0,12	cyclictest	4909	-21	grep	10:45:16	0
30256	99	13	0,12	cyclictest	27415	-21	sed	10:25:19	0
30265	99	12	9,2	cyclictest	7855	-21	tr	09:45:14	3
30265	99	12	9,2	cyclictest	7697	-21	head	11:55:26	3
30265	99	12	9,2	cyclictest	4955	-21	head	10:45:16	3
30265	99	12	9,2	cyclictest	4230	-21	cut	11:50:19	3
30265	99	12	9,2	cyclictest	38	1	rcuc/3	09:00:13	3
30265	99	12	9,2	cyclictest	3400	-21	grep	07:20:16	3
30265	99	12	9,2	cyclictest	32232	-21	cut	10:35:16	3
30265	99	12	9,2	cyclictest	2898	-21	wc	08:25:24	3
30265	99	12	9,2	cyclictest	23948	-21	apt	10:20:00	3
30265	99	12	9,2	cyclictest	23287	-21	sed	08:00:20	3
30265	99	12	9,2	cyclictest	21934	-21	tail	12:25:25	3
30265	99	12	9,2	cyclictest	19132	-21	tr	11:15:13	3
30265	99	12	9,2	cyclictest	18649	-21	grep	12:20:17	3
30265	99	12	9,2	cyclictest	14258	-21	cut	12:10:20	3
30265	99	12	9,2	cyclictest	13641	-21	wc	07:40:24	3
30265	99	12	8,3	cyclictest	10693	-21	wc	12:05:00	3
30265	99	12	0,8	cyclictest	0	-21	swapper/3	09:50:14	3
30265	99	12	0,11	cyclictest	2922	-21	grep	10:40:21	3
30263	99	12	9,2	cyclictest	6297	-21	cron	10:50:01	2
30263	99	12	9,2	cyclictest	4513	-21	grep	07:20:27	2
30263	99	12	9,2	cyclictest	3923	-21	cron	10:45:00	2
30263	99	12	9,2	cyclictest	3352	-21	grep	09:35:16	2
30263	99	12	9,2	cyclictest	31480	-21	grep	09:25:22	2
30263	99	12	9,2	cyclictest	31009	-21	sh	11:39:59	2
30263	99	12	9,2	cyclictest	27172	-21	cut	08:10:17	2
30263	99	12	9,2	cyclictest	25402	-21	uname	08:05:19	2
30263	99	12	9,2	cyclictest	24591	-21	grep	11:25:19	2
30263	99	12	9,2	cyclictest	24507	-21	grep	10:20:12	2
30263	99	12	9,2	cyclictest	23765	-21	tail	08:00:25	2
30263	99	12	9,2	cyclictest	21515	-21	tr	11:20:15	2
30263	99	12	9,2	cyclictest	18844	-21	grep	09:00:15	2
30263	99	12	9,2	cyclictest	16150	-21	tr	08:55:16	2
30263	99	12	9,2	cyclictest	16150	-21	tr	08:55:16	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17