OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Tue Dec 30, 2025 00:43:46)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
31092	2	92	42,45	sleep1	0	-21	swapper/1	19:05:42	1
31362	2	90	63,9	sleep2	0	-21	swapper/2	19:09:12	2
31397	2	88	75,9	sleep0	0	-21	swapper/0	19:09:38	0
31104	2	85	60,9	sleep3	0	-21	swapper/3	19:05:52	3
31519	99	18	2,15	cyclictest	18100	-21	apt-get	19:50:02	3
31511	99	18	0,17	cyclictest	0	-21	swapper/1	21:35:15	1
31509	99	18	14,3	cyclictest	13	-21	ksoftirqd/0	21:35:14	0
31511	99	16	0,0	cyclictest	0	-21	swapper/1	21:57:01	1
31511	99	15	1,1	cyclictest	24936	-21	grep	20:00:25	1
31519	99	14	11,2	cyclictest	13703	-21	head	23:00:17	3
31519	99	14	10,3	cyclictest	23045	-21	cron	20:00:00	3
31519	99	14	10,3	cyclictest	18342	-21	wc	22:00:26	3
31515	99	14	11,2	cyclictest	29068	-21	turbostat.cron	00:40:00	2
31515	99	14	10,3	cyclictest	21818	-21	grep	23:15:28	2
31515	99	14	10,3	cyclictest	17588	-21	grep	19:45:25	2
31515	99	14	10,2	cyclictest	11455	-21	tr	19:35:17	2
31511	99	14	9,3	cyclictest	14287	-21	cat	21:55:01	1
31511	99	14	2,8	cyclictest	27054	-21	apache2	20:05:14	1
31511	99	14	10,3	cyclictest	19956	-21	sed	22:05:19	1
31509	99	14	9,2	cyclictest	21659	-21	tail	19:55:17	0
31509	99	14	7,5	cyclictest	13	-21	ksoftirqd/0	21:15:19	0
31509	99	14	10,2	cyclictest	17426	-21	cut	22:00:17	0
31509	99	14	10,2	cyclictest	13494	-21	cat	19:40:00	0
31509	99	14	0,7	cyclictest	0	-21	swapper/0	22:35:14	0
31519	99	13	9,3	cyclictest	27492	-21	sed	20:05:27	3
31519	99	13	9,3	cyclictest	11706	-21	wc	20:40:23	3
31519	99	13	9,2	cyclictest	5179	-21	tr	21:35:18	3
31519	99	13	11,1	cyclictest	28152	-21	apt-get	20:10:15	3
31519	99	13	10,2	cyclictest	31848	-21	grep	22:30:18	3
31519	99	13	10,2	cyclictest	26993	-21	latency_hist	23:30:00	3
31519	99	13	10,2	cyclictest	23764	-21	cat	22:15:00	3
31519	99	13	10,2	cyclictest	17762	-21	sed	00:15:19	3
31515	99	13	9,3	cyclictest	8086	-21	grep	19:25:28	2
31515	99	13	9,3	cyclictest	6449	-21	cut	20:30:15	2
31515	99	13	10,2	cyclictest	19864	-21	perf	23:15:00	2
31515	99	13	10,2	cyclictest	15240	-21	sort	00:10:18	2
31515	99	13	10,2	cyclictest	12209	-21	grep	22:55:28	2
31515	99	13	10,2	cyclictest	11193	-21	grep	20:40:18	2
31515	99	13	0,2	cyclictest	0	-21	swapper/2	23:30:16	2
31515	99	13	0,12	cyclictest	0	-21	swapper/2	23:40:14	2
31511	99	13	9,3	cyclictest	27604	-21	tr	21:15:15	1
31511	99	13	9,3	cyclictest	21786	-21	tr	22:10:15	1
31511	99	13	9,3	cyclictest	21216	-21	grep	19:55:15	1
31511	99	13	9,3	cyclictest	15942	-21	cut	20:50:17	1
31511	99	13	1,7	cyclictest	381	50	irq/131-snd_hda	00:05:14	1
31511	99	13	10,2	cyclictest	24542	-21	kworker/1:1	23:25:20	1
31509	99	13	9,3	cyclictest	23276	-21	sed	23:20:18	0
31509	99	13	9,3	cyclictest	2007	-21	sleep	23:45:00	0
31509	99	13	9,3	cyclictest	18082	-21	cron	19:50:00	0
31509	99	13	8,3	cyclictest	4471	-21	wc	22:40:19	0
31509	99	13	3,9	cyclictest	32682	-21	sendmail-msp	20:20:00	0
31509	99	13	10,2	cyclictest	8956	-21	head	22:50:17	0
31509	99	13	10,2	cyclictest	22241	-21	grep	00:25:15	0
31509	99	13	10,2	cyclictest	18455	-21	tail	23:10:17	0
31509	99	13	10,2	cyclictest	15961	-21	grep	20:50:17	0
31509	99	13	10,2	cyclictest	15106	-21	dump-pmu-power	23:05:00	0
31509	99	13	10,2	cyclictest	13694	-21	grep	21:50:25	0
31519	99	12	9,2	cyclictest	8829	-21	grep	23:55:29	3
31519	99	12	9,2	cyclictest	8594	-21	basename	19:30:00	3
31519	99	12	9,2	cyclictest	7107	-21	grep	22:45:22	3
31519	99	12	9,2	cyclictest	3667	-21	awk	21:30:22	3
31519	99	12	9,2	cyclictest	29853	-21	grep	22:25:21	3
31519	99	12	9,2	cyclictest	24210	-21	sed	23:20:27	3
31519	99	12	9,2	cyclictest	2111	-21	cat	21:30:00	3
31519	99	12	9,2	cyclictest	18436	-21	cut	23:10:17	3
31519	99	12	8,3	cyclictest	2959	-21	wc	19:15:23	3
31519	99	12	8,3	cyclictest	20828	-21	grep	23:15:18	3
31519	99	12	8,3	cyclictest	19203	-21	sed	20:55:27	3
31519	99	12	8,3	cyclictest	13048	-21	wc	00:05:20	3
31519	99	12	8,2	cyclictest	598	-21	head	23:40:17	3
31519	99	12	8,2	cyclictest	27103	-21	head	22:20:19	3
31519	99	12	8,2	cyclictest	10704	-21	grep	22:55:15	3
31519	99	12	8,2	cyclictest	10426	-21	grep	21:45:15	3
31519	99	12	3,4	cyclictest	4642	-21	apt-get	23:50:14	3
31519	99	12	3,4	cyclictest	4642	-21	apt-get	23:50:14	3
31519	99	12	0,1	cyclictest	0	-21	swapper/3	22:50:14	3
31515	99	12	9,2	cyclictest	9557	-21	head	19:30:18	2
31515	99	12	9,2	cyclictest	9337	-21	awk	00:00:00	2
31515	99	12	9,2	cyclictest	5893	-21	wc	23:50:22	2
31515	99	12	9,2	cyclictest	5893	-21	wc	23:50:22	2
31515	99	12	9,2	cyclictest	4029	-21	df	22:40:16	2
31515	99	12	9,2	cyclictest	30362	-21	cat	20:15:00	2
31515	99	12	9,2	cyclictest	26600	-21	tr	22:20:17	2
31515	99	12	9,2	cyclictest	24947	-21	sed	20:00:25	2
31515	99	12	9,2	cyclictest	24637	-21	basename	23:25:00	2
31515	99	12	9,2	cyclictest	24466	-21	awk	21:10:00	2
31515	99	12	9,2	cyclictest	23651	-21	cron	22:15:00	2
31515	99	12	9,2	cyclictest	14869	-21	wc	19:40:21	2
31515	99	12	9,2	cyclictest	12539	-21	grep	00:05:16	2
31515	99	12	9,2	cyclictest	12171	-21	grep	21:50:14	2
31515	99	12	8,2	cyclictest	3771	-21	cut	20:25:18	2
31515	99	12	3,2	cyclictest	0	-21	swapper/2	22:05:25	2
31515	99	12	0,11	cyclictest	0	-21	swapper/2	00:25:14	2
31511	99	12	9,2	cyclictest	20206	-21	grep	19:50:27	1
31511	99	12	9,2	cyclictest	1752	-21	wc	20:20:20	1
31511	99	12	9,2	cyclictest	15907	-21	wc	19:45:00	1
31511	99	12	9,2	cyclictest	1553	-21	grep	21:25:28	1
31511	99	12	9,2	cyclictest	15473	-21	tr	23:05:14	1
31511	99	12	8,3	cyclictest	27798	-21	wc	22:20:28	1
31511	99	12	8,3	cyclictest	1446	-21	perf	19:15:01	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17