OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot4s.osadl.org (updated Wed Dec 24, 2025 00:45:14)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
30292	99	53	33,15	cyclictest	283	66	irq/58-eth0	21:55:13	3
30279	99	53	2,28	cyclictest	28337	-21	ssh	21:35:12	2
30279	99	51	20,26	cyclictest	27	-21	ksoftirqd/2	21:20:13	2
30292	99	50	34,11	cyclictest	283	66	irq/58-eth0	00:35:11	3
30292	99	50	24,21	cyclictest	283	66	irq/58-eth0	22:00:13	3
30268	99	43	2,11	cyclictest	24582	-21	apt-get	22:10:14	1
30259	99	43	1,14	cyclictest	12336	-21	apt-get	19:25:03	0
30279	99	42	1,18	cyclictest	0	-21	swapper/2	19:25:14	2
30279	99	42	0,27	cyclictest	0	-21	swapper/2	21:45:12	2
30279	99	42	0,27	cyclictest	0	-21	swapper/2	21:45:12	2
30292	99	41	4,26	cyclictest	283	66	irq/58-eth0	21:15:12	3
30292	99	41	2,27	cyclictest	283	66	irq/58-eth0	23:40:13	3
30279	99	41	3,25	cyclictest	30717	-21	ssh	21:55:13	2
30279	99	41	20,17	cyclictest	10882	-21	apt-get	21:25:12	2
30268	99	41	2,4	cyclictest	0	-21	swapper/1	20:30:13	1
30259	99	41	2,36	cyclictest	23957	-21	grep	20:10:20	0
30268	99	40	3,6	cyclictest	0	-21	swapper/1	20:35:14	1
30292	99	39	3,21	cyclictest	283	66	irq/58-eth0	00:05:14	3
30268	99	39	1,35	cyclictest	26904	-21	kthreadcore	23:25:18	1
30292	99	38	6,20	cyclictest	0	-21	swapper/3	22:25:14	3
30279	99	38	3,14	cyclictest	30245	-21	cyclictest	20:50:13	2
30279	99	38	2,28	cyclictest	28374	-21	tr	21:15:20	2
30268	99	38	3,32	cyclictest	28692	-21	kthreadcore	22:30:22	1
30268	99	38	3,32	cyclictest	28692	-21	kthreadcore	22:30:21	1
30268	99	38	1,5	cyclictest	14722	-21	ssh	23:00:15	1
30259	99	38	3,32	cyclictest	16057	-21	ps	22:40:27	0
30292	99	37	6,22	cyclictest	283	66	irq/58-eth0	22:05:17	3
30279	99	37	2,18	cyclictest	6185	-21	tr	21:40:19	2
30259	99	37	2,32	cyclictest	7497	-21	hwlatdetect	21:00:22	0
30268	99	36	3,11	cyclictest	13669	-21	kthreadcore	22:40:21	1
30259	99	36	2,30	cyclictest	22712	-21	tail	19:35:17	0
30259	99	36	1,32	cyclictest	26722	-21	cat	00:00:32	0
30259	99	36	1,11	cyclictest	11981	-21	ssh	23:15:23	0
30268	99	35	1,13	cyclictest	10193	-21	/usr/sbin/munin	00:30:16	1
30268	99	35	1,10	cyclictest	12704	-21	cpuspeed	19:25:17	1
30259	99	35	4,27	cyclictest	4963	-21	tr	21:20:22	0
30292	99	34	16,10	cyclictest	0	-21	swapper/3	20:15:14	3
30279	99	34	3,9	cyclictest	0	-21	swapper/2	22:12:20	2
30279	99	34	3,12	cyclictest	30245	-21	cyclictest	21:10:15	2
30279	99	34	11,4	cyclictest	0	-21	swapper/2	23:45:17	2
30268	99	34	5,7	cyclictest	26362	-21	irqrtprio	21:15:18	1
30268	99	34	1,10	cyclictest	3178	-21	mailstats	19:45:32	1
30268	99	34	1,10	cyclictest	26255	-21	grep	00:20:20	1
30259	99	34	3,3	cyclictest	13	1	rcu_preempt	20:15:15	0
30259	99	34	2,9	cyclictest	754	-21	kthreadcore	20:50:22	0
30259	99	34	1,9	cyclictest	25139	-21	latency_hist	20:45:02	0
30259	99	34	1,9	cyclictest	13381	-21	proc_pri	22:20:27	0
30259	99	34	1,8	cyclictest	6604	-21	kthreadcore	20:25:19	0
30259	99	34	1,8	cyclictest	28659	-21	/usr/sbin/munin	23:45:16	0
30259	99	34	1,8	cyclictest	22425	-21	ntp_192.168.114	20:05:23	0
30259	99	34	1,8	cyclictest	20262	-21	kthreadcore	22:25:21	0
30259	99	34	1,8	cyclictest	14142	-21	ntp_192.168.114	20:30:25	0
30259	99	34	1,8	cyclictest	13953	-21	sed	00:30:30	0
30259	99	34	1,6	cyclictest	28513	-21	memory	20:45:22	0
30279	99	33	4,20	cyclictest	223	46	irq/40-dwc2_hso	00:20:14	2
30279	99	33	22,8	cyclictest	15543	-21	apt-get	20:35:14	2
30279	99	33	19,10	cyclictest	7272	-21	apt-get	21:00:17	2
30279	99	33	11,17	cyclictest	0	-21	swapper/2	22:18:24	2
30268	99	33	6,5	cyclictest	22908	-21	cut	00:00:21	1
30268	99	33	3,27	cyclictest	32514	-21	cut	00:05:22	1
30268	99	33	1,9	cyclictest	31836	-21	awk	23:10:01	1
30268	99	33	1,9	cyclictest	29278	-21	sed	23:45:20	1
30268	99	33	1,9	cyclictest	24545	-21	sendmail	00:20:02	1
30268	99	33	1,9	cyclictest	18656	-21	ssh	00:35:13	1
30268	99	33	1,9	cyclictest	13553	-21	sort	21:45:16	1
30268	99	33	1,9	cyclictest	13553	-21	sort	21:45:15	1
30268	99	33	1,9	cyclictest	1057	-21	/usr/sbin/munin	20:20:29	1
30268	99	33	1,8	cyclictest	5142	-21	diskstats	23:50:16	1
30268	99	33	1,8	cyclictest	5142	-21	diskstats	23:50:15	1
30268	99	33	1,8	cyclictest	27482	-21	ssh	23:43:59	1
30268	99	33	1,8	cyclictest	17175	-21	kthreadcore	22:05:22	1
30268	99	33	1,13	cyclictest	29444	-21	kthreadcore	21:35:19	1
30268	99	33	1,11	cyclictest	5132	-21	kthreadcore	20:55:20	1
30268	99	33	1,11	cyclictest	5132	-21	kthreadcore	20:55:19	1
30268	99	33	1,10	cyclictest	8186	-21	kthreadcore	22:00:19	1
30268	99	33	1,10	cyclictest	31771	-21	kthreadcore	22:50:21	1
30259	99	33	7,4	cyclictest	14828	-21	kthreadcore	20:00:20	0
30259	99	33	2,11	cyclictest	4735	-21	kthreadcore	19:15:21	0
30259	99	33	1,9	cyclictest	4948	-21	ssh	21:40:16	0
30259	99	33	1,9	cyclictest	4556	-21	kthreadcore	23:30:22	0
30259	99	33	1,9	cyclictest	20820	-21	/usr/sbin/munin	19:30:22	0
30259	99	33	1,9	cyclictest	16492	-21	fschecks_time	00:15:16	0
30259	99	33	1,9	cyclictest	14839	-21	df	23:00:19	0
30259	99	33	1,8	cyclictest	835	-21	kthreadcore	23:10:22	0
30259	99	33	1,8	cyclictest	3175	-21	kthreadcore	20:20:24	0
30259	99	33	1,8	cyclictest	2118	-21	latency_hist	23:30:02	0
30259	99	33	1,8	cyclictest	19752	-21	cat	22:05:30	0
30259	99	33	1,7	cyclictest	755	-21	tr	19:10:23	0
30259	99	33	1,7	cyclictest	5090	-21	kthreadcore	19:50:21	0
30259	99	33	1,7	cyclictest	18015	-21	ps	23:20:19	0
30259	99	33	1,7	cyclictest	17998	-21	cut	21:10:21	0
30259	99	33	1,13	cyclictest	5989	-21	usage_EthcTask	00:25:32	0
30259	99	33	1,10	cyclictest	13994	-21	kthreadcore	21:05:21	0
30279	99	32	9,7	cyclictest	27	-21	ksoftirqd/2	20:15:15	2
30279	99	32	17,10	cyclictest	22394	-21	apt-get	19:35:12	2
30268	99	32	2,11	cyclictest	3153	-21	kthreadcore	00:25:22	1
30268	99	32	1,9	cyclictest	9617	-21	ssh	22:20:13	1
30268	99	32	1,9	cyclictest	32760	-21	ssh	23:10:20	1
30268	99	32	1,9	cyclictest	23334	-21	ssh	21:30:28	1
30268	99	32	1,8	cyclictest	22331	-21	cut	22:45:21	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17