OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot4s.osadl.org (updated Mon May 25, 2026 00:45:15)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
9507	99	88	49,25	cyclictest	359	66	irq/58-eth0	19:35:14	3
9507	99	60	48,6	cyclictest	359	66	irq/58-eth0	23:05:11	3
9507	99	46	7,20	cyclictest	359	66	irq/58-eth0	00:15:14	3
9507	99	46	28,9	cyclictest	359	66	irq/58-eth0	21:55:12	3
9507	99	45	4,28	cyclictest	359	66	irq/58-eth0	21:25:13	3
9507	99	45	1,31	cyclictest	359	66	irq/58-eth0	19:25:13	3
9507	99	44	5,25	cyclictest	359	66	irq/58-eth0	23:45:12	3
9507	99	43	3,25	cyclictest	359	66	irq/58-eth0	22:50:15	3
9507	99	42	3,23	cyclictest	359	66	irq/58-eth0	00:20:13	3
9507	99	42	3,21	cyclictest	359	66	irq/58-eth0	21:05:13	3
9507	99	41	2,24	cyclictest	359	66	irq/58-eth0	21:30:12	3
9496	99	38	9,9	cyclictest	5294	-21	latency_hist	22:45:02	2
9496	99	38	9,9	cyclictest	5294	-21	latency_hist	22:45:01	2
9496	99	37	1,9	cyclictest	15519	-21	fschecks_time	00:20:18	2
9496	99	37	0,13	cyclictest	31536	66	ssh	23:51:24	2
9488	99	37	1,29	cyclictest	31912	-21	rm	21:30:12	1
9488	99	37	1,22	cyclictest	28083	-21	ssh	21:10:15	1
9488	99	37	0,30	cyclictest	2534	-21	ssh	23:55:12	1
9476	99	37	1,13	cyclictest	6255	-21	apt-get	00:15:13	0
9488	99	36	1,17	cyclictest	10917	-21	apt-get	19:45:12	1
9507	99	35	4,21	cyclictest	359	66	irq/58-eth0	23:40:13	3
9488	99	35	1,6	cyclictest	29022	-21	apt-get	22:40:12	1
9488	99	35	1,6	cyclictest	29022	-21	apt-get	22:40:12	1
9488	99	35	1,12	cyclictest	26491	-21	ntp_192.168.114	21:25:24	1
9488	99	35	0,10	cyclictest	5827	-21	ssh	00:30:36	1
9476	99	35	9,5	cyclictest	0	-21	swapper/0	20:45:27	0
9488	99	34	8,23	cyclictest	13773	-21	basename	00:35:30	1
9488	99	34	8,23	cyclictest	13773	-21	basename	00:35:29	1
9488	99	34	1,9	cyclictest	9880	-21	ntp_192.168.114	00:15:25	1
9488	99	34	1,7	cyclictest	21989	-21	missed_timers	23:10:23	1
9488	99	34	1,11	cyclictest	31360	-21	/usr/sbin/munin	23:35:18	1
9488	99	34	1,10	cyclictest	4825	-21	df_inode	21:15:17	1
9488	99	34	0,6	cyclictest	18859	-21	kthreadcore	19:50:23	1
9488	99	34	0,6	cyclictest	18859	-21	kthreadcore	19:50:22	1
9507	99	33	4,21	cyclictest	359	66	irq/58-eth0	19:55:14	3
9507	99	33	19,8	cyclictest	359	66	irq/58-eth0	22:20:14	3
9488	99	33	6,4	cyclictest	0	-21	swapper/1	23:50:22	1
9488	99	33	1,8	cyclictest	27133	-21	ssh	22:38:12	1
9488	99	33	1,8	cyclictest	15914	-21	cut	19:15:19	1
9488	99	33	1,7	cyclictest	31178	-21	ntp_states	23:15:27	1
9488	99	33	1,7	cyclictest	18211	-21	ssh	22:33:18	1
9488	99	33	1,7	cyclictest	12029	-21	ntp_states	20:50:26	1
9488	99	33	1,5	cyclictest	16079	-21	ntp_states	21:55:26	1
9488	99	33	1,3	cyclictest	12157	-21	apt-get	20:20:14	1
9476	99	33	14,3	cyclictest	13	1	rcu_preempt	00:10:11	0
9476	99	33	1,13	cyclictest	6230	-21	ntp_states	23:55:26	0
9476	99	33	1,10	cyclictest	16868	-21	ssh	23:45:12	0
9507	99	32	2,18	cyclictest	359	66	irq/58-eth0	20:40:13	3
9507	99	32	1,21	cyclictest	359	66	irq/58-eth0	23:25:12	3
9496	99	32	0,11	cyclictest	930	-21	snmpd	19:30:13	2
9488	99	32	8,21	cyclictest	27636	-21	diskmemload	22:00:27	1
9488	99	32	1,9	cyclictest	26539	-21	kthreadcore	22:55:21	1
9488	99	32	1,9	cyclictest	23169	-21	df	21:05:16	1
9488	99	32	1,8	cyclictest	27580	-21	ntp_states	19:25:26	1
9488	99	32	1,8	cyclictest	24331	-21	ntp_states	00:05:30	1
9488	99	32	1,8	cyclictest	22391	-21	ssh	23:46:16	1
9488	99	32	1,8	cyclictest	17528	-21	cpu	21:40:13	1
9488	99	32	1,7	cyclictest	9841	-21	kthreadcore	21:35:20	1
9488	99	32	1,7	cyclictest	7824	-21	proc_pri	23:20:26	1
9488	99	32	1,7	cyclictest	21683	-21	nfs_client	21:00:23	1
9488	99	32	1,6	cyclictest	6483	-21	kthreadcore	22:45:20	1
9488	99	32	1,6	cyclictest	3072	-21	cut	19:35:21	1
9488	99	32	1,6	cyclictest	24635	-21	cut	23:30:22	1
9488	99	32	1,6	cyclictest	24331	-21	unixbench_multi	19:55:28	1
9488	99	32	1,6	cyclictest	18243	-21	tr	22:15:20	1
9488	99	32	1,6	cyclictest	14187	-21	cut	21:20:18	1
9488	99	32	1,6	cyclictest	14187	-21	cut	21:20:17	1
9488	99	32	1,5	cyclictest	9204	-21	cat	23:05:01	1
9488	99	32	1,5	cyclictest	7049	-21	memory	20:45:23	1
9488	99	32	1,4	cyclictest	1866	-21	ntp_192.168.114	22:05:25	1
9476	99	32	8,5	cyclictest	14326	-21	tr	21:55:21	0
9476	99	32	11,3	cyclictest	16965	-21	apt	20:25:12	0
9507	99	31	3,13	cyclictest	359	66	irq/58-eth0	00:05:16	3
9507	99	31	1,15	cyclictest	359	66	irq/58-eth0	20:10:12	3
9488	99	31	1,8	cyclictest	26623	-21	df_inode	20:35:15	1
9488	99	31	1,7	cyclictest	4909	-21	tr	21:50:22	1
9488	99	31	1,6	cyclictest	7150	-21	/usr/sbin/munin	22:10:12	1
9488	99	31	1,6	cyclictest	32137	-21	ssh	21:46:20	1
9488	99	31	1,6	cyclictest	27297	-21	cut	00:25:19	1
9488	99	31	1,6	cyclictest	2604	-21	cut	22:25:18	1
9488	99	31	1,6	cyclictest	21254	-21	ssh	00:21:52	1
9488	99	31	1,6	cyclictest	15469	-21	ssh	22:50:21	1
9488	99	31	1,6	cyclictest	13502	-21	grep	23:05:27	1
9488	99	31	1,6	cyclictest	13417	-21	cpuspeed	20:55:14	1
9488	99	31	1,6	cyclictest	11585	-21	ssh	00:00:15	1
9488	99	31	1,6	cyclictest	11585	-21	ssh	00:00:14	1
9488	99	31	1,5	cyclictest	9616	-21	kthreadcore	23:40:20	1
9488	99	31	0,7	cyclictest	21782	-21	apt-get	20:30:14	1
9476	99	31	11,3	cyclictest	12707	-21	latency_hist	23:25:02	0
9476	99	31	1,11	cyclictest	25287	66	ssh	22:20:13	0
9476	99	31	1,10	cyclictest	12091	-21	ntp_192.168.114	21:35:25	0
9496	99	30	12,13	cyclictest	628	-21	tcf-agent	23:10:12	2
9488	99	30	1,8	cyclictest	29652	-21	cpu	00:10:14	1
9488	99	30	1,7	cyclictest	9641	-21	awk	19:10:13	1
9488	99	30	1,7	cyclictest	3497	-21	idleruntime-cro	20:45:01	1
9488	99	30	1,6	cyclictest	7127	-21	cut	20:15:01	1
9488	99	30	1,6	cyclictest	30569	-21	ssh	22:21:00	1
9488	99	30	1,6	cyclictest	19872	-21	cut	19:20:18	1
9488	99	30	1,6	cyclictest	19872	-21	cut	19:20:17	1
9488	99	30	1,6	cyclictest	11924	-21	cut	20:20:01	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17