OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of all highest latencies:

System rackeslot6s.osadl.org (updated Wed Apr 01, 2026 00:44:46)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2090004	99	82	1,40	cyclictest	2119041	-21	apt-get	22:00:02	3
2089997	99	81	10,14	cyclictest	2144797	-21	apt-get	00:35:02	0
2089997	99	79	2,5	cyclictest	2111233	-21	tail	21:10:27	0
2089997	99	75	6,2	cyclictest	0	-21	swapper/0	19:55:01	0
2089997	99	71	35,32	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:05:20	0
2089997	99	70	32,30	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	00:35:18	0
2089997	99	70	2,3	cyclictest	2137376	-21	meminfo	23:45:24	0
2089997	99	69	1,38	cyclictest	2137064	-21	latency_hist	23:45:01	0
2089997	99	68	31,30	cyclictest	208	50	irq/208-8000000.ethernet	23:05:00	0
2090001	99	67	61,4	cyclictest	23	1	ktimers/1	22:50:00	1
2089997	99	67	2,8	cyclictest	2127984	-21	sshd	22:50:21	0
2089997	99	67	2,8	cyclictest	2127984	-21	sshd	22:50:21	0
2089997	99	67	1,2	cyclictest	0	-21	swapper/0	21:50:20	0
2089997	99	65	36,26	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:15:17	0
2089997	99	65	3,5	cyclictest	2133553	-21	sed	23:25:02	0
2089997	99	65	32,29	cyclictest	0	-21	swapper/0	20:05:29	0
2089997	99	65	29,9	cyclictest	0	-21	swapper/0	19:35:22	0
2089997	99	65	24,10	cyclictest	0	-21	swapper/0	23:25:21	0
2089997	99	64	28,8	cyclictest	0	-21	swapper/0	19:10:29	0
2089997	99	64	28,7	cyclictest	0	-21	swapper/0	00:15:28	0
2089997	99	64	1,37	cyclictest	2143076	-21	users	00:20:27	0
2089997	99	63	28,31	cyclictest	0	-21	swapper/0	21:15:28	0
2089997	99	63	2,4	cyclictest	2102874	-21	mailstats	20:20:29	0
2089997	99	63	2,4	cyclictest	2102874	-21	mailstats	20:20:29	0
2089997	99	61	37,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	00:25:13	0
2089997	99	61	1,4	cyclictest	2110032	-21	df_inode	21:05:15	0
2089997	99	61	1,10	cyclictest	2115354	-21	tail	21:35:26	0
2089997	99	58	25,7	cyclictest	0	-21	swapper/0	22:30:27	0
2089997	99	58	22,8	cyclictest	0	-21	swapper/0	22:00:28	0
2089997	99	57	26,7	cyclictest	0	-21	swapper/0	20:25:30	0
2089997	99	56	33,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:35:29	0
2089997	99	56	33,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:35:29	0
2089997	99	56	24,8	cyclictest	0	-21	swapper/0	21:30:28	0
2089997	99	55	30,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:00:14	0
2089997	99	55	30,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:00:14	0
2090002	99	53	44,5	cyclictest	2137716	-21	munin-plugin-st	23:50:00	2
2090002	99	53	24,18	cyclictest	2120186	-21	/usr/sbin/munin	22:05:20	2
2089997	99	52	44,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:25:19	0
2089997	99	52	39,10	cyclictest	2138555	-21	munin-plugin-st	23:55:00	0
2090001	99	51	39,9	cyclictest	23	1	ktimers/1	22:25:14	1
2090001	99	51	12,6	cyclictest	14	-21	rcu_preempt	19:55:01	1
2089997	99	51	34,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:40:19	0
2090001	99	50	36,10	cyclictest	2115669	-21	sendmail	21:40:00	1
2089997	99	50	42,4	cyclictest	0	-21	swapper/0	21:55:13	0
2089997	99	48	41,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:35:15	0
2089997	99	48	41,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:45:22	0
2089997	99	47	40,5	cyclictest	2097979	-21	latency_hist	19:55:01	0
2089997	99	47	39,5	cyclictest	2129266	-21	munin-plugin-st	22:59:59	0
2090001	99	46	33,8	cyclictest	0	-21	swapper/1	20:15:01	1
2090001	99	46	26,16	cyclictest	23	1	ktimers/1	22:30:10	1
2089997	99	46	39,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:20:27	0
2090001	99	45	28,11	cyclictest	23	1	ktimers/1	19:20:17	1
2089997	99	45	1,9	cyclictest	0	-21	swapper/0	19:10:00	0
2090002	99	44	35,6	cyclictest	0	-21	swapper/2	20:35:28	2
2090002	99	44	1,17	cyclictest	0	-21	swapper/2	00:20:02	2
2090001	99	44	32,6	cyclictest	0	-21	swapper/1	23:20:26	1
2090001	99	44	23,14	cyclictest	23	1	ktimers/1	23:55:13	1
2090001	99	44	12,26	cyclictest	2129710	-21	if_eth0	23:00:24	1
2089997	99	44	39,3	cyclictest	2126744	-21	munin-plugin-st	22:45:00	0
2089997	99	44	12,21	cyclictest	2115882	-21	/usr/sbin/munin	21:40:18	0
2090002	99	43	5,12	cyclictest	0	-21	swapper/2	20:30:01	2
2090001	99	43	36,4	cyclictest	0	-21	swapper/1	23:10:02	1
2090001	99	43	32,7	cyclictest	2117363	-21	munin-plugin-st	21:50:00	1
2090001	99	43	28,10	cyclictest	0	-21	swapper/1	23:25:28	1
2089997	99	43	1,6	cyclictest	12	-21	ksoftirqd/0	20:30:18	0
2090002	99	42	28,8	cyclictest	0	-21	swapper/2	23:20:28	2
2090002	99	42	25,11	cyclictest	0	-21	swapper/2	21:35:19	2
2090001	99	42	6,3	cyclictest	0	-21	swapper/1	19:35:00	1
2090001	99	42	34,4	cyclictest	0	-21	swapper/1	20:55:17	1
2090001	99	42	28,7	cyclictest	0	-21	swapper/1	19:10:00	1
2090001	99	42	20,16	cyclictest	23	1	ktimers/1	23:50:11	1
2089997	99	42	34,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:15:25	0
2089997	99	42	29,9	cyclictest	0	-21	swapper/0	19:30:29	0
2089997	99	42	26,12	cyclictest	0	-21	swapper/0	20:00:28	0
2089997	99	42	2,5	cyclictest	2130624	-21	/usr/sbin/munin	23:05:25	0
2090002	99	41	31,5	cyclictest	0	-21	swapper/2	00:30:28	2
2090002	99	41	26,10	cyclictest	0	-21	swapper/2	23:35:28	2
2090002	99	41	1,2	cyclictest	0	-21	swapper/2	22:30:01	2
2090001	99	41	31,7	cyclictest	2111394	-21	unixbench_singl	21:10:29	1
2090001	99	41	30,5	cyclictest	0	-21	swapper/1	20:35:28	1
2090001	99	41	29,8	cyclictest	0	-21	swapper/1	22:35:28	1
2090001	99	41	29,8	cyclictest	0	-21	swapper/1	22:35:27	1
2090001	99	41	29,7	cyclictest	0	-21	swapper/1	22:10:29	1
2090001	99	41	23,8	cyclictest	0	-21	swapper/1	21:55:29	1
2089997	99	41	2,4	cyclictest	2109024	-21	cat	21:00:01	0
2089997	99	41	2,4	cyclictest	2107685	-21	irqstats	20:50:23	0
2090002	99	40	32,5	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	20:20:00	2
2090002	99	40	31,5	cyclictest	2136787	-21	grep	23:40:30	2
2090002	99	40	30,7	cyclictest	2096062	-21	unixbench_multi	19:40:29	2
2090001	99	40	29,8	cyclictest	0	-21	swapper/1	21:45:01	1
2090001	99	40	28,8	cyclictest	0	-21	swapper/1	00:30:28	1
2090001	99	40	27,8	cyclictest	0	-21	swapper/1	23:15:26	1
2090001	99	40	24,8	cyclictest	0	-21	swapper/1	21:20:29	1
2089997	99	40	35,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:30:15	0
2089997	99	40	35,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:30:15	0
2089997	99	40	34,2	cyclictest	0	-21	swapper/0	19:40:11	0
2089997	99	40	26,10	cyclictest	0	-21	swapper/0	00:00:28	0
2089997	99	40	2,3	cyclictest	2127202	-21	irqstats	22:45:23	0
2090002	99	39	30,5	cyclictest	0	-21	swapper/2	22:44:59	2
2090002	99	39	29,5	cyclictest	0	-21	swapper/2	19:30:28	2
2090002	99	39	28,8	cyclictest	0	-21	swapper/2	23:20:00	2
2090002	99	39	25,8	cyclictest	2124984	-21	grep	22:30:26	2
2090002	99	39	15,6	cyclictest	2109124	-21	/usr/sbin/munin	21:00:21	2
2090002	99	39	15,6	cyclictest	2109124	-21	/usr/sbin/munin	21:00:21	2
2090001	99	39	30,5	cyclictest	0	-21	swapper/1	22:05:29	1
2090001	99	39	25,7	cyclictest	0	-21	swapper/1	00:20:28	1
2090001	99	39	22,8	cyclictest	0	-21	swapper/1	20:20:20	1
2090001	99	39	22,8	cyclictest	0	-21	swapper/1	20:20:20	1
2090001	99	39	1,4	cyclictest	0	-21	swapper/1	20:20:00	1
2089997	99	39	24,10	cyclictest	0	-21	swapper/0	19:20:31	0
2089997	99	39	1,9	cyclictest	0	-21	swapper/0	21:30:00	0
2089997	99	39	1,36	cyclictest	0	-21	swapper/0	22:23:21	0
2089997	99	39	1,36	cyclictest	0	-21	swapper/0	21:45:17	0
2089997	99	39	1,36	cyclictest	0	-21	swapper/0	00:10:13	0
2090002	99	38	29,6	cyclictest	2143042	-21	unixbench_singl	00:20:28	2
2090001	99	38	28,6	cyclictest	2093526	-21	unixbench_multi	19:25:29	1
2090001	99	38	26,8	cyclictest	2090709	-21	grep	19:10:30	1
2090001	99	38	25,9	cyclictest	0	-21	swapper/1	00:05:01	1
2090001	99	38	1,8	cyclictest	0	-21	swapper/1	20:30:00	1
2089997	99	38	1,2	cyclictest	0	-21	swapper/0	20:15:02	0
2090002	99	37	30,4	cyclictest	2112296	-21	sendmail	21:20:01	2
2090001	99	37	28,4	cyclictest	0	-21	swapper/1	19:15:04	1
2090001	99	37	22,9	cyclictest	0	-21	swapper/1	00:10:24	1
2089997	99	37	21,14	cyclictest	0	-21	swapper/0	22:19:12	0
2089997	99	37	1,34	cyclictest	0	-21	swapper/0	22:10:20	0
2089997	99	37	13,13	cyclictest	2106585	-21	/usr/sbin/munin	20:45:18	0
2090002	99	36	29,5	cyclictest	14	-21	rcu_preempt	20:55:00	2
2090002	99	36	27,4	cyclictest	0	-21	swapper/2	22:10:17	2
2090001	99	36	27,5	cyclictest	0	-21	swapper/1	21:30:01	1
2090001	99	36	26,7	cyclictest	0	-21	swapper/1	22:00:28	1
2090001	99	36	18,9	cyclictest	0	-21	swapper/1	19:40:20	1
2090001	99	36	1,30	cyclictest	0	-21	swapper/1	21:30:26	1
2089997	99	36	27,6	cyclictest	0	-21	swapper/0	23:35:27	0
2089997	99	36	25,8	cyclictest	0	-21	swapper/0	20:15:29	0
2089997	99	36	1,33	cyclictest	0	-21	swapper/0	23:55:14	0
2089997	99	36	1,28	cyclictest	0	-21	swapper/0	23:15:01	0
2090002	99	35	29,3	cyclictest	2095311	-21	sh	19:40:00	2
2090002	99	35	22,8	cyclictest	0	-21	swapper/2	00:00:26	2
2090001	99	35	24,5	cyclictest	0	-21	swapper/1	22:15:25	1
2090001	99	35	23,5	cyclictest	0	-21	swapper/1	21:00:21	1
2090001	99	35	23,5	cyclictest	0	-21	swapper/1	21:00:21	1
2090001	99	35	22,8	cyclictest	0	-21	swapper/1	19:45:19	1
2090001	99	35	20,10	cyclictest	0	-21	swapper/1	23:40:31	1
2090001	99	35	17,11	cyclictest	0	-21	swapper/1	22:40:26	1
2090001	99	35	1,28	cyclictest	0	-21	swapper/1	20:00:12	1
2090001	99	35	11,9	cyclictest	2137074	-21	/usr/sbin/munin	23:45:18	1
2089997	99	35	1,4	cyclictest	0	-21	swapper/0	00:08:26	0
2090002	99	34	2,26	cyclictest	0	-21	swapper/2	21:10:24	2
2090001	99	34	21,8	cyclictest	2098781	-21	sendmail	20:00:01	1
2090001	99	34	20,7	cyclictest	0	-21	swapper/1	20:30:27	1
2089997	99	34	1,3	cyclictest	0	-21	swapper/0	22:25:46	0
2090002	99	33	25,4	cyclictest	0	-21	swapper/2	19:20:23	2
2090002	99	33	24,4	cyclictest	0	-21	swapper/2	20:45:19	2
2090002	99	33	23,7	cyclictest	0	-21	swapper/2	22:55:22	2
2090002	99	33	20,9	cyclictest	0	-21	swapper/2	19:55:31	2
2090001	99	33	1,26	cyclictest	0	-21	swapper/1	20:40:11	1
2090002	99	32	26,4	cyclictest	0	-21	swapper/2	20:55:16	2
2090002	99	32	24,4	cyclictest	0	-21	swapper/2	19:10:27	2
2090002	99	32	23,5	cyclictest	2135059	-21	grep	23:30:28	2
2090002	99	32	23,5	cyclictest	2135059	-21	grep	23:30:28	2
2090002	99	32	23,5	cyclictest	0	-21	swapper/2	20:05:26	2
2090002	99	32	20,8	cyclictest	2127576	-21	cron	22:50:00	2
2090001	99	32	24,4	cyclictest	0	-21	swapper/1	22:55:19	1
2090001	99	32	2,3	cyclictest	2106952	-21	ps	20:45:26	1
2090001	99	32	22,4	cyclictest	0	-21	swapper/1	23:10:21	1
2090001	99	32	21,7	cyclictest	0	-21	swapper/1	22:50:29	1
2090001	99	32	21,7	cyclictest	0	-21	swapper/1	22:50:28	1
2090001	99	32	20,7	cyclictest	0	-21	swapper/1	21:50:28	1
2090002	99	31	9,4	cyclictest	1653	-21	lldpd	00:08:49	2
2090002	99	31	2,4	cyclictest	1660	-21	snmpd	21:56:22	2
2090002	99	31	24,5	cyclictest	2143944	-21	munin-run	00:30:00	2
2090002	99	31	23,5	cyclictest	0	-21	swapper/2	21:20:29	2
2090002	99	31	23,5	cyclictest	0	-21	swapper/2	19:50:19	2
2090002	99	31	20,5	cyclictest	0	-21	swapper/2	19:25:27	2
2090002	99	31	19,8	cyclictest	0	-21	swapper/2	23:30:00	2
2090001	99	31	23,5	cyclictest	0	-21	swapper/1	19:35:17	1
2090001	99	31	22,6	cyclictest	0	-21	swapper/1	00:40:01	1
2090001	99	31	21,7	cyclictest	0	-21	swapper/1	00:15:29	1
2090001	99	31	1,7	cyclictest	24	-21	ksoftirqd/1	20:10:02	1
2090002	99	30	22,6	cyclictest	0	-21	swapper/2	23:59:59	2
2090002	99	30	20,6	cyclictest	0	-21	swapper/2	23:05:27	2
2090002	99	30	1,27	cyclictest	0	-21	swapper/2	21:25:17	2
2090001	99	30	22,4	cyclictest	0	-21	swapper/1	21:05:24	1
2090001	99	30	21,4	cyclictest	0	-21	swapper/1	22:20:27	1
2090001	99	30	20,7	cyclictest	0	-21	swapper/1	20:50:28	1
2090002	99	29	21,5	cyclictest	0	-21	swapper/2	00:35:27	2
2090002	99	29	21,4	cyclictest	0	-21	swapper/2	20:20:17	2
2090002	99	29	21,4	cyclictest	0	-21	swapper/2	20:20:17	2
2090002	99	29	20,5	cyclictest	0	-21	swapper/2	22:15:17	2
2090002	99	29	19,7	cyclictest	0	-21	swapper/2	19:15:03	2
2090002	99	29	19,5	cyclictest	0	-21	swapper/2	22:35:27	2
2090002	99	29	19,5	cyclictest	0	-21	swapper/2	22:35:27	2
2090002	99	29	18,7	cyclictest	0	-21	swapper/2	00:10:24	2
2090002	99	29	18,6	cyclictest	0	-21	swapper/2	20:40:28	2
2090002	99	29	1,26	cyclictest	0	-21	swapper/2	21:50:16	2
2090001	99	29	20,6	cyclictest	0	-21	swapper/1	23:35:28	1
2090002	99	28	22,3	cyclictest	0	-21	swapper/2	23:50:25	2
2090002	99	28	19,5	cyclictest	0	-21	swapper/2	21:45:28	2
2090002	99	28	1,3	cyclictest	2119524	-21	perl	22:00:21	2
2090002	99	28	1,25	cyclictest	0	-21	swapper/2	22:50:26	2
2090002	99	28	1,25	cyclictest	0	-21	swapper/2	22:50:26	2
2090002	99	28	1,25	cyclictest	0	-21	swapper/2	20:10:27	2
2090001	99	28	17,8	cyclictest	23	1	ktimers/1	21:15:13	1
2090002	99	27	1,24	cyclictest	0	-21	swapper/2	23:02:00	2
2090001	99	27	15,8	cyclictest	0	-21	swapper/1	23:30:25	1
2090001	99	27	15,8	cyclictest	0	-21	swapper/1	23:30:24	1
2090001	99	27	1,2	cyclictest	0	-21	swapper/1	00:30:01	1
2090002	99	26	20,4	cyclictest	31	1	ktimers/2	19:45:25	2
2090002	99	26	19,5	cyclictest	0	-21	swapper/2	22:20:28	2
2090002	99	26	18,5	cyclictest	0	-21	swapper/2	23:10:27	2
2090002	99	26	17,5	cyclictest	0	-21	swapper/2	20:30:28	2
2090002	99	25	18,4	cyclictest	0	-21	swapper/2	21:05:19	2
2090002	99	25	16,7	cyclictest	31	1	ktimers/2	21:30:11	2
2090002	99	25	14,8	cyclictest	1660	-21	snmpd	20:00:59	2
2090002	99	25	13,9	cyclictest	0	-21	swapper/2	21:40:19	2
2090001	99	23	14,6	cyclictest	2140555	-21	sshd	00:09:27	1
2090002	99	19	4,8	cyclictest	2090092	-21	wc	19:10:01	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2020-10-04 - 12:21