OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot6s.osadl.org (updated Tue Feb 03, 2026 00:44:50)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1989526	99	59	49,6	cyclictest	15	-21	rcuc/0	00:20:13	0
1989526	99	58	12,42	cyclictest	2032222	-21	phc2sys	23:15:26	0
1989526	99	57	5,48	cyclictest	2030669	-21	grep	23:05:25	0
1989526	99	53	46,4	cyclictest	208	50	irq/208-8000000.ethernet	23:30:21	0
1989529	99	52	43,5	cyclictest	2012090	-21	sendmail	21:20:00	1
1989530	99	51	40,7	cyclictest	2039284	-21	cut	00:00:00	2
1989526	99	50	43,5	cyclictest	1995692	-21	munin-plugin-st	19:45:01	0
1989526	99	50	42,5	cyclictest	2007859	-21	munin-run	20:54:59	0
1989526	99	50	42,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:20:00	0
1989526	99	50	42,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:20:00	0
1989526	99	50	33,12	cyclictest	13	1	ktimers/0	19:35:10	0
1989526	99	49	41,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:45:23	0
1989526	99	49	41,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:45:23	0
1989526	99	49	41,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:40:31	0
1989526	99	49	41,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:40:31	0
1989526	99	49	2,6	cyclictest	2038100	-21	meminfo	23:50:24	0
1989526	99	49	1,7	cyclictest	2027901	-21	/usr/sbin/munin	22:50:27	0
1989530	99	48	35,9	cyclictest	2012087	-21	sendmail	21:19:59	2
1989530	99	48	32,9	cyclictest	0	-21	swapper/2	20:05:00	2
1989526	99	48	31,15	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:40:24	0
1989530	99	47	39,4	cyclictest	2004308	-21	munin-plugin-st	20:34:59	2
1989530	99	47	34,9	cyclictest	0	-21	swapper/2	00:05:26	2
1989530	99	47	28,9	cyclictest	0	-21	swapper/2	20:50:28	2
1989526	99	47	4,10	cyclictest	2028600	-21	cut	22:55:21	0
1989526	99	47	34,9	cyclictest	1999798	-21	grep	20:05:29	0
1989526	99	47	32,10	cyclictest	0	-21	swapper/0	23:40:19	0
1989530	99	46	31,7	cyclictest	0	-21	swapper/2	21:30:01	2
1989526	99	46	38,5	cyclictest	208	50	irq/208-8000000.ethernet	19:25:00	0
1989526	99	46	3,6	cyclictest	1261	-21	rs:main		0
1989526	99	46	33,7	cyclictest	0	-21	swapper/0	20:00:27	0
1989526	99	46	29,15	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:15:21	0
1989526	99	46	29,14	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:25:14	0
1989526	99	46	28,15	cyclictest	2004569	-21	date	20:35:02	0
1989526	99	46	25,18	cyclictest	2045161	-21	ptp4l-jitter	00:30:27	0
1989526	99	46	22,8	cyclictest	1597374	-1	kworker/0:2H+mmc_complete	19:15:02	0
1989530	99	45	32,9	cyclictest	0	-21	swapper/2	00:15:29	2
1989526	99	45	2,5	cyclictest	2011699	-21	irqstats	21:15:23	0
1989526	99	45	1,41	cyclictest	0	-21	swapper/0	20:45:26	0
1989526	99	45	1,38	cyclictest	0	-21	swapper/0	23:25:02	0
1989526	99	45	10,17	cyclictest	12	-21	ksoftirqd/0	23:40:02	0
1989530	99	44	34,7	cyclictest	2031504	-21	unixbench_multi	23:10:30	2
1989530	99	44	34,7	cyclictest	2031504	-21	unixbench_multi	23:10:30	2
1989530	99	44	34,6	cyclictest	0	-21	swapper/2	21:30:22	2
1989526	99	44	29,10	cyclictest	2040065	-21	grep	00:00:30	0
1989526	99	44	26,12	cyclictest	2044308	-21	ptp4l	00:25:29	0
1989526	99	44	1,38	cyclictest	0	-21	swapper/0	22:50:02	0
1989530	99	43	34,6	cyclictest	2018744	-21	grep	21:55:28	2
1989530	99	43	1,12	cyclictest	0	-21	swapper/2	19:50:01	2
1989529	99	43	33,5	cyclictest	0	-21	swapper/1	19:15:29	1
1989529	99	43	1,30	cyclictest	0	-21	swapper/1	19:50:02	1
1989526	99	43	35,5	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:05:29	0
1989526	99	43	31,8	cyclictest	2000517	-21	phc2sys	20:10:25	0
1989526	99	43	31,7	cyclictest	0	-21	swapper/0	00:10:22	0
1989526	99	43	30,9	cyclictest	0	-21	swapper/0	22:35:01	0
1989526	99	43	30,9	cyclictest	0	-21	swapper/0	22:35:01	0
1989530	99	42	33,5	cyclictest	274	-21	systemd-journal	19:15:00	2
1989530	99	42	32,7	cyclictest	1993155	-21	munin-plugin-st	19:30:00	2
1989530	99	42	31,5	cyclictest	0	-21	swapper/2	20:35:28	2
1989530	99	42	30,8	cyclictest	0	-21	swapper/2	23:00:00	2
1989530	99	42	27,8	cyclictest	0	-21	swapper/2	22:55:01	2
1989529	99	42	32,5	cyclictest	2032440	-21	unixbench_multi	23:15:29	1
1989529	99	42	30,6	cyclictest	1993062	-21	grep	19:25:28	1
1989526	99	42	3,6	cyclictest	2042749	-21	apt-get	00:20:01	0
1989526	99	42	29,10	cyclictest	0	-21	swapper/0	21:45:29	0
1989526	99	42	28,9	cyclictest	0	-21	swapper/0	21:50:19	0
1989526	99	42	2,37	cyclictest	0	-21	swapper/0	21:55:24	0
1989530	99	41	33,4	cyclictest	0	-21	swapper/2	20:05:21	2
1989530	99	41	32,5	cyclictest	0	-21	swapper/2	19:35:01	2
1989530	99	41	32,5	cyclictest	0	-21	swapper/2	19:35:01	2
1989530	99	41	30,7	cyclictest	0	-21	swapper/2	20:50:00	2
1989529	99	41	33,4	cyclictest	0	-21	swapper/1	22:20:01	1
1989529	99	41	33,4	cyclictest	0	-21	swapper/1	22:20:00	1
1989529	99	41	32,4	cyclictest	0	-21	swapper/1	20:15:02	1
1989529	99	41	28,9	cyclictest	0	-21	swapper/1	19:20:28	1
1989526	99	41	34,4	cyclictest	197	50	irq/27-20000000.i2c	20:55:28	0
1989526	99	41	34,4	cyclictest	0	-21	swapper/0	21:30:12	0
1989526	99	41	32,3	cyclictest	0	-21	swapper/0	21:40:26	0
1989526	99	41	30,8	cyclictest	1998152	-21	grep	19:55:30	0
1989526	99	41	30,8	cyclictest	1998152	-21	grep	19:55:30	0
1989530	99	40	34,4	cyclictest	2019698	-21	munin-plugin-st	22:04:59	2
1989530	99	40	28,8	cyclictest	2008708	-21	sendmail	21:00:01	2
1989530	99	40	28,5	cyclictest	0	-21	swapper/2	20:25:30	2
1989530	99	40	27,8	cyclictest	0	-21	swapper/2	21:00:29	2
1989529	99	40	8,26	cyclictest	0	-21	swapper/1	21:20:22	1
1989529	99	40	2,31	cyclictest	1660	-21	snmpd	19:45:01	1
1989526	99	40	34,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:35:15	0
1989526	99	40	34,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:35:15	0
1989526	99	40	33,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:20:17	0
1989526	99	40	31,5	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:45:24	0
1989526	99	40	27,9	cyclictest	2001682	-21	grep	20:15:29	0
1989526	99	40	26,9	cyclictest	0	-21	swapper/0	19:50:28	0
1989526	99	40	25,11	cyclictest	0	-21	swapper/0	23:25:27	0
1989530	99	39	30,7	cyclictest	2037546	-21	grep	23:45:30	2
1989530	99	39	30,7	cyclictest	2037546	-21	grep	23:45:30	2
1989530	99	39	30,5	cyclictest	0	-21	swapper/2	19:40:21	2
1989530	99	39	28,6	cyclictest	0	-21	swapper/2	23:05:23	2
1989530	99	39	1,30	cyclictest	1660	-21	snmpd	22:05:29	2
1989529	99	39	30,5	cyclictest	0	-21	swapper/1	23:05:26	1
1989529	99	39	28,8	cyclictest	0	-21	swapper/1	23:44:59	1
1989529	99	39	27,8	cyclictest	0	-21	swapper/1	22:40:00	1
1989529	99	39	27,7	cyclictest	0	-21	swapper/1	00:05:01	1
1989529	99	39	22,11	cyclictest	0	-21	swapper/1	23:30:27	1
1989526	99	39	31,5	cyclictest	14	-21	rcu_preempt	22:25:20	0
1989526	99	39	25,10	cyclictest	0	-21	swapper/0	22:20:28	0
1989526	99	39	24,7	cyclictest	0	-21	swapper/0	22:35:20	0
1989526	99	39	1,36	cyclictest	0	-21	swapper/0	23:55:17	0
1989526	99	39	1,36	cyclictest	0	-21	swapper/0	23:00:28	0
1989526	99	39	1,36	cyclictest	0	-21	swapper/0	23:00:28	0
1989526	99	39	1,36	cyclictest	0	-21	swapper/0	22:00:16	0
1989526	99	39	1,36	cyclictest	0	-21	swapper/0	19:25:25	0
1989530	99	38	29,6	cyclictest	0	-21	swapper/2	19:20:29	2
1989530	99	38	27,7	cyclictest	0	-21	swapper/2	22:25:29	2
1989530	99	38	27,7	cyclictest	0	-21	swapper/2	20:40:31	2
1989530	99	38	27,7	cyclictest	0	-21	swapper/2	20:40:31	2
1989530	99	38	26,9	cyclictest	2036681	-21	grep	23:40:28	2
1989530	99	38	26,8	cyclictest	0	-21	swapper/2	23:15:28	2
1989529	99	38	29,4	cyclictest	2005242	-21	munin-plugin-st	20:39:59	1
1989529	99	38	24,7	cyclictest	0	-21	swapper/1	22:20:29	1
1989529	99	38	1,29	cyclictest	2017888	-21	unixbench_multi	21:50:26	1
1989526	99	38	24,9	cyclictest	2012858	-21	grep	21:20:28	0
1989526	99	38	1,35	cyclictest	0	-21	swapper/0	21:00:19	0
1989530	99	37	27,7	cyclictest	0	-21	swapper/2	21:10:28	2
1989530	99	37	23,7	cyclictest	0	-21	swapper/2	23:50:28	2
1989529	99	37	30,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	23:00:31	1
1989529	99	37	30,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	23:00:31	1
1989529	99	37	29,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	21:55:30	1
1989529	99	37	29,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	21:35:27	1
1989529	99	37	29,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	21:35:26	1
1989529	99	37	28,5	cyclictest	0	-21	swapper/1	22:50:27	1
1989529	99	37	28,5	cyclictest	0	-21	swapper/1	20:25:31	1
1989529	99	37	28,4	cyclictest	0	-21	swapper/1	19:15:01	1
1989529	99	37	23,10	cyclictest	0	-21	swapper/1	00:30:20	1
1989526	99	37	30,4	cyclictest	208	50	irq/208-8000000.ethernet	00:40:00	0
1989526	99	37	27,7	cyclictest	2011155	-21	grep	21:10:29	0
1989526	99	37	26,7	cyclictest	0	-21	swapper/0	20:30:01	0
1989530	99	36	26,5	cyclictest	0	-21	swapper/2	20:15:19	2
1989530	99	36	25,6	cyclictest	0	-21	swapper/2	23:25:29	2
1989530	99	36	23,6	cyclictest	0	-21	swapper/2	22:45:29	2
1989530	99	36	22,9	cyclictest	0	-21	swapper/2	22:10:25	2
1989529	99	36	28,5	cyclictest	0	-21	swapper/1	21:10:30	1
1989529	99	36	26,6	cyclictest	0	-21	swapper/1	23:40:01	1
1989529	99	36	26,5	cyclictest	0	-21	swapper/1	22:05:29	1
1989529	99	36	24,8	cyclictest	0	-21	swapper/1	20:45:00	1
1989529	99	36	24,8	cyclictest	0	-21	swapper/1	20:45:00	1
1989529	99	36	22,9	cyclictest	0	-21	swapper/1	23:25:21	1
1989529	99	36	1,3	cyclictest	2004476	-21	latency_hist	20:35:01	1
1989526	99	36	29,4	cyclictest	0	-21	swapper/0	19:30:19	0
1989526	99	36	29,4	cyclictest	0	-21	swapper/0	19:30:19	0
1989526	99	36	28,4	cyclictest	206	50	irq/252-8000000.ethernet	00:05:22	0
1989526	99	36	23,7	cyclictest	0	-21	swapper/0	23:10:28	0
1989526	99	36	23,7	cyclictest	0	-21	swapper/0	23:10:28	0
1989526	99	36	23,10	cyclictest	13	1	ktimers/0	22:10:12	0
1989530	99	35	27,5	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	21:45:25	2
1989530	99	35	25,7	cyclictest	0	-21	swapper/2	19:20:00	2
1989530	99	35	24,7	cyclictest	0	-21	swapper/2	22:40:28	2
1989530	99	35	21,9	cyclictest	0	-21	swapper/2	21:20:27	2
1989529	99	35	27,5	cyclictest	0	-21	swapper/1	23:50:28	1
1989529	99	35	22,9	cyclictest	0	-21	swapper/1	00:20:27	1
1989529	99	35	19,8	cyclictest	0	-21	swapper/1	22:55:21	1
1989530	99	34	26,4	cyclictest	0	-21	swapper/2	21:35:30	2
1989530	99	34	26,4	cyclictest	0	-21	swapper/2	21:35:30	2
1989530	99	34	26,4	cyclictest	0	-21	swapper/2	21:05:26	2
1989530	99	34	22,9	cyclictest	0	-21	swapper/2	19:40:00	2
1989530	99	34	2,26	cyclictest	0	-21	swapper/2	00:25:29	2
1989530	99	34	21,10	cyclictest	2000999	-21	cat	20:15:02	2
1989530	99	34	11,10	cyclictest	2029282	-21	/usr/sbin/munin	23:00:20	2
1989530	99	34	11,10	cyclictest	2029282	-21	/usr/sbin/munin	23:00:20	2
1989529	99	34	27,4	cyclictest	0	-21	swapper/1	20:55:00	1
1989529	99	34	26,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	21:40:23	1
1989529	99	34	25,6	cyclictest	2031483	-21	grep	23:10:30	1
1989529	99	34	25,6	cyclictest	2031483	-21	grep	23:10:30	1
1989529	99	34	25,5	cyclictest	0	-21	swapper/1	21:45:29	1
1989529	99	34	24,7	cyclictest	0	-21	swapper/1	21:25:30	1
1989529	99	34	22,8	cyclictest	0	-21	swapper/1	22:10:25	1
1989529	99	34	21,8	cyclictest	0	-21	swapper/1	23:25:02	1
1989530	99	33	26,3	cyclictest	0	-21	swapper/2	22:39:59	2
1989530	99	33	24,5	cyclictest	0	-21	swapper/2	23:35:17	2
1989530	99	33	22,7	cyclictest	0	-21	swapper/2	00:30:28	2
1989530	99	33	21,8	cyclictest	0	-21	swapper/2	21:50:24	2
1989530	99	33	20,9	cyclictest	0	-21	swapper/2	23:30:29	2
1989529	99	33	25,5	cyclictest	0	-21	swapper/1	23:55:28	1
1989529	99	33	21,9	cyclictest	2002180	-21	sh	20:20:24	1
1989529	99	33	20,7	cyclictest	0	-21	swapper/1	20:10:00	1
1989529	99	33	1,5	cyclictest	1996962	-21	if_eth0	19:50:21	1
1989529	99	33	1,29	cyclictest	1660	-21	snmpd	21:00:16	1
1989526	99	33	26,4	cyclictest	14	-21	rcu_preempt	22:05:21	0
1989530	99	32	23,6	cyclictest	2002574	-21	grep	20:20:30	2
1989530	99	32	23,5	cyclictest	0	-21	swapper/2	19:55:30	2
1989530	99	32	23,5	cyclictest	0	-21	swapper/2	19:55:29	2
1989529	99	32	21,8	cyclictest	23	1	ktimers/1	21:30:15	1
1989529	99	32	1,28	cyclictest	0	-21	swapper/1	00:15:27	1
1989526	99	32	3,19	cyclictest	1989605	-21	latency_hist	19:10:02	0
1989530	99	31	21,6	cyclictest	0	-21	swapper/2	00:00:30	2
1989530	99	31	1,28	cyclictest	31	1	ktimers/2	23:20:26	2
1989529	99	31	24,4	cyclictest	0	-21	swapper/1	22:50:00	1
1989529	99	31	24,4	cyclictest	0	-21	swapper/1	22:30:28	1
1989529	99	31	24,4	cyclictest	0	-21	swapper/1	22:30:28	1
1989529	99	31	23,5	cyclictest	2046181	-21	grep	00:35:30	1
1989529	99	31	23,4	cyclictest	0	-21	swapper/1	23:45:28	1
1989529	99	31	23,4	cyclictest	0	-21	swapper/1	23:45:28	1
1989529	99	31	23,4	cyclictest	0	-21	swapper/1	19:40:01	1
1989529	99	31	23,4	cyclictest	0	-21	swapper/1	00:05:25	1
1989529	99	31	21,7	cyclictest	0	-21	swapper/1	19:55:28	1
1989529	99	31	21,7	cyclictest	0	-21	swapper/1	19:55:28	1
1989529	99	31	21,6	cyclictest	2026398	-21	grep	22:40:29	1
1989529	99	31	19,8	cyclictest	0	-21	swapper/1	00:10:30	1
1989529	99	31	1,27	cyclictest	0	-21	swapper/1	20:45:30	1
1989530	99	30	21,6	cyclictest	0	-21	swapper/2	22:15:30	2
1989530	99	30	21,6	cyclictest	0	-21	swapper/2	22:15:30	2
1989530	99	29	22,4	cyclictest	0	-21	swapper/2	22:30:26	2
1989530	99	29	22,4	cyclictest	0	-21	swapper/2	22:30:25	2
1989530	99	29	19,6	cyclictest	0	-21	swapper/2	00:35:30	2
1989529	99	29	18,8	cyclictest	0	-21	swapper/1	22:25:29	1
1989529	99	29	18,8	cyclictest	0	-21	swapper/1	19:30:29	1
1989529	99	29	18,8	cyclictest	0	-21	swapper/1	19:30:29	1
1989529	99	29	14,11	cyclictest	1989528	-21	date	19:10:00	1
1989530	99	28	22,4	cyclictest	0	-21	swapper/2	22:20:12	2
1989530	99	28	18,5	cyclictest	0	-21	swapper/2	19:50:27	2
1989529	99	28	20,4	cyclictest	0	-21	swapper/1	22:00:26	1
1989530	99	27	18,5	cyclictest	0	-21	swapper/2	21:40:24	2
1989530	99	27	16,7	cyclictest	0	-21	swapper/2	00:10:22	2
1989529	99	27	7,14	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	20:00:03	1
1989529	99	27	17,7	cyclictest	0	-21	swapper/1	20:55:29	1
1989529	99	27	1,2	cyclictest	0	-21	swapper/1	21:10:01	1
1989530	99	26	16,7	cyclictest	0	-21	swapper/2	00:20:20	2
1989529	99	26	17,7	cyclictest	0	-21	swapper/1	20:15:29	1
1989530	99	25	17,4	cyclictest	0	-21	swapper/2	19:10:00	2
1989529	99	25	17,6	cyclictest	23	1	ktimers/1	00:25:17	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2020-10-04 - 12:21