OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot6s.osadl.org (updated Sun Aug 09, 2026 00:44:48)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3609921	99	99	0,47	cyclictest	3614582	-21	/usr/sbin/munin	19:35:20	2
3609918	99	85	47,34	cyclictest	3660351	-21	sshd	00:05:00	0
3609918	99	75	3,32	cyclictest	208	50	irq/208-8000000.ethernet	23:50:01	0
3609918	99	74	32,9	cyclictest	0	-21	swapper/0	00:25:30	0
3609918	99	74	32,9	cyclictest	0	-21	swapper/0	00:25:30	0
3609918	99	72	38,5	cyclictest	3627140	-21	munin-plugin-st	20:49:59	0
3609921	99	68	6,27	cyclictest	880	-21	avahi-daemon	00:35:19	2
3609921	99	68	39,19	cyclictest	32	-21	ksoftirqd/2	22:05:18	2
3609921	99	65	59,3	cyclictest	32	-21	ksoftirqd/2	22:10:22	2
3609918	99	62	51,8	cyclictest	0	-21	swapper/0	20:35:11	0
3609921	99	60	2,38	cyclictest	3611202	-21	/usr/sbin/munin	19:15:20	2
3609921	99	59	1,38	cyclictest	0	-21	swapper/2	23:05:01	2
3609921	99	58	1,37	cyclictest	0	-21	swapper/2	20:50:01	2
3609919	99	58	48,4	cyclictest	23	1	ktimers/1	20:05:02	1
3609919	99	58	48,4	cyclictest	23	1	ktimers/1	20:05:02	1
3609921	99	55	42,8	cyclictest	3623616	-21	grep	20:25:30	2
3609921	99	54	45,6	cyclictest	3639900	-21	munin-plugin-st	22:05:00	2
3609921	99	54	13,21	cyclictest	3609913	-21	cyclictest	20:35:14	2
3609919	99	54	44,5	cyclictest	3656115	-21	latency_hist	23:40:00	1
3609919	99	54	36,13	cyclictest	23	1	ktimers/1	19:55:14	1
3609918	99	53	49,2	cyclictest	0	-21	swapper/0	22:40:12	0
3609921	99	52	35,12	cyclictest	3618600	-21	sendmail	20:00:01	2
3609918	99	52	40,4	cyclictest	0	-21	swapper/0	20:25:20	0
3609919	99	51	44,3	cyclictest	3663051	-21	sendmail	00:20:01	1
3609919	99	51	42,5	cyclictest	3640741	-21	munin-plugin-st	22:10:00	1
3609919	99	51	16,19	cyclictest	3660512	-21	gpg-connect-age	00:05:01	1
3609918	99	51	32,13	cyclictest	13	1	ktimers/0	00:15:13	0
3609921	99	50	8,17	cyclictest	0	-21	swapper/2	19:15:01	2
3609921	99	49	8,18	cyclictest	3612064	-21	/usr/sbin/munin	19:20:18	2
3609918	99	49	37,4	cyclictest	13	1	ktimers/0	21:35:13	0
3609918	99	49	1,8	cyclictest	0	-21	swapper/0	19:30:02	0
3609918	99	49	1,8	cyclictest	0	-21	swapper/0	19:30:02	0
3609921	99	48	34,10	cyclictest	3662284	-21	latency_hist	00:15:01	2
3609921	99	48	34,10	cyclictest	3662284	-21	latency_hist	00:15:01	2
3609919	99	48	30,14	cyclictest	23	1	ktimers/1	21:35:14	1
3609919	99	48	30,11	cyclictest	23	1	ktimers/1	20:00:15	1
3609918	99	48	1,4	cyclictest	3643449	-21	latency_hist	22:25:01	0
3609918	99	47	6,15	cyclictest	15	-21	rcuc/0	19:15:00	0
3609918	99	47	2,41	cyclictest	1660	-21	snmpd	22:55:17	0
3609921	99	46	13,16	cyclictest	3645046	-21	apt-get	22:35:02	2
3609919	99	46	38,5	cyclictest	3621074	-21	unixbench_singl	20:10:29	1
3609919	99	46	34,9	cyclictest	23	1	ktimers/1	21:55:14	1
3609919	99	46	31,11	cyclictest	23	1	ktimers/1	22:20:14	1
3609919	99	46	31,10	cyclictest	23	1	ktimers/1	23:20:12	1
3609918	99	46	27,13	cyclictest	13	1	ktimers/0	23:55:16	0
3609918	99	46	21,23	cyclictest	0	-21	swapper/0	20:10:16	0
3609921	99	45	33,7	cyclictest	3628827	-21	sendmail	21:00:00	2
3609921	99	45	26,11	cyclictest	0	-21	swapper/2	22:30:00	2
3609921	99	45	23,15	cyclictest	30	-21	rcuc/2	21:45:01	2
3609919	99	45	30,12	cyclictest	23	1	ktimers/1	21:15:15	1
3609919	99	45	29,12	cyclictest	23	1	ktimers/1	20:30:16	1
3609919	99	45	29,12	cyclictest	23	1	ktimers/1	00:35:13	1
3609919	99	45	2,23	cyclictest	0	-21	swapper/1	20:35:05	1
3609918	99	45	28,15	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:15:27	0
3609918	99	45	2,40	cyclictest	0	-21	swapper/0	22:25:14	0
3609921	99	44	36,5	cyclictest	3609913	-21	cyclictest	23:54:59	2
3609921	99	44	34,7	cyclictest	0	-21	swapper/2	19:30:27	2
3609921	99	44	31,9	cyclictest	3649359	-21	munin-plugin-st	23:00:00	2
3609921	99	44	18,12	cyclictest	0	-21	swapper/2	20:50:13	2
3609919	99	44	7,4	cyclictest	3645213	-21	rm	22:35:01	1
3609919	99	44	31,10	cyclictest	23	1	ktimers/1	19:45:13	1
3609919	99	44	30,9	cyclictest	23	1	ktimers/1	23:30:16	1
3609919	99	44	30,11	cyclictest	23	1	ktimers/1	21:10:14	1
3609919	99	44	28,10	cyclictest	23	1	ktimers/1	00:05:14	1
3609918	99	44	1,3	cyclictest	0	-21	swapper/0	21:50:01	0
3609921	99	43	24,12	cyclictest	0	-21	swapper/2	23:14:59	2
3609919	99	43	32,8	cyclictest	22	-21	rcuc/1	23:25:01	1
3609919	99	43	32,8	cyclictest	22	-21	rcuc/1	23:25:00	1
3609919	99	43	28,10	cyclictest	23	1	ktimers/1	19:25:15	1
3609919	99	43	28,10	cyclictest	23	1	ktimers/1	19:25:15	1
3609919	99	43	24,15	cyclictest	23	1	ktimers/1	20:55:14	1
3609918	99	43	36,4	cyclictest	0	-21	swapper/0	00:25:00	0
3609918	99	43	2,10	cyclictest	0	-21	swapper/0	20:20:01	0
3609921	99	42	29,8	cyclictest	3638968	-21	grep	21:55:30	2
3609921	99	42	23,10	cyclictest	0	-21	swapper/2	19:30:00	2
3609921	99	42	23,10	cyclictest	0	-21	swapper/2	19:30:00	2
3609921	99	42	16,13	cyclictest	0	-21	swapper/2	21:05:14	2
3609921	99	42	16,13	cyclictest	0	-21	swapper/2	21:05:14	2
3609919	99	42	9,7	cyclictest	3634733	-21	apt-get	21:35:02	1
3609919	99	42	33,6	cyclictest	3639899	-21	munin-run	22:05:00	1
3609919	99	42	28,9	cyclictest	0	-21	swapper/1	19:10:31	1
3609919	99	42	27,10	cyclictest	23	1	ktimers/1	22:45:13	1
3609919	99	42	27,10	cyclictest	23	1	ktimers/1	22:45:13	1
3609919	99	42	25,11	cyclictest	0	-21	swapper/1	23:15:01	1
3609919	99	42	22,13	cyclictest	23	1	ktimers/1	23:40:11	1
3609918	99	42	6,21	cyclictest	208	50	irq/208-8000000.ethernet	23:05:20	0
3609918	99	42	36,4	cyclictest	0	-21	swapper/0	22:50:11	0
3609918	99	42	28,9	cyclictest	0	-21	swapper/0	22:35:01	0
3609918	99	42	2,4	cyclictest	3631012	-21	meminfo	21:10:25	0
3609918	99	42	1,5	cyclictest	3640740	-21	munin-run	22:10:00	0
3609918	99	42	1,39	cyclictest	0	-21	swapper/0	22:15:01	0
3609921	99	41	26,9	cyclictest	0	-21	swapper/2	23:20:26	2
3609921	99	41	24,9	cyclictest	0	-21	swapper/2	19:40:22	2
3609919	99	41	4,31	cyclictest	24	-21	ksoftirqd/1	22:15:19	1
3609919	99	41	35,4	cyclictest	3662112	-21	apt-get	00:15:01	1
3609919	99	41	35,4	cyclictest	3662112	-21	apt-get	00:15:01	1
3609919	99	41	27,10	cyclictest	23	1	ktimers/1	22:40:15	1
3609919	99	41	27,10	cyclictest	23	1	ktimers/1	21:50:09	1
3609919	99	41	26,11	cyclictest	23	1	ktimers/1	19:40:14	1
3609918	99	41	1,4	cyclictest	3642426	-21	munin-run	22:20:00	0
3609918	99	41	1,4	cyclictest	3624516	-21	munin-run	20:35:00	0
3609918	99	41	1,38	cyclictest	0	-21	swapper/0	19:15:21	0
3609921	99	40	3,31	cyclictest	0	-21	swapper/2	21:45:25	2
3609921	99	40	28,6	cyclictest	3616920	-21	munin-plugin-st	19:50:00	2
3609921	99	40	23,5	cyclictest	3624514	-21	apt-get	20:35:02	2
3609919	99	40	31,5	cyclictest	3614288	-21	unixbench_multi	19:30:27	1
3609919	99	40	27,9	cyclictest	0	-21	swapper/1	20:25:19	1
3609919	99	40	23,13	cyclictest	23	1	ktimers/1	21:45:11	1
3609919	99	40	1,7	cyclictest	0	-21	swapper/1	23:10:01	1
3609919	99	40	1,4	cyclictest	0	-21	swapper/1	20:25:01	1
3609918	99	40	28,9	cyclictest	0	-21	swapper/0	23:30:27	0
3609918	99	40	25,11	cyclictest	13	1	ktimers/0	20:55:12	0
3609918	99	40	1,4	cyclictest	3632617	-21	fw_packets	21:20:22	0
3609918	99	40	1,37	cyclictest	0	-21	swapper/0	22:00:26	0
3609918	99	40	1,37	cyclictest	0	-21	swapper/0	21:40:22	0
3609918	99	40	1,2	cyclictest	0	-21	swapper/0	19:45:20	0
3609921	99	39	31,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	20:00:22	2
3609921	99	39	24,8	cyclictest	0	-21	swapper/2	20:20:29	2
3609921	99	39	20,9	cyclictest	0	-21	swapper/2	20:05:21	2
3609921	99	39	20,9	cyclictest	0	-21	swapper/2	20:05:20	2
3609919	99	39	7,13	cyclictest	3609913	-21	cyclictest	19:10:00	1
3609919	99	39	3,19	cyclictest	22	-21	rcuc/1	21:45:02	1
3609919	99	39	30,5	cyclictest	3659392	-21	grep	23:55:29	1
3609919	99	39	29,4	cyclictest	14	-21	rcu_preempt	21:25:21	1
3609919	99	39	27,8	cyclictest	0	-21	swapper/1	21:00:22	1
3609919	99	39	22,13	cyclictest	23	1	ktimers/1	00:20:14	1
3609919	99	39	22,11	cyclictest	23	1	ktimers/1	23:50:12	1
3609919	99	39	20,14	cyclictest	23	1	ktimers/1	22:35:15	1
3609918	99	39	1,4	cyclictest	3661235	-21	sed	00:05:32	0
3609918	99	39	1,4	cyclictest	3629104	-21	/usr/sbin/munin	21:00:19	0
3609918	99	39	1,36	cyclictest	0	-21	swapper/0	23:50:14	0
3609918	99	39	1,36	cyclictest	0	-21	swapper/0	20:40:18	0
3609918	99	39	1,36	cyclictest	0	-21	swapper/0	19:20:27	0
3609921	99	38	28,6	cyclictest	3654373	-21	grep	23:25:28	2
3609921	99	38	28,6	cyclictest	3654373	-21	grep	23:25:28	2
3609921	99	38	27,7	cyclictest	3660562	-21	grep	00:05:01	2
3609921	99	38	23,8	cyclictest	0	-21	swapper/2	19:50:30	2
3609921	99	38	23,11	cyclictest	3645980	-21	sendmail	22:40:01	2
3609919	99	38	28,5	cyclictest	0	-21	swapper/1	22:55:00	1
3609919	99	38	25,9	cyclictest	3657706	-21	grep	23:45:30	1
3609919	99	38	22,10	cyclictest	0	-21	swapper/1	21:05:18	1
3609919	99	38	22,10	cyclictest	0	-21	swapper/1	21:05:18	1
3609918	99	38	8,4	cyclictest	0	-21	swapper/0	21:10:01	0
3609918	99	38	8,4	cyclictest	0	-21	swapper/0	21:10:01	0
3609918	99	38	31,5	cyclictest	0	-21	swapper/0	19:50:24	0
3609918	99	38	30,5	cyclictest	0	-21	swapper/0	23:05:01	0
3609918	99	38	27,7	cyclictest	3666431	-21	cron	00:40:00	0
3609918	99	38	23,11	cyclictest	0	-21	swapper/0	22:35:22	0
3609918	99	38	22,14	cyclictest	0	-21	swapper/0	21:55:22	0
3609918	99	38	22,10	cyclictest	0	-21	swapper/0	00:15:01	0
3609918	99	38	22,10	cyclictest	0	-21	swapper/0	00:15:01	0
3609918	99	38	1,3	cyclictest	3634137	-21	cpu	21:30:16	0
3609921	99	37	27,7	cyclictest	3625660	-21	/usr/sbin/munin	20:40:11	2
3609921	99	37	24,8	cyclictest	0	-21	swapper/2	23:30:18	2
3609921	99	37	22,9	cyclictest	0	-21	swapper/2	21:20:28	2
3609921	99	37	21,9	cyclictest	0	-21	swapper/2	21:15:28	2
3609921	99	37	13,13	cyclictest	0	-21	swapper/2	22:50:14	2
3609919	99	37	25,8	cyclictest	0	-21	swapper/1	22:55:28	1
3609919	99	37	21,13	cyclictest	23	1	ktimers/1	21:20:13	1
3609919	99	37	18,13	cyclictest	23	1	ktimers/1	00:30:12	1
3609918	99	37	31,4	cyclictest	0	-21	swapper/0	19:35:21	0
3609918	99	37	24,8	cyclictest	0	-21	swapper/0	00:30:29	0
3609918	99	37	1,4	cyclictest	0	-21	swapper/0	20:05:00	0
3609918	99	37	1,3	cyclictest	0	-21	swapper/0	20:05:28	0
3609918	99	37	1,3	cyclictest	0	-21	swapper/0	20:05:28	0
3609918	99	37	1,34	cyclictest	0	-21	swapper/0	23:20:12	0
3609918	99	37	1,34	cyclictest	0	-21	swapper/0	21:50:22	0
3609921	99	36	26,6	cyclictest	3661220	-21	grep	00:05:32	2
3609921	99	36	20,9	cyclictest	1660	-21	snmpd	20:10:23	2
3609919	99	36	21,10	cyclictest	0	-21	swapper/1	20:20:00	1
3609918	99	36	30,4	cyclictest	208	50	irq/208-8000000.ethernet	20:50:22	0
3609918	99	36	2,32	cyclictest	0	-21	swapper/0	23:25:12	0
3609918	99	36	2,32	cyclictest	0	-21	swapper/0	23:25:12	0
3609921	99	35	1,28	cyclictest	3621398	-21	df	20:15:19	2
3609919	99	35	7,4	cyclictest	0	-21	swapper/1	19:55:01	1
3609919	99	35	31,2	cyclictest	0	-21	swapper/1	22:10:22	1
3609919	99	35	23,7	cyclictest	3626249	-21	grep	20:40:28	1
3609919	99	35	1,2	cyclictest	0	-21	swapper/1	19:40:00	1
3609919	99	35	1,2	cyclictest	0	-21	swapper/1	00:30:01	1
3609919	99	35	1,2	cyclictest	0	-21	swapper/1	00:30:01	1
3609918	99	35	24,8	cyclictest	0	-21	swapper/0	23:10:28	0
3609918	99	35	1,32	cyclictest	0	-21	swapper/0	23:18:59	0
3609921	99	34	28,4	cyclictest	31	1	ktimers/2	21:10:24	2
3609921	99	34	25,7	cyclictest	31	1	ktimers/2	22:15:19	2
3609921	99	34	25,6	cyclictest	3657808	-21	munin-plugin-st	23:50:00	2
3609921	99	34	24,6	cyclictest	0	-21	swapper/2	23:55:29	2
3609921	99	34	2,25	cyclictest	0	-21	swapper/2	21:51:18	2
3609921	99	34	2,25	cyclictest	0	-21	swapper/2	00:16:22	2
3609921	99	34	18,9	cyclictest	0	-21	swapper/2	22:45:20	2
3609921	99	34	18,9	cyclictest	0	-21	swapper/2	22:45:20	2
3609921	99	34	11,13	cyclictest	0	-21	swapper/2	22:20:15	2
3609919	99	34	14,8	cyclictest	3627337	-21	/usr/sbin/munin	20:50:20	1
3609918	99	34	28,4	cyclictest	208	50	irq/208-8000000.ethernet	19:40:18	0
3609918	99	34	26,5	cyclictest	0	-21	swapper/0	21:25:30	0
3609918	99	34	2,4	cyclictest	3622999	-21	grep	20:25:02	0
3609918	99	34	24,7	cyclictest	0	-21	swapper/0	19:30:28	0
3609918	99	34	21,8	cyclictest	0	-21	swapper/0	19:55:29	0
3609918	99	34	1,31	cyclictest	0	-21	swapper/0	23:38:37	0
3609918	99	34	1,28	cyclictest	0	-21	swapper/0	23:40:04	0
3609921	99	33	25,4	cyclictest	0	-21	swapper/2	23:05:17	2
3609921	99	33	10,14	cyclictest	0	-21	swapper/2	23:15:12	2
3609919	99	33	1,4	cyclictest	3612676	-21	users	19:20:29	1
3609919	99	33	1,27	cyclictest	0	-21	swapper/1	22:26:50	1
3609918	99	33	4,4	cyclictest	0	-21	swapper/0	19:10:00	0
3609921	99	32	5,21	cyclictest	2	-21	kthreadd	23:41:21	2
3609921	99	32	26,4	cyclictest	31	1	ktimers/2	21:35:01	2
3609921	99	32	26,4	cyclictest	31	1	ktimers/2	00:25:01	2
3609921	99	32	22,5	cyclictest	3665023	-21	df	00:30:17	2
3609921	99	32	16,8	cyclictest	30	-21	rcuc/2	22:40:02	2
3609921	99	32	1,26	cyclictest	0	-21	swapper/2	21:30:16	2
3609919	99	32	23,4	cyclictest	0	-21	swapper/1	19:15:28	1
3609919	99	32	16,11	cyclictest	0	-21	swapper/1	20:50:01	1
3609918	99	32	25,3	cyclictest	0	-21	swapper/0	22:45:23	0
3609918	99	32	25,3	cyclictest	0	-21	swapper/0	22:45:23	0
3609921	99	31	2,23	cyclictest	0	-21	swapper/2	21:29:23	2
3609921	99	31	17,8	cyclictest	3656003	-21	grep	23:35:30	2
3609921	99	31	1,25	cyclictest	0	-21	swapper/2	00:28:20	2
3609921	99	31	1,25	cyclictest	0	-21	swapper/2	00:28:20	2
3609919	99	29	17,8	cyclictest	0	-21	swapper/1	23:15:29	1
3609921	99	28	17,7	cyclictest	0	-21	swapper/2	19:10:01	2
3609921	99	28	17,6	cyclictest	31	1	ktimers/2	21:00:32	2
3609919	99	28	19,5	cyclictest	0	-21	swapper/1	23:00:18	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2020-10-04 - 12:21