OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot6s.osadl.org (updated Mon Mar 09, 2026 00:44:49)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2725669	99	85	5,11	cyclictest	2752361	-21	apt-get	21:45:02	3
2725666	99	74	18,42	cyclictest	30	-21	rcuc/2	20:50:01	2
2725663	99	70	1,66	cyclictest	0	-21	swapper/0	20:40:14	0
2725663	99	59	2,4	cyclictest	0	-21	swapper/0	23:45:14	0
2725663	99	58	24,29	cyclictest	0	-21	swapper/0	00:25:26	0
2725666	99	57	3,38	cyclictest	2777301	-21	cmp	00:10:00	2
2725666	99	57	24,22	cyclictest	30	-21	rcuc/2	20:20:00	2
2725663	99	55	2,10	cyclictest	0	-21	swapper/0	22:45:02	0
2725664	99	54	3,9	cyclictest	2753290	-21	apt-get	21:50:02	1
2725664	99	54	3,9	cyclictest	2753290	-21	apt-get	21:50:01	1
2725664	99	53	38,10	cyclictest	0	-21	swapper/1	23:05:00	1
2725664	99	53	1,37	cyclictest	0	-21	swapper/1	21:20:02	1
2725664	99	50	1,8	cyclictest	0	-21	swapper/1	23:15:20	1
2725663	99	50	42,5	cyclictest	2744754	-21	run-parts	21:00:00	0
2725663	99	50	2,10	cyclictest	0	-21	swapper/0	23:15:00	0
2725666	99	49	35,9	cyclictest	2730127	-21	sh	19:34:59	2
2725664	99	49	2,33	cyclictest	0	-21	swapper/1	21:50:01	1
2725663	99	49	42,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:40:15	0
2725666	99	48	35,10	cyclictest	31	1	ktimers/2	22:20:12	2
2725666	99	48	29,13	cyclictest	31	1	ktimers/2	23:35:11	2
2725664	99	48	24,19	cyclictest	23	1	ktimers/1	20:15:16	1
2725666	99	47	38,5	cyclictest	2778985	-21	sendmail	00:20:01	2
2725666	99	47	38,5	cyclictest	2778985	-21	sendmail	00:20:00	2
2725666	99	47	29,13	cyclictest	31	1	ktimers/2	00:20:12	2
2725664	99	47	1,8	cyclictest	0	-21	swapper/1	22:50:02	1
2725666	99	46	39,4	cyclictest	2751505	-21	sendmail	21:39:59	2
2725666	99	46	37,5	cyclictest	2729187	-21	tail	19:25:30	2
2725663	99	46	39,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:50:01	0
2725663	99	46	39,3	cyclictest	0	-21	swapper/0	21:15:01	0
2725663	99	46	26,16	cyclictest	14	-21	rcu_preempt	00:10:01	0
2725664	99	45	35,5	cyclictest	0	-21	swapper/1	23:55:01	1
2725663	99	45	37,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:40:27	0
2725663	99	45	33,8	cyclictest	0	-21	swapper/0	22:10:28	0
2725663	99	45	29,14	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	23:25:24	0
2725664	99	44	7,15	cyclictest	2772895	-21	apt-get	23:45:01	1
2725664	99	44	34,6	cyclictest	0	-21	swapper/1	00:15:00	1
2725664	99	44	29,8	cyclictest	0	-21	swapper/1	20:40:30	1
2725663	99	44	38,2	cyclictest	0	-21	swapper/0	20:00:20	0
2725663	99	44	33,7	cyclictest	13	1	ktimers/0	22:25:28	0
2725663	99	44	32,9	cyclictest	0	-21	swapper/0	23:55:28	0
2725663	99	44	1,40	cyclictest	2767356	-21	lsblk	23:10:20	0
2725663	99	44	1,38	cyclictest	0	-21	swapper/0	22:55:01	0
2725666	99	43	33,7	cyclictest	2755067	-21	sendmail	22:00:00	2
2725666	99	43	28,8	cyclictest	31	1	ktimers/2	22:00:13	2
2725666	99	43	28,10	cyclictest	31	1	ktimers/2	19:35:12	2
2725663	99	43	37,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:30:14	0
2725663	99	43	36,4	cyclictest	2750427	-21	/usr/sbin/munin	21:30:25	0
2725663	99	43	35,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:55:28	0
2725663	99	43	30,8	cyclictest	0	-21	swapper/0	23:05:28	0
2725663	99	43	30,8	cyclictest	0	-21	swapper/0	23:05:28	0
2725663	99	43	2,6	cyclictest	2755578	-21	memory	22:00:25	0
2725663	99	43	1,5	cyclictest	274	-21	systemd-journal	20:30:00	0
2725663	99	43	1,39	cyclictest	2778800	-21	mailstats	00:15:28	0
2725663	99	43	1,39	cyclictest	2778800	-21	mailstats	00:15:28	0
2725666	99	42	28,8	cyclictest	0	-21	swapper/2	23:10:19	2
2725666	99	42	26,11	cyclictest	31	1	ktimers/2	23:05:11	2
2725666	99	42	26,11	cyclictest	31	1	ktimers/2	23:05:11	2
2725666	99	42	25,11	cyclictest	31	1	ktimers/2	21:00:13	2
2725666	99	42	23,11	cyclictest	14	-21	rcu_preempt	20:35:01	2
2725664	99	42	6,18	cyclictest	0	-21	swapper/1	22:40:01	1
2725664	99	42	29,8	cyclictest	0	-21	swapper/1	20:45:29	1
2725664	99	42	26,8	cyclictest	0	-21	swapper/1	20:55:28	1
2725664	99	42	23,10	cyclictest	0	-21	swapper/1	20:35:20	1
2725663	99	42	36,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	21:35:12	0
2725663	99	42	36,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:30:11	0
2725663	99	42	32,6	cyclictest	2749307	-21	sh	21:25:21	0
2725663	99	42	32,6	cyclictest	2749307	-21	sh	21:25:20	0
2725663	99	42	31,7	cyclictest	0	-21	swapper/0	20:15:31	0
2725663	99	42	30,8	cyclictest	0	-21	swapper/0	00:00:29	0
2725663	99	42	30,8	cyclictest	0	-21	swapper/0	00:00:28	0
2725663	99	42	28,10	cyclictest	0	-21	swapper/0	19:35:29	0
2725663	99	42	2,6	cyclictest	2781789	-21	diskstats	00:35:18	0
2725663	99	42	1,39	cyclictest	0	-21	swapper/0	19:15:14	0
2725666	99	41	32,5	cyclictest	2733418	-21	grep	19:50:29	2
2725666	99	41	32,5	cyclictest	2733418	-21	grep	19:50:29	2
2725666	99	41	27,11	cyclictest	31	1	ktimers/2	23:25:13	2
2725666	99	41	26,11	cyclictest	31	1	ktimers/2	21:05:12	2
2725664	99	41	30,7	cyclictest	0	-21	swapper/1	19:10:29	1
2725664	99	41	29,8	cyclictest	0	-21	swapper/1	21:39:59	1
2725664	99	41	28,8	cyclictest	0	-21	swapper/1	22:05:29	1
2725664	99	41	27,9	cyclictest	0	-21	swapper/1	23:55:29	1
2725664	99	41	27,10	cyclictest	0	-21	swapper/1	22:55:30	1
2725664	99	41	27,10	cyclictest	0	-21	swapper/1	22:55:29	1
2725664	99	41	25,11	cyclictest	0	-21	swapper/1	23:20:19	1
2725664	99	41	2,17	cyclictest	2740448	-21	cmp	20:35:01	1
2725663	99	41	35,4	cyclictest	2759300	-21	munin-plugin-st	22:25:00	0
2725663	99	41	29,8	cyclictest	0	-21	swapper/0	21:05:27	0
2725663	99	41	2,4	cyclictest	2745266	-21	/usr/sbin/munin	21:00:25	0
2725663	99	41	1,4	cyclictest	2772205	-21	latency_hist	23:40:02	0
2725663	99	41	1,37	cyclictest	2729364	-21	date	19:30:02	0
2725666	99	40	29,7	cyclictest	0	-21	swapper/2	22:10:27	2
2725666	99	40	23,13	cyclictest	31	1	ktimers/2	21:25:12	2
2725666	99	40	23,13	cyclictest	31	1	ktimers/2	21:25:11	2
2725664	99	40	5,5	cyclictest	0	-21	swapper/1	21:45:01	1
2725664	99	40	29,6	cyclictest	2763401	-21	grep	22:45:29	1
2725664	99	40	29,4	cyclictest	0	-21	swapper/1	20:10:25	1
2725664	99	40	28,7	cyclictest	0	-21	swapper/1	19:25:31	1
2725664	99	40	27,8	cyclictest	0	-21	swapper/1	23:25:29	1
2725664	99	40	26,9	cyclictest	0	-21	swapper/1	20:05:28	1
2725663	99	40	5,5	cyclictest	0	-21	swapper/0	23:00:20	0
2725663	99	40	33,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:45:13	0
2725663	99	40	33,4	cyclictest	13	1	ktimers/0	22:15:23	0
2725663	99	40	33,4	cyclictest	13	1	ktimers/0	22:15:23	0
2725663	99	40	33,4	cyclictest	0	-21	swapper/0	21:15:22	0
2725663	99	40	27,10	cyclictest	0	-21	swapper/0	19:20:28	0
2725663	99	40	23,15	cyclictest	0	-21	swapper/0	21:50:02	0
2725663	99	40	23,15	cyclictest	0	-21	swapper/0	21:50:02	0
2725666	99	39	7,21	cyclictest	2748343	-21	/usr/sbin/munin	21:20:18	2
2725666	99	39	30,5	cyclictest	2769415	-21	grep	23:20:28	2
2725666	99	39	30,5	cyclictest	0	-21	swapper/2	20:10:28	2
2725666	99	39	28,8	cyclictest	31	1	ktimers/2	19:45:15	2
2725666	99	39	26,9	cyclictest	31	1	ktimers/2	00:30:12	2
2725664	99	39	28,6	cyclictest	0	-21	swapper/1	00:00:29	1
2725664	99	39	28,6	cyclictest	0	-21	swapper/1	00:00:29	1
2725664	99	39	26,9	cyclictest	2755688	-21	ptp4l	22:00:28	1
2725664	99	39	26,8	cyclictest	2727494	-21	grep	19:15:30	1
2725663	99	39	33,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	20:50:27	0
2725663	99	39	32,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:50:12	0
2725663	99	39	32,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	19:50:12	0
2725663	99	39	29,7	cyclictest	0	-21	swapper/0	21:20:26	0
2725663	99	39	27,8	cyclictest	0	-21	swapper/0	21:45:28	0
2725663	99	39	26,9	cyclictest	0	-21	swapper/0	19:15:00	0
2725663	99	39	24,8	cyclictest	0	-21	swapper/0	22:30:20	0
2725663	99	39	1,36	cyclictest	0	-21	swapper/0	00:10:19	0
2725666	99	38	28,6	cyclictest	2747182	-21	grep	21:10:34	2
2725666	99	38	27,8	cyclictest	31	1	ktimers/2	20:35:12	2
2725666	99	38	27,6	cyclictest	2756705	-21	grep	22:05:28	2
2725666	99	38	25,8	cyclictest	0	-21	swapper/2	22:50:19	2
2725664	99	38	28,6	cyclictest	2758402	-21	grep	22:15:28	1
2725664	99	38	28,6	cyclictest	2758402	-21	grep	22:15:27	1
2725664	99	38	25,7	cyclictest	0	-21	swapper/1	23:45:28	1
2725664	99	38	1,19	cyclictest	0	-21	swapper/1	21:30:02	1
2725664	99	38	1,19	cyclictest	0	-21	swapper/1	21:30:02	1
2725663	99	38	22,14	cyclictest	0	-21	swapper/0	00:30:15	0
2725663	99	38	1,35	cyclictest	0	-21	swapper/0	23:50:14	0
2725663	99	38	1,35	cyclictest	0	-21	swapper/0	22:05:13	0
2725663	99	38	1,34	cyclictest	13	1	ktimers/0	19:55:28	0
2725666	99	37	29,4	cyclictest	0	-21	swapper/2	21:00:00	2
2725666	99	37	27,6	cyclictest	0	-21	swapper/2	20:50:27	2
2725666	99	37	24,7	cyclictest	0	-21	swapper/2	22:40:29	2
2725666	99	37	23,10	cyclictest	0	-21	swapper/2	20:40:27	2
2725664	99	37	24,8	cyclictest	0	-21	swapper/1	00:30:28	1
2725664	99	37	0,30	cyclictest	0	-21	swapper/1	23:35:26	1
2725663	99	37	27,6	cyclictest	2779736	-21	grep	00:20:28	0
2725663	99	37	1,34	cyclictest	0	-21	swapper/0	20:35:15	0
2725666	99	36	27,6	cyclictest	0	-21	swapper/2	00:00:18	2
2725666	99	36	27,6	cyclictest	0	-21	swapper/2	00:00:18	2
2725666	99	36	26,6	cyclictest	2782304	-21	grep	00:35:29	2
2725666	99	36	25,8	cyclictest	0	-21	swapper/2	21:15:35	2
2725666	99	36	24,8	cyclictest	0	-21	swapper/2	00:25:29	2
2725664	99	36	28,4	cyclictest	0	-21	swapper/1	19:40:15	1
2725664	99	36	25,7	cyclictest	0	-21	swapper/1	23:05:28	1
2725664	99	36	25,7	cyclictest	0	-21	swapper/1	23:05:27	1
2725664	99	36	24,8	cyclictest	0	-21	swapper/1	22:40:29	1
2725664	99	36	23,8	cyclictest	0	-21	swapper/1	19:35:28	1
2725664	99	36	15,14	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	23:32:52	1
2725663	99	36	2,4	cyclictest	2751862	-21	fschecks_time	21:40:21	0
2725663	99	36	1,33	cyclictest	0	-21	swapper/0	22:35:23	0
2725666	99	35	26,7	cyclictest	31	1	ktimers/2	19:10:13	2
2725666	99	35	26,5	cyclictest	2766017	-21	grep	23:00:28	2
2725666	99	35	25,7	cyclictest	2772817	-21	grep	23:40:26	2
2725666	99	35	25,6	cyclictest	0	-21	swapper/2	23:50:30	2
2725666	99	35	24,6	cyclictest	2727540	-21	grep	19:15:30	2
2725666	99	35	23,8	cyclictest	2735959	-21	grep	20:05:27	2
2725666	99	35	22,8	cyclictest	0	-21	swapper/2	22:25:28	2
2725664	99	35	25,7	cyclictest	2781715	-21	/usr/sbin/munin	00:35:20	1
2725664	99	35	25,7	cyclictest	0	-21	swapper/1	19:20:31	1
2725664	99	35	20,9	cyclictest	0	-21	swapper/1	21:10:30	1
2725664	99	35	1,2	cyclictest	0	-21	swapper/1	00:30:00	1
2725664	99	35	1,28	cyclictest	0	-21	swapper/1	00:20:21	1
2725663	99	35	29,4	cyclictest	0	-21	swapper/0	20:20:10	0
2725663	99	35	28,5	cyclictest	0	-21	swapper/0	20:07:19	0
2725663	99	35	28,4	cyclictest	0	-21	swapper/0	22:55:34	0
2725663	99	35	28,4	cyclictest	0	-21	swapper/0	22:55:34	0
2725663	99	35	17,16	cyclictest	0	-21	swapper/0	20:10:25	0
2725663	99	35	1,32	cyclictest	0	-21	swapper/0	23:30:14	0
2725666	99	34	18,11	cyclictest	0	-21	swapper/2	00:10:19	2
2725664	99	34	27,4	cyclictest	2748976	-21	cron	21:25:00	1
2725664	99	34	21,9	cyclictest	0	-21	swapper/1	21:05:27	1
2725664	99	34	21,8	cyclictest	2732574	-21	tail	19:45:28	1
2725664	99	34	20,6	cyclictest	0	-21	swapper/1	00:05:28	1
2725664	99	34	19,3	cyclictest	0	-21	swapper/1	21:05:01	1
2725666	99	33	24,5	cyclictest	0	-21	swapper/2	21:30:17	2
2725666	99	33	23,8	cyclictest	31	1	ktimers/2	19:55:11	2
2725666	99	33	23,7	cyclictest	2731284	-21	grep	19:40:19	2
2725666	99	33	22,7	cyclictest	0	-21	swapper/2	22:55:21	2
2725666	99	33	22,7	cyclictest	0	-21	swapper/2	22:55:21	2
2725666	99	33	20,9	cyclictest	0	-21	swapper/2	22:35:29	2
2725666	99	33	20,6	cyclictest	0	-21	swapper/2	21:50:01	2
2725666	99	33	20,10	cyclictest	0	-21	swapper/2	23:15:28	2
2725666	99	33	1,26	cyclictest	0	-21	swapper/2	23:55:28	2
2725664	99	33	23,5	cyclictest	0	-21	swapper/1	22:15:00	1
2725664	99	33	21,8	cyclictest	2767814	-21	cron	23:15:00	1
2725664	99	33	14,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	00:15:18	1
2725664	99	33	14,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	00:15:17	1
2725664	99	33	1,27	cyclictest	0	-21	swapper/1	19:50:28	1
2725664	99	33	1,27	cyclictest	0	-21	swapper/1	19:50:28	1
2725664	99	33	1,26	cyclictest	0	-21	swapper/1	20:20:14	1
2725663	99	33	26,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	22:40:16	0
2725666	99	32	15,12	cyclictest	31	1	ktimers/2	20:00:11	2
2725666	99	32	1,25	cyclictest	0	-21	swapper/2	20:25:18	2
2725664	99	32	18,7	cyclictest	0	-21	swapper/1	22:25:27	1
2725664	99	32	18,5	cyclictest	0	-21	swapper/1	19:10:00	1
2725664	99	32	1,3	cyclictest	0	-21	swapper/1	20:05:01	1
2725663	99	32	1,2	cyclictest	0	-21	swapper/0	23:25:02	0
2725666	99	31	25,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	23:47:16	2
2725666	99	31	21,7	cyclictest	0	-21	swapper/2	21:50:28	2
2725666	99	31	21,7	cyclictest	0	-21	swapper/2	21:50:28	2
2725666	99	31	1,3	cyclictest	2751982	-21	irqstats	21:40:22	2
2725666	99	31	1,24	cyclictest	0	-21	swapper/2	22:30:11	2
2725664	99	31	23,4	cyclictest	0	-21	swapper/1	19:30:27	1
2725666	99	30	20,7	cyclictest	0	-21	swapper/2	23:30:28	2
2725666	99	30	1,27	cyclictest	0	-21	swapper/2	22:45:29	2
2725664	99	30	26,2	cyclictest	0	-21	swapper/1	20:25:19	1
2725663	99	30	1,25	cyclictest	15	-21	rcuc/0	19:10:00	0
2725666	99	29	5,8	cyclictest	2728390	-21	grep	19:20:30	2
2725664	99	29	20,6	cyclictest	0	-21	swapper/1	21:55:30	1
2725664	99	29	19,7	cyclictest	2734296	-21	grep	19:55:29	1
2725666	99	28	22,4	cyclictest	0	-21	swapper/2	22:19:54	2
2725666	99	28	22,4	cyclictest	0	-21	swapper/2	22:19:54	2
2725666	99	28	14,9	cyclictest	0	-21	swapper/2	20:20:19	2
2725664	99	28	21,5	cyclictest	23	1	ktimers/1	21:30:22	1
2725664	99	28	19,3	cyclictest	2760337	-21	/usr/sbin/munin	22:30:15	1
2725664	99	25	8,13	cyclictest	0	-21	swapper/1	22:20:17	1
2725666	99	24	19,3	cyclictest	14	-21	rcu_preempt	19:10:01	2
2725664	99	24	19,3	cyclictest	0	-21	swapper/1	20:50:15	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2020-10-04 - 12:21