OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rack3slot5s.osadl.org (updated Tue Mar 24, 2026 00:43:31)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
31698	2	116	0	sleep1	1215	50	irq/30-eth0_g0_	00:00:41	1
31147	2	114	0	chrt	1215	50	irq/30-eth0_g0_	21:46:58	1
12	1	89	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:05:13	1
4799	2	76	0	sleep1	1214	50	irq/29-eth0_g0_	22:11:17	1
4831	2	65	0	sleep0	1215	50	irq/30-eth0_g0_	22:11:39	0
12009	2	55	0	sleep1	1215	50	irq/30-eth0_g0_	22:40:22	1
8733	2	52	0	sleep0	1214	50	irq/29-eth0_g0_	22:29:16	0
26303	2	51	0	sleep1	1215	50	irq/30-eth0_g0_	19:15:17	1
12	1	49	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	00:19:40	1
12	1	48	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	21:54:37	1
24393	2	46	0	sleep1	1215	50	irq/30-eth0_g0_	23:31:12	1
12	1	46	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:36:28	1
12	1	46	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:30:44	1
12	1	46	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:02:13	1
12	1	46	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	20:05:22	0
1214	50	45	0	irq/29-eth0_g0_	1947	-21	runrttasks	19:07:01	0
12	1	45	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:50:05	1
1214	50	44	0	irq/29-eth0_g0_	12	1	ksoftirqd/1	20:55:18	1
1214	50	44	0	irq/29-eth0_g0_	11208	-21	sendmail_mailqu	20:25:20	0
1214	50	44	0	irq/29-eth0_g0_	11208	-21	sendmail_mailqu	20:25:20	0
12	1	44	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:45:16	0
12	1	44	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:38:16	0
1215	50	43	0	irq/30-eth0_g0_	1227	-21	kworker/1:2	23:15:20	1
12	1	43	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:42:47	1
12	1	43	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:48:25	0
12	1	43	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	00:35:13	1
12	1	43	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	00:25:12	0
12	1	43	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	21:25:00	1
4503	2	42	0	sleep1	1215	50	irq/30-eth0_g0_	00:24:24	1
24609	99	42	8	cyclictest	29564	-21	latency_hist	19:30:00	1
24609	99	42	2	cyclictest	936	-21	sshd	00:10:12	1
1214	50	42	0	irq/29-eth0_g0_	12	1	ksoftirqd/1	22:01:52	0
12	1	42	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	23:51:42	0
12	1	42	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	23:20:45	1
1214	50	41	0	irq/29-eth0_g0_	563	-21	seq	00:06:48	0
1214	50	41	0	irq/29-eth0_g0_	3	1	ksoftirqd/0	23:11:54	0
1214	50	41	0	irq/29-eth0_g0_	27076	-21	proc_pri	21:30:20	0
1214	50	41	0	irq/29-eth0_g0_	27076	-21	proc_pri	21:30:20	0
1214	50	41	0	irq/29-eth0_g0_	13	-21	kworker/0:1	22:55:50	0
1214	50	41	0	irq/29-eth0_g0_	13	-21	kworker/0:1	22:44:18	0
12	1	41	0	ksoftirqd/1	22600	-21	runrttasks	21:12:15	1
12	1	41	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:55:55	0
12	1	41	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:20:00	0
12	1	41	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	00:36:41	0
12	1	41	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	21:09:59	1
24609	99	40	1	cyclictest	6905	-21	ps	22:20:19	1
24609	99	40	1	cyclictest	3697	-21	ntp_kernel_pll_	19:55:16	1
1215	50	40	0	irq/30-eth0_g0_	1947	-21	runrttasks	19:50:12	0
1215	50	40	0	irq/30-eth0_g0_	12	1	ksoftirqd/1	23:57:14	1
1215	50	40	0	irq/30-eth0_g0_	12	1	ksoftirqd/1	21:58:57	1
1214	50	40	0	irq/29-eth0_g0_	28527	-21	grep	21:35:22	0
1214	50	40	0	irq/29-eth0_g0_	13	-21	kworker/0:1	23:16:27	0
1214	50	40	0	irq/29-eth0_g0_	13	-21	kworker/0:1	21:45:47	0
1214	50	40	0	irq/29-eth0_g0_	13	-21	kworker/0:1	21:25:05	0
1214	50	40	0	irq/29-eth0_g0_	13	-21	kworker/0:1	21:18:33	0
1214	50	40	0	irq/29-eth0_g0_	13	-21	kworker/0:1	00:31:58	0
1214	50	40	0	irq/29-eth0_g0_	13	-21	kworker/0:1	00:31:58	0
1214	50	40	0	irq/29-eth0_g0_	12	1	ksoftirqd/1	00:24:34	0
12	1	40	0	ksoftirqd/1	23846	-21	/usr/sbin/munin	19:05:17	1
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:30:54	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:31:53	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:21:57	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	21:56:07	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	21:43:51	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	21:00:29	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	20:57:11	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	19:57:18	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	19:43:26	0
12	1	40	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	19:37:57	0
12	1	40	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	00:27:34	1
24609	99	39	1	cyclictest	2723	-21	grep	19:50:20	1
24609	99	39	1	cyclictest	21612	-21	diskmemload	21:30:20	1
24609	99	39	1	cyclictest	21612	-21	diskmemload	21:30:20	1
1215	50	39	0	irq/30-eth0_g0_	22364	-21	/usr/sbin/munin	23:25:13	1
1215	50	39	0	irq/30-eth0_g0_	12725	-21	df_inode	22:45:15	1
1215	50	39	0	irq/30-eth0_g0_	1227	-21	kworker/1:2	22:09:42	1
1214	50	39	0	irq/29-eth0_g0_	3	1	ksoftirqd/0	20:20:00	0
1214	50	39	0	irq/29-eth0_g0_	13	-21	kworker/0:1	23:25:52	0
1214	50	39	0	irq/29-eth0_g0_	13	-21	kworker/0:1	00:19:25	0
1214	50	39	0	irq/29-eth0_g0_	13	-21	kworker/0:1	00:10:16	0
1214	50	39	0	irq/29-eth0_g0_	13	-21	kworker/0:1	00:01:15	0
12	1	39	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:06:21	0
12	1	39	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	23:00:08	0
12	1	39	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	22:06:58	0
12	1	39	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	21:21:41	0
12	1	39	0	ksoftirqd/1	1215	50	irq/30-eth0_g0_	20:01:00	0
12	1	39	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	21:18:50	1
24609	99	38	2	cyclictest	8335	-21	grep	22:25:22	1
24609	99	38	1	cyclictest	6228	-21	proc_pri	20:05:19	1
1215	50	38	0	irq/30-eth0_g0_	1227	-21	kworker/1:2	23:46:02	1
1215	50	38	0	irq/30-eth0_g0_	1227	-21	kworker/1:2	22:37:23	1
1215	50	38	0	irq/30-eth0_g0_	0	-21	swapper/1	21:35:21	1
1214	50	38	0	irq/29-eth0_g0_	21708	-21	/usr/sbin/munin	21:10:13	0
1214	50	38	0	irq/29-eth0_g0_	1947	-21	runrttasks	20:20:30	0
1214	50	38	0	irq/29-eth0_g0_	1405	-21	mailstats	19:45:22	0
1214	50	38	0	irq/29-eth0_g0_	1405	-21	mailstats	19:45:22	0
1214	50	38	0	irq/29-eth0_g0_	13	-21	kworker/0:1	23:42:54	0
1214	50	38	0	irq/29-eth0_g0_	13	-21	kworker/0:1	21:50:58	0
24609	99	37	2	cyclictest	8602	2	sleep1	20:15:17	1
24609	99	37	2	cyclictest	334	-21	grep	00:05:23	1
24609	99	37	2	cyclictest	19635	-21	awk	21:00:17	1
24609	99	37	2	cyclictest	10590	-21	latency_hist	20:25:00	1
24609	99	37	1	cyclictest	26057	-21	grep	21:25:22	1
24609	99	37	1	cyclictest	16376	2	sleep1	20:46:10	1
24609	99	37	1	cyclictest	14333	-21	taskset	20:40:00	1
24609	99	37	1	cyclictest	12203	-21	missed_timers	20:30:17	1
1214	50	37	0	irq/29-eth0_g0_	30557	2	sleep0	19:32:56	0
1214	50	37	0	irq/29-eth0_g0_	1947	-21	runrttasks	21:09:02	0
1214	50	37	0	irq/29-eth0_g0_	1947	-21	runrttasks	20:53:05	0
1214	50	37	0	irq/29-eth0_g0_	16264	-21	grep	20:45:24	0
1214	50	37	0	irq/29-eth0_g0_	13	-21	kworker/0:1	23:22:16	0
1214	50	37	0	irq/29-eth0_g0_	13	-21	kworker/0:1	22:51:44	0
1214	50	37	0	irq/29-eth0_g0_	12542	-21	unixbench_multi	20:30:23	0
1214	50	37	0	irq/29-eth0_g0_	1227	-21	kworker/1:2	23:04:40	1
1214	50	37	0	irq/29-eth0_g0_	12	1	ksoftirqd/1	22:17:37	1
12	1	37	0	ksoftirqd/1	1214	50	irq/29-eth0_g0_	22:55:15	1
24609	99	36	1	cyclictest	30064	-21	ps	19:30:18	1
24609	99	36	1	cyclictest	27847	-21	unixbench_singl	19:20:22	1
24609	99	36	1	cyclictest	0	-21	swapper/1	21:40:15	1
24609	99	36	0	cyclictest	0	-21	swapper/1	20:15:00	1
1215	50	36	0	irq/30-eth0_g0_	3	1	ksoftirqd/0	23:35:10	0
1215	50	36	0	irq/30-eth0_g0_	18876	-21	if_err_eth0	23:10:11	1
1214	50	36	0	irq/29-eth0_g0_	28909	-21	sendmail_mailqu	19:25:19	0
24609	99	35	1	cyclictest	6274	-21	ntp_states	00:30:16	1
24609	99	35	1	cyclictest	6274	-21	ntp_states	00:30:16	1
24609	99	35	1	cyclictest	0	-21	swapper/1	20:26:49	1
24609	99	35	1	cyclictest	0	-21	swapper/1	20:26:49	1
1215	50	35	0	irq/30-eth0_g0_	12	1	ksoftirqd/1	22:50:16	1
1214	50	35	0	irq/29-eth0_g0_	8189	-21	sort	20:15:00	0
1214	50	35	0	irq/29-eth0_g0_	27697	-21	/usr/sbin/munin	19:20:20	0
1214	50	35	0	irq/29-eth0_g0_	25934	-21	cpu	19:15:11	0
1214	50	35	0	irq/29-eth0_g0_	25341	-21	unixbench_multi	19:10:18	0
1214	50	35	0	irq/29-eth0_g0_	1947	-21	runrttasks	20:42:15	0
1214	50	35	0	irq/29-eth0_g0_	13584	-21	taskset	20:35:20	0
24609	99	34	1	cyclictest	30824	2	sleep1	19:35:10	1
24609	99	34	1	cyclictest	0	-21	swapper/1	20:52:37	1
24609	99	33	18	cyclictest	25863	-21	wc	19:15:01	1
1214	50	33	0	irq/29-eth0_g0_	5226	-21	sed	20:00:23	1
12	1	33	0	ksoftirqd/1	14212	-21	taskset	20:39:29	1
1215	50	32	0	irq/30-eth0_g0_	1473	-21	unixbench_multi	19:45:23	1
1215	50	32	0	irq/30-eth0_g0_	1473	-21	unixbench_multi	19:45:23	1
1215	50	32	0	irq/30-eth0_g0_	0	-21	swapper/1	19:40:11	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17