OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Sat Jun 06, 2026 00:45:18)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3653782	99	87	23,19	cyclictest	3696523	-21	ssh	22:31:52	2
3653782	99	86	28,20	cyclictest	3703621	-21	ssh	22:47:43	2
3653782	99	78	31,8	cyclictest	3743307	-21	ssh	00:15:58	2
3653782	99	78	25,19	cyclictest	3656632	-21	sed	20:05:02	2
3653782	99	78	24,19	cyclictest	3678965	-21	ssh	21:52:36	2
3653782	99	78	23,14	cyclictest	3736129	-21	ssh	00:00:04	2
3653782	99	76	30,15	cyclictest	3670664	-21	ssh	21:34:03	2
3653782	99	76	30,13	cyclictest	3722078	-21	sh	23:29:20	2
3653782	99	75	25,19	cyclictest	3467602	-21	systemd-journal	21:37:46	2
3653782	99	75	25,19	cyclictest	3467602	-21	systemd-journal	21:37:45	2
3653782	99	75	23,14	cyclictest	3728458	-21	ssh	23:43:18	2
3653782	99	75	22,19	cyclictest	3659905	-21	sed	21:05:03	2
3653782	99	75	22,19	cyclictest	3658259	-21	cut	20:35:03	2
3653782	99	75	22,18	cyclictest	3689187	-21	ssh	22:15:16	2
3653782	99	74	23,19	cyclictest	3708197	-21	ssh	22:58:12	2
3653782	99	74	23,17	cyclictest	3747672	-21	rm	00:25:42	2
3653782	99	74	22,20	cyclictest	3658998	-21	latency_hist	20:50:02	2
3653782	99	74	22,19	cyclictest	3698789	-21	sh	22:36:55	2
3653782	99	74	22,15	cyclictest	3732383	-21	ssh	23:51:53	2
3653782	99	73	23,16	cyclictest	3739029	-21	rm	00:06:37	2
3653782	99	73	21,17	cyclictest	3658742	-21	latency_hist	20:45:01	2
3653782	99	72	24,18	cyclictest	3686523	-21	rm	22:09:38	2
3653782	99	72	19,18	cyclictest	3674742	-21	ssh	21:43:03	2
3653782	99	71	18,13	cyclictest	3717481	-21	ssh	23:18:50	2
3653782	99	70	25,15	cyclictest	3467602	-21	systemd-journal	23:03:51	2
3653782	99	70	24,16	cyclictest	3683800	-21	sh	22:03:32	2
3653782	99	70	20,15	cyclictest	3725937	-21	ssh	23:37:48	2
3653782	99	70	20,15	cyclictest	3725937	-21	ssh	23:37:47	2
3653775	99	70	31,9	cyclictest	12	-21	ksoftirqd/0	22:57:27	0
3653782	99	69	22,15	cyclictest	3742350	-21	ssh	00:14:10	2
3653782	99	69	22,15	cyclictest	3687194	-21	ssh	22:10:44	2
3653782	99	68	30,15	cyclictest	3712175	-21	ssh	23:06:59	2
3653782	99	68	25,15	cyclictest	3750412	-21	ssh	00:32:07	2
3653782	99	68	23,14	cyclictest	3752949	-21	ssh	00:37:50	2
3653782	99	68	22,13	cyclictest	3666869	-21	ssh	21:25:04	2
3653782	99	68	21,17	cyclictest	3655189	-21	sed	19:35:02	2
3653782	99	68	19,14	cyclictest	3693898	-21	ssh	22:25:46	2
3653782	99	67	59,6	cyclictest	30	-21	ksoftirqd/2	23:25:02	2
3653782	99	66	50,13	cyclictest	3665255	-21	ssh	21:21:35	2
3653782	99	66	27,9	cyclictest	3724624	-21	munin-run	23:35:01	2
3653782	99	66	20,17	cyclictest	3661304	-21	ssh	21:12:47	2
3653775	99	66	8,28	cyclictest	13	1	rcu_preempt	22:00:04	0
3653775	99	66	57,6	cyclictest	169	50	irq/224-8000000	22:05:03	0
3653782	99	65	42,19	cyclictest	3467602	-21	systemd-journal	21:20:01	2
3653782	99	64	49,12	cyclictest	3729993	-21	ssh	23:46:34	2
3653782	99	64	43,16	cyclictest	3701239	-21	ssh	22:42:22	2
3653782	99	64	23,14	cyclictest	3735026	-21	rm	23:57:59	2
3653782	99	64	23,14	cyclictest	3735026	-21	rm	23:57:59	2
3653782	99	63	44,16	cyclictest	3714912	-21	ssh	23:13:07	2
3653782	99	63	42,17	cyclictest	3680526	-21	ssh	21:55:58	2
3653775	99	63	8,29	cyclictest	13	1	rcu_preempt	21:10:12	0
3653775	99	63	4,9	cyclictest	12	-21	ksoftirqd/0	23:50:00	0
3653775	99	63	41,19	cyclictest	168	50	irq/134-8000000	00:30:00	0
3653782	99	61	42,16	cyclictest	3704784	-21	ssh	22:50:12	2
3653782	99	61	40,17	cyclictest	3745627	-21	ssh	00:21:09	2
3653775	99	61	8,5	cyclictest	13	1	rcu_preempt	23:22:30	0
3653782	99	60	45,12	cyclictest	3691501	-21	ssh	22:20:24	2
3653782	99	60	45,12	cyclictest	3676301	-21	ssh	21:46:20	2
3653782	99	60	42,15	cyclictest	3657856	-21	sshd	20:27:27	2
3653778	99	60	38,9	cyclictest	23	-21	ksoftirqd/1	22:59:04	1
3653775	99	60	2,9	cyclictest	12	-21	ksoftirqd/0	00:00:02	0
3653775	99	60	2,9	cyclictest	12	-21	ksoftirqd/0	00:00:02	0
3653787	99	59	3,3	cyclictest	13	1	rcu_preempt	22:02:53	3
3653775	99	59	25,31	cyclictest	12	-21	ksoftirqd/0	23:04:08	0
3653775	99	59	14,6	cyclictest	13	1	rcu_preempt	21:30:55	0
3653787	99	58	2,41	cyclictest	3700159	-21	ssh	22:40:02	3
3653782	99	58	44,11	cyclictest	3657607	-21	sshd	20:22:28	2
3653775	99	57	7,9	cyclictest	12	-21	ksoftirqd/0	22:25:02	0
3653775	99	57	27,14	cyclictest	3680799	-21	ssh	21:56:46	0
3653775	99	57	16,20	cyclictest	168	50	irq/134-8000000	22:20:01	0
3653775	99	57	13,10	cyclictest	12	-21	ksoftirqd/0	21:27:51	0
3653775	99	57	12,5	cyclictest	13	1	rcu_preempt	20:35:03	0
3653787	99	56	32,18	cyclictest	275	-21	dbus-daemon	23:20:00	3
3653782	99	56	43,9	cyclictest	3658366	-21	sshd	20:38:29	2
3653782	99	56	40,10	cyclictest	3653963	-21	cron	19:15:01	2
3653782	99	56	36,16	cyclictest	364	-21	snmpd	19:47:41	2
3653782	99	56	34,17	cyclictest	364	-21	snmpd	19:51:08	2
3653778	99	56	1,9	cyclictest	3659960	-21	diskmemload	23:30:00	1
3653775	99	56	8,6	cyclictest	13	1	rcu_preempt	22:13:28	0
3653775	99	56	27,14	cyclictest	3727799	-21	ssh	23:41:42	0
3653775	99	56	22,31	cyclictest	12	-21	ksoftirqd/0	23:28:43	0
3653775	99	56	1,38	cyclictest	3723257	-21	ssh	23:31:39	0
3653782	99	55	48,4	cyclictest	3656853	-21	munin-run	20:10:01	2
3653782	99	55	46,5	cyclictest	3659145	-21	sshd	20:51:27	2
3653782	99	55	44,8	cyclictest	3655518	-21	sshd	19:40:27	2
3653782	99	55	42,10	cyclictest	3659928	-21	sshd	21:06:27	2
3653782	99	55	40,12	cyclictest	3654734	-21	sshd	19:25:26	2
3653782	99	55	33,16	cyclictest	3655243	-21	modprobe	19:36:28	2
3653778	99	55	5,6	cyclictest	13	1	rcu_preempt	22:10:30	1
3653775	99	55	3,21	cyclictest	12	-21	ksoftirqd/0	23:54:12	0
3653775	99	55	28,11	cyclictest	3712014	-21	ssh	23:06:36	0
3653775	99	55	27,12	cyclictest	3716446	-21	ssh	23:16:28	0
3653775	99	55	27,11	cyclictest	3749913	-21	ssh	00:30:50	0
3653775	99	55	11,6	cyclictest	13	1	rcu_preempt	22:52:30	0
3653775	99	55	10,5	cyclictest	13	1	rcu_preempt	22:33:01	0
3653782	99	54	37,12	cyclictest	364	-21	snmpd	19:15:33	2
3653782	99	54	24,25	cyclictest	3657361	-21	munin-plugin-st	20:20:01	2
3653778	99	54	37,13	cyclictest	1	-21	systemd	21:20:01	1
3653775	99	54	8,18	cyclictest	13	1	rcu_preempt	22:35:52	0
3653775	99	54	18,8	cyclictest	13	1	rcu_preempt	23:11:56	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33