OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Wed Jul 29, 2026 00:45:31)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
4015404	99	97	1,8	cyclictest	13	1	rcu_preempt	21:55:02	3
4015390	99	86	53,23	cyclictest	12	-21	ksoftirqd/0	21:15:03	0
4015404	99	75	24,46	cyclictest	48	-21	kcompactd0	00:05:09	3
4015390	99	70	30,35	cyclictest	1	-21	systemd	00:40:02	0
4015404	99	67	22,39	cyclictest	4073473	-21	sh	22:58:47	3
4015390	99	67	31,32	cyclictest	12	-21	ksoftirqd/0	22:10:00	0
4015404	99	65	2,12	cyclictest	4053255	-21	ssh	22:16:12	3
4015399	99	65	19,9	cyclictest	30	-21	ksoftirqd/2	00:00:01	2
4015395	99	65	29,32	cyclictest	22	1	rcuc/1	21:15:02	1
4015404	99	64	31,30	cyclictest	4084654	-21	sshd	23:21:58	3
4015404	99	64	2,13	cyclictest	4016096	-21	sshd	19:21:19	3
4015390	99	64	35,4	cyclictest	12	-21	ksoftirqd/0	23:14:33	0
4015404	99	63	3,13	cyclictest	4033478	-21	munin-run	21:35:02	3
4015404	99	63	26,32	cyclictest	4093141	-21	cron	23:39:59	3
4015404	99	63	1,10	cyclictest	4016850	-21	sshd	19:36:22	3
4015399	99	63	13,9	cyclictest	30	-21	ksoftirqd/2	00:05:02	2
4015399	99	63	13,9	cyclictest	30	-21	ksoftirqd/2	00:05:01	2
4015390	99	63	1,46	cyclictest	4056471	1	ssh	22:23:02	0
4015404	99	62	2,7	cyclictest	4110986	-21	cron	00:17:00	3
4015404	99	62	24,34	cyclictest	4030511	-21	sh	21:28:46	3
4015404	99	62	1,18	cyclictest	4117120	-21	munin-plugin-st	00:30:00	3
4015404	99	62	1,14	cyclictest	4018217	-21	munin-run	20:05:01	3
4015404	99	62	1,13	cyclictest	4079901	-21	ssh	23:12:00	3
4015404	99	62	1,13	cyclictest	4044837	-21	ssh	21:58:46	3
4015404	99	62	1,13	cyclictest	4044837	-21	ssh	21:58:45	3
4015404	99	62	1,12	cyclictest	4068430	-21	ssh	22:47:59	3
4015404	99	61	2,14	cyclictest	4035984	-21	latency_hist	21:40:01	3
4015404	99	61	2,13	cyclictest	4090800	-21	munin-run	23:35:01	3
4015404	99	61	2,13	cyclictest	4090800	-21	munin-run	23:35:01	3
4015404	99	61	2,13	cyclictest	4021287	-21	sshd	21:00:54	3
4015404	99	61	2,13	cyclictest	4021287	-21	sshd	21:00:53	3
4015404	99	61	2,12	cyclictest	4104233	-21	ssh	00:02:53	3
4015404	99	61	2,12	cyclictest	4104233	-21	ssh	00:02:53	3
4015404	99	61	1,14	cyclictest	364	-21	snmpd	22:34:47	3
4015404	99	61	1,13	cyclictest	4017386	-21	sshd	19:46:22	3
4015399	99	61	6,47	cyclictest	30	-21	ksoftirqd/2	20:05:01	2
4015399	99	61	12,8	cyclictest	30	-21	ksoftirqd/2	21:50:02	2
4015399	99	61	12,8	cyclictest	30	-21	ksoftirqd/2	21:50:01	2
4015390	99	61	3,38	cyclictest	13	1	rcu_preempt	21:13:30	0
4015404	99	60	3,13	cyclictest	4114721	-21	munin-run	00:25:01	3
4015404	99	60	3,13	cyclictest	4114721	-21	munin-run	00:25:01	3
4015404	99	60	2,14	cyclictest	4017265	-21	cat	19:45:02	3
4015404	99	60	1,9	cyclictest	364	-21	snmpd	22:43:11	3
4015404	99	60	1,18	cyclictest	4059156	-21	ssh	22:28:47	3
4015404	99	60	1,18	cyclictest	4059156	-21	ssh	22:28:47	3
4015404	99	60	1,11	cyclictest	4120549	-21	sshd	00:37:01	3
4015404	99	60	1,10	cyclictest	4108562	-21	ssh	00:11:42	3
4015399	99	60	19,11	cyclictest	30	-21	ksoftirqd/2	22:15:01	2
4015395	99	60	26,14	cyclictest	23	-21	ksoftirqd/1	22:10:01	1
4015404	99	59	26,29	cyclictest	4015812	-21	sshd	19:16:53	3
4015404	99	59	23,32	cyclictest	4100328	-21	cron	23:55:01	3
4015404	99	59	2,14	cyclictest	4020537	-21	munin-run	20:50:00	3
4015404	99	59	2,12	cyclictest	4102743	-40	mandb	00:00:00	3
4015404	99	59	1,15	cyclictest	4074551	-21	ssh	23:00:39	3
4015399	99	59	15,9	cyclictest	30	-21	ksoftirqd/2	21:20:41	2
4015390	99	59	29,4	cyclictest	4091620	1	ssh	23:36:31	0
4015390	99	59	29,14	cyclictest	4032335	-21	ssh	21:32:27	0
4015404	99	58	3,7	cyclictest	4016367	-21	cron	19:30:00	3
4015404	99	58	2,13	cyclictest	4048303	-21	ssh	22:05:37	3
4015404	99	58	2,11	cyclictest	364	-21	snmpd	20:58:27	3
4015404	99	58	1,13	cyclictest	4118148	-21	rm	00:31:49	3
4015399	99	58	20,31	cyclictest	310	-21	in:imklog	19:30:02	2
4015399	99	58	16,8	cyclictest	30	-21	ksoftirqd/2	21:55:48	2
4015399	99	58	16,8	cyclictest	30	-21	ksoftirqd/2	21:55:48	2
4015399	99	58	14,10	cyclictest	30	-21	ksoftirqd/2	21:45:02	2
4015390	99	58	3,37	cyclictest	12	-21	ksoftirqd/0	22:46:24	0
4015390	99	58	28,14	cyclictest	4113711	-21	ssh	00:22:38	0
4015390	99	58	28,14	cyclictest	4113711	-21	ssh	00:22:38	0
4015390	99	58	28,14	cyclictest	4039050	-21	ssh	21:46:28	0
4015390	99	58	28,14	cyclictest	4039050	-21	ssh	21:46:27	0
4015390	99	58	26,15	cyclictest	4089398	-21	ssh	23:31:49	0
4015390	99	58	26,15	cyclictest	4089398	-21	ssh	23:31:49	0
4015390	99	58	14,26	cyclictest	13	1	rcu_preempt	23:58:30	0
4015404	99	57	3,14	cyclictest	4021556	-21	sshd	21:09:53	3
4015404	99	57	2,7	cyclictest	4095547	-21	cron	23:45:00	3
4015404	99	57	2,7	cyclictest	4095547	-21	cron	23:45:00	3
4015404	99	57	2,6	cyclictest	4019740	-21	cron	20:35:00	3
4015404	99	57	23,29	cyclictest	4078830	-21	cron	23:10:00	3
4015404	99	57	2,19	cyclictest	4052553	-21	cut	22:15:00	3
4015404	99	57	2,14	cyclictest	4020969	-21	latency_hist	20:55:03	3
4015404	99	57	1,14	cyclictest	364	-21	snmpd	20:26:21	3
4015404	99	57	1,12	cyclictest	364	-21	snmpd	22:04:26	3
4015404	99	57	1,11	cyclictest	4055647	-21	sh	22:21:06	3
4015390	99	57	30,17	cyclictest	12	-21	ksoftirqd/0	23:28:46	0
4015390	99	57	23,30	cyclictest	12	-21	ksoftirqd/0	22:43:21	0
4015390	99	57	1,39	cyclictest	4084733	-21	ssh	23:22:08	0
4015404	99	56	9,11	cyclictest	4070732	-21	sh	22:52:47	3
4015404	99	56	3,7	cyclictest	4016615	-21	sshd	19:34:52	3
4015404	99	56	2,18	cyclictest	4038256	-21	ssh	21:45:02	3
4015404	99	56	2,16	cyclictest	4023882	-21	ssh	21:15:00	3
4015404	99	56	2,16	cyclictest	4017684	-21	munin-run	19:55:01	3
4015404	99	56	2,14	cyclictest	4020255	-21	sshd	20:41:30	3
4015404	99	56	2,11	cyclictest	364	-21	snmpd	23:25:04	3
4015404	99	56	1,10	cyclictest	4063085	-21	ssh	22:36:54	3
4015404	99	56	1,10	cyclictest	4063085	-21	ssh	22:36:54	3
4015399	99	56	15,9	cyclictest	30	-21	ksoftirqd/2	21:30:02	2
4015399	99	56	10,8	cyclictest	30	-21	ksoftirqd/2	00:39:39	2
4015390	99	56	8,30	cyclictest	13	1	rcu_preempt	23:50:08	0
4015390	99	56	20,32	cyclictest	12	-21	ksoftirqd/0	22:12:01	0
4015404	99	55	2,8	cyclictest	4017892	-21	sshd	19:56:22	3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33