OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Sat Apr 04, 2026 12:45:19)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
634010	99	76	6,51	cyclictest	717909	-21	ssh	12:10:01	1
634022	99	67	1,12	cyclictest	3467602	-21	systemd-journal	09:25:41	3
634022	99	66	1,12	cyclictest	48	-21	kcompactd0	08:45:25	3
634022	99	65	35,25	cyclictest	275	-21	dbus-daemon	10:20:01	3
634022	99	65	2,18	cyclictest	703144	-21	ssh	11:35:31	3
634022	99	65	1,14	cyclictest	681827	-21	ssh	10:46:14	3
634022	99	64	1,13	cyclictest	649152	-21	ssh	09:30:40	3
634022	99	63	2,18	cyclictest	676447	-21	sshd	10:34:16	3
634022	99	63	2,12	cyclictest	364	-21	snmpd	10:38:58	3
634022	99	63	2,11	cyclictest	364	-21	snmpd	11:55:48	3
634022	99	63	1,9	cyclictest	150	-21	pr/ttyS2	09:47:13	3
634022	99	62	1,14	cyclictest	642406	-21	munin-run	09:15:01	3
634022	99	62	1,13	cyclictest	671489	-21	ssh	10:22:27	3
634022	99	62	1,10	cyclictest	653515	-21	ssh	09:40:43	3
634022	99	61	2,14	cyclictest	665957	-21	munin-run	10:10:01	3
634022	99	61	1,9	cyclictest	686851	-21	ssh	10:58:05	3
634022	99	61	1,13	cyclictest	720749	-21	sh	12:16:32	3
634022	99	61	1,13	cyclictest	639395	-21	munin-run	08:55:01	3
634022	99	61	1,13	cyclictest	639395	-21	munin-run	08:55:00	3
634022	99	61	1,13	cyclictest	637036	-21	sshd	08:06:16	3
634022	99	61	1,13	cyclictest	637036	-21	sshd	08:06:16	3
634022	99	61	1,10	cyclictest	659115	-21	ssh	09:54:09	3
634008	99	61	26,17	cyclictest	713533	-21	ssh	12:00:00	0
634008	99	61	21,36	cyclictest	704886	-21	latency_hist	11:40:01	0
634008	99	61	1,57	cyclictest	0	-21	swapper/0	09:35:02	0
634022	99	60	31,24	cyclictest	693551	-21	sh	11:13:47	3
634022	99	60	2,6	cyclictest	635512	-21	munin-run	07:40:00	3
634022	99	60	22,33	cyclictest	37	-21	ksoftirqd/3	12:32:16	3
634022	99	60	1,9	cyclictest	695047	-21	ssh	11:16:55	3
634022	99	60	1,9	cyclictest	639894	-21	sshd	09:00:37	3
634022	99	60	1,12	cyclictest	707655	-21	sshd	11:46:06	3
634008	99	60	35,20	cyclictest	657317	-21	munin-run	09:49:59	0
634008	99	60	32,12	cyclictest	685220	-21	ssh	10:54:26	0
634022	99	59	9,46	cyclictest	0	-21	swapper/3	07:18:19	3
634022	99	59	2,20	cyclictest	644264	-21	ssh	09:19:19	3
634022	99	59	2,13	cyclictest	48	-21	kcompactd0	12:27:03	3
634022	99	59	2,12	cyclictest	364	-21	snmpd	10:14:51	3
634022	99	59	1,13	cyclictest	688329	-21	sshd	11:01:16	3
634022	99	59	10,44	cyclictest	0	-21	swapper/3	07:43:17	3
634022	99	58	2,22	cyclictest	3467602	-21	systemd-journal	11:05:15	3
634022	99	58	2,16	cyclictest	680719	-21	sh	10:44:00	3
634022	99	58	2,15	cyclictest	652196	-21	ssh	09:37:53	3
634022	99	58	15,37	cyclictest	645042	-21	sh	09:20:42	3
634022	99	58	13,40	cyclictest	636256	-21	sshd	07:52:00	3
634022	99	58	1,14	cyclictest	364	-21	snmpd	08:00:59	3
634022	99	58	1,11	cyclictest	699626	-21	ssh	11:27:30	3
634008	99	58	22,32	cyclictest	12	-21	ksoftirqd/0	09:45:01	0
634008	99	58	15,4	cyclictest	13	1	rcu_preempt	09:20:52	0
634022	99	57	6,4	cyclictest	0	-21	swapper/3	07:57:14	3
634022	99	57	3,9	cyclictest	639660	-21	grep	09:00:00	3
634022	99	57	3,16	cyclictest	640205	-21	munin-run	09:10:01	3
634022	99	57	2,17	cyclictest	685389	-21	munin-run	10:55:01	3
634022	99	57	2,13	cyclictest	723054	-21	ssh	12:21:54	3
634022	99	57	2,13	cyclictest	723054	-21	ssh	12:21:54	3
634022	99	57	2,11	cyclictest	698416	-21	munin-run	11:25:01	3
634022	99	57	12,40	cyclictest	719126	-21	journal-offline	12:12:53	3
634010	99	57	25,21	cyclictest	22	1	rcuc/1	10:55:02	1
634010	99	57	13,8	cyclictest	23	-21	ksoftirqd/1	12:38:38	1
634008	99	57	36,15	cyclictest	1	-21	systemd	09:15:00	0
634008	99	57	23,28	cyclictest	12	-21	ksoftirqd/0	12:15:32	0
634008	99	57	1,40	cyclictest	699011	-21	ssh	11:25:55	0
634022	99	56	7,20	cyclictest	634346	-21	cat	07:15:02	3
634022	99	56	3,7	cyclictest	637849	-21	modprobe	08:25:00	3
634022	99	56	2,9	cyclictest	364	-21	snmpd	07:46:47	3
634022	99	56	2,8	cyclictest	364	-21	snmpd	08:32:51	3
634022	99	56	1,7	cyclictest	635238	-21	cron	07:35:00	3
634022	99	56	17,35	cyclictest	663534	-21	sh	10:04:18	3
634008	99	56	8,30	cyclictest	169	50	irq/224-8000000	12:35:02	0
634022	99	55	3,10	cyclictest	661625	-21	munin-run	10:00:00	3
634022	99	55	2,18	cyclictest	672855	-21	ssh	10:25:29	3
634022	99	55	2,13	cyclictest	364	-21	snmpd	12:37:57	3
634016	99	55	9,30	cyclictest	643188	-21	kworker/u8:0	09:20:03	2
634010	99	55	15,10	cyclictest	0	-21	swapper/1	11:06:31	1
634010	99	55	10,8	cyclictest	23	-21	ksoftirqd/1	11:46:50	1
634008	99	55	31,17	cyclictest	169	50	irq/224-8000000	11:50:02	0
634008	99	55	28,12	cyclictest	671697	-21	ssh	10:23:06	0
634008	99	55	23,15	cyclictest	662073	-21	ssh	10:00:29	0
634008	99	55	18,35	cyclictest	12	-21	ksoftirqd/0	10:10:02	0
634022	99	54	3,8	cyclictest	717159	-21	ssh	12:08:24	3
634022	99	54	3,6	cyclictest	284	-21	systemd-logind	08:30:01	3
634022	99	54	30,20	cyclictest	702622	-21	sh	11:34:48	3
634022	99	54	2,6	cyclictest	637317	-21	munin-run	08:15:00	3
634022	99	54	1,14	cyclictest	364	-21	snmpd	11:51:48	3
634022	99	54	1,12	cyclictest	364	-21	snmpd	12:04:01	3
634022	99	54	1,12	cyclictest	364	-21	snmpd	12:04:00	3
634010	99	54	4,12	cyclictest	23	-21	ksoftirqd/1	10:18:22	1
634010	99	54	26,19	cyclictest	709239	-21	kworker/1:0	12:30:02	1
634010	99	54	20,5	cyclictest	0	-21	swapper/1	09:45:01	1
634008	99	54	3,37	cyclictest	13	1	rcu_preempt	12:14:12	0
634008	99	54	27,12	cyclictest	701469	-21	ssh	11:31:42	0
634008	99	54	1,34	cyclictest	13	1	rcu_preempt	10:57:49	0
634022	99	53	24,23	cyclictest	634996	-21	latency_hist	07:30:02	3
634022	99	53	1,13	cyclictest	364	-21	snmpd	07:21:16	3
634016	99	53	42,7	cyclictest	30	-21	ksoftirqd/2	11:45:03	2
634016	99	53	3,3	cyclictest	13	1	rcu_preempt	11:27:15	2
634010	99	53	38,13	cyclictest	23	-21	ksoftirqd/1	12:18:28	1
634010	99	53	10,6	cyclictest	0	-21	swapper/1	10:31:08	1
634008	99	53	44,5	cyclictest	169	50	irq/224-8000000	07:50:00	0
634008	99	53	26,12	cyclictest	690020	-21	ssh	11:05:07	0
634008	99	53	26,12	cyclictest	674586	-21	ssh	10:29:53	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33