OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Fri May 22, 2026 00:45:30)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3024385	99	90	27,11	cyclictest	3084233	-21	ssh	23:00:56	0
3024385	99	83	27,13	cyclictest	3117751	-21	ssh	00:10:27	0
3024385	99	82	1,37	cyclictest	3110727	-21	ssh	23:55:50	0
3024385	99	82	1,37	cyclictest	3110727	-21	ssh	23:55:50	0
3024385	99	76	27,12	cyclictest	3047874	-21	ssh	21:45:25	0
3024385	99	75	24,13	cyclictest	3093222	-21	run-parts	23:20:00	0
3024385	99	74	23,17	cyclictest	3131872	-21	ssh	00:40:01	0
3024385	99	72	35,4	cyclictest	169	50	irq/224-8000000	22:55:00	0
3024385	99	72	2,11	cyclictest	3090951	-21	ls	23:15:02	0
3024385	99	70	42,24	cyclictest	169	50	irq/224-8000000	21:40:00	0
3024385	99	70	29,12	cyclictest	3088429	-21	ssh	23:09:59	0
3024385	99	67	26,12	cyclictest	3051552	-21	ssh	21:53:20	0
3024385	99	67	26,10	cyclictest	3120859	-21	ssh	00:16:55	0
3024390	99	66	24,37	cyclictest	3062130	-21	sh	22:15:04	1
3024385	99	66	29,12	cyclictest	3116145	-21	ssh	00:07:17	0
3024385	99	66	29,12	cyclictest	3116145	-21	ssh	00:07:17	0
3024390	99	65	3,13	cyclictest	3059516	-21	munin-run	22:10:01	1
3024390	99	65	25,35	cyclictest	3090671	-21	sh	23:14:41	1
3024385	99	65	18,25	cyclictest	169	50	irq/224-8000000	23:45:03	0
3024395	99	64	26,27	cyclictest	37	-21	ksoftirqd/3	23:55:02	3
3024390	99	64	2,14	cyclictest	3027204	-21	munin-run	20:05:01	1
3024385	99	64	21,11	cyclictest	3064865	-21	ssh	22:20:43	0
3024385	99	64	15,14	cyclictest	3061539	-21	ssh	22:14:09	0
3024385	99	64	14,29	cyclictest	12	-21	ksoftirqd/0	22:46:01	0
3024390	99	63	26,33	cyclictest	3038657	-21	sh	21:26:25	1
3024390	99	63	24,35	cyclictest	23	-21	ksoftirqd/1	00:01:03	1
3024390	99	63	2,14	cyclictest	3074012	-21	munin-run	22:40:01	1
3024390	99	63	1,15	cyclictest	3034319	-21	ssh	21:17:36	1
3024390	99	63	1,14	cyclictest	364	-21	snmpd	23:09:34	1
3024390	99	62	1,16	cyclictest	3024588	-21	munin-run	19:15:02	1
3024390	99	62	1,16	cyclictest	3024588	-21	munin-run	19:15:01	1
3024390	99	62	0,25	cyclictest	3049170	-21	ssh	21:48:25	1
3024390	99	61	2,18	cyclictest	3026923	-21	sendmail	20:00:00	1
3024390	99	61	21,36	cyclictest	23	-21	ksoftirqd/1	23:55:17	1
3024390	99	61	21,36	cyclictest	23	-21	ksoftirqd/1	23:55:17	1
3024390	99	61	21,35	cyclictest	3068626	-21	sh	22:28:42	1
3024390	99	61	1,15	cyclictest	48	-21	kcompactd0	22:50:37	1
3024390	99	61	1,13	cyclictest	3467602	-21	systemd-journal	22:24:41	1
3024390	99	60	9,4	cyclictest	0	-21	swapper/1	21:59:50	1
3024390	99	60	18,36	cyclictest	275	-21	dbus-daemon	00:15:01	1
3024390	99	60	17,39	cyclictest	3467562	-21	sshd	21:38:07	1
3024390	99	60	1,16	cyclictest	3050741	-21	ssh	21:51:26	1
3024395	99	59	7,6	cyclictest	13	1	rcu_preempt	22:56:32	3
3024392	99	59	50,6	cyclictest	3011910	-21	kworker/2:1	20:55:00	2
3024390	99	59	2,17	cyclictest	3030036	-21	munin-run	21:00:00	1
3024390	99	59	2,15	cyclictest	3130888	-21	ssh	00:37:52	1
3024390	99	59	2,14	cyclictest	364	-21	snmpd	21:00:05	1
3024390	99	59	1,16	cyclictest	3120248	-21	ssh	00:15:35	1
3024390	99	59	1,16	cyclictest	3081511	-21	ssh	22:55:10	1
3024390	99	59	1,13	cyclictest	3031881	-21	sh	21:12:24	1
3024390	99	59	10,4	cyclictest	0	-21	swapper/1	21:08:46	1
3024390	99	58	9,4	cyclictest	0	-21	swapper/1	20:42:54	1
3024390	99	58	9,3	cyclictest	0	-21	swapper/1	23:46:48	1
3024390	99	58	2,9	cyclictest	3076393	-21	modprobe	22:45:00	1
3024390	99	58	25,29	cyclictest	23	-21	ksoftirqd/1	23:17:22	1
3024390	99	58	2,14	cyclictest	3084971	-21	sshd	23:02:40	1
3024390	99	58	2,11	cyclictest	3026420	-21	munin-run	19:50:01	1
3024390	99	58	14,39	cyclictest	3055385	-21	sshd	22:01:08	1
3024390	99	58	13,12	cyclictest	23	-21	ksoftirqd/1	22:33:22	1
3024390	99	58	1,14	cyclictest	48	-21	kcompactd0	00:25:19	1
3024390	99	58	1,12	cyclictest	3037602	-21	sh	21:24:30	1
3024390	99	57	2,14	cyclictest	3104789	-21	ssh	23:43:41	1
3024390	99	57	21,33	cyclictest	23	-21	ksoftirqd/1	19:29:59	1
3024390	99	57	19,33	cyclictest	3128530	-21	sh	00:32:59	1
3024390	99	57	1,16	cyclictest	3467602	-21	systemd-journal	23:39:53	1
3024390	99	57	1,16	cyclictest	3026295	-21	latency_hist	19:45:01	1
3024385	99	57	33,11	cyclictest	12	-21	ksoftirqd/0	22:40:01	0
3024395	99	56	39,12	cyclictest	3467602	-21	systemd-journal	22:10:01	3
3024390	99	56	5,8	cyclictest	0	-21	swapper/1	20:15:01	1
3024390	99	56	3,19	cyclictest	3047458	-21	ssh	21:45:00	1
3024390	99	56	2,8	cyclictest	3027945	-21	cron	20:20:00	1
3024390	99	56	2,8	cyclictest	3027945	-21	cron	20:19:59	1
3024390	99	56	2,17	cyclictest	3040493	-21	ssh	21:30:05	1
3024390	99	56	2,15	cyclictest	3060005	-21	ssh	22:10:37	1
3024390	99	56	13,6	cyclictest	3030561	-21	diskmemload	23:26:51	1
3024390	99	56	1,11	cyclictest	3100528	-21	cron	23:35:00	1
3024390	99	56	1,10	cyclictest	3093647	-21	ssh	23:20:24	1
3024390	99	56	1,10	cyclictest	3093647	-21	ssh	23:20:23	1
3024385	99	56	26,15	cyclictest	3054150	-21	ssh	21:58:44	0
3024385	99	56	2,49	cyclictest	3037969	-21	cat	21:25:02	0
3024395	99	55	36,14	cyclictest	3467602	-21	systemd-journal	00:31:37	3
3024392	99	55	46,5	cyclictest	3011910	-21	kworker/2:1	21:10:01	2
3024392	99	55	45,6	cyclictest	3011910	-21	kworker/2:1	21:15:18	2
3024390	99	55	12,37	cyclictest	275	-21	dbus-daemon	20:10:00	1
3024390	99	55	1,12	cyclictest	3077468	-21	ssh	22:46:57	1
3024390	99	55	1,11	cyclictest	3123324	-21	ssh	00:22:01	1
3024390	99	55	1,11	cyclictest	3123324	-21	ssh	00:22:01	1
3024385	99	55	28,23	cyclictest	3075250	-21	ssh	22:42:19	0
3024385	99	55	26,14	cyclictest	3097768	-21	ssh	23:29:16	0
3024385	99	55	26,13	cyclictest	3055065	-21	ssh	22:00:17	0
3024385	99	55	25,14	cyclictest	3062912	-21	ssh	22:16:53	0
3024385	99	55	1,3	cyclictest	13	1	rcu_preempt	00:31:14	0
3024385	99	55	1,38	cyclictest	3105853	-21	ssh	23:45:40	0
3024392	99	54	46,5	cyclictest	3045103	-21	kworker/2:3	23:31:28	2
3024392	99	54	45,6	cyclictest	3122213	-21	kworker/2:0	00:36:32	2
3024390	99	54	2,15	cyclictest	3108659	-21	ssh	23:51:40	1
3024390	99	54	2,15	cyclictest	3029789	-21	munin-run	20:55:01	1
3024390	99	54	2,11	cyclictest	3467602	-21	systemd-journal	20:37:37	1
3024390	99	54	10,5	cyclictest	0	-21	swapper/1	19:20:01	1
3024385	99	54	28,22	cyclictest	3095025	-21	ssh	23:23:32	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33