OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot3s.osadl.org (updated Fri Jun 19, 2026 00:45:22)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3545522	99	81	29,15	cyclictest	3546519	-21	munin-run	19:30:01	1
3545522	99	80	29,16	cyclictest	3589705	-21	ssh	22:34:09	1
3545531	99	73	26,42	cyclictest	275	-21	dbus-daemon	00:15:02	3
3545526	99	73	26,14	cyclictest	3551537	-21	cat	21:05:01	2
3545526	99	72	30,13	cyclictest	3600564	-21	sh	22:58:08	2
3545522	99	72	28,13	cyclictest	3640394	-21	ssh	00:26:29	1
3545526	99	71	20,14	cyclictest	3601913	-21	ssh	23:00:47	2
3545531	99	70	20,45	cyclictest	275	-21	dbus-daemon	00:10:01	3
3545526	99	70	28,14	cyclictest	3560263	-21	ssh	21:28:53	2
3545526	99	70	24,15	cyclictest	3605265	-21	sh	23:08:33	2
3545526	99	70	24,15	cyclictest	3605265	-21	sh	23:08:33	2
3545526	99	70	22,13	cyclictest	3606323	-21	ssh	23:10:33	2
3545522	99	69	26,37	cyclictest	3549907	-21	cron	20:35:01	1
3545522	99	68	28,13	cyclictest	3613174	-21	ssh	23:25:34	1
3545531	99	67	11,53	cyclictest	3589042	-21	ssh	22:32:27	3
3545522	99	67	25,13	cyclictest	150	-21	pr/ttyS2	23:24:55	1
3545531	99	66	13,49	cyclictest	3555715	-21	ssh	21:18:42	3
3545526	99	66	30,7	cyclictest	3633649	-21	ssh	00:11:42	2
3545531	99	65	7,46	cyclictest	3592351	-21	sh	22:40:01	3
3545531	99	65	20,40	cyclictest	37	-21	ksoftirqd/3	19:55:00	3
3545531	99	64	5,50	cyclictest	0	-21	swapper/3	20:14:59	3
3545531	99	64	3,17	cyclictest	3566104	-21	ssh	21:41:31	3
3545531	99	64	2,6	cyclictest	3623023	-21	ssh	23:48:03	3
3545522	99	64	25,11	cyclictest	150	-21	pr/ttyS2	21:57:43	1
3545531	99	63	5,54	cyclictest	3610418	-21	modprobe	23:19:59	3
3545531	99	63	3,19	cyclictest	3563009	-21	latency_hist	21:35:02	3
3545531	99	63	3,19	cyclictest	3563009	-21	latency_hist	21:35:01	3
3545531	99	63	3,17	cyclictest	3553983	-21	modprobe	21:15:01	3
3545531	99	63	2,7	cyclictest	3568637	-21	sshd	21:47:23	3
3545531	99	63	2,16	cyclictest	3578745	-21	cron	22:10:00	3
3545531	99	63	1,10	cyclictest	3636442	-21	ssh	00:18:00	3
3545522	99	63	41,18	cyclictest	3575103	-21	ssh	22:01:28	1
3545522	99	63	24,14	cyclictest	150	-21	pr/ttyS2	23:13:29	1
3545517	99	63	24,35	cyclictest	12	-21	ksoftirqd/0	20:25:00	0
3545531	99	62	2,19	cyclictest	3574229	-21	munin-run	22:00:01	3
3545531	99	62	2,16	cyclictest	3557150	-21	ssh	21:21:41	3
3545531	99	62	21,36	cyclictest	275	-21	dbus-daemon	23:45:01	3
3545526	99	62	46,13	cyclictest	3591460	-21	ssh	22:38:04	2
3545522	99	62	24,13	cyclictest	150	-21	pr/ttyS2	00:30:47	1
3545531	99	61	4,49	cyclictest	3545715	-21	munin-run	19:15:01	3
3545531	99	61	3,48	cyclictest	3617191	-21	modprobe	23:34:59	3
3545531	99	61	3,15	cyclictest	3619366	-21	true	23:40:01	3
3545531	99	61	26,32	cyclictest	3586209	-21	sh	22:26:10	3
3545531	99	61	23,34	cyclictest	37	-21	ksoftirqd/3	23:11:18	3
3545531	99	61	2,14	cyclictest	3597647	-21	ssh	22:51:35	3
3545531	99	61	21,35	cyclictest	3581434	-21	sh	22:15:26	3
3545531	99	61	13,45	cyclictest	3576001	-21	ssh	22:03:46	3
3545531	99	61	13,44	cyclictest	3604564	-21	ssh	23:06:51	3
3545531	99	61	13,44	cyclictest	3604564	-21	ssh	23:06:51	3
3545531	99	61	1,13	cyclictest	3596522	-21	ssh	22:49:12	3
3545526	99	61	42,16	cyclictest	3611703	-21	ssh	23:22:31	2
3545526	99	61	24,9	cyclictest	0	-21	swapper/2	21:30:40	2
3545526	99	61	24,9	cyclictest	0	-21	swapper/2	21:30:40	2
3545526	99	61	22,13	cyclictest	3596502	-21	ssh	22:49:09	2
3545522	99	61	40,16	cyclictest	3622937	-21	ssh	23:47:52	1
3545522	99	61	24,10	cyclictest	23	-21	ksoftirqd/1	23:09:09	1
3545522	99	61	24,10	cyclictest	23	-21	ksoftirqd/1	23:09:09	1
3545522	99	61	23,13	cyclictest	150	-21	pr/ttyS2	21:15:24	1
3545522	99	61	23,12	cyclictest	150	-21	pr/ttyS2	22:08:15	1
3545517	99	61	29,15	cyclictest	3628089	-21	ssh	23:59:24	0
3545531	99	60	9,46	cyclictest	309	-21	in:imuxsock	22:45:01	3
3545531	99	60	9,46	cyclictest	309	-21	in:imuxsock	22:45:00	3
3545531	99	60	5,48	cyclictest	3548336	-21	systemd-run	20:05:01	3
3545531	99	60	5,48	cyclictest	3548336	-21	systemd-run	20:05:00	3
3545531	99	60	4,17	cyclictest	3639552	-21	grep	00:25:00	3
3545531	99	60	4,12	cyclictest	3547561	-21	cat	19:50:01	3
3545531	99	60	2,8	cyclictest	3563455	-21	ssh	21:35:44	3
3545531	99	60	2,8	cyclictest	120	-21	kswapd0	00:00:02	3
3545531	99	60	2,14	cyclictest	3629376	-21	ssh	00:02:08	3
3545531	99	60	1,9	cyclictest	3579648	-21	ssh	22:11:41	3
3545531	99	60	1,18	cyclictest	3611648	-21	ssh	23:22:22	3
3545531	99	60	1,14	cyclictest	3614581	-21	ssh	23:29:13	3
3545531	99	60	1,10	cyclictest	3559807	-21	ssh	21:27:42	3
3545526	99	60	44,13	cyclictest	3614561	-21	ssh	23:29:10	2
3545526	99	60	44,13	cyclictest	3586233	-21	ssh	22:26:14	2
3545526	99	60	44,12	cyclictest	3646008	-21	ssh	00:39:14	2
3545526	99	60	44,12	cyclictest	3640553	-21	ssh	00:26:50	2
3545526	99	60	43,14	cyclictest	3632515	-21	ssh	00:09:16	2
3545526	99	60	43,14	cyclictest	3584271	-21	ssh	22:21:51	2
3545526	99	60	43,13	cyclictest	3626977	-21	ssh	23:56:46	2
3545526	99	60	42,14	cyclictest	3552008	-21	ssh	21:10:05	2
3545526	99	60	41,16	cyclictest	3618398	-21	ssh	23:37:49	2
3545526	99	60	40,16	cyclictest	3593110	-21	ssh	22:41:25	2
3545526	99	60	40,16	cyclictest	3593110	-21	ssh	22:41:24	2
3545526	99	60	40,16	cyclictest	3580981	-21	ssh	22:14:57	2
3545526	99	60	39,17	cyclictest	3572636	-21	ssh	21:56:01	2
3545526	99	60	36,19	cyclictest	284	-21	systemd-logind	22:10:01	2
3545522	99	60	40,16	cyclictest	3599913	-21	ssh	22:56:32	1
3545522	99	60	24,11	cyclictest	150	-21	pr/ttyS2	19:11:45	1
3545522	99	60	23,11	cyclictest	150	-21	pr/ttyS2	21:04:30	1
3545531	99	59	3,17	cyclictest	3643358	-21	ssh	00:33:16	3
3545531	99	59	13,39	cyclictest	3467602	-21	systemd-journal	23:02:17	3
3545531	99	59	1,11	cyclictest	3585469	-21	ssh	22:24:48	3
3545526	99	59	44,12	cyclictest	3574650	-21	ssh	22:00:25	2
3545526	99	59	43,13	cyclictest	3637737	-21	ssh	00:20:30	2
3545526	99	59	43,13	cyclictest	3581376	-21	ssh	22:15:19	2
3545526	99	59	42,14	cyclictest	3621377	-21	ssh	23:44:32	2
3545526	99	59	42,14	cyclictest	3565650	-21	ssh	21:40:31	2
3545526	99	59	41,14	cyclictest	3597307	-21	ssh	22:50:37	2
3545526	99	59	41,14	cyclictest	3569495	-21	ssh	21:49:24	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:33