OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack3slot6.osadl.org (updated Wed Sep 23, 2026 00:44:57)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
15583	2	108	26,69	sleep1	0	-21	swapper/1	19:09:33	1
14856	2	91	35,41	sleep0	0	-21	swapper/0	19:05:29	0
15760	99	87	54,27	cyclictest	14	-21	ksoftirqd/0	21:50:12	0
15760	99	87	51,24	cyclictest	5027	-21	apt-get	21:40:12	0
15760	99	84	33,44	cyclictest	15817	-21	apt-get	20:40:12	0
15765	99	83	44,26	cyclictest	26	-21	ksoftirqd/1	22:10:17	1
15760	99	83	52,15	cyclictest	6776	-21	apt-get	21:45:14	0
15760	99	82	59,12	cyclictest	0	-21	swapper/0	00:10:14	0
15765	99	81	44,25	cyclictest	26	-21	ksoftirqd/1	21:15:15	1
15760	99	79	49,15	cyclictest	12260	-21	apt-get	20:30:14	0
15760	99	79	48,16	cyclictest	1476	-21	apt-get	21:30:13	0
15760	99	77	54,11	cyclictest	21604	-21	apt-get	19:25:13	0
15760	99	77	49,15	cyclictest	10519	-21	apt-get	20:25:13	0
15760	99	77	24,23	cyclictest	4594	-21	apt-get	23:10:00	0
15760	99	76	50,14	cyclictest	18103	-21	apt-get	19:15:13	0
15760	99	76	47,15	cyclictest	31902	-21	apt-get	22:55:16	0
15760	99	76	45,22	cyclictest	31994	-21	apt-get	00:25:14	0
15760	99	74	50,13	cyclictest	3218	-21	apt-get	21:35:14	0
15760	99	74	48,12	cyclictest	17584	-21	apt-get	22:15:23	0
15760	99	74	47,14	cyclictest	32165	-21	apt-get	21:25:14	0
15760	99	74	46,14	cyclictest	15546	-21	apt-get	22:10:19	0
15760	99	74	39,25	cyclictest	30430	-21	apt-get	21:20:15	0
15760	99	74	35,28	cyclictest	25138	-21	apt-get	21:05:17	0
15760	99	73	40,17	cyclictest	12351	-21	apt-get	23:30:12	0
15760	99	73	40,17	cyclictest	12351	-21	apt-get	23:30:12	0
15760	99	72	31,36	cyclictest	22164	-21	ntp_states	23:55:30	0
15760	99	72	24,42	cyclictest	21669	-21	/usr/sbin/munin	22:25:27	0
15760	99	72	15,15	cyclictest	0	-21	swapper/0	23:50:14	0
15760	99	72	14,20	cyclictest	0	-21	swapper/0	22:00:15	0
15760	99	72	13,52	cyclictest	0	-21	swapper/0	19:45:11	0
15760	99	72	13,51	cyclictest	0	-21	swapper/0	20:50:15	0
15760	99	71	7,20	cyclictest	10613	-21	apt-get	23:25:13	0
15760	99	71	50,10	cyclictest	10249	-21	apt-get	21:55:11	0
15760	99	71	46,14	cyclictest	1303	-21	apt-get	00:30:11	0
15760	99	70	37,16	cyclictest	15909	-21	apt-get	23:40:13	0
15760	99	70	29,16	cyclictest	28954	-21	/usr/sbin/munin	00:15:24	0
1249	2	70	0,9	sleep1	26	-21	ksoftirqd/1	23:00:16	1
15760	99	69	48,11	cyclictest	28668	-21	apt-get	21:15:15	0
15760	99	69	46,17	cyclictest	8872	-21	apt-get	23:20:12	0
15760	99	69	44,12	cyclictest	7020	-21	apt-get	20:15:12	0
15760	99	69	41,20	cyclictest	4657	-21	apt	00:40:00	0
15760	99	69	39,23	cyclictest	9136	-21	cat	20:20:22	0
15760	99	69	24,38	cyclictest	27275	-21	df_inode	21:10:25	0
15760	99	69	20,44	cyclictest	0	-21	swapper/0	22:05:56	0
15760	99	69	16,47	cyclictest	0	-21	swapper/0	00:05:12	0
25094	2	68	25,21	sleep0	0	-21	swapper/0	19:35:16	0
15760	99	68	35,24	cyclictest	1826	-21	missed_timers	23:00:27	0
15760	99	68	34,24	cyclictest	26378	-21	apt-get	22:40:10	0
15760	99	68	31,31	cyclictest	25043	-21	latency_hist	19:35:01	0
15760	99	68	28,35	cyclictest	14615	-21	grep	20:35:25	0
15760	99	67	48,10	cyclictest	14	-21	ksoftirqd/0	00:20:13	0
15760	99	67	37,16	cyclictest	32415	-21	apt-get	19:55:26	0
15760	99	67	37,16	cyclictest	21336	-21	apt-get	20:55:22	0
15760	99	67	13,15	cyclictest	0	-21	swapper/0	23:15:15	0
15765	99	66	35,18	cyclictest	26996	-21	grep	22:40:25	1
15760	99	66	33,22	cyclictest	1	-21	systemd	22:30:17	0
15760	99	66	30,9	cyclictest	31939	-21	munin-run	19:55:00	0
15760	99	66	28,32	cyclictest	27952	-21	apt-get	22:45:15	0
1299	2	66	13,28	sleep1	0	-21	swapper/1	00:30:11	1
15765	99	65	34,11	cyclictest	26	-21	ksoftirqd/1	20:30:15	1
15765	99	65	27,29	cyclictest	0	-21	swapper/1	20:00:13	1
15760	99	65	36,24	cyclictest	24345	-21	grep	00:00:38	0
15760	99	65	36,15	cyclictest	1751	-21	apt-get	20:00:15	0
15760	99	65	34,24	cyclictest	5538	-21	/usr/sbin/munin	23:10:28	0
15760	99	65	22,8	cyclictest	5622	-21	diskstats	20:10:21	0
15760	99	64	35,24	cyclictest	20922	-21	idleruntime-cro	22:25:00	0
15760	99	64	34,25	cyclictest	28456	-21	cron	19:45:01	0
15760	99	64	34,24	cyclictest	1	-21	systemd	23:46:52	0
15760	99	64	34,16	cyclictest	14094	-21	apt-get	23:35:13	0
15760	99	64	23,35	cyclictest	243	-21	systemd-journal	22:51:42	0
15765	99	63	32,17	cyclictest	26	-21	ksoftirqd/1	19:40:07	1
15760	99	63	35,21	cyclictest	16389	-21	ls	19:10:33	0
15760	99	63	35,20	cyclictest	24961	-21	munin-run	21:05:00	0
15760	99	63	32,24	cyclictest	18776	-21	runrttasks	20:46:19	0
15760	99	63	28,13	cyclictest	3528	-21	taskset	20:05:16	0
15760	99	63	14,17	cyclictest	0	-21	swapper/0	19:20:13	0
15760	99	62	40,18	cyclictest	503	-21	snmpd	22:35:37	0
15765	99	61	28,26	cyclictest	0	-21	swapper/1	23:05:14	1
15765	99	61	23,15	cyclictest	19634	-21	cut	23:50:01	1
15765	99	61	16,19	cyclictest	28989	-21	cut	00:15:25	1
15765	99	61	16,10	cyclictest	0	-21	swapper/1	23:10:43	1
15765	99	61	14,12	cyclictest	0	-21	swapper/1	20:56:03	1
15765	99	60	22,15	cyclictest	24826	-21	cat	00:05:01	1
15765	99	59	5,8	cyclictest	0	-21	swapper/1	23:37:32	1
15765	99	59	29,9	cyclictest	26	-21	ksoftirqd/1	19:56:24	1
15765	99	59	29,19	cyclictest	24963	-21	apt-get	00:05:13	1
15765	99	59	25,28	cyclictest	0	-21	swapper/1	21:55:11	1
15765	99	58	15,19	cyclictest	26	-21	ksoftirqd/1	00:00:01	1
15765	99	58	10,42	cyclictest	0	-21	swapper/1	21:30:29	1
15765	99	57	35,16	cyclictest	10649	-21	aten2.4_r3power	20:25:16	1
15765	99	57	26,23	cyclictest	0	-21	swapper/1	21:45:13	1
15765	99	57	24,20	cyclictest	5278	-21	apt-get	20:10:13	1
15765	99	57	18,20	cyclictest	26	-21	ksoftirqd/1	20:38:29	1
15765	99	56	33,8	cyclictest	26	-21	ksoftirqd/1	19:29:24	1
15765	99	56	31,18	cyclictest	26	-21	ksoftirqd/1	23:50:33	1
15765	99	56	25,18	cyclictest	28633	-21	apt-get	19:45:11	1
15765	99	56	24,25	cyclictest	0	-21	swapper/1	21:50:13	1
15765	99	56	24,23	cyclictest	25087	-21	apt-get	19:35:13	1
15765	99	56	18,20	cyclictest	26	-21	ksoftirqd/1	21:07:59	1
15765	99	56	10,13	cyclictest	31223	-21	tail	22:50:36	1
15765	99	55	9,33	cyclictest	0	-21	swapper/1	19:20:12	1
15765	99	55	6,43	cyclictest	0	-21	swapper/1	22:15:43	1
15765	99	55	39,9	cyclictest	26	-21	ksoftirqd/1	23:15:20	1
15765	99	55	32,8	cyclictest	26	-21	ksoftirqd/1	20:19:52	1
15765	99	55	30,15	cyclictest	31902	-21	apt-get	22:55:15	1
15765	99	55	22,21	cyclictest	8872	-21	apt-get	23:20:24	1
15765	99	55	18,20	cyclictest	26	-21	ksoftirqd/1	22:26:48	1
15765	99	55	18,20	cyclictest	26	-21	ksoftirqd/1	19:17:49	1
15765	99	55	17,21	cyclictest	26	-21	ksoftirqd/1	22:09:44	1
15765	99	55	15,17	cyclictest	0	-21	swapper/1	22:00:21	1
15765	99	55	10,40	cyclictest	0	-21	swapper/1	23:30:24	1
15765	99	55	10,40	cyclictest	0	-21	swapper/1	23:30:24	1
15765	99	54	29,15	cyclictest	26757	-21	apt-get	00:10:14	1
15765	99	54	26,14	cyclictest	30254	-21	apt-get	00:20:13	1
15765	99	54	25,17	cyclictest	0	-21	swapper/1	21:40:12	1
15765	99	54	18,20	cyclictest	26	-21	ksoftirqd/1	22:34:21	1
15765	99	54	18,20	cyclictest	26	-21	ksoftirqd/1	21:39:50	1
15765	99	54	16,20	cyclictest	26	-21	ksoftirqd/1	21:25:48	1
15765	99	54	16,20	cyclictest	26	-21	ksoftirqd/1	21:24:36	1
15765	99	53	9,8	cyclictest	0	-21	swapper/1	19:10:43	1
15765	99	53	30,17	cyclictest	26	-21	ksoftirqd/1	22:20:27	1
15765	99	53	27,20	cyclictest	457	-21	dbus-daemon	20:09:01	1
15765	99	53	20,21	cyclictest	0	-21	swapper/1	00:25:14	1
15765	99	53	17,20	cyclictest	26	-21	ksoftirqd/1	20:49:44	1
15765	99	53	16,21	cyclictest	26	-21	ksoftirqd/1	20:22:29	1
15765	99	52	6,41	cyclictest	0	-21	swapper/1	21:01:17	1
15765	99	52	26,15	cyclictest	15909	-21	apt-get	23:40:12	1
15765	99	52	15,15	cyclictest	26	-21	ksoftirqd/1	21:10:28	1
15765	99	52	14,21	cyclictest	26	-21	ksoftirqd/1	19:31:16	1
15765	99	51	9,9	cyclictest	24561	-21	kworker/1:2	22:35:10	1
15765	99	51	37,8	cyclictest	26	-21	ksoftirqd/1	22:46:10	1
15765	99	51	24,14	cyclictest	10613	-21	apt-get	23:25:14	1
15765	99	51	18,24	cyclictest	19298	-21	apt-get	20:50:15	1
15765	99	51	14,21	cyclictest	26	-21	ksoftirqd/1	00:35:59	1
15765	99	50	36,9	cyclictest	26	-21	ksoftirqd/1	19:52:06	1
15765	99	50	23,15	cyclictest	15817	-21	apt-get	20:40:13	1
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	3
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
none	08:08:41	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17