OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #7, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rack7slot2.osadl.org (updated Fri Dec 19, 2025 12:44:17)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
981630	99	109	95,11	cyclictest	53	-21	kswapd0	10:30:08	0
981630	99	90	76,11	cyclictest	53	-21	kswapd0	12:00:12	0
981630	99	89	80,6	cyclictest	1134221	-21	kworker/u4:11+events_unbound	11:50:08	0
981630	99	87	9,33	cyclictest	1081542	-21	ps	10:35:36	0
981630	99	87	76,7	cyclictest	1114099	-21	kworker/u4:8+events_unbound	11:20:05	0
981630	99	84	74,6	cyclictest	0	-21	swapper/0	07:45:10	0
981630	99	84	74,6	cyclictest	0	-21	swapper/0	07:45:09	0
981630	99	83	71,9	cyclictest	1051054	-21	kworker/u4:7+events_unbound	10:25:13	0
981630	99	80	30,11	cyclictest	256	50	irq/23-42890000.ethernet	07:35:10	0
981630	99	73	43,4	cyclictest	981626	-21	cyclictest	11:15:03	0
981630	99	71	62,6	cyclictest	1051054	-21	kworker/u4:7+events_unbound	10:14:51	0
981630	99	69	61,6	cyclictest	53	-21	kswapd0	09:30:31	0
981630	99	69	61,6	cyclictest	53	-21	kswapd0	09:30:30	0
981631	99	68	60,5	cyclictest	27	1	ktimers/1	12:01:55	1
981631	99	68	54,6	cyclictest	965230	-21	kworker/u4:11+events_unbound	07:15:06	1
981630	99	68	59,6	cyclictest	991991	-21	kworker/u4:14+events_unbound	08:04:25	0
981630	99	67	59,6	cyclictest	1090093	-21	kworker/u4:5+events_unbound	11:05:13	0
981630	99	64	2,16	cyclictest	1003606	-21	apt-get	08:35:10	0
981631	99	63	54,6	cyclictest	1011412	-21	kworker/u4:11+events_unbound	09:05:02	1
981630	99	62	53,7	cyclictest	1010117	-21	kworker/u4:7+events_unbound	09:09:38	0
981630	99	61	52,6	cyclictest	1114096	-21	kworker/u4:5+events_unbound	11:35:01	0
981630	99	61	52,6	cyclictest	1019411	-21	kworker/u4:6+events_unbound	09:24:43	0
981630	99	60	50,6	cyclictest	1085849	-21	/usr/sbin/munin	10:40:39	0
981631	99	59	54,4	cyclictest	27	1	ktimers/1	07:27:12	1
981631	99	59	45,10	cyclictest	994534	-21	kworker/u4:21-rpciod	09:40:17	1
981631	99	59	45,10	cyclictest	994534	-21	kworker/u4:21-rpciod	09:40:17	1
981631	99	57	50,5	cyclictest	1027414	-21	kworker/u4:5+events_unbound	09:32:45	1
981631	99	57	50,5	cyclictest	1027414	-21	kworker/u4:5+events_unbound	09:32:44	1
981630	99	57	49,5	cyclictest	1126185	-21	kworker/u4:8+events_unbound	11:45:09	0
981630	99	57	48,7	cyclictest	315	-21	systemd-journal	11:40:07	0
981631	99	56	49,5	cyclictest	963855	-21	kworker/u4:6+events_unbound	07:10:43	1
981631	99	56	41,10	cyclictest	0	-21	swapper/1	09:15:36	1
981631	99	55	41,6	cyclictest	1153504	-21	kworker/u4:10+events_unbound	12:24:44	1
981631	99	55	41,6	cyclictest	1153504	-21	kworker/u4:10+events_unbound	12:24:43	1
981630	99	55	46,7	cyclictest	1051054	-21	kworker/u4:7+events_unbound	10:19:53	0
981631	99	54	40,9	cyclictest	53	-21	kswapd0	11:30:07	1
981631	99	54	40,7	cyclictest	0	-21	swapper/1	10:10:05	1
981630	99	54	47,5	cyclictest	1110039	-21	kworker/u4:14+events_unbound	11:15:01	0
981630	99	54	34,18	cyclictest	1092109	-21	munin-plugin-st	10:50:00	0
981630	99	53	45,6	cyclictest	1171778	-21	sh	12:30:25	0
981630	99	53	45,6	cyclictest	1171778	-21	sh	12:30:25	0
981630	99	53	45,5	cyclictest	1018013	-21	/usr/sbin/munin	09:15:37	0
981630	99	53	44,7	cyclictest	1167860	-21	/usr/sbin/munin	12:25:23	0
981630	99	53	44,7	cyclictest	1167860	-21	/usr/sbin/munin	12:25:22	0
981630	99	53	44,6	cyclictest	997188	-21	kworker/u4:1+events_unbound	08:44:32	0
981631	99	52	44,5	cyclictest	985098	-21	sendmail_mailqu	07:20:34	1
981631	99	52	37,11	cyclictest	27	1	ktimers/1	12:10:28	1
981630	99	52	45,5	cyclictest	1003571	-21	kworker/u4:6+events_unbound	08:50:01	0
981630	99	52	43,6	cyclictest	984134	-21	kworker/u4:13+events_unbound	07:25:01	0
981630	99	52	42,8	cyclictest	1033399	-21	fschecks_count	09:35:19	0
981630	99	52	10,22	cyclictest	1012724	-21	diskmemload	11:35:24	0
981631	99	51	44,5	cyclictest	1057600	-21	sh	10:05:30	1
981631	99	51	44,5	cyclictest	1057600	-21	sh	10:05:30	1
981631	99	51	44,5	cyclictest	1043035	-21	kworker/u4:10+events_unbound	09:50:01	1
981631	99	51	38,8	cyclictest	0	-21	swapper/1	11:50:26	1
981630	99	51	44,5	cyclictest	1094595	-21	kworker/u4:8+events_unbound	10:55:04	0
981630	99	51	33,16	cyclictest	1046131	-21	timerwakeupswit	09:50:38	0
981630	99	51	11,30	cyclictest	1144464	-21	ssh	11:55:36	0
981631	99	50	43,5	cyclictest	986709	-21	kworker/u4:5+events_unbound	08:18:44	1
981631	99	50	42,5	cyclictest	1074044	-21	unixbench_singl	10:25:41	1
981631	99	50	39,9	cyclictest	994571	-21	apt-get	08:00:10	1
981630	99	50	42,5	cyclictest	1152779	-21	vmstat	12:05:42	0
981630	99	50	41,7	cyclictest	981626	-21	cyclictest	09:43:53	0
981630	99	50	41,7	cyclictest	981626	-21	cyclictest	09:43:53	0
981630	99	50	29,18	cyclictest	0	-21	swapper/0	09:50:01	0
981631	99	49	42,5	cyclictest	1128606	-21	ntp_kernel_pll_	11:35:27	1
981631	99	49	42,5	cyclictest	1096148	-21	latency_hist	10:55:01	1
981631	99	49	39,7	cyclictest	1050207	-21	unixbench_singl	09:55:39	1
981630	99	49	39,8	cyclictest	397	-21	systemd-logind	09:56:42	0
981631	99	48	42,4	cyclictest	1033467	-21	fschecks_time	09:35:21	1
981631	99	48	42,4	cyclictest	1025361	-21	/usr/sbin/munin	09:25:19	1
981631	99	48	41,4	cyclictest	1147402	-21	latency_hist	12:00:02	1
981631	99	48	41,4	cyclictest	1103827	-21	kworker/u4:12	11:05:01	1
981631	99	48	40,5	cyclictest	1119842	-21	latency_hist	11:25:01	1
981631	99	48	17,18	cyclictest	0	-21	swapper/1	09:50:24	1
981630	99	48	40,6	cyclictest	53	-21	kswapd0	11:00:12	0
981630	99	48	37,8	cyclictest	996312	-21	df_abs	08:05:20	0
981630	99	48	3,26	cyclictest	1175636	-21	ssh	12:35:24	0
981630	99	48	26,6	cyclictest	0	-21	swapper/0	10:10:02	0
981630	99	48	26,6	cyclictest	0	-21	swapper/0	10:10:02	0
981631	99	47	40,5	cyclictest	991663	-21	sendmail_mailqu	07:45:39	1
981631	99	47	40,5	cyclictest	991663	-21	sendmail_mailqu	07:45:38	1
981631	99	47	32,10	cyclictest	1006334	-21	modprobe	08:45:01	1
981630	99	47	40,5	cyclictest	0	-21	swapper/0	12:10:20	0
981630	99	47	39,6	cyclictest	989340	-21	kworker/u4:12+events_unbound	08:29:28	0
981630	99	47	39,5	cyclictest	1053248	-21	fschecks_time	10:00:19	0
981631	99	46	40,4	cyclictest	1065129	-21	fschecks_count	10:15:18	1
981631	99	46	37,6	cyclictest	1007334	-21	switchtime	08:45:36	1
981631	99	46	36,7	cyclictest	1002075	-21	threads	08:25:36	1
981631	99	46	34,9	cyclictest	0	-21	swapper/1	08:35:00	1
981631	99	46	34,9	cyclictest	0	-21	swapper/1	08:35:00	1
981631	99	46	27,17	cyclictest	27	1	ktimers/1	10:48:08	1
981631	99	46	17,18	cyclictest	1108991	-21	open_files	11:10:38	1
981630	99	46	38,6	cyclictest	1013436	-21	gltestperf	09:10:21	0
981630	99	46	38,5	cyclictest	1028767	-21	ssh	09:30:01	0
981630	99	46	36,7	cyclictest	999981	-21	aten2__	08:20:17	0
981630	99	46	29,8	cyclictest	0	-21	swapper/0	12:15:36	0
981631	99	45	39,5	cyclictest	986709	-21	kworker/u4:5+events_unbound	07:34:43	1
981631	99	45	39,4	cyclictest	1132048	-21	aten_r7power_vo	11:40:17	1
981631	99	45	38,4	cyclictest	993808	-21	fschecks_time	07:55:18	1
981631	99	45	36,7	cyclictest	1120208	-21	aten_r7power_vo	11:25:19	1
981631	99	45	36,7	cyclictest	1054186	-21	/usr/sbin/munin	10:00:39	1
981631	99	45	36,6	cyclictest	0	-21	swapper/1	12:35:18	1
981631	99	45	15,18	cyclictest	0	-21	swapper/1	11:45:18	1
981631	99	45	14,6	cyclictest	27	1	ktimers/1	11:05:33	1
981630	99	45	38,5	cyclictest	990681	-21	idleruntime-cro	07:44:59	0
981630	99	45	38,5	cyclictest	982039	-21	fschecks_count	07:10:20	0
981630	99	45	37,6	cyclictest	985922	-21	fschecks_count	07:25:19	0
981630	99	45	35,8	cyclictest	1093579	-21	unixbench_multi	10:50:39	0
981630	99	45	35,7	cyclictest	993509	-21	aten_r7power_cu	07:55:14	0
981630	99	45	34,8	cyclictest	984149	-21	cron	07:19:59	0
981630	99	45	33,7	cyclictest	256	50	irq/23-42890000.ethernet	08:55:00	0
981631	99	44	38,4	cyclictest	1112348	-21	fschecks_count	11:15:18	1
981631	99	44	37,5	cyclictest	1151614	-21	aten_r7power_vo	12:05:16	1
981631	99	44	35,6	cyclictest	1076726	-21	/usr/sbin/munin	10:30:11	1
981631	99	44	34,6	cyclictest	0	-21	swapper/1	09:10:15	1
981631	99	44	32,9	cyclictest	0	-21	swapper/1	12:25:34	1
981631	99	44	32,9	cyclictest	0	-21	swapper/1	12:25:33	1
981631	99	44	14,5	cyclictest	27	1	ktimers/1	10:35:17	1
981631	99	44	13,6	cyclictest	27	1	ktimers/1	10:25:00	1
981630	99	44	38,4	cyclictest	966596	-21	kworker/u4:15+events_unbound	07:35:01	0
981630	99	44	37,5	cyclictest	993302	-21	idleruntime-cro	07:54:59	0
981630	99	44	36,6	cyclictest	1010404	-21	aten_r7power_en	09:00:16	0
981631	99	43	37,4	cyclictest	1171429	-21	fschecks_count	12:30:20	1
981631	99	43	37,4	cyclictest	1171429	-21	fschecks_count	12:30:19	1
981631	99	43	36,5	cyclictest	1097244	-21	grep	10:55:38	1
981631	99	43	34,6	cyclictest	1084822	-21	df_abs	10:40:17	1
981631	99	43	33,7	cyclictest	0	-21	swapper/1	12:15:30	1
981631	99	43	32,6	cyclictest	0	-21	swapper/1	09:21:40	1
981630	99	43	35,6	cyclictest	1121088	-21	switchtime	11:25:40	0
981630	99	43	32,9	cyclictest	1069229	-21	fschecks_time	10:20:20	0
981630	99	43	28,11	cyclictest	1157499	-21	kworker/u4:0+events_unbound	12:20:15	0
981630	99	43	28,11	cyclictest	1157499	-21	kworker/u4:0+events_unbound	12:20:15	0
981631	99	42	36,4	cyclictest	996360	-21	fschecks_count	08:05:22	1
981631	99	42	34,5	cyclictest	1004098	-21	fschecks_count	08:35:23	1
981631	99	41	35,5	cyclictest	989340	-21	kworker/u4:12+events_unbound	08:58:59	1
981631	99	41	33,5	cyclictest	998190	-21	switchtime	08:10:36	1
981631	99	41	31,8	cyclictest	990811	-21	latency_hist	07:45:01	1
981631	99	41	27,6	cyclictest	1003570	-21	kworker/u4:2+events_unbound	08:53:13	1
981630	99	41	32,7	cyclictest	1009057	-21	aten2__	08:55:17	0
981630	99	41	28,10	cyclictest	0	-21	swapper/0	08:15:00	0
981631	99	40	32,5	cyclictest	999738	-21	kworker/u4:7-rpciod	08:20:38	1
981630	99	40	32,5	cyclictest	989334	-21	kworker/u4:8-nfsiod	08:15:25	0
981631	99	39	32,4	cyclictest	992508	-21	fschecks_time	07:50:20	1
981631	99	39	31,5	cyclictest	1012505	-21	timerwakeupswit	09:05:37	1
981631	99	38	11,3	cyclictest	27	1	ktimers/1	07:35:32	1
981630	99	38	19,17	cyclictest	0	-21	swapper/0	08:35:01	0
981630	99	38	19,17	cyclictest	0	-21	swapper/0	08:35:00	0
981631	99	33	25,6	cyclictest	0	-21	swapper/1	07:10:01	1
981630	99	33	26,5	cyclictest	0	-21	swapper/0	07:10:02	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17