OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #7, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of all highest latencies:

System rack7slot2.osadl.org (updated Mon Dec 08, 2025 12:44:06)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3002125	99	140	102,28	cyclictest	3023983	-21	kworker/u4:12+events_unbound	08:50:11	1
3002123	99	122	111,7	cyclictest	3061141	-21	kworker/u4:0+events_unbound	09:55:17	0
3002123	99	117	105,8	cyclictest	3009796	-21	kworker/u4:3+events_unbound	07:50:37	0
3002125	99	114	102,10	cyclictest	53	-21	kswapd0	11:40:34	1
3002123	99	108	97,8	cyclictest	3047963	-21	kworker/u4:4+events_unbound	10:10:17	0
3002123	99	104	93,7	cyclictest	3061144	-21	kworker/u4:5+events_unbound	10:05:18	0
3002125	99	96	88,6	cyclictest	3095478	-21	kworker/u4:12+events_unbound	10:46:17	1
3002125	99	95	89,4	cyclictest	3074667	-21	kworker/u4:3+events_unbound	10:15:19	1
3002125	99	95	85,7	cyclictest	3152416	-21	kworker/u4:8+events_unbound	12:16:31	1
3002123	99	91	86,3	cyclictest	3159951	-21	kworker/u4:3+events_unbound	12:25:41	0
3002125	99	90	82,5	cyclictest	3064685	-21	kworker/u4:1+events_unbound	10:00:17	1
3002123	99	87	82,3	cyclictest	3095474	-21	kworker/u4:7+events_unbound	11:01:21	0
3002125	99	85	77,6	cyclictest	2792902	-21	kworker/u4:1+events_unbound	08:35:57	1
3002125	99	85	74,8	cyclictest	3005949	-21	kworker/u4:15+events_unbound	07:40:36	1
3002125	99	83	76,5	cyclictest	3013665	-21	kworker/u4:12+events_unbound	07:55:38	1
3002125	99	83	75,6	cyclictest	2792902	-21	kworker/u4:1+events_unbound	08:15:01	1
3002125	99	83	42,33	cyclictest	3002122	-21	cyclictest	09:46:11	1
3002123	99	82	74,6	cyclictest	3167951	-21	kworker/u4:5+events_unbound	12:36:34	0
3002123	99	82	74,6	cyclictest	3026568	-21	kworker/u4:11+events_unbound	09:01:06	0
3002123	99	81	71,8	cyclictest	3031807	-21	kworker/u4:20+events_unbound	09:10:14	0
3002123	99	81	71,8	cyclictest	3031807	-21	kworker/u4:20+events_unbound	09:10:13	0
3002125	99	80	72,6	cyclictest	3018812	-21	kworker/u4:14+events_unbound	08:25:53	1
3002123	99	80	68,9	cyclictest	3035140	-21	kworker/u4:4+events_unbound	09:15:14	0
3002125	99	79	70,7	cyclictest	2991912	-21	kworker/u4:4+events_unbound	07:45:37	1
3002123	99	79	71,6	cyclictest	3050773	-21	kworker/u4:12+events_unbound	09:41:12	0
3002125	99	78	69,6	cyclictest	3132508	-21	kworker/u4:10+events_unbound	11:51:27	1
3002125	99	78	68,7	cyclictest	2993290	-21	kworker/u4:13+events_unbound	08:10:02	1
3002125	99	77	58,7	cyclictest	3068222	-21	kworker/u4:2+events_unbound	10:35:25	1
3002125	99	75	66,7	cyclictest	2792902	-21	kworker/u4:1+events_unbound	07:35:35	1
3002125	99	75	65,8	cyclictest	2986721	-21	kworker/u4:9+events_unbound	08:05:00	1
3002125	99	75	56,6	cyclictest	3099069	-21	kworker/u4:13+events_unbound	10:55:26	1
3002125	99	75	40,5	cyclictest	3108003	-21	sh	11:00:42	1
3002123	99	75	67,6	cyclictest	3035137	-21	kworker/u4:0+events_unbound	09:26:09	0
3002123	99	75	67,6	cyclictest	3026567	-21	kworker/u4:6+events_unbound	09:00:00	0
3002123	99	75	67,6	cyclictest	2989354	-21	kworker/u4:3+events_unbound	07:24:49	0
3002125	99	74	65,6	cyclictest	3159955	-21	kworker/u4:10+events_unbound	12:21:31	1
3002125	99	74	39,8	cyclictest	3131173	-21	sshd	11:35:33	1
3002123	99	74	68,4	cyclictest	3018812	-21	kworker/u4:14+events_unbound	08:25:04	0
3002123	99	74	66,6	cyclictest	3125286	-21	kworker/u4:2+events_unbound	11:31:25	0
3002123	99	72	36,32	cyclictest	3156129	-21	kworker/u4:5-xprtiod	12:20:32	0
3002125	99	71	62,7	cyclictest	3007232	-21	kworker/u4:17+events_unbound	07:30:35	1
3002125	99	71	62,7	cyclictest	3007232	-21	kworker/u4:17+events_unbound	07:30:35	1
3002125	99	71	60,9	cyclictest	3108754	-21	kworker/u4:12+events_unbound	11:25:31	1
3002125	99	71	39,7	cyclictest	3062794	-21	systemd	09:55:17	1
3002125	99	70	56,7	cyclictest	3074668	-21	kworker/u4:5+events_unbound	10:20:19	1
3002125	99	70	38,5	cyclictest	3169589	-21	modprobe	12:30:00	1
3002123	99	70	60,7	cyclictest	3152417	-21	kworker/u4:9+events_unbound	12:11:31	0
3002123	99	70	35,32	cyclictest	0	-21	swapper/0	12:15:28	0
3002123	99	70	34,32	cyclictest	0	-21	swapper/0	08:15:01	0
3002125	99	69	59,7	cyclictest	2986721	-21	kworker/u4:9+events_unbound	07:14:45	1
3002123	99	69	39,28	cyclictest	3121027	-21	mailstats	11:20:41	0
3002123	99	69	37,29	cyclictest	3079197	-21	latency_hist	10:20:02	0
3002125	99	68	35,7	cyclictest	3151533	-21	processes	12:05:40	1
3002123	99	68	60,6	cyclictest	3132507	-21	kworker/u4:9+events_unbound	12:00:38	0
3002123	99	68	32,33	cyclictest	0	-21	swapper/0	09:35:21	0
3002125	99	67	58,6	cyclictest	3035139	-21	kworker/u4:3+events_unbound	09:16:07	1
3002123	99	67	59,6	cyclictest	3005949	-21	kworker/u4:15+events_unbound	09:21:10	0
3002123	99	67	36,29	cyclictest	3010263	-21	fschecks_count	07:40:18	0
3002123	99	67	36,28	cyclictest	3044593	-21	kworker/u4:10+rpciod	09:46:11	0
3002123	99	67	31,5	cyclictest	3089850	-21	sh	10:35:02	0
3002123	99	67	30,6	cyclictest	3094076	-21	systemd	10:40:23	0
3002123	99	67	28,7	cyclictest	3090312	-21	ssh	10:35:20	0
3002123	99	67	25,39	cyclictest	0	-21	swapper/0	11:55:01	0
3002123	99	66	1,24	cyclictest	3137836	-21	tail	11:45:22	0
3002125	99	65	57,6	cyclictest	3023983	-21	kworker/u4:12+events_unbound	08:55:10	1
3002123	99	65	56,6	cyclictest	53	-21	kswapd0	10:20:09	0
3002123	99	65	30,32	cyclictest	0	-21	swapper/0	10:00:42	0
3002123	99	65	25,7	cyclictest	0	-21	swapper/0	12:30:29	0
3002123	99	65	10,40	cyclictest	3002101	-21	munin-run	07:10:01	0
3002123	99	64	31,30	cyclictest	0	-21	swapper/0	07:39:54	0
3002123	99	64	29,7	cyclictest	3083677	-21	open_files	10:25:37	0
3002123	99	64	29,31	cyclictest	0	-21	swapper/0	11:15:36	0
3002123	99	64	26,7	cyclictest	3098081	-21	timerandwakeup	10:45:42	0
3002123	99	63	55,6	cyclictest	3005949	-21	kworker/u4:15+events_unbound	07:49:58	0
3002123	99	63	27,6	cyclictest	3131023	-21	/usr/sbin/munin	11:35:25	0
3002123	99	63	26,6	cyclictest	3022987	-21	aten_r7power_en	08:30:19	0
3002125	99	62	51,9	cyclictest	3026569	-21	kworker/u4:13+events_unbound	09:20:14	1
3002123	99	62	26,6	cyclictest	0	-21	swapper/0	11:10:15	0
3002123	99	62	25,6	cyclictest	0	-21	swapper/0	07:15:01	0
3002123	99	61	51,7	cyclictest	3017533	-21	kworker/u4:8+events_unbound	08:20:02	0
3002123	99	61	30,4	cyclictest	3100517	-21	aten_r7power_en	10:50:16	0
3002123	99	61	28,5	cyclictest	3044586	-21	kworker/u4:2-rpciod	09:50:25	0
3002123	99	61	27,4	cyclictest	0	-21	swapper/0	07:55:38	0
3002123	99	61	26,8	cyclictest	3154041	-21	latency_hist	12:10:01	0
3002123	99	61	26,31	cyclictest	0	-21	swapper/0	09:05:36	0
3002123	99	61	24,7	cyclictest	0	-21	swapper/0	07:30:01	0
3002123	99	60	50,7	cyclictest	2989354	-21	kworker/u4:3+events_unbound	07:19:45	0
3002123	99	60	29,5	cyclictest	0	-21	swapper/0	11:40:39	0
3002123	99	60	28,5	cyclictest	0	-21	swapper/0	08:05:37	0
3002123	99	60	24,33	cyclictest	0	-21	swapper/0	08:15:20	0
3002125	99	59	48,8	cyclictest	3125292	-21	kworker/u4:10+events_unbound	11:30:31	1
3002123	99	59	50,7	cyclictest	3132504	-21	kworker/u4:5+events_unbound	11:56:29	0
3002123	99	58	26,6	cyclictest	3123925	-21	ssh	11:25:22	0
3002125	99	56	47,7	cyclictest	3074666	-21	kworker/u4:0+events_unbound	10:41:17	1
3002125	99	56	46,7	cyclictest	3099069	-21	kworker/u4:13+events_unbound	10:50:27	1
3002125	99	55	49,4	cyclictest	0	-21	swapper/1	12:35:41	1
3002125	99	55	48,5	cyclictest	0	-21	swapper/1	10:25:19	1
3002123	99	54	45,7	cyclictest	3109141	-21	kworker/u4:16+events_unbound	11:05:28	0
3002125	99	53	48,3	cyclictest	3003350	-21	kworker/u4:12+events_unbound	07:20:34	1
3002125	99	53	43,7	cyclictest	3018952	-21	latency_hist	08:15:01	1
3002125	99	52	44,6	cyclictest	3030787	-21	aten_r7power_cu	09:00:18	1
3002125	99	52	44,6	cyclictest	0	-21	swapper/1	07:50:38	1
3002125	99	52	44,5	cyclictest	3044593	-21	kworker/u4:10+events_unbound	09:30:01	1
3002125	99	52	44,5	cyclictest	0	-21	swapper/1	10:30:32	1
3002125	99	51	43,5	cyclictest	0	-21	swapper/1	12:30:25	1
3002125	99	51	41,7	cyclictest	3020529	-21	sed	08:20:18	1
3002125	99	50	43,5	cyclictest	3110892	-21	ntp_offset	11:05:32	1
3002125	99	49	42,5	cyclictest	3147676	-21	fschecks_count	12:00:21	1
3002125	99	49	41,5	cyclictest	0	-21	swapper/1	11:55:45	1
3002125	99	49	39,6	cyclictest	0	-21	swapper/1	07:25:35	1
3002125	99	49	38,8	cyclictest	3036419	-21	modprobe	09:15:01	1
3002125	99	49	38,8	cyclictest	3036419	-21	modprobe	09:15:00	1
3002123	99	49	29,18	cyclictest	3027038	-21	fschecks_count	08:45:23	0
3002125	99	48	41,5	cyclictest	3138114	-21	ssh	11:45:32	1
3002125	99	48	41,5	cyclictest	3116958	-21	fschecks_count	11:15:18	1
3002125	99	48	41,5	cyclictest	3108753	-21	kworker/u4:10-nfsiod	11:20:30	1
3002125	99	48	41,5	cyclictest	3052445	-21	aten_r7power_vo	09:40:18	1
3002125	99	48	41,5	cyclictest	0	-21	swapper/1	09:50:33	1
3002125	99	48	38,7	cyclictest	0	-21	swapper/1	10:05:24	1
3002123	99	48	25,21	cyclictest	3028435	-21	fschecks_time	08:50:23	0
3002125	99	47	37,8	cyclictest	3116413	-21	latency_hist	11:15:02	1
3002123	99	47	40,5	cyclictest	3046737	-21	ld	09:30:30	0
3002123	99	47	39,6	cyclictest	3025118	-21	unixbench-2d	08:35:42	0
3002125	99	46	39,5	cyclictest	0	-21	swapper/1	08:30:44	1
3002125	99	45	38,5	cyclictest	0	-21	swapper/1	12:10:29	1
3002125	99	45	37,5	cyclictest	3002208	-21	latency_hist	07:10:01	1
3002125	99	45	37,5	cyclictest	0	-21	swapper/1	10:10:26	1
3002125	99	45	36,7	cyclictest	0	-21	swapper/1	09:35:34	1
3002123	99	45	33,9	cyclictest	3104204	-21	/usr/sbin/munin	10:55:22	0
3002125	99	44	37,5	cyclictest	3046955	-21	proc_pri	09:30:41	1
3002125	99	44	37,5	cyclictest	3027865	-21	cron	08:50:00	1
3002123	99	44	37,5	cyclictest	3025730	-21	fschecks_count	08:40:19	0
3002123	99	44	37,5	cyclictest	3015369	-21	df_abs	08:00:17	0
3002125	99	43	31,9	cyclictest	0	-21	swapper/1	07:15:37	1
3002125	99	42	34,6	cyclictest	0	-21	swapper/1	08:40:21	1
3002125	99	37	25,9	cyclictest	0	-21	swapper/1	09:05:21	1
3002123	99	37	28,7	cyclictest	0	-21	swapper/0	07:30:18	0
3002123	99	37	28,7	cyclictest	0	-21	swapper/0	07:30:18	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17