OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #a, slot #6

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rackaslot6.osadl.org (updated Thu Jun 12, 2025 06:45:38)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3271519	99	65	61,2	cyclictest	3607321	-21	Run	05:49:39	7
3271509	99	55	50,3	cyclictest	4082595	-21	spawn	05:58:23	1
3271514	99	54	50,2	cyclictest	4184436	-21	spawn	05:58:39	4
3271514	99	51	47,2	cyclictest	3607318	-21	Run	05:49:38	4
3271510	99	51	45,3	cyclictest	110085	-21	spawn	05:58:51	2
3271513	99	50	47,2	cyclictest	3607324	-21	Run	05:49:39	3
3271513	99	50	37,10	cyclictest	3674799	-21	spawn	05:57:39	3
3271519	99	47	43,2	cyclictest	139396	-21	spawn	05:58:53	7
3271518	99	46	41,3	cyclictest	4160041	-21	spawn	05:58:31	6
3271515	99	46	41,3	cyclictest	3609960	-21	spawn	05:57:51	5
3271510	99	45	41,2	cyclictest	3607273	-21	Run	05:48:49	2
3271518	99	43	37,4	cyclictest	3274824	-21	Run	05:46:21	6
3271506	99	42	36,3	cyclictest	3767915	-21	spawn	05:57:47	0
3271518	99	39	35,2	cyclictest	3605103	-21	Run	05:42:50	6
3271509	99	39	35,2	cyclictest	3607315	-21	Run	05:49:39	1
3271519	99	38	33,2	cyclictest	354061	-21	looper	06:05:48	7
3271519	99	38	33,2	cyclictest	354061	-21	looper	06:05:48	7
3271515	99	38	34,2	cyclictest	3607250	-21	execl	05:47:41	5
3271515	99	38	34,2	cyclictest	3605106	-21	Run	05:42:51	5
3271515	99	38	34,2	cyclictest	354073	-21	looper	06:05:48	5
3271515	99	38	34,2	cyclictest	354073	-21	looper	06:05:47	5
3271514	99	38	34,2	cyclictest	3607344	-21	Run	05:50:27	4
3271506	99	38	32,4	cyclictest	3607230	-21	execl	05:47:24	0
3271518	99	37	33,2	cyclictest	324444	-21	sh	06:03:00	6
3271518	99	37	33,2	cyclictest	324444	-21	sh	06:03:00	6
3271518	99	37	33,2	cyclictest	324444	-21	sh	06:02:59	6
3271506	99	37	33,2	cyclictest	3604989	-21	Run	05:41:55	0
3271509	99	36	32,2	cyclictest	364236	-21	sh	06:07:48	1
3271509	99	36	32,2	cyclictest	364236	-21	sh	06:07:48	1
3271509	99	36	31,2	cyclictest	311109	-21	grep	06:02:05	1
3271509	99	36	31,2	cyclictest	311109	-21	grep	06:02:05	1
3271509	99	36	31,2	cyclictest	311109	-21	grep	06:02:04	1
3271513	99	34	30,2	cyclictest	347391	-21	sh	06:04:41	3
3271513	99	34	30,2	cyclictest	347391	-21	sh	06:04:41	3
3271513	99	34	30,2	cyclictest	347391	-21	sh	06:04:40	3
3271513	99	34	29,3	cyclictest	3607897	-21	Run	05:54:28	3
3271510	99	34	30,1	cyclictest	3607882	-21	Run	05:54:28	2
3271515	99	33	25,4	cyclictest	349117	-21	sh	06:04:53	5
3271515	99	33	25,4	cyclictest	349117	-21	sh	06:04:53	5
3271515	99	33	25,4	cyclictest	349117	-21	sh	06:04:52	5
3271515	99	33	12,17	cyclictest	3599071	-21	sort	05:39:22	5
3271515	99	33	12,17	cyclictest	3599071	-21	sort	05:39:21	5
3271514	99	33	28,3	cyclictest	347373	-21	sh	06:04:41	4
3271514	99	33	28,3	cyclictest	347373	-21	sh	06:04:41	4
3271514	99	33	28,3	cyclictest	347373	-21	sh	06:04:40	4
3271514	99	33	28,2	cyclictest	359913	-21	sh	06:06:54	4
3271514	99	33	28,2	cyclictest	359913	-21	sh	06:06:53	4
3271519	99	32	27,2	cyclictest	3605116	-21	Run	05:42:50	7
3271510	99	32	27,2	cyclictest	354166	-21	sh	06:05:48	2
3271510	99	32	27,2	cyclictest	354166	-21	sh	06:05:47	2
3271510	99	32	25,3	cyclictest	311111	-21	tee	06:02:05	2
3271510	99	32	25,3	cyclictest	311111	-21	tee	06:02:05	2
3271510	99	32	25,3	cyclictest	311111	-21	tee	06:02:04	2
3271506	99	32	27,3	cyclictest	3593669	-21	grep	05:38:06	0
3271506	99	32	27,3	cyclictest	3593669	-21	grep	05:38:05	0
3271519	99	31	26,3	cyclictest	303686	-21	sh	06:01:52	7
3271519	99	31	26,3	cyclictest	303686	-21	sh	06:01:52	7
3271519	99	31	26,3	cyclictest	303686	-21	sh	06:01:51	7
3271518	99	31	25,3	cyclictest	364387	-21	sh	06:07:50	6
3271518	99	31	25,3	cyclictest	364387	-21	sh	06:07:50	6
3271514	99	31	28,2	cyclictest	3604993	-21	Run	05:41:56	4
3271514	99	31	25,3	cyclictest	3595333	-21	sh	05:38:23	4
3271514	99	31	25,3	cyclictest	3595333	-21	sh	05:38:22	4
3271513	99	31	26,3	cyclictest	3604985	-21	Run	05:41:56	3
3271515	99	30	27,2	cyclictest	3607858	-21	Run	05:53:40	5
3271510	99	30	25,2	cyclictest	3597386	-21	tee	05:38:50	2
3271510	99	30	25,2	cyclictest	3597386	-21	tee	05:38:49	2
3271510	99	30	23,4	cyclictest	3602227	-21	sort	05:40:12	2
3271509	99	30	24,3	cyclictest	3607880	-21	Run	05:54:29	1
3271519	99	29	26,1	cyclictest	3607357	-21	Run	05:50:27	7
3271513	99	29	25,2	cyclictest	3596767	-21	sh	05:38:44	3
3271513	99	29	25,2	cyclictest	3596767	-21	sh	05:38:43	3
3271513	99	29	22,3	cyclictest	280	50	irq/117-46000000.ethernet	05:28:35	3
3271506	99	29	20,4	cyclictest	340761	-21	sh	06:04:08	0
3271506	99	29	20,4	cyclictest	340761	-21	sh	06:04:08	0
3271506	99	29	20,4	cyclictest	340761	-21	sh	06:04:07	0
3271518	99	28	22,2	cyclictest	3607847	-21	Run	05:53:40	6
3271513	99	28	22,3	cyclictest	352444	-21	grep	06:05:25	3
3271513	99	28	22,3	cyclictest	352444	-21	grep	06:05:24	3
3271509	99	28	20,4	cyclictest	3598183	-21	sh	05:39:04	1
3271509	99	28	20,4	cyclictest	3598183	-21	sh	05:39:04	1
3271506	99	28	25,2	cyclictest	3607878	-21	Run	05:54:29	0
3271519	99	27	23,2	cyclictest	3594448	-21	grep	05:38:13	7
3271519	99	27	23,2	cyclictest	3594448	-21	grep	05:38:13	7
3271515	99	27	21,4	cyclictest	3273978	-21	gltestperf	05:10:16	5
3271515	99	27	21,4	cyclictest	3273978	-21	gltestperf	05:10:16	5
3271515	99	27	19,3	cyclictest	3276578	-21	kworker/u16:9+rpciod	05:27:37	5
3271509	99	27	23,2	cyclictest	3604919	-21	sh	05:41:16	1
3271519	99	26	15,6	cyclictest	0	-21	swapper/7	05:23:52	7
3271513	99	26	13,9	cyclictest	38	-21	rcuc/3	05:34:57	3
3271510	99	26	6,16	cyclictest	0	-21	swapper/2	05:05:26	2
3271510	99	26	6,16	cyclictest	0	-21	swapper/2	05:05:25	2
3271518	99	25	21,2	cyclictest	3599209	-21	od	05:39:25	6
3271518	99	25	21,2	cyclictest	3599209	-21	od	05:39:25	6
3271514	99	25	21,2	cyclictest	367598	-21	grep	06:20:18	4
3271510	99	25	2,19	cyclictest	3297419	-21	spawn	05:31:11	2
3271506	99	25	21,1	cyclictest	0	-21	swapper/0	05:28:34	0
3271506	99	25	19,4	cyclictest	359897	-21	expr	06:06:52	0
3271506	99	25	19,4	cyclictest	359897	-21	expr	06:06:52	0
3271515	99	24	20,2	cyclictest	3271859	-21	/usr/sbin/munin	05:05:13	5
3271515	99	24	20,2	cyclictest	3271859	-21	/usr/sbin/munin	05:05:12	5
3271514	99	24	20,2	cyclictest	3273975	-21	grep	05:10:17	4
3271514	99	24	20,2	cyclictest	3273975	-21	grep	05:10:16	4
3271510	99	24	1,19	cyclictest	0	-21	swapper/2	05:30:01	2
3271509	99	24	3,17	cyclictest	0	-21	swapper/1	05:05:18	1
3271509	99	24	3,17	cyclictest	0	-21	swapper/1	05:05:17	1
3271515	99	23	3,15	cyclictest	366669	-21	apt-get	06:20:01	5
3271514	99	23	19,2	cyclictest	3567954	-21	grep	05:35:01	4
3271514	99	23	16,3	cyclictest	3278852	-21	sendmail-mta	05:27:28	4
3271513	99	23	3,17	cyclictest	40	-21	ksoftirqd/3	06:25:02	3
3271513	99	23	3,17	cyclictest	0	-21	swapper/3	05:19:32	3
3271510	99	23	18,3	cyclictest	3273984	-21	cut	05:10:17	2
3271510	99	23	18,3	cyclictest	3273984	-21	cut	05:10:17	2
3271509	99	23	17,3	cyclictest	64539	-21	containerd	05:27:05	1
3271506	99	23	15,4	cyclictest	3274819	-21	latency_hist	05:15:00	0
3271506	99	23	15,4	cyclictest	3274819	-21	latency_hist	05:14:59	0
3271519	99	22	4,1	cyclictest	3396573	-21	spawn	05:31:45	7
3271519	99	22	18,2	cyclictest	3276445	-21	unixbench_multi	05:15:26	7
3271515	99	22	17,3	cyclictest	3276470	-21	unixbench_singl	05:15:27	5
3271514	99	22	4,16	cyclictest	47	1	ktimers/4	05:17:40	4
3271513	99	22	17,3	cyclictest	372558	-21	/usr/sbin/munin	06:35:16	3
3271513	99	22	17,2	cyclictest	365937	-21	/usr/sbin/munin	06:15:14	3
3271513	99	22	1,4	cyclictest	38	-21	rcuc/3	05:10:20	3
3271513	99	22	1,4	cyclictest	38	-21	rcuc/3	05:10:19	3
3271510	99	22	1,17	cyclictest	0	-21	swapper/2	05:20:47	2
3271509	99	22	2,16	cyclictest	0	-21	swapper/1	05:30:02	1
3271506	99	22	1,16	cyclictest	771179	-21	in:imuxsock	05:30:02	0
3271519	99	21	1,3	cyclictest	0	-21	swapper/7	06:20:02	7
3271519	99	21	1,19	cyclictest	3278851	-21	fstime	05:26:52	7
3271519	99	21	1,16	cyclictest	366035	-21	ntpq	06:15:18	7
3271518	99	21	16,3	cyclictest	367923	-21	grep	06:20:25	6
3271518	99	21	1,3	cyclictest	1166	-21	dockerd	05:16:55	6
3271518	99	21	1,16	cyclictest	0	-21	swapper/6	05:28:02	6
3271515	99	21	18,1	cyclictest	3452845	-21	spawn	05:32:03	5
3271515	99	21	17,2	cyclictest	0	-21	swapper/5	06:25:11	5
3271515	99	21	16,3	cyclictest	54	-21	rcuc/5	05:20:12	5
3271514	99	21	18,1	cyclictest	365357	-21	/usr/sbin/munin	06:15:22	4
3271514	99	21	16,2	cyclictest	3198453	-21	dockerd	06:25:11	4
3271514	99	21	0,2	cyclictest	0	-21	swapper/4	05:20:03	4
3271513	99	21	19,1	cyclictest	0	-21	swapper/3	06:27:21	3
3271513	99	21	18,2	cyclictest	0	-21	swapper/3	06:30:15	3
3271513	99	21	17,2	cyclictest	0	-21	swapper/3	06:10:58	3
3271513	99	21	17,2	cyclictest	0	-21	swapper/3	06:10:58	3
3271513	99	21	17,2	cyclictest	0	-21	swapper/3	05:06:29	3
3271513	99	21	17,2	cyclictest	0	-21	swapper/3	05:06:29	3
3271513	99	21	15,3	cyclictest	3277080	-21	execl	05:20:10	3
3271510	99	21	16,3	cyclictest	370400	-21	wc	06:30:02	2
3271510	99	21	1,16	cyclictest	0	-21	swapper/2	05:05:01	2
3271509	99	21	16,3	cyclictest	3680002	-21	containerd	05:20:35	1
3271506	99	21	5,2	cyclictest	3275378	-21	/usr/sbin/munin	05:15:14	0
3271506	99	21	3,2	cyclictest	3271602	-21	latency_hist	05:05:00	0
3271506	99	21	15,3	cyclictest	0	-21	swapper/0	06:28:11	0
3271506	99	21	1,2	cyclictest	1151	-21	containerd	05:20:29	0
3271519	99	20	16,2	cyclictest	0	-21	swapper/7	06:35:18	7
3271519	99	20	16,2	cyclictest	0	-21	swapper/7	06:30:26	7
3271519	99	20	1,16	cyclictest	3274472	-21	wc	05:10:26	7
3271519	99	20	1,16	cyclictest	3274472	-21	wc	05:10:25	7
3271518	99	20	15,3	cyclictest	366059	-21	ntp_states	06:15:19	6
3271518	99	20	15,3	cyclictest	0	-21	swapper/6	05:30:02	6
3271518	99	20	15,2	cyclictest	3274825	-21	munin-run	05:15:00	6
3271518	99	20	15,2	cyclictest	3274825	-21	munin-run	05:14:59	6
3271518	99	20	0,15	cyclictest	0	-21	swapper/6	06:35:12	6
3271515	99	20	2,14	cyclictest	0	-21	swapper/5	06:35:19	5
3271515	99	20	15,1	cyclictest	0	-21	swapper/5	06:30:20	5
3271515	99	20	0,17	cyclictest	0	-21	swapper/5	06:18:02	5
3271514	99	20	15,2	cyclictest	0	-21	swapper/4	06:30:14	4
3271514	99	20	1,3	cyclictest	372603	-21	grep	06:35:16	4
3271510	99	20	5,3	cyclictest	368274	-21	/usr/sbin/munin	06:20:27	2
3271510	99	20	17,1	cyclictest	3276697	-21	192.168.115.60-manager	05:19:32	2
3271510	99	20	0,17	cyclictest	372367	-21	cpu	06:35:14	2
3271510	99	20	0,17	cyclictest	371117	-21	ntpq	06:30:17	2
3271509	99	20	16,2	cyclictest	366641	-21	sendmail-mta	06:17:27	1
3271509	99	20	15,3	cyclictest	861	-21	irqbalance	05:14:54	1
3271509	99	20	15,3	cyclictest	861	-21	irqbalance	05:14:53	1
3271509	99	20	1,16	cyclictest	0	-21	swapper/1	06:35:19	1
3271506	99	20	2,3	cyclictest	3607202	-21	kworker/u16:4+rpciod	06:18:31	0
3271506	99	20	16,2	cyclictest	0	-21	swapper/0	06:37:34	0
3271506	99	20	15,3	cyclictest	3930851	-21	dockerd	06:21:59	0
3271506	99	20	15,2	cyclictest	0	-21	swapper/0	05:05:21	0
3271506	99	20	15,2	cyclictest	0	-21	swapper/0	05:05:21	0
3271519	99	19	15,2	cyclictest	370488	-21	sed	06:30:02	7
3271519	99	19	1,2	cyclictest	3272516	-21	ntpq	05:05:21	7
3271519	99	19	1,2	cyclictest	3272516	-21	ntpq	05:05:20	7
3271518	99	19	16,2	cyclictest	0	-21	swapper/6	06:28:10	6
3271518	99	19	15,2	cyclictest	3277080	-21	execl	05:20:25	6
3271518	99	19	13,4	cyclictest	371820	-21	cron	06:35:00	6
3271518	99	19	0,16	cyclictest	0	-21	swapper/6	05:07:38	6
3271518	99	19	0,16	cyclictest	0	-21	swapper/6	05:07:38	6
3271514	99	19	1,2	cyclictest	0	-21	swapper/4	05:05:23	4
3271514	99	19	1,2	cyclictest	0	-21	swapper/4	05:05:22	4
3271509	99	19	15,2	cyclictest	0	-21	swapper/1	06:20:08	1
3271509	99	19	1,2	cyclictest	1164	-21	containerd	06:25:17	1
3271509	99	19	0,15	cyclictest	3275301	-21	apt-get	05:15:13	1
3271519	99	18	15,1	cyclictest	3198453	-21	dockerd	06:14:34	7
3271519	99	18	15,1	cyclictest	3198453	-21	dockerd	06:14:33	7
3271510	99	18	0,14	cyclictest	365417	-21	apt-get	06:15:07	2
3271509	99	18	1,2	cyclictest	0	-21	swapper/1	06:30:05	1
3271506	99	18	15,1	cyclictest	0	-21	swapper/0	06:30:18	0
3271519	99	17	15,1	cyclictest	0	-21	swapper/7	05:05:00	7
3271515	99	17	1,13	cyclictest	0	-21	swapper/5	06:15:00	5
3271515	99	17	0,15	cyclictest	3271478	-21	latency_hist	05:05:00	5
3271514	99	17	0,1	cyclictest	0	-21	swapper/4	06:10:20	4
3271514	99	17	0,1	cyclictest	0	-21	swapper/4	06:10:19	4
3271509	99	17	13,2	cyclictest	853	-21	dbus-daemon	06:10:10	1
3271509	99	17	13,2	cyclictest	853	-21	dbus-daemon	06:10:09	1
3271506	99	17	0,2	cyclictest	0	-21	swapper/0	06:13:37	0
3271506	99	17	0,2	cyclictest	0	-21	swapper/0	06:13:36	0
3271518	99	16	0,2	cyclictest	0	-21	swapper/6	06:13:52	6
3271518	99	16	0,2	cyclictest	0	-21	swapper/6	06:13:51	6
3271515	99	16	0,1	cyclictest	0	-21	swapper/5	06:10:20	5
3271510	99	16	0,1	cyclictest	0	-21	swapper/2	06:10:55	2
3271510	99	16	0,1	cyclictest	0	-21	swapper/2	06:10:55	2
3271509	99	16	13,2	cyclictest	0	-21	swapper/1	05:05:00	1
3271518	99	15	1,1	cyclictest	0	-21	swapper/6	05:05:00	6
3271513	99	15	0,1	cyclictest	0	-21	swapper/3	05:05:01	3
3271514	99	14	9,2	cyclictest	3271786	-21	latency_hist	05:05:01	4
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17