OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #7, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rack7slot2.osadl.org (updated Fri May 09, 2025 00:44:07)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
257764	99	108	64,30	cyclictest	54	-21	kswapd0	22:05:09	0
257764	99	108	64,30	cyclictest	54	-21	kswapd0	22:05:09	0
257767	99	72	59,9	cyclictest	390604	-21	kernelversion	23:30:28	1
257767	99	64	23,15	cyclictest	277548	-21	ntpq	20:25:28	1
257764	99	62	46,5	cyclictest	391733	-21	kworker/u4:13+events_unbound	23:50:59	0
257764	99	62	44,6	cyclictest	410023	-21	kworker/u4:3+events_unbound	00:17:00	0
257767	99	61	49,8	cyclictest	0	-21	swapper/1	21:50:34	1
257767	99	61	48,10	cyclictest	428971	-21	/usr/sbin/munin	00:25:38	1
257767	99	61	45,9	cyclictest	0	-21	swapper/1	21:40:37	1
257767	99	60	9,45	cyclictest	418946	-21	ssh	00:10:26	1
257767	99	59	5,48	cyclictest	405086	-21	ssh	23:50:24	1
257767	99	59	50,6	cyclictest	281829	-21	kworker/u4:12+events_unbound	20:45:01	1
257764	99	59	42,5	cyclictest	366392	-21	kworker/u4:11+events_unbound	23:06:59	0
257767	99	58	23,9	cyclictest	351579	-21	vmstat	22:35:43	1
257767	99	58	22,6	cyclictest	376339	-21	ntpq	23:10:31	1
257767	99	58	13,14	cyclictest	316757	-21	ssh	21:48:26	1
257764	99	58	40,16	cyclictest	289603	-21	unixbench-2d	21:10:40	0
257767	99	57	49,6	cyclictest	409242	-21	unixbench-2d	23:55:42	1
257767	99	57	17,15	cyclictest	340148	-21	ntpq	22:20:27	1
257764	99	57	46,7	cyclictest	257763	-21	cyclictest	23:59:27	0
257767	99	56	19,15	cyclictest	295276	-21	ssh	21:19:11	1
257767	99	56	19,14	cyclictest	0	-21	swapper/1	21:55:29	1
257767	99	56	17,14	cyclictest	346094	-21	ssh	22:29:31	1
257767	99	56	15,16	cyclictest	348623	-21	ssh	22:32:15	1
257767	99	56	14,16	cyclictest	364264	-21	ssh	22:54:55	1
257764	99	56	49,5	cyclictest	323524	-21	kworker/u4:14+events_unbound	22:01:59	0
257764	99	56	49,5	cyclictest	323524	-21	kworker/u4:14+events_unbound	22:01:59	0
257764	99	56	39,6	cyclictest	316232	-21	kworker/u4:12+events_unbound	21:59:11	0
257764	99	56	32,21	cyclictest	0	-21	swapper/0	22:50:36	0
257767	99	55	16,14	cyclictest	389719	-21	ssh	23:29:55	1
257767	99	55	16,11	cyclictest	399031	-21	ssh	23:41:38	1
257767	99	55	16,11	cyclictest	308318	-21	ssh	21:35:59	1
257767	99	55	15,15	cyclictest	374166	-21	ssh	23:07:39	1
257767	99	55	14,15	cyclictest	395455	-21	ssh	23:36:47	1
257767	99	55	14,15	cyclictest	304694	-21	ssh	21:30:58	1
257764	99	55	47,6	cyclictest	395390	-21	kworker/u4:15+events_unbound	23:38:10	0
257764	99	55	46,6	cyclictest	364330	-21	apt-get	22:55:09	0
257764	99	55	38,15	cyclictest	397396	-21	aten_r7power_en	23:40:17	0
257764	99	55	31,7	cyclictest	275497	-21	kworker/u4:11+events_unbound	20:25:35	0
257767	99	54	41,8	cyclictest	0	-21	swapper/1	22:45:31	1
257767	99	54	17,11	cyclictest	300315	-21	ssh	21:25:30	1
257767	99	54	17,11	cyclictest	300315	-21	ssh	21:25:30	1
257767	99	54	16,15	cyclictest	426381	-21	ssh	00:21:28	1
257767	99	54	16,13	cyclictest	366548	-21	ssh	22:57:05	1
257767	99	54	16,13	cyclictest	298292	-21	ssh	21:22:49	1
257767	99	54	15,13	cyclictest	417401	-21	ssh	00:08:15	1
257767	99	54	15,13	cyclictest	288374	-21	ssh	21:10:05	1
257767	99	54	14,16	cyclictest	380990	-21	ssh	23:16:57	1
257767	99	54	14,15	cyclictest	403816	-21	ssh	23:49:12	1
257767	99	54	14,11	cyclictest	338497	-21	ssh	22:18:30	1
257767	99	54	14,11	cyclictest	338497	-21	ssh	22:18:30	1
257767	99	54	13,16	cyclictest	369570	-21	ssh	23:00:47	1
257764	99	54	43,5	cyclictest	410023	-21	kworker/u4:3+events_unbound	00:20:11	0
257764	99	54	41,6	cyclictest	330318	-21	kworker/u4:2+events_unbound	22:24:00	0
257764	99	54	36,7	cyclictest	262871	-21	kworker/u4:5+events_unbound	19:33:58	0
257767	99	53	16,12	cyclictest	434466	-21	ntpq	00:35:28	1
257767	99	53	15,13	cyclictest	424437	-21	ssh	00:19:51	1
257767	99	53	15,11	cyclictest	384708	-21	ssh	23:22:02	1
257767	99	53	15,10	cyclictest	432443	-21	ssh	00:31:11	1
257767	99	53	13,15	cyclictest	329252	-21	ntpq	22:05:32	1
257767	99	53	13,15	cyclictest	329252	-21	ntpq	22:05:32	1
257764	99	53	42,5	cyclictest	281829	-21	kworker/u4:12+events_unbound	20:55:10	0
257764	99	53	38,6	cyclictest	337735	-21	kworker/u4:5+events_unbound	22:40:09	0
257764	99	53	37,7	cyclictest	327153	-21	kworker/u4:0+events_unbound	22:46:58	0
257767	99	52	14,14	cyclictest	332051	-21	ssh	22:10:02	1
257767	99	52	14,10	cyclictest	327762	-21	ssh	22:03:59	1
257767	99	52	14,10	cyclictest	327762	-21	ssh	22:03:58	1
257767	99	52	13,14	cyclictest	355888	-21	ssh	22:42:22	1
257764	99	52	44,5	cyclictest	300238	-21	/usr/sbin/munin	21:25:27	0
257764	99	52	44,5	cyclictest	300238	-21	/usr/sbin/munin	21:25:27	0
257764	99	51	44,5	cyclictest	381123	-21	kworker/u4:2+events_unbound	23:48:59	0
257764	99	51	43,6	cyclictest	297767	-21	kworker/u4:9+events_unbound	21:22:10	0
257764	99	51	40,9	cyclictest	434748	-21	/usr/sbin/munin	00:35:39	0
257764	99	51	36,6	cyclictest	316231	-21	kworker/u4:11+events_unbound	21:51:10	0
257764	99	51	32,16	cyclictest	339174	-21	sh	22:20:00	0
257764	99	51	32,16	cyclictest	339174	-21	sh	22:20:00	0
257767	99	50	29,17	cyclictest	259258	-21	apt-get	19:15:15	1
257767	99	50	29,17	cyclictest	259258	-21	apt-get	19:15:15	1
257767	99	50	14,13	cyclictest	286366	-21	ntpq	21:00:27	1
257767	99	50	13,13	cyclictest	413071	-21	ssh	00:00:52	1
257767	99	50	13,12	cyclictest	273048	-21	latency_hist	20:10:01	1
257764	99	50	4,8	cyclictest	249107	-21	kworker/u4:11+rpciod	19:10:08	0
257764	99	50	43,4	cyclictest	0	-21	swapper/0	23:25:34	0
257764	99	50	42,6	cyclictest	430904	-21	munin-plugin-st	00:30:00	0
257764	99	50	42,5	cyclictest	391110	-21	xargs	23:30:44	0
257764	99	50	38,5	cyclictest	256	50	irq/23-42890000.ethernet	19:10:00	0
257764	99	50	19,21	cyclictest	270429	-21	kworker/u4:6-nfsiod	20:00:31	0
257767	99	49	38,7	cyclictest	285653	-21	sendmail	21:00:01	1
257767	99	48	38,8	cyclictest	273423	-21	fschecks_count	20:10:20	1
257767	99	48	18,12	cyclictest	288161	-21	modprobe	21:10:01	1
257767	99	48	18,12	cyclictest	288161	-21	modprobe	21:10:00	1
257764	99	48	41,5	cyclictest	307171	-21	/usr/sbin/munin	21:35:19	0
257764	99	48	40,6	cyclictest	410026	-21	kworker/u4:10+events_unbound	00:10:58	0
257764	99	48	31,6	cyclictest	249107	-21	kworker/u4:11+events_unbound	19:27:28	0
257764	99	48	28,18	cyclictest	303683	-21	fschecks_time	21:30:21	0
257767	99	47	18,7	cyclictest	263689	-21	ld	19:30:31	1
257764	99	47	41,5	cyclictest	337735	-21	kworker/u4:5+events_unbound	22:32:09	0
257764	99	47	40,5	cyclictest	310529	-21	/usr/sbin/munin	21:40:12	0
257764	99	47	40,5	cyclictest	275003	-21	ld	20:15:27	0
257764	99	47	37,8	cyclictest	267949	-21	munin-run	19:50:00	0
257764	99	47	29,7	cyclictest	366392	-21	kworker/u4:11+events_unbound	23:04:31	0
257764	99	47	27,8	cyclictest	305166	-21	kworker/u4:10+events_unbound	22:10:50	0
257767	99	46	32,10	cyclictest	262445	-21	ntpq	19:25:35	1
257767	99	46	15,11	cyclictest	258604	-21	collect2	19:10:32	1
257767	99	46	12,10	cyclictest	278801	-21	ntpq	20:30:28	1
257764	99	46	40,4	cyclictest	265688	-21	aten_r7power_po	19:40:15	0
257764	99	46	39,5	cyclictest	312521	-21	kworker/u4:2+events_unbound	21:49:39	0
257764	99	46	39,4	cyclictest	292355	-21	aten_r7power_cu	21:15:20	0
257764	99	46	38,6	cyclictest	411713	-21	munin-node	00:00:14	0
257764	99	46	25,18	cyclictest	0	-21	swapper/0	21:10:01	0
257764	99	46	25,18	cyclictest	0	-21	swapper/0	21:10:01	0
257767	99	45	33,8	cyclictest	274971	-21	ntpq	20:15:24	1
257767	99	45	16,10	cyclictest	265416	-21	munin-plugin-st	19:40:00	1
257767	99	45	14,5	cyclictest	261183	-21	ntpq	19:20:28	1
257764	99	45	38,5	cyclictest	282283	-21	fschecks_count	20:45:21	0
257764	99	45	37,5	cyclictest	431171	-21	aten_r7power_cu	00:30:16	0
257764	99	45	37,5	cyclictest	418083	-21	idleruntime-cro	00:10:00	0
257764	99	45	37,5	cyclictest	261379	-21	switchtime	19:20:33	0
257764	99	45	36,5	cyclictest	376089	-21	/usr/sbin/munin	23:10:21	0
257764	99	45	32,11	cyclictest	270474	-21	run-parts	20:00:00	0
257767	99	44	34,7	cyclictest	269299	-21	latency_hist	19:55:01	1
257767	99	44	27,14	cyclictest	269365	-21	apt-get	19:55:15	1
257767	99	44	27,14	cyclictest	266290	-21	ps	19:40:35	1
257767	99	44	13,5	cyclictest	276365	-21	ntp_states	20:20:31	1
257767	99	44	11,9	cyclictest	284376	-21	munin-run	20:55:01	1
257764	99	44	37,5	cyclictest	352342	-21	kworker/u4:11+events_unbound	22:38:00	0
257764	99	44	37,5	cyclictest	272973	-21	kworker/u4:1+events_unbound	20:40:09	0
257764	99	44	37,5	cyclictest	259490	-21	sort	19:15:20	0
257764	99	44	37,5	cyclictest	259490	-21	sort	19:15:19	0
257764	99	44	36,5	cyclictest	344129	-21	timerandwakeup	22:25:41	0
257764	99	44	34,7	cyclictest	280652	-21	latency_hist	20:40:01	0
257767	99	43	8,20	cyclictest	271242	-21	sshd	20:00:27	1
257764	99	43	33,6	cyclictest	383070	-21	/usr/sbin/munin	23:20:23	0
257764	99	43	32,5	cyclictest	377535	-21	kworker/u4:4+events_unbound	23:15:11	0
257764	99	43	29,6	cyclictest	274231	-21	kworker/u4:7+events_unbound	20:22:01	0
257767	99	42	28,10	cyclictest	282646	-21	ntpq	20:45:32	1
257764	99	42	34,6	cyclictest	286100	-21	fschecks_count	21:00:21	0
257767	99	41	27,11	cyclictest	267496	-21	ntpq	19:45:33	1
257767	99	41	26,12	cyclictest	280179	-21	ps	20:35:33	1
257764	99	41	34,5	cyclictest	273863	-21	grep	20:10:36	0
257764	99	41	29,6	cyclictest	271715	-21	kworker/u4:9+events_unbound	20:05:14	0
257764	99	39	33,4	cyclictest	281829	-21	kworker/u4:12-rpciod	20:55:00	0
257764	99	39	29,8	cyclictest	269096	-21	sed	19:50:37	0
257764	99	39	16,6	cyclictest	17	1	rcu_preempt	20:30:31	0
257764	99	38	32,4	cyclictest	262865	-21	kworker/u4:0-rpciod	19:40:00	0
257767	99	37	29,6	cyclictest	0	-21	swapper/1	19:10:00	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17