OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #c, slot #4

Intel(R) Celeron(R) CPU N3350 @ 1.10GHz, Linux 5.9.1-rt18 (Profile)

Latency plot of system in rack #c, slot #4

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the alll highest latencies:

System rackcslot4.osadl.org (updated Fri Sep 10, 2021 12:43:49)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
21376	99	35	3,12	cyclictest	906	-21	apt-get	11:30:27	1
21376	99	34	3,11	cyclictest	14526	-21	apt-get	08:35:22	1
21376	99	32	5,7	cyclictest	30237	-21	apt-get	09:30:18	1
21376	99	32	2,10	cyclictest	24629	-21	apt-get	09:10:15	1
21376	99	32	0,11	cyclictest	17507	-21	apt-get	12:25:18	1
21376	99	30	2,10	cyclictest	5538	-21	apt-get	08:05:16	1
21376	99	30	0,15	cyclictest	21786	-21	/usr/sbin/munin	10:50:22	1
21376	99	30	0,10	cyclictest	4106	-21	apt-get	08:00:17	1
21376	99	29	2,8	cyclictest	11325	-21	apt-get	10:15:13	1
21376	99	28	4,9	cyclictest	30992	-21	sed	11:20:25	1
21376	99	28	2,12	cyclictest	2230	-21	sed	09:40:29	1
21376	99	27	4,9	cyclictest	4153	-21	dpkg	11:40:02	1
21376	99	27	4,8	cyclictest	28049	-21	latency	09:20:22	1
21376	99	27	1,12	cyclictest	20817	-21	cat	10:45:27	1
21376	99	26	4,7	cyclictest	15346	-21	grep	12:15:26	1
21376	99	26	3,8	cyclictest	4734	-21	sed	09:50:24	1
21376	99	25	4,6	cyclictest	23047	-21	munin-run	09:04:59	1
21376	99	25	3,6	cyclictest	27107	-21	cat	11:05:29	1
21376	99	25	3,6	cyclictest	17765	-21	tune2fs	10:35:22	1
21376	99	25	3,15	cyclictest	23225	-21	apt-esm-hook	09:05:19	1
21376	99	25	2,8	cyclictest	19899	-21	sed	10:40:27	1
21376	99	25	2,7	cyclictest	19	-21	ksoftirqd/1	07:50:18	1
21376	99	25	2,6	cyclictest	19	-21	ksoftirqd/1	07:40:33	1
21376	99	24	4,6	cyclictest	13793	-21	sed	08:30:34	1
21376	99	24	3,14	cyclictest	12796	-21	apt-esm-hook	10:20:18	1
21376	99	24	2,8	cyclictest	9424	-21	sed	08:15:25	1
21376	99	24	2,8	cyclictest	7453	-21	cat	10:00:21	1
21376	99	24	2,8	cyclictest	17267	-21	unixbench_singl	12:20:31	1
21376	99	24	2,7	cyclictest	717	-21	sed	07:45:30	1
21376	99	24	2,7	cyclictest	19	-21	ksoftirqd/1	11:15:26	1
21376	99	24	2,6	cyclictest	6608	-21	cat	11:45:27	1
21376	99	24	2,6	cyclictest	23924	-21	sed	07:15:29	1
21376	99	24	1,9	cyclictest	20168	-21	mailstats	12:30:26	1
21376	99	24	1,9	cyclictest	20168	-21	mailstats	12:30:26	1
21376	99	24	1,10	cyclictest	2642	-21	/usr/sbin/munin	07:55:30	1
21376	99	24	0,9	cyclictest	26279	-21	/usr/sbin/munin	07:25:25	1
21376	99	24	0,9	cyclictest	19881	-21	cat	08:50:30	1
21375	99	24	9,9	cyclictest	21910	-21	apt-get	07:10:19	0
21375	99	24	2,15	cyclictest	32187	-21	apt-get	09:35:18	0
21375	99	24	2,15	cyclictest	16048	-40	apt-check	12:20:11	0
21375	99	24	10,9	cyclictest	21763	-21	apt-get	09:00:19	0
21376	99	23	2,7	cyclictest	19	-21	ksoftirqd/1	11:50:26	1
21376	99	23	2,6	cyclictest	19	-21	ksoftirqd/1	08:25:31	1
21376	99	23	2,5	cyclictest	16966	-21	sed	08:40:33	1
21376	99	23	2,14	cyclictest	20754	-21	apt-esm-hook	12:35:16	1
21376	99	23	2,14	cyclictest	14230	-21	apt-esm-hook	10:25:15	1
21376	99	23	1,9	cyclictest	19	-21	ksoftirqd/1	07:35:16	1
21376	99	23	0,9	cyclictest	19	-21	ksoftirqd/1	12:10:25	1
21376	99	23	0,6	cyclictest	23252	-21	dpkg	07:15:00	1
21375	99	23	8,9	cyclictest	26065	-21	apt-get	09:15:18	0
21375	99	23	8,9	cyclictest	26065	-21	apt-get	09:15:17	0
21375	99	23	3,14	cyclictest	5564	-21	apt-get	09:55:16	0
21375	99	23	2,15	cyclictest	24849	-21	apt-get	07:20:22	0
21375	99	23	2,15	cyclictest	20346	-21	apt-get	10:45:16	0
21375	99	23	2,15	cyclictest	17457	-21	apt-esm-hook	08:45:15	0
21375	99	23	2,14	cyclictest	5549	-21	apt-esm-hook	08:05:16	0
21375	99	23	14,7	cyclictest	1386	-21	cat	11:30:31	0
21376	99	22	2,7	cyclictest	19	-21	ksoftirqd/1	09:35:25	1
21376	99	22	2,6	cyclictest	3713	-21	sed	09:45:27	1
21376	99	22	2,6	cyclictest	21271	-21	cat	08:55:27	1
21376	99	22	2,6	cyclictest	19	-21	ksoftirqd/1	10:55:26	1
21376	99	22	2,6	cyclictest	19	-21	ksoftirqd/1	08:45:27	1
21376	99	22	2,14	cyclictest	9942	-21	apt-get	12:00:17	1
21376	99	22	2,14	cyclictest	9902	-21	apt-esm-hook	10:10:12	1
21376	99	22	2,14	cyclictest	8469	-21	apt-esm-hook	10:05:15	1
21376	99	22	2,13	cyclictest	24689	-21	apt-esm-hook	11:00:13	1
21376	99	22	1,6	cyclictest	9087	-21	sed	11:55:25	1
21376	99	22	1,6	cyclictest	19	-21	ksoftirqd/1	11:35:26	1
21376	99	22	1,6	cyclictest	19	-21	ksoftirqd/1	11:25:19	1
21376	99	22	1,6	cyclictest	16694	-21	cat	10:30:30	1
21376	99	22	1,5	cyclictest	6118	-21	cat	09:55:21	1
21376	99	22	0,8	cyclictest	28538	-21	switchtime	11:10:27	1
21376	99	22	0,7	cyclictest	29561	-21	cat	09:25:22	1
21376	99	22	0,7	cyclictest	26594	-21	latency	09:15:23	1
21376	99	22	0,7	cyclictest	26594	-21	latency	09:15:22	1
21376	99	22	0,7	cyclictest	10945	-21	cat	08:20:29	1
21375	99	22	9,9	cyclictest	15985	-21	apt-get	08:40:18	0
21375	99	22	2,14	cyclictest	7048	-21	apt-get	11:50:14	0
21375	99	22	2,14	cyclictest	30736	-40	apt-check	09:30:10	0
21375	99	22	2,14	cyclictest	20295	-21	apt-get	08:55:15	0
21375	99	22	2,14	cyclictest	16944	-21	apt-get	10:35:18	0
21375	99	22	2,14	cyclictest	12785	-21	apt-get	10:20:17	0
21375	99	22	2,14	cyclictest	11354	-21	apt-esm-hook	10:15:13	0
21375	99	22	2,14	cyclictest	11299	-21	apt-get	08:25:14	0
21375	99	22	0,6	cyclictest	10	1	rcuc/0	11:25:01	0
21376	99	21	3,12	cyclictest	24877	-21	apt-esm-hook	07:20:22	1
21376	99	21	2,13	cyclictest	11408	-21	apt-esm-hook	12:05:16	1
21376	99	21	1,14	cyclictest	27768	-21	apt-esm-hook	07:30:19	1
21375	99	21	4,10	cyclictest	23223	-21	apt-get	10:55:12	0
21375	99	21	3,13	cyclictest	31875	-21	apt-get	11:25:15	0
21375	99	21	2,14	cyclictest	27740	-21	apt-get	07:30:19	0
21375	99	21	2,13	cyclictest	9972	-21	apt-esm-hook	12:00:17	0
21375	99	21	2,13	cyclictest	8406	-21	apt-get	08:15:12	0
21375	99	21	2,13	cyclictest	4179	-21	apt-get	11:40:21	0
21375	99	21	2,13	cyclictest	29900	-21	snmp_o5powerctl	09:25:27	0
21375	99	21	2,13	cyclictest	29269	-21	apt-get	07:35:14	0
21375	99	21	2,13	cyclictest	2691	-21	apt-get	09:45:15	0
21375	99	21	2,13	cyclictest	2361	-21	apt-get	11:35:17	0
21375	99	21	2,13	cyclictest	18908	-21	apt-get	10:40:15	0
21375	99	21	2,13	cyclictest	18790	-21	apt-get	12:30:06	0
21375	99	21	2,13	cyclictest	18790	-21	apt-get	12:30:06	0
21375	99	21	2,13	cyclictest	12812	-21	apt-get	12:10:16	0
21375	99	21	2,13	cyclictest	1247	-21	apt-get	09:40:16	0
21375	99	21	2,13	cyclictest	1201	-21	apt-get	07:50:15	0
21375	99	21	2,12	cyclictest	23390	-21	apt-get	07:15:19	0
21376	99	20	2,13	cyclictest	6982	-21	apt-esm-hook	08:10:14	1
21375	99	20	5,10	cyclictest	21789	-21	apt-get	10:50:16	0
21375	99	20	3,11	cyclictest	14217	-21	apt-get	10:25:15	0
21375	99	20	3,10	cyclictest	26112	-21	apt-get	11:05:14	0
21375	99	20	2,13	cyclictest	24678	-21	apt-get	11:00:14	0
21375	99	20	2,13	cyclictest	18861	-21	apt-get	08:50:18	0
21375	99	20	2,12	cyclictest	9891	-21	apt-get	10:10:14	0
21375	99	20	2,12	cyclictest	28981	-21	apt-get	11:15:15	0
21375	99	20	2,12	cyclictest	23197	-21	apt-get	09:05:18	0
21375	99	20	2,12	cyclictest	17537	-21	apt-esm-hook	12:25:18	0
21375	99	20	1,14	cyclictest	9865	-21	apt-get	08:20:16	0
21375	99	20	1,14	cyclictest	2645	-21	apt-get	07:55:16	0
21375	99	20	1,14	cyclictest	26282	-21	apt-get	07:25:21	0
21375	99	20	1,13	cyclictest	8456	-21	apt-get	10:05:15	0
21375	99	20	1,13	cyclictest	7034	-21	apt-esm-hook	10:00:18	0
21375	99	20	1,13	cyclictest	15654	-21	apt-get	10:30:19	0
21375	99	20	1,13	cyclictest	14009	-21	apt-get	08:35:16	0
21375	99	19	2,13	cyclictest	13671	-21	snmp_o5powerctl	08:30:31	0
21375	99	19	2,12	cyclictest	4134	-21	apt-esm-hook	08:00:16	0
21375	99	19	2,12	cyclictest	27573	-21	apt-esm-hook	11:10:15	0
21375	99	19	2,10	cyclictest	27525	-21	apt-get	09:20:18	0
21375	99	19	1,14	cyclictest	24642	-21	apt-esm-hook	09:10:15	0
21375	99	19	1,13	cyclictest	6971	-21	apt-get	08:10:15	0
21375	99	19	1,13	cyclictest	32160	-21	apt-get	07:45:16	0
21375	99	19	1,13	cyclictest	11378	-21	apt-get	12:05:15	0
21375	99	19	1,12	cyclictest	20725	-21	apt-get	12:35:16	0
21375	99	19	1,12	cyclictest	14246	-21	apt-get	12:15:14	0
21375	99	18	7,8	cyclictest	4132	-21	apt-get	09:50:14	0
21375	99	18	4,8	cyclictest	30727	-21	apt-get	07:40:16	0
21375	99	18	2,10	cyclictest	5614	-21	apt-get	11:45:16	0
21375	99	18	1,12	cyclictest	8484	-21	apt-get	11:55:16	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17