OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack2slot7s.osadl.org (updated Sat Apr 18, 2026 00:43:19)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
955913	99	54	40,9	cyclictest	1147567	-21	ssh	00:25:37	1
955913	99	54	40,9	cyclictest	1147567	-21	ssh	00:25:37	1
955912	99	51	23,22	cyclictest	1151383	-21	ssh	00:30:43	0
955913	99	50	41,6	cyclictest	421	-21	dbus-daemon	23:40:31	1
955913	99	50	41,6	cyclictest	421	-21	dbus-daemon	23:40:31	1
955912	99	50	16,26	cyclictest	1041649	-21	ssh	22:10:38	0
955914	99	49	41,5	cyclictest	1097007	-21	munin-plugin-st	23:20:00	2
955913	99	48	1,41	cyclictest	1020188	-21	ssh	21:45:00	1
955912	99	48	40,5	cyclictest	955911	-21	cyclictest	22:35:43	0
955913	99	47	4,36	cyclictest	1001294	-21	sh	21:20:45	1
955912	99	47	37,7	cyclictest	15	1	ktimers/0	20:35:38	0
955912	99	47	37,6	cyclictest	1003984	-21	latency_hist	21:25:01	0
955914	99	46	37,6	cyclictest	1053147	-21	aten_r2power_cu	22:25:27	2
955914	99	46	36,6	cyclictest	3345305	-21	cups-browsed	21:54:49	2
955913	99	45	36,6	cyclictest	1105040	-21	ssh	23:30:01	1
955912	99	45	3,32	cyclictest	1086408	-21	grep	23:05:45	0
955912	99	45	3,32	cyclictest	1086408	-21	grep	23:05:45	0
955912	99	45	17,21	cyclictest	1009157	-21	ssh	21:30:38	0
955914	99	44	36,5	cyclictest	955917	-21	systemd-run	19:10:01	2
955914	99	44	34,6	cyclictest	3345305	-21	cups-browsed	22:10:46	2
955913	99	44	35,6	cyclictest	1054941	-21	sshd	22:26:37	1
955913	99	43	37,3	cyclictest	0	-21	swapper/1	22:50:33	1
955913	99	43	36,3	cyclictest	0	-21	swapper/1	22:10:01	1
955913	99	43	36,3	cyclictest	0	-21	swapper/1	22:10:01	1
955913	99	43	34,5	cyclictest	1093027	-21	munin-plugin-st	23:15:00	1
955913	99	43	34,5	cyclictest	1052691	-21	systemd-run	22:25:01	1
955912	99	43	15,23	cyclictest	1005150	-21	ssh	21:25:39	0
955912	99	42	5,27	cyclictest	1106210	-21	ssh	23:30:39	0
955912	99	42	33,5	cyclictest	1056	-21	mta-sts-daemon	21:11:20	0
955914	99	41	32,6	cyclictest	1061530	-21	fschecks_count	22:35:30	2
955914	99	41	30,8	cyclictest	1017652	-21	switchtime	21:40:47	2
955912	99	41	31,7	cyclictest	421	-21	dbus-daemon	22:20:40	0
955912	99	41	27,11	cyclictest	3345305	-21	cups-browsed	21:55:44	0
955912	99	41	27,11	cyclictest	1117453	-21	aten_r2power_po	23:45:28	0
955912	99	41	2,32	cyclictest	1098290	-21	ssh	23:20:40	0
955912	99	41	23,14	cyclictest	420	-21	avahi-daemon	22:00:52	0
955912	99	41	23,14	cyclictest	420	-21	avahi-daemon	22:00:52	0
955914	99	40	32,5	cyclictest	1139441	-21	systemd-run	00:15:01	2
955914	99	40	31,6	cyclictest	1	-21	systemd	23:05:01	2
955914	99	40	31,6	cyclictest	1157492	-21	munin-plugin-st	00:40:00	2
955914	99	40	2,32	cyclictest	1037699	-21	ssh	22:05:43	2
955914	99	40	2,32	cyclictest	1037699	-21	ssh	22:05:43	2
955913	99	40	32,5	cyclictest	988879	-21	cron	21:00:01	1
955913	99	40	32,5	cyclictest	1113072	-21	latency_hist	23:40:03	1
955913	99	40	28,7	cyclictest	0	-21	swapper/1	23:55:37	1
955912	99	40	28,10	cyclictest	1057608	-21	sh	22:30:33	0
955914	99	39	31,5	cyclictest	1147215	-21	uniq	00:25:32	2
955914	99	39	31,5	cyclictest	1147215	-21	uniq	00:25:32	2
955913	99	39	31,5	cyclictest	1136800	-21	kernelversion	00:10:33	1
955913	99	39	2,33	cyclictest	1041727	-21	ssh	22:10:40	1
955912	99	39	32,4	cyclictest	984430	-21	kworker/u8:14-xprtiod	21:05:37	0
955912	99	39	31,5	cyclictest	1157522	-21	sendmail-msp	00:40:01	0
955912	99	39	29,7	cyclictest	1128531	-21	cups-browsed	00:25:43	0
955912	99	39	29,7	cyclictest	1128531	-21	cups-browsed	00:25:43	0
955912	99	39	29,6	cyclictest	421	-21	dbus-daemon	22:53:30	0
955912	99	39	28,8	cyclictest	1081943	-21	latency	23:00:33	0
955912	99	39	27,9	cyclictest	1	-21	systemd	21:15:31	0
955912	99	39	27,9	cyclictest	1	-21	systemd	21:15:31	0
955912	99	39	27,8	cyclictest	1054356	-21	timerwakeupswit	22:25:49	0
955914	99	38	32,3	cyclictest	0	-21	swapper/2	23:25:35	2
955914	99	38	30,5	cyclictest	1089021	-21	idleruntime-cro	23:10:01	2
955914	99	38	30,5	cyclictest	1089021	-21	idleruntime-cro	23:10:01	2
955914	99	38	25,10	cyclictest	1113729	-21	/usr/sbin/munin	23:40:32	2
955914	99	38	25,10	cyclictest	1113729	-21	/usr/sbin/munin	23:40:32	2
955914	99	38	24,10	cyclictest	3345305	-21	cups-browsed	22:45:37	2
955913	99	38	8,5	cyclictest	17	1	rcu_preempt	22:00:41	1
955913	99	38	8,5	cyclictest	17	1	rcu_preempt	22:00:41	1
955913	99	38	33,2	cyclictest	0	-21	swapper/1	00:20:01	1
955913	99	38	32,3	cyclictest	0	-21	swapper/1	21:40:01	1
955913	99	38	30,5	cyclictest	976904	-21	systemctl	20:20:01	1
955913	99	38	2,31	cyclictest	1078157	-21	ssh	22:55:39	1
955913	99	38	2,31	cyclictest	1078157	-21	ssh	22:55:39	1
955912	99	38	30,5	cyclictest	974045	-21	latency_hist	20:10:02	0
955912	99	38	2,27	cyclictest	1013156	-21	sh	21:35:37	0
955914	99	37	29,5	cyclictest	1029128	-21	fschecks_time	21:55:30	2
955913	99	37	29,6	cyclictest	1048589	-21	sendmail	22:20:01	1
955913	99	37	29,5	cyclictest	1153742	-21	cron	00:35:00	1
955913	99	37	28,4	cyclictest	0	-21	swapper/1	22:35:43	1
955912	99	37	3,27	cyclictest	991896	-21	diskmemload	22:05:51	0
955912	99	37	3,27	cyclictest	991896	-21	diskmemload	22:05:51	0
955912	99	37	28,4	cyclictest	0	-21	swapper/0	00:20:42	0
955912	99	37	27,6	cyclictest	620	-21	sshd	00:04:37	0
955912	99	37	2,27	cyclictest	1133507	-21	ssh	00:05:43	0
955912	99	37	13,14	cyclictest	0	-21	swapper/0	23:10:56	0
955914	99	36	3,28	cyclictest	1054	-21	mta-sts-daemon	22:00:41	2
955914	99	36	3,28	cyclictest	1054	-21	mta-sts-daemon	22:00:41	2
955914	99	36	30,3	cyclictest	981375	-21	kworker/u8:5-xprtiod	21:40:01	2
955914	99	36	28,6	cyclictest	976897	-21	/usr/sbin/munin	20:20:02	2
955914	99	36	28,5	cyclictest	1121385	-21	aten_r2power_vo	23:50:27	2
955914	99	36	28,5	cyclictest	1060812	-21	latency_hist	22:35:02	2
955914	99	36	27,7	cyclictest	964125	-21	missed_timers	19:35:37	2
955914	99	36	27,6	cyclictest	1	-21	systemd	00:00:00	2
955914	99	36	27,6	cyclictest	1093014	-21	cron	23:15:00	2
955913	99	36	28,5	cyclictest	963416	-21	systemd-run	19:35:01	1
955913	99	36	27,6	cyclictest	3345305	-21	cups-browsed	22:49:17	1
955913	99	36	26,6	cyclictest	993078	-21	/usr/sbin/munin	21:10:37	1
955912	99	36	25,6	cyclictest	123	50	irq/32-eth0	23:50:36	0
955912	99	36	2,12	cyclictest	1070066	-21	ssh	22:45:38	0
955914	99	35	28,4	cyclictest	970878	-21	cron	20:00:00	2
955914	99	35	28,4	cyclictest	970878	-21	cron	20:00:00	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17