OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack7slot7s.osadl.org (updated Wed Dec 17, 2025 12:43:47)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
32699	2	90	76,9	sleep2	0	-21	swapper/2	07:08:13	2
30467	2	88	73,10	sleep0	0	-21	swapper/0	07:05:01	0
375	2	81	64,13	sleep1	0	-21	swapper/1	07:10:01	1
32531	2	78	65,9	sleep3	0	-21	swapper/3	07:06:05	3
479	99	22	0,1	cyclictest	0	-21	swapper/2	10:40:14	2
479	99	20	18,1	cyclictest	3055	-21	/usr/sbin/munin	09:30:19	2
472	99	19	3,12	cyclictest	0	-21	swapper/1	09:50:12	1
486	99	18	15,2	cyclictest	39	-21	ksoftirqd/3	07:54:59	3
479	99	17	1,14	cyclictest	0	-21	swapper/2	07:25:13	2
472	99	17	16,1	cyclictest	0	-21	swapper/1	09:27:11	1
469	99	17	5,6	cyclictest	12562	-21	munin-run	12:05:00	0
479	99	16	9,6	cyclictest	455	-21	dbus-daemon	07:30:22	2
479	99	16	9,6	cyclictest	0	-21	swapper/2	07:35:16	2
472	99	16	3,2	cyclictest	225	-21	systemd-journal	08:14:59	1
479	99	15	3,4	cyclictest	225	-21	systemd-journal	08:05:00	2
479	99	15	11,3	cyclictest	584	-21	ls	09:25:00	2
479	99	15	0,14	cyclictest	565	50	irq/124-enp0s31	10:56:29	2
469	99	15	6,8	cyclictest	9991	-21	apt-get	07:30:00	0
469	99	15	1,7	cyclictest	0	-21	swapper/0	07:10:21	0
479	99	14	1,12	cyclictest	29365	-21	latency_hist	08:09:59	2
479	99	14	11,2	cyclictest	18144	-21	head	10:00:16	2
479	99	14	10,3	cyclictest	24589	-21	wc	09:05:20	2
479	99	14	0,1	cyclictest	0	-21	swapper/2	12:35:13	2
472	99	14	8,5	cyclictest	11788	-21	fgrep	07:30:22	1
472	99	14	10,3	cyclictest	26119	-21	wc	10:15:26	1
469	99	14	9,3	cyclictest	8084	-21	grep	11:55:13	0
469	99	14	9,3	cyclictest	14126	-21	basename	11:00:14	0
469	99	14	12,1	cyclictest	13	-21	ksoftirqd/0	10:40:18	0
469	99	14	11,2	cyclictest	13	-21	ksoftirqd/0	07:15:11	0
469	99	14	10,3	cyclictest	16513	-21	basename	08:50:15	0
469	99	14	10,3	cyclictest	16513	-21	basename	08:50:14	0
469	99	14	0,7	cyclictest	22169	-21	awk	07:55:00	0
486	99	13	9,3	cyclictest	14717	-21	perf	09:54:59	3
486	99	13	9,2	cyclictest	30479	-21	turbostat.cron	09:20:01	3
486	99	13	2,8	cyclictest	0	-21	swapper/3	08:05:00	3
486	99	13	10,2	cyclictest	30188	-21	sort	08:10:15	3
486	99	13	10,2	cyclictest	29260	-21	basename	12:40:01	3
486	99	13	10,2	cyclictest	28246	-21	wc	08:05:18	3
486	99	13	10,2	cyclictest	23555	-21	grep	11:20:01	3
486	99	13	10,2	cyclictest	18044	-21	grep	08:50:26	3
486	99	13	10,2	cyclictest	18044	-21	grep	08:50:26	3
486	99	13	10,2	cyclictest	14951	-21	cat	12:09:59	3
486	99	13	10,2	cyclictest	12695	-21	cat	10:55:19	3
479	99	13	9,3	cyclictest	29337	-21	grep	11:30:17	2
479	99	13	9,2	cyclictest	29682	-21	grep	10:25:12	2
479	99	13	4,8	cyclictest	17166	-21	latency_hist	10:00:00	2
479	99	13	3,9	cyclictest	0	-21	swapper/2	12:00:00	2
479	99	13	10,2	cyclictest	640	-21	munin-plugin-st	11:39:59	2
479	99	13	10,2	cyclictest	5784	-21	tr	07:20:15	2
479	99	13	10,2	cyclictest	21934	-21	grep	09:00:17	2
479	99	13	10,2	cyclictest	19788	-21	wc	07:50:00	2
479	99	13	10,2	cyclictest	15906	-21	grep	12:10:16	2
479	99	13	0,3	cyclictest	0	-21	swapper/2	08:50:14	2
479	99	13	0,3	cyclictest	0	-21	swapper/2	08:50:14	2
479	99	13	0,12	cyclictest	0	-21	swapper/2	07:40:13	2
479	99	13	0,11	cyclictest	15246	-21	pmu-power	08:45:21	2
472	99	13	9,3	cyclictest	9949	-21	cron	09:44:59	1
472	99	13	9,3	cyclictest	1509	-21	grep	07:10:24	1
472	99	13	9,3	cyclictest	12609	-21	wc	08:40:19	1
472	99	13	1,2	cyclictest	0	-21	swapper/1	07:25:12	1
472	99	13	10,2	cyclictest	6416	-21	sleep	08:30:00	1
472	99	13	10,2	cyclictest	22927	-21	tail	10:10:17	1
472	99	13	10,2	cyclictest	17587	-21	tr	10:00:14	1
469	99	13	9,3	cyclictest	7116	-21	grep	07:20:26	0
469	99	13	6,6	cyclictest	6417	-21	apt-get	08:30:00	0
469	99	13	5,7	cyclictest	29266	-21	munin-run	12:40:00	0
469	99	13	10,2	cyclictest	979	-21	head	10:30:21	0
486	99	12	9,2	cyclictest	9985	-21	grep	10:50:16	3
486	99	12	9,2	cyclictest	5401	-21	cron	11:50:00	3
486	99	12	9,2	cyclictest	415	-21	grep	10:30:16	3
486	99	12	9,2	cyclictest	22646	-21	grep	07:55:16	3
486	99	12	9,2	cyclictest	20576	-21	grep	12:20:15	3
486	99	12	9,2	cyclictest	19539	-21	grep	08:55:17	3
486	99	12	9,2	cyclictest	16133	-21	cron	11:05:01	3
486	99	12	9,2	cyclictest	14075	-21	grep	07:35:23	3
486	99	12	9,2	cyclictest	12054	-21	grep	12:00:25	3
486	99	12	9,2	cyclictest	10723	-21	tr	07:30:14	3
486	99	12	10,1	cyclictest	31717	-21	grep	11:35:17	3
486	99	12	10,1	cyclictest	25548	-21	grep	12:30:16	3
486	99	12	10,1	cyclictest	23038	-21	grep	12:25:18	3
479	99	12	9,2	cyclictest	8809	-21	cat	09:40:18	2
479	99	12	9,2	cyclictest	3819	-21	grep	07:15:16	2
479	99	12	9,2	cyclictest	343	-21	cut	10:30:16	2
479	99	12	9,2	cyclictest	21501	-21	wc	12:20:23	2
479	99	12	9,2	cyclictest	20733	-21	grep	07:50:17	2
479	99	12	9,2	cyclictest	20509	-21	grep	10:05:16	2
479	99	12	9,2	cyclictest	18840	-21	grep	08:55:14	2
479	99	12	9,2	cyclictest	14460	-21	grep	12:05:26	2
479	99	12	9,2	cyclictest	12424	-21	grep	08:40:17	2
479	99	12	8,3	cyclictest	5691	-21	grep	09:35:13	2
479	99	12	8,3	cyclictest	29998	-21	sed	09:15:25	2
479	99	12	8,3	cyclictest	25769	-21	grep	10:15:21	2
479	99	12	8,3	cyclictest	10378	-21	tr	08:35:19	2
479	99	12	0,3	cyclictest	4047	-21	perf	08:24:59	2
479	99	12	0,11	cyclictest	12564	-21	perf	12:05:00	2
472	99	12	9,2	cyclictest	7111	-21	grep	09:35:24	1
472	99	12	9,2	cyclictest	32229	-21	tr	08:15:15	1
472	99	12	9,2	cyclictest	21064	-21	head	12:20:18	1
472	99	12	9,2	cyclictest	18398	-21	cut	07:45:17	1
472	99	12	9,2	cyclictest	17167	-21	cut	11:05:17	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17