OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of the 792 highest latencies:

System rackeslot4s.osadl.org (updated Thu Sep 24, 2026 00:44:05)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1421757	99	65	59,4	cyclictest	1551227	-21	latency_hist	21:15:00	2
1421762	99	47	41,4	cyclictest	1551865	-21	cp	21:15:01	3
1421750	99	46	41,4	cyclictest	1515064	-21	kworker/u8:3+flush-259:0	21:40:24	0
1421752	99	44	38,4	cyclictest	1702866	-21	latency_hist	23:40:01	1
1421762	99	41	34,5	cyclictest	1457181	-21	ntpq	19:40:23	3
1421750	99	41	35,4	cyclictest	1453142	-21	kworker/u8:2+flush-259:0	19:50:22	0
1421752	99	38	34,3	cyclictest	1515064	-21	kworker/u8:3+flush-259:0	22:10:27	1
1421750	99	38	32,4	cyclictest	1630605	-21	/usr/sbin/munin	22:30:15	0
1421752	99	37	32,3	cyclictest	1515303	-21	apt-get	20:40:11	1
1421750	99	36	32,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	21:45:24	0
1421762	99	35	31,3	cyclictest	1677220	-21	kworker/u8:3+flush-259:0	23:30:22	3
1421750	99	35	27,6	cyclictest	1514147	-21	ntpq	20:35:18	0
1421762	99	34	31,2	cyclictest	1515064	-21	kworker/u8:3+events_unbound	21:20:27	3
1421762	99	34	28,4	cyclictest	1572765	-21	latency_hist	21:35:01	3
1421750	99	34	27,5	cyclictest	1614025	-21	ntpq	22:10:27	0
1421762	99	33	27,4	cyclictest	1447731	-21	latency_hist	19:35:01	3
1421762	99	32	26,4	cyclictest	335557	-21	systemd-journal	21:50:01	3
1421752	99	32	28,2	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	00:25:17	1
1421750	99	32	28,3	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:25:20	0
1421750	99	31	24,5	cyclictest	1696871	-21	ntpq	23:30:21	0
1421757	99	30	26,3	cyclictest	335557	-21	systemd-journal	22:10:43	2
1421752	99	30	27,2	cyclictest	1515064	-21	kworker/u8:3+events_unbound	22:35:15	1
1421752	99	30	25,4	cyclictest	1594232	-21	latency_hist	21:55:00	1
1421750	99	30	27,2	cyclictest	1468464	-21	kworker/u8:1+flush-259:0	20:10:23	0
1421750	99	30	26,2	cyclictest	1468464	-21	kworker/u8:1+flush-259:0	21:10:42	0
1421750	99	30	24,4	cyclictest	1525629	-21	/usr/sbin/munin	20:50:23	0
1421752	99	29	26,2	cyclictest	1453142	-21	kworker/u8:2+flush-259:0	19:50:26	1
1421750	99	29	26,2	cyclictest	1567611	-21	kworker/u8:0+flush-259:0	21:55:01	0
1421750	99	29	24,4	cyclictest	1515064	-21	kworker/u8:3+events_unbound	22:22:43	0
1421750	99	29	24,3	cyclictest	1468464	-21	kworker/u8:1+events_unbound	20:45:11	0
1421757	99	28	25,2	cyclictest	709034	-21	kworker/u8:1+flush-259:0	19:10:12	2
1421757	99	28	25,2	cyclictest	1468464	-21	kworker/u8:1+flush-259:0	20:30:18	2
1421757	99	28	24,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	21:45:19	2
1421752	99	28	24,3	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:30:28	1
1421752	99	28	23,4	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	22:50:29	1
1421750	99	28	24,3	cyclictest	1630595	-21	kworker/u8:2+events_unbound	22:55:00	0
1421750	99	28	22,4	cyclictest	1473317	-21	kworker/u8:4+events_unbound	20:05:25	0
1421750	99	27	23,3	cyclictest	1708041	-21	kworker/u8:0+events_unbound	00:20:23	0
1421750	99	27	23,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	20:55:23	0
1421750	99	27	22,3	cyclictest	1703134	-21	apt-get	23:40:15	0
1421762	99	26	22,3	cyclictest	1599630	-21	kworker/u8:1+flush-259:0	22:20:12	3
1421762	99	26	21,4	cyclictest	1515064	-21	kworker/u8:3+flush-259:0	22:35:25	3
1421757	99	26	23,2	cyclictest	1677220	-21	kworker/u8:3+events_unbound	00:05:19	2
1421757	99	26	22,3	cyclictest	1437230	-21	kworker/u8:3+flush-259:0	19:30:18	2
1421752	99	26	22,3	cyclictest	1427128	-21	kworker/u8:0+flush-259:0	19:45:20	1
1421750	99	26	22,3	cyclictest	1708041	-21	kworker/u8:0+events_unbound	23:55:15	0
1421750	99	26	22,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	22:45:22	0
1421762	99	25	21,3	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	23:25:26	3
1421757	99	25	21,3	cyclictest	1437230	-21	kworker/u8:3+flush-259:0	19:25:22	2
1421752	99	25	20,4	cyclictest	1640972	-21	apt-get	22:40:12	1
1421752	99	25	20,3	cyclictest	1473317	-21	kworker/u8:4+flush-259:0	20:05:19	1
1421750	99	25	21,3	cyclictest	709034	-21	kworker/u8:1+events_unbound	19:11:43	0
1421750	99	25	21,3	cyclictest	1493978	-21	kworker/u8:0+events_unbound	20:20:18	0
1421750	99	25	0,23	cyclictest	1	-21	systemd	22:18:42	0
1421762	99	24	20,3	cyclictest	1416362	-21	kworker/u8:2+events_unbound	19:15:16	3
1421762	99	24	19,3	cyclictest	1739694	-21	/usr/sbin/munin	00:15:14	3
1421752	99	24	20,3	cyclictest	335557	-21	systemd-journal	21:35:02	1
1421750	99	24	20,3	cyclictest	1697979	-21	apt-get	23:35:12	0
1421750	99	24	20,3	cyclictest	1437230	-21	kworker/u8:3+flush-259:0	19:35:21	0
1421750	99	24	19,4	cyclictest	1630595	-21	kworker/u8:2+events_unbound	23:25:24	0
1421757	99	23	20,2	cyclictest	1594246	-21	kworker/u8:2+flush-259:0	21:55:15	2
1421757	99	23	20,2	cyclictest	1416362	-21	kworker/u8:2+events_unbound	19:15:17	2
1421757	99	23	18,4	cyclictest	1697714	-21	latency_hist	23:35:01	2
1421752	99	23	20,2	cyclictest	1708041	-21	kworker/u8:0+events_unbound	23:55:01	1
1421752	99	23	20,2	cyclictest	1416362	-21	kworker/u8:2+flush-259:0	19:10:21	1
1421752	99	23	19,3	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:25:25	1
1421752	99	23	16,5	cyclictest	1468703	-21	apt-get	19:55:11	1
1421750	99	23	20,2	cyclictest	1473317	-21	kworker/u8:4+flush-259:0	20:15:20	0
1421750	99	23	19,3	cyclictest	1630595	-21	kworker/u8:2+events_unbound	23:45:12	0
1421750	99	23	19,3	cyclictest	1468464	-21	kworker/u8:1+events_unbound	21:15:30	0
1421762	99	22	19,2	cyclictest	1708041	-21	kworker/u8:0+flush-259:0	00:20:16	3
1421762	99	22	19,2	cyclictest	1599630	-21	kworker/u8:1+flush-259:0	22:05:24	3
1421762	99	22	19,2	cyclictest	1473317	-21	kworker/u8:4+flush-259:0	20:15:25	3
1421762	99	22	18,3	cyclictest	1760267	-21	kworker/u8:0+events_unbound	00:35:20	3
1421762	99	22	17,3	cyclictest	1666901	-21	apt-get	23:05:12	3
1421762	99	22	17,3	cyclictest	1504977	-21	apt-get	20:30:14	3
1421762	99	22	16,4	cyclictest	335557	-21	systemd-journal	22:18:42	3
1421757	99	22	18,3	cyclictest	335557	-21	systemd-journal	23:26:40	2
1421757	99	22	18,3	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	22:40:00	2
1421757	99	22	17,4	cyclictest	335557	-21	systemd-journal	19:55:21	2
1421757	99	22	16,5	cyclictest	1499508	-21	latency_hist	20:25:01	2
1421752	99	22	19,2	cyclictest	1468464	-21	kworker/u8:1+flush-259:0	21:10:26	1
1421752	99	22	18,3	cyclictest	1677220	-21	kworker/u8:3+events_unbound	00:00:01	1
1421750	99	22	18,3	cyclictest	1754834	-21	kworker/u8:1+events_unbound	00:30:21	0
1421750	99	22	18,3	cyclictest	1666658	-21	kworker/u8:0+events_unbound	23:11:42	0
1421762	99	21	19,1	cyclictest	1468464	-21	kworker/u8:1+events_unbound	21:05:18	3
1421762	99	21	18,2	cyclictest	1645906	-21	kworker/u8:4+events_unbound	22:55:30	3
1421762	99	21	16,3	cyclictest	335557	-21	systemd-journal	00:00:51	3
1421757	99	21	18,2	cyclictest	1697736	-21	kworker/u8:1+flush-259:0	23:35:16	2
1421752	99	21	17,3	cyclictest	1599630	-21	kworker/u8:1+flush-259:0	22:05:25	1
1421750	99	21	18,2	cyclictest	709034	-21	kworker/u8:1+flush-259:0	19:30:22	0
1421750	99	21	18,2	cyclictest	1453142	-21	kworker/u8:2+events_unbound	20:00:27	0
1421750	99	21	17,3	cyclictest	1468464	-21	kworker/u8:1+events_unbound	20:30:17	0
1421750	99	21	14,5	cyclictest	1437418	-21	latency_hist	19:25:02	0
1421762	99	20	16,3	cyclictest	1468464	-21	kworker/u8:1+events_unbound	20:10:24	3
1421762	99	20	15,4	cyclictest	1422105	-21	cat	19:10:01	3
1421757	99	20	17,2	cyclictest	1677220	-21	kworker/u8:3+events_unbound	23:50:22	2
1421757	99	20	17,2	cyclictest	1427128	-21	kworker/u8:0+flush-259:0	19:45:19	2
1421757	99	20	16,3	cyclictest	1630595	-21	kworker/u8:2+events_unbound	22:40:23	2
1421750	99	20	14,5	cyclictest	1453176	-21	apt-get	19:40:13	0
1421762	99	19	15,3	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	23:00:25	3
1421757	99	19	16,2	cyclictest	1677220	-21	kworker/u8:3+flush-259:0	23:20:15	2
1421757	99	19	16,2	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:40:15	2
1421757	99	19	16,2	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:40:00	2
1421757	99	19	15,3	cyclictest	1515064	-21	kworker/u8:3+flush-259:0	21:20:20	2
1421757	99	19	14,4	cyclictest	335557	-21	systemd-journal	22:09:40	2
1421752	99	19	15,3	cyclictest	1473317	-21	kworker/u8:4+events_unbound	20:15:16	1
1421752	99	19	14,4	cyclictest	1453142	-21	kworker/u8:2+events_unbound	20:05:00	1
1421750	99	19	16,2	cyclictest	0	-21	swapper/0	21:09:59	0
1421750	99	19	15,3	cyclictest	1630595	-21	kworker/u8:2+events_unbound	00:10:10	0
1421750	99	19	13,4	cyclictest	1630595	-21	kworker/u8:2+events_unbound	00:15:02	0
1421762	99	18	15,2	cyclictest	0	-21	swapper/3	23:50:00	3
1421762	99	18	13,3	cyclictest	335557	-21	systemd-journal	23:14:41	3
1421757	99	18	15,2	cyclictest	1677220	-21	kworker/u8:3+events_unbound	00:00:11	2
1421757	99	18	15,2	cyclictest	1645906	-21	kworker/u8:4+events_unbound	23:00:24	2
1421757	99	18	15,2	cyclictest	1	-21	systemd	19:35:18	2
1421757	99	18	14,3	cyclictest	1754834	-21	kworker/u8:1+flush-259:0	00:30:01	2
1421757	99	18	13,4	cyclictest	1723947	-21	latency_hist	00:00:01	2
1421752	99	18	15,2	cyclictest	1677220	-21	kworker/u8:3+events_unbound	23:29:43	1
1421752	99	18	14,3	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:20:26	1
1421750	99	18	13,4	cyclictest	1630595	-21	kworker/u8:2+events_unbound	23:17:43	0
1421750	99	18	12,4	cyclictest	1661909	-21	diskstats	23:00:15	0
1421762	99	17	14,2	cyclictest	1453142	-21	kworker/u8:2+events_unbound	20:00:24	3
1421762	99	17	14,2	cyclictest	1453142	-21	kworker/u8:2+events_unbound	19:55:25	3
1421762	99	17	14,1	cyclictest	1	-21	systemd	19:30:00	3
1421762	99	17	13,3	cyclictest	509	-21	rs:main		3
1421762	99	17	12,3	cyclictest	1421869	-21	base64	19:10:00	3
1421757	99	17	14,2	cyclictest	1630595	-21	kworker/u8:2+events_unbound	22:45:14	2
1421757	99	17	14,2	cyclictest	1437230	-21	kworker/u8:3+flush-259:0	19:52:47	2
1421757	99	17	11,4	cyclictest	335557	-21	systemd-journal	23:50:01	2
1421757	99	17	0,16	cyclictest	1760229	-21	turbostat	00:35:01	2
1421752	99	17	14,2	cyclictest	1677220	-21	kworker/u8:3+flush-259:0	00:00:02	1
1421752	99	17	14,2	cyclictest	1515046	-21	turbostat.cron	20:39:59	1
1421752	99	17	13,3	cyclictest	1744848	-21	/usr/sbin/munin	00:20:13	1
1421752	99	17	13,3	cyclictest	1734539	-21	apt-get	00:10:09	1
1421750	99	17	14,2	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	00:25:24	0
1421750	99	17	13,3	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	22:59:51	0
1421750	99	17	13,3	cyclictest	1630595	-21	kworker/u8:2+events_unbound	22:40:16	0
1421750	99	17	13,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	21:00:15	0
1421750	99	17	11,4	cyclictest	1573276	-21	base64	21:35:01	0
1421750	99	17	11,4	cyclictest	1453142	-21	kworker/u8:2+events_unbound	19:55:22	0
1421762	99	16	13,3	cyclictest	1556840	-21	kworker/u8:2+flush-259:0	21:25:24	3
1421757	99	16	12,3	cyclictest	1666658	-21	kworker/u8:0+events_unbound	23:10:19	2
1421757	99	16	11,4	cyclictest	1529641	-21	ntpq	20:50:22	2
1421752	99	16	9,6	cyclictest	1442581	-21	sh	19:30:01	1
1421752	99	16	14,1	cyclictest	0	-21	swapper/1	21:20:01	1
1421752	99	16	13,2	cyclictest	0	-21	swapper/1	19:10:00	1
1421752	99	16	12,3	cyclictest	1567857	-21	apt-get	21:30:16	1
1421750	99	16	14,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	21:55:23	0
1421750	99	16	14,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	00:35:24	0
1421750	99	16	13,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	22:09:16	0
1421750	99	16	12,3	cyclictest	603	50	irq/132-enp0s29f1:rx-0	00:05:22	0
1421750	99	16	12,3	cyclictest	1677220	-21	kworker/u8:3+flush-259:0	00:03:14	0
1421750	99	16	12,3	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:43:57	0
1421750	99	16	11,4	cyclictest	1515064	-21	kworker/u8:3+events_unbound	22:01:42	0
1421762	99	15	12,2	cyclictest	709034	-21	kworker/u8:1-writeback	19:40:00	3
1421762	99	15	12,2	cyclictest	1718376	-21	gpgv	23:55:00	3
1421762	99	15	12,2	cyclictest	1567611	-21	kworker/u8:0+flush-259:0	21:50:26	3
1421762	99	15	11,3	cyclictest	335557	-21	systemd-journal	20:55:01	3
1421762	99	15	11,3	cyclictest	1625473	-21	/usr/sbin/munin	22:25:14	3
1421762	99	15	11,3	cyclictest	1468464	-21	kworker/u8:1+flush-259:0	21:10:26	3
1421757	99	15	11,3	cyclictest	1760267	-21	kworker/u8:0+events_unbound	00:35:17	2
1421757	99	15	11,3	cyclictest	1473860	-21	/usr/sbin/munin	20:00:15	2
1421757	99	15	11,3	cyclictest	1468464	-21	kworker/u8:1+flush-259:0	20:05:27	2
1421757	99	15	11,2	cyclictest	500	-21	systemd-logind	23:20:00	2
1421752	99	15	11,3	cyclictest	1594487	-21	apt-get	21:55:10	1
1421752	99	15	11,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	21:45:20	1
1421750	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	23:20:18	0
1421750	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	22:35:14	0
1421750	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	21:25:25	0
1421750	99	15	13,1	cyclictest	603	50	irq/132-enp0s29f1:rx-0	19:25:18	0
1421750	99	15	12,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	21:21:13	0
1421750	99	15	12,2	cyclictest	603	50	irq/132-enp0s29f1:rx-0	19:16:46	0
1421750	99	15	12,2	cyclictest	1515064	-21	kworker/u8:3+events_unbound	22:25:15	0
1421750	99	15	11,3	cyclictest	1515064	-21	kworker/u8:3+events_unbound	21:30:41	0
1421750	99	15	10,3	cyclictest	603	50	irq/132-enp0s29f1:rx-0	19:45:20	0
1421762	99	14	12,1	cyclictest	0	-21	swapper/3	21:05:01	3
1421762	99	14	11,2	cyclictest	1677220	-21	kworker/u8:3+flush-259:0	23:20:21	3
1421757	99	14	10,3	cyclictest	1515064	-21	kworker/u8:3+flush-259:0	20:55:25	2
1421752	99	14	9,4	cyclictest	509	-21	rs:main		1
1421752	99	14	11,2	cyclictest	1672304	-21	fschecks_time	23:10:18	1
1421752	99	14	11,2	cyclictest	1646408	-21	fschecks_time	22:45:14	1
1421752	99	14	10,3	cyclictest	1760491	-21	apt-get	00:35:13	1
1421762	99	13	9,3	cyclictest	1630595	-21	kworker/u8:2+flush-259:0	22:30:17	3
1421762	99	13	11,1	cyclictest	1760208	-21	sh	00:35:01	3
1421762	99	13	10,1	cyclictest	0	-21	swapper/3	20:10:00	3
1421757	99	13	9,3	cyclictest	335557	-21	systemd-journal	00:20:01	2
1421757	99	13	8,4	cyclictest	335557	-21	systemd-journal	21:29:41	2
1421757	99	13	10,2	cyclictest	1524895	-21	unixbench_multi	20:45:24	2
1421752	99	13	9,1	cyclictest	0	-21	swapper/1	00:20:00	1
1421752	99	13	10,2	cyclictest	1692866	-21	cpuspeed_turbos	23:30:13	1
1421752	99	13	10,2	cyclictest	1493978	-21	kworker/u8:0+flush-259:0	20:45:24	1
1421750	99	13	10,2	cyclictest	1630595	-21	kworker/u8:2+events_unbound	23:05:15	0
1421750	99	12	8,2	cyclictest	1421484	-21	latency_hist	19:10:00	0
1421762	99	10	6,3	cyclictest	1514245	-21	open_inodes	20:35:21	3
1421762	99	10	5,3	cyclictest	1677673	-21	grep	23:15:15	3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2021-09-01 - 00:34