OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sat Feb 07, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1332298	2	52	42,7	sleep4	0	-21	swapper/4	07:02:49	4
1332254	2	49	15,29	sleep7	0	-21	swapper/7	07:02:12	7
1332846	99	48	46,2	cyclictest	1510512	-21	kworker/u16:2+flush-8:0	08:51:10	6
1332262	2	47	37,7	sleep6	0	-21	swapper/6	07:02:19	6
1332848	99	44	42,1	cyclictest	1568296	-21	diskmemload	11:10:27	7
1332848	99	43	40,2	cyclictest	1738124	-21	ssh	10:16:11	7
1332848	99	43	36,5	cyclictest	650	-21	systemd-journal	10:36:10	7
1332848	99	43	0,41	cyclictest	1989161	-21	sh	11:57:11	7
1332842	99	41	36,4	cyclictest	1642326	-21	iostat_ios	09:36:27	5
1332848	99	40	38,1	cyclictest	1568296	-21	diskmemload	09:34:16	7
1332848	99	39	37,1	cyclictest	1568296	-21	diskmemload	11:34:19	7
1332846	99	39	35,3	cyclictest	1641489	-21	munin-node	09:36:23	6
1332846	99	38	36,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	11:37:15	6
1332846	99	38	34,3	cyclictest	1666191	-21	diskstats	09:46:23	6
1332842	99	37	34,2	cyclictest	1665218	-21	kworker/u16:1+events_unbound	10:11:27	5
1332848	99	36	33,2	cyclictest	1857173	-21	ssh	11:03:40	7
1332848	99	35	29,4	cyclictest	1689350	-21	latency_hist	09:56:10	7
1332842	99	35	33,2	cyclictest	1471969	-21	kworker/u16:1+flush-8:0	09:16:31	5
1332335	2	35	24,7	sleep5	0	-21	swapper/5	07:03:23	5
1332846	99	34	32,2	cyclictest	1568337	-21	kworker/u16:0+flush-8:0	09:11:36	6
1332846	99	34	30,3	cyclictest	1689840	-21	munin-node	09:56:23	6
1332842	99	34	31,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	12:21:09	5
1332848	99	33	31,1	cyclictest	0	-21	swapper/7	12:21:11	7
1332846	99	33	30,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	10:26:11	6
1332837	99	33	31,2	cyclictest	1627225	-21	kworker/u16:2+flush-8:0	09:51:26	4
1332848	99	31	31,0	cyclictest	0	-21	swapper/7	09:38:16	7
1332846	99	31	29,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	10:46:51	6
1332846	99	31	29,2	cyclictest	1259537	-21	kworker/u16:0+flush-8:0	07:51:10	6
1332837	99	31	29,2	cyclictest	1665218	-21	kworker/u16:1+flush-8:0	09:56:35	4
1332848	99	30	28,1	cyclictest	1834311	-21	ssh	10:56:09	7
1332846	99	30	28,2	cyclictest	1844130	-21	kworker/u16:1+events_unbound	11:07:27	6
1332846	99	30	28,2	cyclictest	1784348	-21	kworker/u16:1+flush-8:0	10:36:44	6
1332837	99	30	28,2	cyclictest	1413890	-21	kworker/u16:3+flush-8:0	07:46:39	4
1332846	99	29	27,2	cyclictest	2014310	-21	kworker/u16:2+flush-8:0	12:31:10	6
1332846	99	29	27,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	11:14:27	6
1712295	2	28	0,0	sleep7	1712296	-21	rm	10:03:45	7
1332848	99	28	24,3	cyclictest	1737332	-21	sh	10:15:24	7
1332848	99	28	0,27	cyclictest	1628443	-21	ssh	09:30:31	7
1332848	99	28	0,27	cyclictest	1628443	-21	ssh	09:30:31	7
1332846	99	28	26,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	11:42:30	6
1332846	99	28	24,3	cyclictest	1259537	-21	kworker/u16:0+flush-8:0	08:26:10	6
1332842	99	28	25,3	cyclictest	1928822	-21	kworker/u16:3+flush-8:0	12:31:31	5
1332842	99	28	25,2	cyclictest	1314573	-21	kworker/u16:1+flush-8:0	07:21:27	5
1332842	99	28	25,2	cyclictest	1314573	-21	kworker/u16:1+flush-8:0	07:21:27	5
1332837	99	28	25,2	cyclictest	2014310	-21	kworker/u16:2+flush-8:0	12:11:26	4
1332837	99	28	25,2	cyclictest	1314573	-21	kworker/u16:1+flush-8:0	07:26:39	4
1332848	99	27	27,0	cyclictest	0	-21	swapper/7	09:09:11	7
1332848	99	27	26,0	cyclictest	0	-21	swapper/7	09:16:40	7
1332848	99	27	25,1	cyclictest	1568296	-21	diskmemload	09:41:20	7
1332846	99	27	25,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	11:18:39	6
1332842	99	27	25,2	cyclictest	1762030	-21	kworker/u16:3+flush-8:0	10:31:39	5
1332848	99	26	26,0	cyclictest	0	-21	swapper/7	11:12:59	7
1332848	99	26	26,0	cyclictest	0	-21	swapper/7	09:49:23	7
1332848	99	26	25,0	cyclictest	0	-21	swapper/7	08:46:35	7
1332848	99	26	22,3	cyclictest	2015182	-21	ssh	12:10:36	7
1332848	99	26	21,4	cyclictest	650	-21	systemd-journal	10:11:09	7
1332846	99	26	24,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	12:08:59	6
1332846	99	26	24,2	cyclictest	1471969	-21	kworker/u16:1+flush-8:0	09:11:10	6
1332842	99	26	24,2	cyclictest	1665218	-21	kworker/u16:1+flush-8:0	09:51:40	5
1332842	99	26	23,2	cyclictest	1807033	-21	kworker/u16:2+flush-8:0	11:16:10	5
1332837	99	26	22,3	cyclictest	1928822	-21	kworker/u16:3+flush-8:0	12:31:28	4
1332848	99	25	23,1	cyclictest	1568296	-21	diskmemload	12:19:14	7
1332846	99	25	23,2	cyclictest	1665218	-21	kworker/u16:1+events_unbound	10:24:27	6
1332846	99	25	23,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	11:50:32	6
1332846	99	25	23,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	11:04:43	6
1332846	99	25	23,2	cyclictest	1627333	-21	kworker/u16:6+events_unbound	09:35:28	6
1332846	99	25	23,2	cyclictest	1452745	-21	kworker/u16:4+flush-8:0	08:11:10	6
1332846	99	25	23,2	cyclictest	1452745	-21	kworker/u16:4+flush-8:0	08:11:10	6
1332842	99	25	23,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	09:56:40	5
1332842	99	25	22,3	cyclictest	1928822	-21	kworker/u16:3+flush-8:0	11:56:25	5
1332837	99	25	23,2	cyclictest	1568337	-21	kworker/u16:0+flush-8:0	09:11:32	4
1332848	99	24	24,0	cyclictest	0	-21	swapper/7	10:36:11	7
1332846	99	24	22,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	11:25:08	6
1332846	99	24	21,2	cyclictest	1762030	-21	kworker/u16:3+flush-8:0	10:31:16	6
1332848	99	23	23,0	cyclictest	0	-21	swapper/7	11:37:44	7
1332848	99	23	23,0	cyclictest	0	-21	swapper/7	11:24:19	7
1332848	99	23	18,4	cyclictest	1616558	-21	latency_hist	09:26:10	7
1332846	99	23	22,1	cyclictest	1653496	-21	kworker/u16:0+events_unbound	10:44:15	6
1332846	99	23	21,2	cyclictest	1807033	-21	kworker/u16:2+events_unbound	10:57:27	6
1332846	99	23	21,2	cyclictest	1627225	-21	kworker/u16:2+events_unbound	10:04:12	6
1332846	99	23	21,2	cyclictest	1604492	-21	kworker/u16:4+events_unbound	09:28:12	6
1332846	99	23	21,2	cyclictest	1604492	-21	kworker/u16:4+events_unbound	09:28:12	6
1332842	99	23	21,2	cyclictest	1604492	-21	kworker/u16:4+flush-8:0	09:41:22	5
1332842	99	23	19,3	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	11:21:29	5
1332837	99	23	21,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	12:26:38	4
1332848	99	22	22,0	cyclictest	0	-21	swapper/7	12:04:11	7
1332848	99	22	21,1	cyclictest	1761178	-21	ssh	10:24:51	7
1332846	99	22	20,2	cyclictest	1844130	-21	kworker/u16:1+events_unbound	11:55:55	6
1332846	99	22	20,2	cyclictest	1665218	-21	kworker/u16:1+events_unbound	10:17:27	6
1332846	99	22	20,2	cyclictest	1314573	-21	kworker/u16:1+events_unbound	07:41:32	6
1332846	99	22	19,2	cyclictest	1665218	-21	kworker/u16:1+flush-8:0	10:06:15	6
1332846	99	22	19,2	cyclictest	1653496	-21	kworker/u16:0+flush-8:0	12:26:16	6
1332842	99	22	20,2	cyclictest	1259537	-21	kworker/u16:0+flush-8:0	07:51:40	5
1332837	99	22	21,1	cyclictest	1844130	-21	kworker/u16:1+events_unbound	12:03:56	4
1332837	99	22	20,2	cyclictest	1653496	-21	kworker/u16:0+events_unbound	10:31:25	4
1332837	99	22	20,2	cyclictest	1394720	-21	kworker/u16:2+flush-8:0	08:11:25	4
1332848	99	21	21,0	cyclictest	0	-21	swapper/7	11:51:47	7
1332848	99	21	20,0	cyclictest	0	-21	swapper/7	12:26:51	7
1332848	99	21	0,20	cyclictest	1965938	-21	ssh	11:49:04	7
1332846	99	21	19,2	cyclictest	1844130	-21	kworker/u16:1+events_unbound	11:34:23	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17