OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Aug 12, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1075327	2	50	40,7	sleep4	0	-21	swapper/4	19:04:33	4
1075346	2	45	35,7	sleep5	0	-21	swapper/5	19:04:50	5
1075865	99	43	38,4	cyclictest	1469389	-21	latency_hist	22:13:07	7
1075853	99	43	40,2	cyclictest	1759568	-21	kworker/u16:3+events_unbound	00:28:09	4
1074759	2	41	4,30	sleep7	0	-21	swapper/7	19:03:40	7
1075320	2	36	26,7	sleep6	0	-21	swapper/6	19:04:27	6
1075865	99	33	31,1	cyclictest	0	-21	swapper/7	21:55:41	7
1443290	2	32	0,1	chrt	1443289	-21	ssh	21:59:12	7
1075865	99	32	26,5	cyclictest	1263978	-21	latency_hist	20:43:08	7
1075857	99	32	30,2	cyclictest	1613704	-21	kworker/u16:1+events_unbound	23:13:57	5
1075857	99	31	29,2	cyclictest	1720654	-21	kworker/u16:4+events_unbound	00:06:00	5
1075857	99	30	28,2	cyclictest	1759568	-21	kworker/u16:3+events_unbound	00:34:20	5
1075857	99	30	28,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	00:01:44	5
1075857	99	29	27,2	cyclictest	1744896	-21	kworker/u16:0+events_unbound	00:25:19	5
1075857	99	29	27,2	cyclictest	1541422	-21	kworker/u16:1+events_unbound	22:44:03	5
1075857	99	29	27,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	23:03:00	5
1075857	99	29	27,2	cyclictest	1407058	-21	kworker/u16:0+events_unbound	22:25:23	5
1075853	99	29	27,2	cyclictest	370161	-21	kworker/u16:3+flush-8:0	20:03:37	4
1075853	99	29	27,2	cyclictest	1720654	-21	kworker/u16:4+flush-8:0	00:03:41	4
1075857	99	28	26,2	cyclictest	1613704	-21	kworker/u16:1+flush-8:0	23:54:23	5
1075857	99	28	26,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	23:33:00	5
1075857	99	28	26,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	22:54:03	5
1075857	99	28	26,2	cyclictest	1455173	-21	kworker/u16:1+events_unbound	22:22:45	5
1075857	99	28	26,2	cyclictest	1407058	-21	kworker/u16:0+events_unbound	22:07:40	5
1075857	99	28	26,2	cyclictest	1292859	-21	kworker/u16:0+events_unbound	21:23:59	5
1075853	99	28	26,2	cyclictest	1348880	-21	kworker/u16:5+events_unbound	21:58:35	4
1075853	99	28	26,2	cyclictest	1098066	-21	kworker/u16:4+flush-8:0	20:43:07	4
1539910	2	27	0,1	chrt	1539908	-21	ssh	22:39:11	7
1075865	99	27	26,0	cyclictest	0	-21	swapper/7	23:48:09	7
1075865	99	27	26,0	cyclictest	0	-21	swapper/7	22:26:41	7
1075865	99	27	25,1	cyclictest	1625853	-21	ssh	23:17:18	7
1075865	99	27	25,1	cyclictest	0	-21	swapper/7	00:33:29	7
1075857	99	27	25,2	cyclictest	1576084	-21	kworker/u16:0+events_unbound	23:07:24	5
1075857	99	27	25,2	cyclictest	1542098	-21	kworker/u16:2+events_unbound	23:24:24	5
1075857	99	27	25,2	cyclictest	1292859	-21	kworker/u16:0+events_unbound	21:17:44	5
1075857	99	27	25,2	cyclictest	1283134	-21	kworker/u16:1+events_unbound	21:54:11	5
1075857	99	27	25,2	cyclictest	1205719	-21	kworker/u16:2+events_unbound	20:52:45	5
1075857	99	27	25,2	cyclictest	1098066	-21	kworker/u16:4+events_unbound	20:29:53	5
1075853	99	27	25,2	cyclictest	1419381	-21	kworker/u16:2+events_unbound	21:53:24	4
1075853	99	27	24,2	cyclictest	1782960	-21	kworker/u16:2+events_unbound	00:23:08	4
1373092	2	26	0,1	chrt	1373093	-21	latency_hist	21:33:07	7
1075865	99	26	24,1	cyclictest	1783313	-21	perf	00:23:06	7
1075865	99	26	24,1	cyclictest	1783313	-21	perf	00:23:06	7
1075857	99	26	24,2	cyclictest	370161	-21	kworker/u16:3+flush-8:0	19:28:09	5
1075857	99	26	24,2	cyclictest	1795470	-21	kworker/u16:5+events_unbound	00:30:27	5
1075857	99	26	24,2	cyclictest	1542098	-21	kworker/u16:2+flush-8:0	23:20:15	5
1075857	99	26	24,2	cyclictest	1542098	-21	kworker/u16:2+events_unbound	00:08:19	5
1075857	99	26	24,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	23:47:20	5
1075857	99	26	24,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	22:35:31	5
1075857	99	26	24,2	cyclictest	1419381	-21	kworker/u16:2+events_unbound	22:00:40	5
1075857	99	26	24,2	cyclictest	1292859	-21	kworker/u16:0+events_unbound	21:30:03	5
1075857	99	26	24,2	cyclictest	1211004	-21	kworker/u16:3+events_unbound	20:22:53	5
1075857	99	26	24,2	cyclictest	1098066	-21	kworker/u16:4+flush-8:0	19:50:05	5
1075857	99	26	23,2	cyclictest	1098066	-21	kworker/u16:4+flush-8:0	19:53:13	5
1075853	99	26	23,2	cyclictest	1720654	-21	kworker/u16:4+flush-8:0	23:58:08	4
1075865	99	25	0,24	cyclictest	1732957	-21	ssh	23:59:09	7
1075857	99	25	23,2	cyclictest	1407058	-21	kworker/u16:0+events_unbound	21:46:00	5
1075857	99	25	23,2	cyclictest	1292859	-21	kworker/u16:0+events_unbound	21:21:15	5
1075857	99	25	23,2	cyclictest	1292859	-21	kworker/u16:0+events_unbound	21:10:20	5
1075857	99	25	23,2	cyclictest	1283134	-21	kworker/u16:1+events_unbound	21:08:07	5
1075857	99	25	23,2	cyclictest	1205719	-21	kworker/u16:2+events_unbound	20:39:57	5
1075857	99	25	23,2	cyclictest	1098066	-21	kworker/u16:4+flush-8:0	20:37:01	5
1075857	99	25	23,2	cyclictest	1098066	-21	kworker/u16:4+events_unbound	20:47:54	5
1075857	99	25	23,2	cyclictest	1098066	-21	kworker/u16:4+events_unbound	20:14:58	5
1075857	99	25	23,2	cyclictest	1098066	-21	kworker/u16:4+events_unbound	20:14:58	5
1075857	99	25	23,2	cyclictest	1098066	-21	kworker/u16:4+events_unbound	19:18:04	5
1075853	99	25	23,2	cyclictest	1613704	-21	kworker/u16:1+events_unbound	23:25:42	4
1075853	99	25	23,2	cyclictest	1348880	-21	kworker/u16:5+events_unbound	21:39:02	4
1075865	99	24	18,5	cyclictest	1157210	-21	latency_hist	19:48:08	7
1075857	99	24	22,2	cyclictest	1613704	-21	kworker/u16:1+events_unbound	00:20:51	5
1075857	99	24	22,2	cyclictest	1613704	-21	kworker/u16:1+events_unbound	00:20:51	5
1075857	99	24	22,2	cyclictest	1527103	-21	kworker/u16:0+flush-8:0	22:39:28	5
1075857	99	24	22,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	23:51:09	5
1075857	99	24	22,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	23:42:35	5
1075857	99	24	22,2	cyclictest	1348880	-21	kworker/u16:5+events_unbound	21:35:35	5
1075857	99	24	22,2	cyclictest	1283134	-21	kworker/u16:1+flush-8:0	21:42:28	5
1075857	99	24	22,2	cyclictest	1283134	-21	kworker/u16:1+events_unbound	20:59:22	5
1075857	99	24	22,2	cyclictest	1283134	-21	kworker/u16:1+events_unbound	20:59:22	5
1075857	99	24	22,2	cyclictest	1156872	-21	kworker/u16:1+events_unbound	19:46:01	5
1075857	99	24	22,2	cyclictest	1108731	-21	kworker/u16:0+flush-8:0	19:23:46	5
1075857	99	24	22,2	cyclictest	1098066	-21	kworker/u16:4+events_unbound	20:08:53	5
1075857	99	24	21,2	cyclictest	1283134	-21	kworker/u16:1+events_unbound	21:48:11	5
1075853	99	24	22,2	cyclictest	1576084	-21	kworker/u16:0+flush-8:0	22:58:09	4
1075853	99	24	22,2	cyclictest	1542098	-21	kworker/u16:2+events_unbound	22:48:21	4
1075853	99	24	22,2	cyclictest	1076077	-21	kworker/u16:2+flush-8:0	19:13:09	4
1075853	99	24	20,3	cyclictest	1638795	-21	kworker/u16:0+flush-8:0	23:33:08	4
1075853	99	24	20,3	cyclictest	1638795	-21	kworker/u16:0+flush-8:0	23:33:08	4
1075865	99	23	21,1	cyclictest	0	-21	swapper/7	22:13:08	7
1075861	99	23	0,16	cyclictest	0	-21	swapper/6	22:28:24	6
1075857	99	23	21,2	cyclictest	370161	-21	kworker/u16:3+flush-8:0	19:43:05	5
1075857	99	23	21,2	cyclictest	1744896	-21	kworker/u16:0+flush-8:0	00:14:31	5
1075857	99	23	21,2	cyclictest	1613704	-21	kworker/u16:1+events_unbound	23:36:35	5
1075857	99	23	21,2	cyclictest	1613704	-21	kworker/u16:1+events_unbound	23:36:35	5
1075857	99	23	21,2	cyclictest	1211004	-21	kworker/u16:3+events_unbound	20:25:23	5
1075857	99	23	21,2	cyclictest	1167084	-21	kworker/u16:0+events_unbound	20:06:01	5
1075857	99	23	21,2	cyclictest	1076077	-21	kworker/u16:2+events_unbound	19:19:26	5
1075853	99	23	21,2	cyclictest	1613704	-21	kworker/u16:1+flush-8:0	00:08:28	4
1075853	99	23	21,2	cyclictest	1481663	-21	kworker/u16:3+events_unbound	22:43:32	4
1075853	99	23	21,2	cyclictest	1283134	-21	kworker/u16:1+events_unbound	21:33:28	4
1075853	99	23	21,2	cyclictest	1254299	-21	kworker/u16:0+flush-8:0	20:43:23	4
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17