OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Jul 28, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2056079	2	146	0,1	sleep7	2056078	-21	ssh	11:06:17	7
1516074	2	56	45,7	sleep5	0	-21	swapper/5	07:04:16	5
1516073	2	56	45,7	sleep4	0	-21	swapper/4	07:04:15	4
1996512	2	54	0,0	sleep7	1996513	-21	ssh	10:43:57	7
1525232	2	54	19,30	sleep7	0	-21	swapper/7	07:07:23	7
1525658	99	49	47,1	cyclictest	1858920	-21	kworker/u16:0+events_unbound	09:54:12	6
1525647	99	43	41,2	cyclictest	1752376	-21	kworker/u16:5+events_unbound	09:34:03	4
1525659	99	41	38,2	cyclictest	1655982	-21	perf	08:14:03	7
1525231	2	40	30,7	sleep6	0	-21	swapper/6	07:07:22	6
1525659	99	39	34,4	cyclictest	2069852	-21	latency_hist	11:14:03	7
1525647	99	38	35,2	cyclictest	1787635	-21	kworker/u16:0+flush-8:0	09:19:06	4
1525652	99	37	35,2	cyclictest	2214741	-21	kworker/u16:3+events_unbound	12:20:23	5
1525652	99	37	35,2	cyclictest	1752376	-21	kworker/u16:5+events_unbound	09:10:47	5
1525647	99	37	34,2	cyclictest	1218428	-21	kworker/u16:0+flush-8:0	07:59:29	4
1525652	99	36	35,1	cyclictest	1957213	-21	kworker/u16:0+flush-8:0	12:18:07	5
1525652	99	36	34,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	10:31:42	5
1525652	99	36	33,2	cyclictest	1996617	-21	kworker/u16:3+events_unbound	11:09:15	5
1525647	99	36	34,2	cyclictest	1752376	-21	kworker/u16:5+events_unbound	09:38:58	4
1525647	99	35	33,2	cyclictest	1957213	-21	kworker/u16:0+flush-8:0	11:46:02	4
1525647	99	35	33,2	cyclictest	1957213	-21	kworker/u16:0+flush-8:0	10:44:05	4
1525647	99	35	33,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	12:04:33	4
1525652	99	34	32,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	11:48:22	5
1525647	99	34	32,2	cyclictest	2214741	-21	kworker/u16:3+events_unbound	12:23:03	4
1525647	99	34	32,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	12:16:03	4
1525647	99	34	32,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	11:33:41	4
1525647	99	34	32,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	10:39:03	4
1525659	99	33	30,2	cyclictest	1775300	-21	munin-run	09:14:03	7
1525652	99	33	31,2	cyclictest	2251717	-21	kworker/u16:1+flush-8:0	12:38:26	5
1525652	99	33	31,2	cyclictest	2021053	-21	kworker/u16:1+events_unbound	11:08:01	5
1525652	99	33	31,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	12:05:18	5
1525652	99	33	31,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	09:57:18	5
1525652	99	33	31,2	cyclictest	1685342	-21	kworker/u16:4+events_unbound	08:49:06	5
1525652	99	33	31,2	cyclictest	1558647	-21	kworker/u16:2+events_unbound	07:45:43	5
1525647	99	33	31,2	cyclictest	2180304	-21	kworker/u16:1+flush-8:0	12:09:05	4
1525647	99	33	31,2	cyclictest	1996617	-21	kworker/u16:3+flush-8:0	11:02:11	4
1525647	99	33	31,2	cyclictest	1996617	-21	kworker/u16:3+flush-8:0	11:02:11	4
1525647	99	33	31,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	10:42:11	4
1525652	99	32	30,2	cyclictest	1996617	-21	kworker/u16:3+events_unbound	11:27:14	5
1525652	99	32	30,2	cyclictest	1940059	-21	kworker/u16:2+flush-8:0	11:17:01	5
1525652	99	32	30,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:43:55	5
1525652	99	32	30,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:43:55	5
1525652	99	32	30,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:29:06	5
1525652	99	32	30,2	cyclictest	1548432	-21	kworker/u16:1+events_unbound	07:28:43	5
1525652	99	32	29,2	cyclictest	1957213	-21	kworker/u16:0+flush-8:0	11:54:04	5
1525647	99	32	31,1	cyclictest	1772170	-21	kworker/u16:3+events_unbound	09:18:50	4
1525647	99	32	30,2	cyclictest	1996617	-21	kworker/u16:3+flush-8:0	11:38:22	4
1525647	99	32	30,2	cyclictest	1957213	-21	kworker/u16:0+flush-8:0	11:53:23	4
1525647	99	32	30,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	10:27:55	4
1525647	99	32	30,2	cyclictest	1940059	-21	kworker/u16:2+flush-8:0	11:15:50	4
1525647	99	32	30,2	cyclictest	1800352	-21	kworker/u16:1+flush-8:0	09:50:30	4
1746906	2	31	0,1	chrt	0	-21	swapper/7	08:59:23	7
1525652	99	31	29,2	cyclictest	2251717	-21	kworker/u16:1+events_unbound	12:30:33	5
1525652	99	31	29,2	cyclictest	2021053	-21	kworker/u16:1+events_unbound	10:57:39	5
1525652	99	31	29,2	cyclictest	1996617	-21	kworker/u16:3+events_unbound	11:31:26	5
1525652	99	31	29,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	10:12:15	5
1525652	99	31	29,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:37:18	5
1525652	99	31	29,2	cyclictest	1772170	-21	kworker/u16:3+events_unbound	09:19:02	5
1525647	99	31	29,2	cyclictest	1940059	-21	kworker/u16:2+flush-8:0	11:19:40	4
1525647	99	31	29,2	cyclictest	1548432	-21	kworker/u16:1+events_unbound	07:46:34	4
1525647	99	31	28,2	cyclictest	1957213	-21	kworker/u16:0+flush-8:0	10:49:33	4
1525647	99	31	28,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	11:58:18	4
2216613	2	30	0,1	chrt	2216611	-21	sh	12:13:17	7
1525659	99	30	29,0	cyclictest	0	-21	swapper/7	12:14:18	7
1525652	99	30	28,2	cyclictest	1996617	-21	kworker/u16:3+flush-8:0	12:03:57	5
1525652	99	30	28,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	12:12:01	5
1525652	99	30	28,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	10:28:50	5
1525652	99	30	28,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	09:51:18	5
1525647	99	30	28,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	11:26:50	4
1525647	99	30	28,2	cyclictest	1940059	-21	kworker/u16:2+flush-8:0	11:05:46	4
1525647	99	30	28,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	10:13:27	4
1525647	99	30	28,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:27:51	4
1525647	99	30	28,2	cyclictest	1743795	-21	kworker/u16:2+events_unbound	09:10:43	4
1525647	99	30	28,2	cyclictest	1548432	-21	kworker/u16:1+events_unbound	07:35:15	4
1525652	99	29	27,2	cyclictest	1996617	-21	kworker/u16:3+events_unbound	10:48:18	5
1525652	99	29	27,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	12:24:22	5
1525652	99	29	27,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	10:50:39	5
1525652	99	29	27,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	10:03:23	5
1525652	99	29	27,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:44:38	5
1525647	99	29	27,2	cyclictest	2214741	-21	kworker/u16:3+flush-8:0	12:26:50	4
1525647	99	29	27,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:57:38	4
1525647	99	29	27,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	09:43:22	4
1525647	99	29	27,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	09:43:22	4
1525647	99	29	27,2	cyclictest	1685342	-21	kworker/u16:4+events_unbound	08:53:14	4
1525652	99	28	26,2	cyclictest	2021053	-21	kworker/u16:1+flush-8:0	11:02:31	5
1525652	99	28	26,2	cyclictest	2021053	-21	kworker/u16:1+flush-8:0	11:02:31	5
1525652	99	28	26,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	10:37:59	5
1525652	99	28	26,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	10:05:38	5
1525652	99	28	26,2	cyclictest	1800352	-21	kworker/u16:1+events_unbound	10:19:58	5
1525652	99	28	26,2	cyclictest	1787635	-21	kworker/u16:0+events_unbound	09:21:48	5
1525647	99	28	26,2	cyclictest	2251717	-21	kworker/u16:1+events_unbound	12:31:10	4
1525647	99	28	26,2	cyclictest	2228922	-21	kworker/u16:2+events_unbound	12:35:34	4
1525647	99	28	26,2	cyclictest	1996617	-21	kworker/u16:3+events_unbound	10:55:03	4
1525647	99	28	26,2	cyclictest	1940059	-21	kworker/u16:2+events_unbound	11:11:42	4
1525647	99	28	26,2	cyclictest	1922786	-21	kworker/u16:4+flush-8:0	10:15:15	4
1525647	99	28	26,2	cyclictest	1858920	-21	kworker/u16:0+events_unbound	10:01:13	4
1525647	99	28	26,2	cyclictest	1800353	-21	kworker/u16:2+events_unbound	09:48:46	4
1525659	99	27	21,5	cyclictest	1743602	-21	latency_hist	08:59:03	7
1525658	99	27	25,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	11:22:05	6
1525652	99	27	25,2	cyclictest	1957213	-21	kworker/u16:0+events_unbound	10:40:22	5
1525652	99	27	25,2	cyclictest	1752376	-21	kworker/u16:5+flush-8:0	09:25:56	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17