OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sun May 10, 2026 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1033102	2	50	40,7	sleep5	0	-21	swapper/5	19:07:40	5
1033435	99	47	45,2	cyclictest	1591794	-21	kworker/u16:3+events_unbound	23:58:46	5
1033435	99	47	44,2	cyclictest	1689376	-21	kworker/u16:2+flush-8:0	00:03:46	5
1033074	2	47	20,8	sleep4	0	-21	swapper/4	19:07:14	4
1033435	99	44	42,2	cyclictest	1134972	-21	kworker/u16:2+flush-8:0	20:23:46	5
1033435	99	43	41,2	cyclictest	1075991	-21	kworker/u16:3+events_unbound	19:48:46	5
1032884	2	40	3,31	sleep7	0	-21	swapper/7	19:04:37	7
1033435	99	39	37,2	cyclictest	1543943	-21	kworker/u16:1+events_unbound	23:02:58	5
1033435	99	39	37,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	22:40:26	5
1033435	99	38	36,2	cyclictest	1591794	-21	kworker/u16:3+events_unbound	23:29:15	5
1033435	99	38	36,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	22:29:31	5
1033435	99	38	36,2	cyclictest	1268537	-21	kworker/u16:1+events_unbound	21:19:43	5
1033435	99	37	35,2	cyclictest	1631027	-21	kworker/u16:0+events_unbound	23:56:07	5
1033435	99	37	35,2	cyclictest	1631027	-21	kworker/u16:0+events_unbound	23:56:07	5
1033435	99	37	35,2	cyclictest	1591794	-21	kworker/u16:3+flush-8:0	23:35:46	5
1033435	99	37	35,2	cyclictest	1591794	-21	kworker/u16:3+events_unbound	00:29:50	5
1033435	99	37	35,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	22:24:55	5
1033435	99	36	34,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	22:19:43	5
1033435	99	36	34,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	21:49:31	5
1033435	99	36	34,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	21:49:31	5
1033435	99	36	34,2	cyclictest	1134972	-21	kworker/u16:2+flush-8:0	20:13:45	5
1033435	99	36	34,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	21:12:47	5
1033435	99	36	34,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	21:12:47	5
1033435	99	36	34,2	cyclictest	1075991	-21	kworker/u16:3+events_unbound	21:37:42	5
1033435	99	35	33,2	cyclictest	1631027	-21	kworker/u16:0+events_unbound	23:46:07	5
1033435	99	35	33,2	cyclictest	1484218	-21	kworker/u16:0+events_unbound	23:12:46	5
1033435	99	35	33,2	cyclictest	1484218	-21	kworker/u16:0+events_unbound	23:05:15	5
1033435	99	35	33,2	cyclictest	1484218	-21	kworker/u16:0+events_unbound	22:37:47	5
1033435	99	35	33,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	23:26:27	5
1033435	99	35	33,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	23:18:59	5
1033435	99	35	33,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	21:58:07	5
1033435	99	35	33,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	21:16:18	5
1033431	99	35	33,2	cyclictest	1304400	-21	kworker/u16:0+events_unbound	21:48:19	4
1033435	99	34	32,2	cyclictest	1714975	-21	kworker/u16:0+events_unbound	00:35:34	5
1033435	99	34	32,2	cyclictest	1714975	-21	kworker/u16:0+events_unbound	00:35:34	5
1033435	99	34	32,2	cyclictest	1591794	-21	kworker/u16:3+events_unbound	00:27:27	5
1033435	99	34	32,2	cyclictest	1498085	-21	kworker/u16:3+events_unbound	22:55:11	5
1033435	99	34	32,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	22:15:27	5
1033435	99	34	32,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	21:42:39	5
1033435	99	34	32,2	cyclictest	1210670	-21	kworker/u16:1+events_unbound	20:53:22	5
1033435	99	33	31,2	cyclictest	1714975	-21	kworker/u16:0+events_unbound	00:15:50	5
1033435	99	33	31,2	cyclictest	1631027	-21	kworker/u16:0+events_unbound	23:52:54	5
1033435	99	33	31,2	cyclictest	1631027	-21	kworker/u16:0+events_unbound	23:43:02	5
1033435	99	33	31,2	cyclictest	1484218	-21	kworker/u16:0+events_unbound	22:45:28	5
1033435	99	33	31,2	cyclictest	1353860	-21	kworker/u16:1+events_unbound	21:45:40	5
1033435	99	33	31,2	cyclictest	1304404	-21	kworker/u16:4+flush-8:0	21:24:11	5
1033435	99	33	31,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	22:02:07	5
1033435	99	33	31,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	21:29:51	5
1033435	99	33	30,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	20:33:22	5
1033435	99	32	30,2	cyclictest	1689376	-21	kworker/u16:2+events_unbound	00:10:47	5
1033431	99	32	29,2	cyclictest	1740021	-21	kworker/u16:1+events_unbound	00:33:31	4
1033431	99	32	29,2	cyclictest	1740021	-21	kworker/u16:1+events_unbound	00:33:31	4
1498078	2	31	0,1	chrt	20	1	rcu_preempt	22:43:20	6
1033435	99	31	29,2	cyclictest	1075991	-21	kworker/u16:3+flush-8:0	20:04:39	5
1033435	99	31	29,2	cyclictest	1033651	-21	kworker/u16:2+events_unbound	19:32:59	5
1033436	99	30	28,2	cyclictest	1484218	-21	kworker/u16:0+events_unbound	23:03:35	6
1033436	99	30	28,2	cyclictest	1210670	-21	kworker/u16:1+events_unbound	20:48:44	6
1033435	99	30	28,2	cyclictest	1689376	-21	kworker/u16:2+events_unbound	00:20:45	5
1033435	99	30	28,2	cyclictest	1210670	-21	kworker/u16:1+events_unbound	20:50:47	5
1033435	99	30	27,2	cyclictest	1304404	-21	kworker/u16:4+flush-8:0	22:08:23	5
1033431	99	30	28,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	21:58:38	4
1025637	2	30	4,7	sleep6	0	-21	swapper/6	19:03:40	6
1033441	99	29	27,1	cyclictest	0	-21	swapper/7	21:38:35	7
1033441	99	29	23,5	cyclictest	1200640	-21	latency_hist	20:33:20	7
1033441	99	29	23,5	cyclictest	1200640	-21	latency_hist	20:33:20	7
1033436	99	29	27,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	20:28:48	6
1033436	99	29	27,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	20:28:48	6
1033435	99	29	27,2	cyclictest	1304404	-21	kworker/u16:4+flush-8:0	22:06:51	5
1033441	99	28	25,2	cyclictest	1543251	-21	ntpq	22:58:45	7
1033441	99	28	0,26	cyclictest	1582680	-21	latency_hist	23:18:19	7
1522711	2	27	0,1	chrt	1522709	-21	ssh	22:53:30	7
1033441	99	27	0,25	cyclictest	1075603	-21	latency_hist	19:28:21	7
1033435	99	27	25,2	cyclictest	1374621	-21	kworker/u16:2+flush-8:0	22:51:23	5
1033435	99	27	25,2	cyclictest	1210670	-21	kworker/u16:1+events_unbound	20:46:47	5
1033435	99	27	25,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	20:10:30	5
1033435	99	27	25,2	cyclictest	1134972	-21	kworker/u16:2+events_unbound	20:01:07	5
1033435	99	27	25,2	cyclictest	1033651	-21	kworker/u16:2+flush-8:0	19:15:00	5
1033435	99	27	25,2	cyclictest	1033651	-21	kworker/u16:2+events_unbound	19:24:56	5
1033435	99	27	25,2	cyclictest	1033651	-21	kworker/u16:2+events_unbound	19:20:40	5
1033431	99	27	24,2	cyclictest	1484218	-21	kworker/u16:0+flush-8:0	23:03:39	4
1033435	99	26	24,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	23:17:26	5
1033435	99	26	24,2	cyclictest	1239536	-21	kworker/u16:0+events_unbound	21:01:03	5
1033435	99	26	24,2	cyclictest	1065630	-21	kworker/u16:1+events_unbound	19:47:03	5
1033435	99	26	24,2	cyclictest	1065630	-21	kworker/u16:1+events_unbound	19:47:03	5
1033436	99	25	23,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	23:08:34	6
1033436	99	25	23,2	cyclictest	1239536	-21	kworker/u16:0+flush-8:0	20:53:36	6
1033436	99	25	22,2	cyclictest	1075991	-21	kworker/u16:3+events_unbound	19:58:33	6
1033441	99	24	23,0	cyclictest	0	-21	swapper/7	19:08:39	7
1033436	99	24	22,2	cyclictest	1304404	-21	kworker/u16:4+flush-8:0	21:28:47	6
1033436	99	24	22,2	cyclictest	1075991	-21	kworker/u16:3+flush-8:0	21:23:42	6
1033436	99	24	21,2	cyclictest	1353860	-21	kworker/u16:1+flush-8:0	22:33:38	6
1033435	99	24	22,2	cyclictest	1075991	-21	kworker/u16:3+events_unbound	20:21:31	5
1033431	99	24	23,1	cyclictest	1689376	-21	kworker/u16:2+events_unbound	00:16:15	4
1033431	99	24	21,2	cyclictest	1033651	-21	kworker/u16:2+events_unbound	19:13:38	4
1033441	99	23	20,2	cyclictest	1407955	-21	ssh	22:03:40	7
1033431	99	23	20,2	cyclictest	1134972	-21	kworker/u16:2+flush-8:0	20:28:35	4
1033431	99	23	20,2	cyclictest	1134972	-21	kworker/u16:2+flush-8:0	20:28:35	4
1033436	99	22	20,2	cyclictest	1374621	-21	kworker/u16:2+events_unbound	21:53:47	6
1033436	99	22	20,2	cyclictest	1304404	-21	kworker/u16:4+events_unbound	22:23:33	6
1033436	99	22	20,2	cyclictest	1075991	-21	kworker/u16:3+flush-8:0	21:18:43	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17