OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed May 06, 2026 00:43:35)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
4158409	2	59	48,8	sleep4	0	-21	swapper/4	19:05:16	4
4158507	2	40	3,31	sleep7	0	-21	swapper/7	19:06:39	7
4149430	2	40	29,7	sleep5	0	-21	swapper/5	19:03:48	5
4158943	99	38	36,2	cyclictest	3894294	-21	kworker/u16:2+flush-8:0	19:13:51	5
4158340	2	35	23,8	sleep6	0	-21	swapper/6	19:04:21	6
4158943	99	34	32,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	23:08:58	5
4158948	99	32	30,2	cyclictest	551527	-21	kworker/u16:1+flush-8:0	00:13:51	6
4158943	99	32	27,4	cyclictest	369796	-21	kworker/u16:0+flush-8:0	22:23:36	5
4158943	99	30	28,2	cyclictest	171525	-21	kworker/u16:1+flush-8:0	21:33:36	5
4158948	99	29	27,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	23:42:43	6
4158935	99	29	25,3	cyclictest	133907	-21	kworker/u16:3+events_unbound	20:38:59	4
117584	2	29	0,1	chrt	0	-21	swapper/7	20:23:54	7
4158948	99	28	26,2	cyclictest	321346	-21	kworker/u16:4+events_unbound	23:14:48	6
4158935	99	28	26,2	cyclictest	4139835	-21	kworker/u16:0+events_unbound	19:09:03	4
4158948	99	27	25,2	cyclictest	4191822	-21	kworker/u16:1+events_unbound	20:00:19	6
4158948	99	27	25,2	cyclictest	321346	-21	kworker/u16:4+events_unbound	22:19:43	6
4158948	99	27	25,2	cyclictest	265507	-21	kworker/u16:3+flush-8:0	22:48:51	6
4158948	99	27	25,2	cyclictest	265507	-21	kworker/u16:3+flush-8:0	22:13:35	6
4158943	99	27	25,2	cyclictest	265507	-21	kworker/u16:3+flush-8:0	21:58:30	5
4158935	99	27	25,2	cyclictest	85075	-21	kworker/u16:0+flush-8:0	20:43:54	4
4158952	99	26	19,6	cyclictest	635	-21	systemd-journal	23:48:36	7
4158948	99	26	24,2	cyclictest	501545	-21	kworker/u16:1+events_unbound	23:19:27	6
4158948	99	26	24,2	cyclictest	4191822	-21	kworker/u16:1+flush-8:0	19:30:35	6
4158948	99	26	24,2	cyclictest	4086117	-21	kworker/u16:3+events_unbound	19:45:39	6
4158943	99	26	24,2	cyclictest	265507	-21	kworker/u16:3+flush-8:0	22:48:50	5
4158952	99	25	18,6	cyclictest	369805	-21	latency_hist	22:18:36	7
4158948	99	25	23,2	cyclictest	3894294	-21	kworker/u16:2+flush-8:0	19:25:07	6
4158948	99	25	23,2	cyclictest	321346	-21	kworker/u16:4+events_unbound	22:02:39	6
4158948	99	25	23,2	cyclictest	133907	-21	kworker/u16:3+events_unbound	20:47:11	6
4158935	99	25	23,2	cyclictest	321346	-21	kworker/u16:4+flush-8:0	22:53:52	4
4158952	99	24	18,5	cyclictest	635	-21	systemd-journal	19:43:36	7
4158948	99	24	22,2	cyclictest	85075	-21	kworker/u16:0+events_unbound	20:41:31	6
4158948	99	24	22,2	cyclictest	3894294	-21	kworker/u16:2+events_unbound	19:17:02	6
4158948	99	24	22,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	22:59:00	6
4158948	99	24	22,2	cyclictest	265507	-21	kworker/u16:3+events_unbound	22:08:13	6
4158948	99	24	22,2	cyclictest	171525	-21	kworker/u16:1+events_unbound	21:32:57	6
4158948	99	24	22,2	cyclictest	171525	-21	kworker/u16:1+events_unbound	21:05:43	6
4158948	99	24	22,2	cyclictest	113863	-21	kworker/u16:2+events_unbound	21:14:42	6
4158948	99	24	22,2	cyclictest	113863	-21	kworker/u16:2+events_unbound	21:01:07	6
4158948	99	23	21,2	cyclictest	4191822	-21	kworker/u16:1+flush-8:0	20:20:35	6
4158948	99	23	21,2	cyclictest	4191822	-21	kworker/u16:1+flush-8:0	19:42:11	6
4158948	99	23	21,2	cyclictest	3894294	-21	kworker/u16:2+flush-8:0	19:53:37	6
4158948	99	23	21,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	22:24:40	6
4158948	99	23	21,2	cyclictest	265507	-21	kworker/u16:3+events_unbound	23:05:27	6
4158948	99	23	21,2	cyclictest	113863	-21	kworker/u16:2+events_unbound	21:10:15	6
4158948	99	23	21,2	cyclictest	113863	-21	kworker/u16:2+events_unbound	20:56:08	6
4158948	99	23	20,2	cyclictest	85075	-21	kworker/u16:0+events_unbound	20:13:46	6
4158943	99	23	21,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	00:03:53	5
4158943	99	23	21,2	cyclictest	321346	-21	kworker/u16:4+flush-8:0	23:14:13	5
4158943	99	23	20,3	cyclictest	321346	-21	kworker/u16:4+flush-8:0	22:04:48	5
4158943	99	23	19,3	cyclictest	171525	-21	kworker/u16:1+flush-8:0	21:33:58	5
4158952	99	22	17,4	cyclictest	575774	-21	latency_hist	23:43:36	7
4158952	99	22	17,4	cyclictest	103893	-21	latency_hist	20:18:36	7
4158948	99	22	20,2	cyclictest	551527	-21	kworker/u16:1+events_unbound	23:36:31	6
4158948	99	22	20,2	cyclictest	527339	-21	kworker/u16:2+events_unbound	00:19:31	6
4158948	99	22	20,2	cyclictest	4139835	-21	kworker/u16:0+events_unbound	19:09:20	6
4158948	99	22	20,2	cyclictest	4086117	-21	kworker/u16:3+events_unbound	19:21:59	6
4158948	99	22	20,2	cyclictest	369796	-21	kworker/u16:0+flush-8:0	22:33:40	6
4158948	99	22	20,2	cyclictest	321346	-21	kworker/u16:4+events_unbound	23:26:16	6
4158948	99	22	20,2	cyclictest	133907	-21	kworker/u16:3+events_unbound	20:36:22	6
4158948	99	22	20,2	cyclictest	133907	-21	kworker/u16:3+events_unbound	20:36:22	6
4158952	99	21	15,5	cyclictest	636335	-21	latency_hist	00:08:35	7
4158948	99	21	19,2	cyclictest	85075	-21	kworker/u16:0+flush-8:0	20:12:54	6
4158948	99	21	19,2	cyclictest	4191822	-21	kworker/u16:1+events_unbound	20:30:52	6
4158948	99	21	19,2	cyclictest	4191822	-21	kworker/u16:1+events_unbound	20:06:46	6
4158948	99	21	19,2	cyclictest	4191822	-21	kworker/u16:1+events_unbound	19:35:24	6
4158948	99	21	19,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	23:52:43	6
4158948	99	21	19,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	23:44:32	6
4158948	99	21	19,2	cyclictest	265507	-21	kworker/u16:3+events_unbound	22:14:15	6
4158948	99	21	18,2	cyclictest	85075	-21	kworker/u16:0+events_unbound	20:23:36	6
4158943	99	21	19,2	cyclictest	4191822	-21	kworker/u16:1+flush-8:0	19:53:51	5
4158943	99	21	19,2	cyclictest	265507	-21	kworker/u16:3+flush-8:0	21:49:13	5
4158943	99	21	18,2	cyclictest	85075	-21	kworker/u16:0+flush-8:0	20:33:51	5
4158943	99	21	18,2	cyclictest	85075	-21	kworker/u16:0+flush-8:0	20:33:51	5
4158952	99	20	14,5	cyclictest	65344	-21	latency_hist	19:58:36	7
4158948	99	20	18,2	cyclictest	527339	-21	kworker/u16:2+flush-8:0	00:29:14	6
4158948	99	20	18,2	cyclictest	527339	-21	kworker/u16:2+events_unbound	23:53:59	6
4158948	99	20	18,2	cyclictest	4191822	-21	kworker/u16:1+events_unbound	19:51:12	6
4158948	99	20	18,2	cyclictest	113863	-21	kworker/u16:2+events_unbound	21:44:23	6
4158948	99	20	17,2	cyclictest	551527	-21	kworker/u16:1+events_unbound	00:03:41	6
4158943	99	20	18,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	22:38:58	5
4158943	99	20	18,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	22:38:58	5
4158943	99	20	18,2	cyclictest	321346	-21	kworker/u16:4+events_unbound	22:33:51	5
4158935	99	20	18,2	cyclictest	171525	-21	kworker/u16:1+flush-8:0	21:14:15	4
4158935	99	20	17,2	cyclictest	4191822	-21	kworker/u16:1+flush-8:0	20:28:51	4
4158952	99	19	0,17	cyclictest	330373	-21	ssh	21:59:01	7
4158948	99	19	17,2	cyclictest	501545	-21	kworker/u16:1+events_unbound	23:12:47	6
4158943	99	19	17,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	23:48:50	5
4158935	99	19	17,2	cyclictest	265507	-21	kworker/u16:3+events_unbound	22:49:02	4
4158952	99	18	9,8	cyclictest	0	-21	swapper/7	19:48:28	7
4158952	99	18	15,2	cyclictest	1234	-21	snmpd	22:21:30	7
4158952	99	18	14,3	cyclictest	1234	-21	snmpd	23:25:52	7
4158948	99	18	16,2	cyclictest	677651	-21	kworker/u16:0+flush-8:0	00:24:01	6
4158948	99	18	16,2	cyclictest	551527	-21	kworker/u16:1+flush-8:0	23:59:19	6
4158948	99	18	16,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	23:31:23	6
4158948	99	18	16,2	cyclictest	113863	-21	kworker/u16:2+events_unbound	21:54:48	6
4158948	99	18	15,2	cyclictest	85075	-21	kworker/u16:0+events_unbound	21:23:36	6
4158948	99	18	15,2	cyclictest	369796	-21	kworker/u16:0+events_unbound	00:08:42	6
4158943	99	18	16,2	cyclictest	527339	-21	kworker/u16:2+flush-8:0	00:19:06	5
4158943	99	18	16,2	cyclictest	4086117	-21	kworker/u16:3+events_unbound	19:19:01	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17