OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri Apr 17, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
625777	2	98	0,1	sleep7	625776	-21	grep	21:19:51	7
1090942	2	94	0,1	sleep4	0	-21	swapper/4	00:30:13	4
1090942	2	94	0,1	sleep4	0	-21	swapper/4	00:30:13	4
367550	2	53	43,7	sleep4	0	-21	swapper/4	19:09:21	4
367550	2	53	43,7	sleep4	0	-21	swapper/4	19:09:21	4
367885	99	42	0,40	cyclictest	988949	-21	latency_hist	23:49:51	7
367286	2	42	32,7	sleep5	0	-21	swapper/5	19:05:37	5
367286	2	42	32,7	sleep5	0	-21	swapper/5	19:05:37	5
367872	99	40	38,2	cyclictest	649475	-21	kworker/u16:3+flush-8:0	21:39:51	5
367373	2	40	5,30	sleep7	0	-21	swapper/7	19:06:51	7
367373	2	40	5,30	sleep7	0	-21	swapper/7	19:06:51	7
367872	99	39	37,2	cyclictest	445376	-21	kworker/u16:0+flush-8:0	20:14:51	5
367872	99	35	33,2	cyclictest	477274	-21	kworker/u16:1+flush-8:0	22:09:30	5
367885	99	34	28,5	cyclictest	572709	-21	latency_hist	20:54:51	7
367872	99	32	30,2	cyclictest	723143	-21	kworker/u16:5+events_unbound	22:49:30	5
367514	2	32	22,7	sleep6	0	-21	swapper/6	19:08:51	6
367514	2	32	22,7	sleep6	0	-21	swapper/6	19:08:51	6
367872	99	31	29,2	cyclictest	477274	-21	kworker/u16:1+events_unbound	22:19:46	5
367872	99	31	28,3	cyclictest	611250	-21	kworker/u16:4+flush-8:0	21:24:51	5
367885	99	30	29,0	cyclictest	0	-21	swapper/7	23:54:58	7
367885	99	30	29,0	cyclictest	0	-21	swapper/7	22:25:06	7
367872	99	30	28,2	cyclictest	723143	-21	kworker/u16:5+events_unbound	22:25:14	5
805029	2	28	0,1	chrt	805030	-21	sh	22:30:27	7
367872	99	28	26,2	cyclictest	592262	-21	kworker/u16:3+flush-8:0	21:15:01	5
367872	99	28	26,2	cyclictest	314801	-21	kworker/u16:3+events_unbound	19:10:14	5
367885	99	27	22,4	cyclictest	635	-21	systemd-journal	21:28:11	7
367872	99	27	25,2	cyclictest	952064	-21	kworker/u16:2+flush-8:0	23:36:01	5
367872	99	27	24,2	cyclictest	850102	-21	kworker/u16:1+events_unbound	23:09:53	5
367872	99	27	24,2	cyclictest	850102	-21	kworker/u16:1+events_unbound	23:09:53	5
367872	99	26	24,2	cyclictest	850102	-21	kworker/u16:1+flush-8:0	22:55:26	5
367872	99	26	24,2	cyclictest	625813	-21	kworker/u16:0+events_unbound	21:41:02	5
367872	99	26	24,2	cyclictest	625813	-21	kworker/u16:0+events_unbound	21:41:02	5
367872	99	26	24,2	cyclictest	572384	-21	kworker/u16:3+flush-8:0	20:55:05	5
367872	99	26	24,2	cyclictest	534549	-21	kworker/u16:0+events_unbound	20:45:05	5
367872	99	26	23,2	cyclictest	445376	-21	kworker/u16:0+flush-8:0	20:04:52	5
367885	99	25	24,0	cyclictest	0	-21	swapper/7	21:40:18	7
367885	99	25	24,0	cyclictest	0	-21	swapper/7	21:40:18	7
367885	99	25	24,0	cyclictest	0	-21	swapper/7	20:00:06	7
367885	99	25	20,4	cyclictest	524861	-21	latency_hist	20:29:51	7
367872	99	25	23,2	cyclictest	496462	-21	kworker/u16:2+events_unbound	20:15:14	5
367872	99	24	22,2	cyclictest	952064	-21	kworker/u16:2+flush-8:0	23:50:41	5
367872	99	24	22,2	cyclictest	952064	-21	kworker/u16:2+flush-8:0	23:50:41	5
367872	99	24	22,2	cyclictest	891501	-21	kworker/u16:2+flush-8:0	23:15:53	5
367872	99	24	22,2	cyclictest	339041	-21	kworker/u16:2+flush-8:0	19:34:51	5
367872	99	24	22,2	cyclictest	339041	-21	kworker/u16:2+flush-8:0	19:34:51	5
367872	99	24	21,3	cyclictest	417489	-21	kworker/u16:4+flush-8:0	19:35:15	5
709637	2	23	0,1	chrt	709638	-21	sh	21:53:09	7
367885	99	23	22,0	cyclictest	0	-21	swapper/7	22:20:02	7
367885	99	23	22,0	cyclictest	0	-21	swapper/7	22:00:50	7
367872	99	23	21,2	cyclictest	850102	-21	kworker/u16:1+flush-8:0	23:00:31	5
367872	99	23	21,2	cyclictest	662430	-21	kworker/u16:2+events_unbound	21:59:34	5
367872	99	23	21,2	cyclictest	477274	-21	kworker/u16:1+flush-8:0	21:52:46	5
367872	99	23	21,2	cyclictest	1002749	-21	kworker/u16:3+flush-8:0	00:23:18	5
367872	99	23	20,2	cyclictest	964768	-21	kworker/u16:1+flush-8:0	00:29:56	5
367872	99	23	20,2	cyclictest	964768	-21	kworker/u16:1+flush-8:0	00:29:56	5
367885	99	22	16,5	cyclictest	438624	-21	latency_hist	19:44:52	7
367885	99	22	0,21	cyclictest	1704	-21	dockerd	22:46:42	7
367885	99	22	0,20	cyclictest	866987	-21	ssh	22:58:05	7
367872	99	22	20,2	cyclictest	964768	-21	kworker/u16:1+flush-8:0	00:05:22	5
367872	99	22	20,2	cyclictest	950296	-21	kworker/u16:1+events_unbound	23:33:01	5
367872	99	22	19,2	cyclictest	477274	-21	kworker/u16:1+events_unbound	22:00:03	5
367885	99	21	20,0	cyclictest	0	-21	swapper/7	21:34:31	7
367885	99	21	20,0	cyclictest	0	-21	swapper/7	19:14:47	7
367877	99	21	19,2	cyclictest	964768	-21	kworker/u16:1+flush-8:0	00:05:16	6
367872	99	21	19,2	cyclictest	780740	-21	kworker/u16:0+flush-8:0	22:21:06	5
367872	99	21	19,2	cyclictest	625813	-21	kworker/u16:0+events_unbound	21:32:01	5
367872	99	21	19,2	cyclictest	592262	-21	kworker/u16:3+flush-8:0	21:13:01	5
367872	99	21	19,2	cyclictest	1002749	-21	kworker/u16:3+events_unbound	00:25:14	5
367885	99	20	19,0	cyclictest	0	-21	swapper/7	23:00:21	7
367885	99	20	18,1	cyclictest	1023105	-21	sed	00:00:50	7
367885	99	20	18,1	cyclictest	0	-21	swapper/7	00:30:57	7
367885	99	20	18,1	cyclictest	0	-21	swapper/7	00:30:57	7
367885	99	20	17,2	cyclictest	1046470	-21	ntpq	00:10:20	7
367885	99	20	15,4	cyclictest	563121	-21	latency_hist	20:49:51	7
367885	99	20	0,18	cyclictest	582278	-21	latency_hist	20:59:51	7
367872	99	20	18,2	cyclictest	964768	-21	kworker/u16:1+flush-8:0	00:39:10	5
367872	99	20	18,2	cyclictest	935640	-21	kworker/u16:4+flush-8:0	23:55:06	5
367872	99	20	18,2	cyclictest	850102	-21	kworker/u16:1+flush-8:0	23:08:22	5
367872	99	20	18,2	cyclictest	850102	-21	kworker/u16:1+flush-8:0	23:08:22	5
367872	99	20	18,2	cyclictest	841915	-21	kworker/u16:0+flush-8:0	23:20:49	5
367872	99	20	18,2	cyclictest	477274	-21	kworker/u16:1+flush-8:0	22:11:22	5
367872	99	19	17,2	cyclictest	935640	-21	kworker/u16:4+flush-8:0	00:03:09	5
367872	99	19	17,2	cyclictest	649475	-21	kworker/u16:3+flush-8:0	21:29:51	5
367872	99	19	17,2	cyclictest	477274	-21	kworker/u16:1+events_unbound	22:29:53	5
367872	99	19	17,2	cyclictest	1059445	-21	kworker/u16:2+events_unbound	00:17:01	5
367870	99	19	12,5	cyclictest	0	-21	swapper/4	21:45:40	4
367885	99	18	17,0	cyclictest	0	-21	swapper/7	22:19:34	7
367885	99	18	17,0	cyclictest	0	-21	swapper/7	22:10:05	7
367885	99	18	14,3	cyclictest	1234	-21	snmpd	23:35:57	7
367885	99	18	14,2	cyclictest	0	-21	swapper/7	19:15:15	7
367885	99	18	0,17	cyclictest	0	-21	swapper/7	22:40:14	7
367872	99	18	16,2	cyclictest	964768	-21	kworker/u16:1+flush-8:0	23:44:33	5
367872	99	18	16,2	cyclictest	534549	-21	kworker/u16:0+flush-8:0	20:40:15	5
367872	99	18	14,3	cyclictest	1025360	-21	kworker/u16:2+flush-8:0	00:10:33	5
367885	99	17	16,0	cyclictest	0	-21	swapper/7	23:20:02	7
367885	99	17	16,0	cyclictest	0	-21	swapper/7	21:11:51	7
367885	99	17	15,1	cyclictest	0	-21	swapper/7	19:25:14	7
367885	99	17	0,15	cyclictest	697382	-21	sh	21:47:52	7
367872	99	17	15,2	cyclictest	696423	-21	kworker/u16:3+flush-8:0	21:48:30	5
367872	99	17	15,2	cyclictest	3844725	-21	kworker/u16:1+events_unbound	19:29:29	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17