OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Mar 25, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2939326	2	61	50,7	sleep6	0	-21	swapper/6	07:00:42	6
2939325	2	54	44,7	sleep5	0	-21	swapper/5	07:00:41	5
2939794	99	43	41,2	cyclictest	2901817	-21	kworker/u16:2+events_unbound	07:23:25	5
2939430	2	42	2,36	sleep7	0	-21	swapper/7	07:02:11	7
6265	89	41	0,34	rtkit-daemon	0	-21	swapper/4	07:01:52	4
2939795	99	40	38,2	cyclictest	3485590	-21	kworker/u16:1+flush-8:0	11:28:39	6
2939794	99	38	36,2	cyclictest	3558320	-21	kworker/u16:3+events_unbound	12:18:31	5
2939795	99	37	35,2	cyclictest	3329606	-21	kworker/u16:3+events_unbound	10:13:24	6
2939794	99	36	34,2	cyclictest	3435267	-21	kworker/u16:0+events_unbound	10:53:20	5
2939799	99	35	34,0	cyclictest	0	-21	swapper/7	07:48:43	7
2939794	99	35	33,2	cyclictest	3435267	-21	kworker/u16:0+events_unbound	12:08:31	5
3523303	2	34	0,1	sleep5	0	-21	swapper/5	11:28:21	5
2939794	99	34	31,3	cyclictest	3485590	-21	kworker/u16:1+events_unbound	11:23:20	5
2939792	99	34	32,2	cyclictest	3521826	-21	kworker/u16:2+events_unbound	11:43:20	4
2939792	99	34	32,2	cyclictest	2991043	-21	kworker/u16:3+events_unbound	07:33:29	4
2939792	99	34	32,2	cyclictest	2920528	-21	kworker/u16:1+events_unbound	07:08:10	4
2939795	99	33	31,2	cyclictest	3438024	-21	kworker/u16:4+events_unbound	11:08:19	6
2939794	99	33	31,2	cyclictest	3020106	-21	kworker/u16:1+events_unbound	07:58:37	5
2939794	99	33	31,2	cyclictest	2920528	-21	kworker/u16:1+events_unbound	07:08:34	5
2939795	99	32	30,2	cyclictest	3435267	-21	kworker/u16:0+events_unbound	12:13:35	6
2939794	99	32	30,2	cyclictest	3020106	-21	kworker/u16:1+events_unbound	09:13:22	5
2939799	99	31	30,0	cyclictest	0	-21	swapper/7	07:08:22	7
2939792	99	31	29,2	cyclictest	3435267	-21	kworker/u16:0+events_unbound	11:28:23	4
2939795	99	30	28,2	cyclictest	3197295	-21	kworker/u16:4+flush-8:0	10:08:38	6
2939795	99	30	27,2	cyclictest	3154031	-21	kworker/u16:2+flush-8:0	09:28:22	6
2939795	99	30	27,2	cyclictest	3154031	-21	kworker/u16:2+flush-8:0	09:28:22	6
2939794	99	30	28,2	cyclictest	3580968	-21	kworker/u16:1+events_unbound	12:08:01	5
2939799	99	29	29,0	cyclictest	0	-21	swapper/7	09:58:41	7
2939795	99	29	27,2	cyclictest	3413993	-21	kworker/u16:3+flush-8:0	10:43:35	6
2939795	99	29	26,2	cyclictest	3435267	-21	kworker/u16:0+flush-8:0	12:08:38	6
2939794	99	29	27,2	cyclictest	3048535	-21	kworker/u16:2+events_unbound	08:08:24	5
2939792	99	29	27,2	cyclictest	3048535	-21	kworker/u16:2+events_unbound	08:28:18	4
2939794	99	28	26,2	cyclictest	3230294	-21	kworker/u16:0+events_unbound	09:48:19	5
2939792	99	28	26,2	cyclictest	3230294	-21	kworker/u16:0+events_unbound	09:43:36	4
2939799	99	27	27,0	cyclictest	0	-21	swapper/7	11:08:41	7
2939799	99	27	26,0	cyclictest	0	-21	swapper/7	12:13:45	7
2939799	99	27	21,5	cyclictest	3173096	-21	latency_hist	09:03:07	7
2939795	99	27	25,2	cyclictest	3001088	-21	kworker/u16:0+events_unbound	08:13:21	6
2939795	99	27	24,3	cyclictest	3558320	-21	kworker/u16:3+flush-8:0	12:33:01	6
2939794	99	27	25,2	cyclictest	3485590	-21	kworker/u16:1+events_unbound	11:34:19	5
2939794	99	27	25,2	cyclictest	3329606	-21	kworker/u16:3+events_unbound	10:23:20	5
2939794	99	27	25,2	cyclictest	3230294	-21	kworker/u16:0+events_unbound	09:43:20	5
2939794	99	27	25,2	cyclictest	3174390	-21	kworker/u16:3+events_unbound	09:13:05	5
2939794	99	27	25,2	cyclictest	3154031	-21	kworker/u16:2+events_unbound	10:42:33	5
2939792	99	27	25,2	cyclictest	3558320	-21	kworker/u16:3+events_unbound	12:18:25	4
2939799	99	26	21,4	cyclictest	2949195	-21	latency_hist	07:08:07	7
2939799	99	26	20,5	cyclictest	3000899	-21	latency_hist	07:33:07	7
2939795	99	26	24,2	cyclictest	2920528	-21	kworker/u16:1+events_unbound	07:08:12	6
2939794	99	26	24,2	cyclictest	3154031	-21	kworker/u16:2+events_unbound	10:23:01	5
2939794	99	26	24,2	cyclictest	3001088	-21	kworker/u16:0+events_unbound	08:38:20	5
2939794	99	26	24,2	cyclictest	2920528	-21	kworker/u16:1+flush-8:0	07:08:02	5
2939799	99	25	22,2	cyclictest	124	-21	kswapd0	12:28:18	7
2939795	99	25	23,2	cyclictest	3230294	-21	kworker/u16:0+flush-8:0	10:33:30	6
2939794	99	25	23,2	cyclictest	3197295	-21	kworker/u16:4+events_unbound	09:29:57	5
2939794	99	25	23,2	cyclictest	3197295	-21	kworker/u16:4+events_unbound	09:29:57	5
2939794	99	25	23,2	cyclictest	3001088	-21	kworker/u16:0+events_unbound	08:33:19	5
2939792	99	25	23,2	cyclictest	3558320	-21	kworker/u16:3+events_unbound	11:54:22	4
2939792	99	25	23,2	cyclictest	3197295	-21	kworker/u16:4+flush-8:0	09:23:27	4
2939792	99	25	23,2	cyclictest	3154031	-21	kworker/u16:2+events_unbound	10:23:38	4
2939792	99	25	23,2	cyclictest	3154031	-21	kworker/u16:2+events_unbound	09:33:18	4
2939792	99	25	23,2	cyclictest	3154031	-21	kworker/u16:2+events_unbound	09:13:26	4
2939792	99	25	23,2	cyclictest	3001088	-21	kworker/u16:0+events_unbound	08:11:43	4
2939799	99	24	24,0	cyclictest	0	-21	swapper/7	11:33:24	7
2939799	99	24	23,0	cyclictest	0	-21	swapper/7	12:03:49	7
2939799	99	24	23,0	cyclictest	0	-21	swapper/7	10:03:20	7
2939799	99	24	22,1	cyclictest	3173097	-21	diskmemload	09:28:12	7
2939799	99	24	22,1	cyclictest	3173097	-21	diskmemload	09:28:12	7
2939795	99	24	22,2	cyclictest	3617905	-21	kworker/u16:2+flush-8:0	12:18:31	6
2939795	99	24	22,2	cyclictest	3230294	-21	kworker/u16:0+flush-8:0	09:38:22	6
2939794	99	24	22,2	cyclictest	3230294	-21	kworker/u16:0+events_unbound	09:23:38	5
2939794	99	24	22,2	cyclictest	3154031	-21	kworker/u16:2+flush-8:0	09:03:02	5
2939794	99	24	22,2	cyclictest	3048535	-21	kworker/u16:2+events_unbound	08:28:22	5
2939794	99	24	22,2	cyclictest	3020106	-21	kworker/u16:1+events_unbound	08:43:21	5
2939799	99	23	22,0	cyclictest	0	-21	swapper/7	07:26:40	7
2939795	99	23	21,2	cyclictest	3048535	-21	kworker/u16:2+flush-8:0	08:28:30	6
2939795	99	23	21,2	cyclictest	2920528	-21	kworker/u16:1+events_unbound	07:03:25	6
2939795	99	23	20,2	cyclictest	3020106	-21	kworker/u16:1+flush-8:0	09:18:07	6
2939794	99	23	21,2	cyclictest	3580968	-21	kworker/u16:1+events_unbound	11:54:48	5
2939794	99	23	21,2	cyclictest	3558320	-21	kworker/u16:3+events_unbound	12:23:35	5
2939794	99	23	21,2	cyclictest	3485590	-21	kworker/u16:1+events_unbound	11:17:03	5
2939794	99	23	21,2	cyclictest	3413993	-21	kworker/u16:3+events_unbound	11:12:04	5
2939794	99	23	21,2	cyclictest	3154031	-21	kworker/u16:2+events_unbound	09:33:56	5
2939794	99	23	21,2	cyclictest	3020106	-21	kworker/u16:1+events_unbound	09:05:12	5
2939792	99	23	21,2	cyclictest	3435267	-21	kworker/u16:0+events_unbound	10:49:04	4
2939792	99	23	21,2	cyclictest	3413993	-21	kworker/u16:3+events_unbound	11:08:29	4
2939792	99	23	21,2	cyclictest	3230294	-21	kworker/u16:0+events_unbound	09:53:30	4
2939792	99	23	21,2	cyclictest	3197295	-21	kworker/u16:4+events_unbound	10:08:25	4
2939792	99	23	21,2	cyclictest	3048535	-21	kworker/u16:2+flush-8:0	08:18:22	4
2939799	99	22	19,2	cyclictest	3206198	-21	ssh	09:13:31	7
2939795	99	22	20,2	cyclictest	3435267	-21	kworker/u16:0+events_unbound	12:03:30	6
2939795	99	22	19,2	cyclictest	3001088	-21	kworker/u16:0+flush-8:0	08:43:06	6
2939794	99	22	20,2	cyclictest	3617905	-21	kworker/u16:2+events_unbound	12:31:48	5
2939794	99	22	20,2	cyclictest	3580968	-21	kworker/u16:1+events_unbound	11:58:35	5
2939794	99	22	20,2	cyclictest	3438024	-21	kworker/u16:4+events_unbound	11:07:32	5
2939794	99	22	20,2	cyclictest	3197295	-21	kworker/u16:4+events_unbound	09:54:56	5
2939794	99	22	20,2	cyclictest	3197295	-21	kworker/u16:4+events_unbound	09:20:17	5
2939794	99	22	20,2	cyclictest	2991044	-21	kworker/u16:4+events_unbound	07:38:35	5
2939792	99	22	20,2	cyclictest	3558320	-21	kworker/u16:3+events_unbound	12:13:20	4
2939792	99	22	20,2	cyclictest	3197295	-21	kworker/u16:4+flush-8:0	09:58:30	4
2939799	99	21	20,0	cyclictest	0	-21	swapper/7	12:18:23	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17