OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue May 26, 2026 12:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
248258	2	100	0,1	sleep7	248256	-21	ssh	11:29:19	7
162281	2	99	0,0	sleep6	0	-21	swapper/6	10:53:04	6
237289	2	97	0,1	sleep6	0	-21	swapper/6	11:26:30	6
3855699	99	58	56,2	cyclictest	298727	-21	kworker/u16:2+events_unbound	11:57:45	5
3855120	2	52	42,7	sleep5	0	-21	swapper/5	07:03:22	5
3855333	2	49	39,7	sleep4	0	-21	swapper/4	07:06:26	4
3855701	99	46	44,2	cyclictest	4173334	-21	kworker/u16:3+events_unbound	10:18:56	6
3855699	99	45	43,2	cyclictest	3870171	-21	kworker/u16:1+flush-8:0	08:37:37	5
3855699	99	41	39,2	cyclictest	4125761	-21	kworker/u16:0+events_unbound	09:43:40	5
3855699	99	41	39,2	cyclictest	4125761	-21	kworker/u16:0+events_unbound	09:43:40	5
3855699	99	41	39,2	cyclictest	3870171	-21	kworker/u16:1+events_unbound	07:22:36	5
3855699	99	40	38,2	cyclictest	3801129	-21	kworker/u16:0+flush-8:0	07:07:39	5
3855699	99	40	38,2	cyclictest	323947	-21	kworker/u16:0+events_unbound	12:06:33	5
3855699	99	40	37,2	cyclictest	323947	-21	kworker/u16:0+events_unbound	12:31:32	5
3854760	2	40	3,31	sleep7	0	-21	swapper/7	07:02:52	7
3855699	99	39	37,2	cyclictest	4022981	-21	kworker/u16:2+events_unbound	09:49:12	5
3855699	99	39	37,2	cyclictest	18792	-21	kworker/u16:1+events_unbound	10:11:56	5
3855699	99	38	36,2	cyclictest	4125761	-21	kworker/u16:0+flush-8:0	09:27:04	5
3855699	99	38	36,2	cyclictest	223004	-21	kworker/u16:0+events_unbound	11:20:49	5
3855699	99	37	35,2	cyclictest	4173334	-21	kworker/u16:3+events_unbound	10:39:00	5
3855699	99	37	35,2	cyclictest	4173334	-21	kworker/u16:3+events_unbound	09:39:16	5
3855699	99	37	35,2	cyclictest	4022981	-21	kworker/u16:2+events_unbound	09:35:21	5
3855699	99	37	35,2	cyclictest	236575	-21	kworker/u16:1+events_unbound	12:10:00	5
3855699	99	37	34,2	cyclictest	323947	-21	kworker/u16:0+events_unbound	12:19:46	5
3855699	99	37	34,2	cyclictest	262477	-21	kworker/u16:0+flush-8:0	11:42:24	5
3855696	99	37	35,2	cyclictest	55006	-21	kworker/u16:2+events_unbound	10:42:29	4
3855699	99	36	34,2	cyclictest	91683	-21	kworker/u16:0+events_unbound	10:31:21	5
3855699	99	36	34,2	cyclictest	4051962	-21	kworker/u16:3+events_unbound	09:27:17	5
3855699	99	36	34,2	cyclictest	4051962	-21	kworker/u16:3+events_unbound	09:18:29	5
3855699	99	36	34,2	cyclictest	392779	-21	kworker/u16:3+events_unbound	12:33:29	5
3855699	99	36	34,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	09:08:21	5
3855699	99	36	34,2	cyclictest	201213	-21	kworker/u16:4+events_unbound	11:24:53	5
3855699	99	36	34,2	cyclictest	186609	-21	kworker/u16:0+events_unbound	11:03:28	5
3855699	99	36	34,2	cyclictest	186609	-21	kworker/u16:0+events_unbound	11:03:28	5
3855699	99	36	34,2	cyclictest	137786	-21	kworker/u16:4+events_unbound	10:54:52	5
3855699	99	36	33,2	cyclictest	55006	-21	kworker/u16:2+flush-8:0	10:42:24	5
3855705	99	35	0,33	cyclictest	635	-21	systemd-journal	10:17:09	7
3855699	99	35	33,2	cyclictest	64709	-21	kworker/u16:4+events_unbound	10:14:44	5
3855699	99	35	33,2	cyclictest	55006	-21	kworker/u16:2+events_unbound	10:24:48	5
3855699	99	35	33,2	cyclictest	55006	-21	kworker/u16:2+events_unbound	10:24:48	5
3855699	99	35	33,2	cyclictest	55006	-21	kworker/u16:2+events_unbound	10:20:44	5
3855699	99	35	33,2	cyclictest	4173334	-21	kworker/u16:3+events_unbound	10:04:05	5
3855699	99	35	33,2	cyclictest	4125761	-21	kworker/u16:0+events_unbound	09:52:49	5
3855699	99	35	33,2	cyclictest	4051962	-21	kworker/u16:3+events_unbound	09:15:37	5
3855699	99	35	33,2	cyclictest	236575	-21	kworker/u16:1+events_unbound	11:54:16	5
3855699	99	35	33,2	cyclictest	201213	-21	kworker/u16:4+events_unbound	11:34:40	5
3855699	99	35	33,2	cyclictest	166350	-21	kworker/u16:2+events_unbound	11:28:40	5
3855699	99	35	33,2	cyclictest	137786	-21	kworker/u16:4+events_unbound	10:48:17	5
3855701	99	34	33,1	cyclictest	347700	-21	kworker/u16:2+events_unbound	12:24:02	6
3855699	99	34	32,2	cyclictest	262477	-21	kworker/u16:0+events_unbound	11:39:33	5
3855699	99	34	32,2	cyclictest	166350	-21	kworker/u16:2+events_unbound	10:58:48	5
3855699	99	34	31,2	cyclictest	4173334	-21	kworker/u16:3+flush-8:0	10:32:16	5
3855699	99	33	31,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	07:32:14	5
3855699	99	33	31,2	cyclictest	236575	-21	kworker/u16:1+events_unbound	12:15:17	5
3855699	99	33	31,2	cyclictest	201213	-21	kworker/u16:4+events_unbound	11:15:33	5
3855696	99	33	31,2	cyclictest	4125761	-21	kworker/u16:0+flush-8:0	09:57:29	4
3855705	99	32	26,5	cyclictest	4138793	-21	latency_hist	09:27:14	7
3855701	99	32	30,2	cyclictest	4173334	-21	kworker/u16:3+flush-8:0	10:32:36	6
3855699	99	32	30,2	cyclictest	4125761	-21	kworker/u16:0+events_unbound	09:58:54	5
3855699	99	32	30,2	cyclictest	4022981	-21	kworker/u16:2+events_unbound	08:48:25	5
3855699	99	32	30,2	cyclictest	236575	-21	kworker/u16:1+events_unbound	11:50:08	5
3855699	99	32	29,2	cyclictest	3955865	-21	kworker/u16:4+flush-8:0	08:07:15	5
3855696	99	32	30,2	cyclictest	201213	-21	kworker/u16:4+flush-8:0	11:52:37	4
3855696	99	32	30,2	cyclictest	166350	-21	kworker/u16:2+flush-8:0	11:32:43	4
3855699	99	31	29,2	cyclictest	347700	-21	kworker/u16:2+events_unbound	12:23:25	5
3855699	99	31	29,2	cyclictest	186609	-21	kworker/u16:0+flush-8:0	11:07:56	5
3855696	99	31	29,2	cyclictest	4125761	-21	kworker/u16:0+flush-8:0	09:22:28	4
3855696	99	31	29,2	cyclictest	4051962	-21	kworker/u16:3+flush-8:0	09:17:41	4
3855705	99	30	25,4	cyclictest	635	-21	systemd-journal	07:22:14	7
3855699	99	30	28,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	07:45:24	5
3855699	99	30	27,2	cyclictest	3926276	-21	kworker/u16:3+events_unbound	08:02:24	5
3855696	99	30	28,2	cyclictest	18792	-21	kworker/u16:1+flush-8:0	10:12:24	4
3855701	99	29	26,2	cyclictest	201213	-21	kworker/u16:4+events_unbound	11:17:14	6
3855696	99	29	27,2	cyclictest	236575	-21	kworker/u16:1+flush-8:0	11:57:29	4
3846264	2	29	19,7	sleep6	0	-21	swapper/6	07:02:27	6
4025987	2	28	0,1	chrt	4025988	-21	kthreadcore	08:32:33	7
3855701	99	28	26,2	cyclictest	55006	-21	kworker/u16:2+events_unbound	10:27:44	6
3855696	99	28	26,2	cyclictest	64709	-21	kworker/u16:4+flush-8:0	10:22:29	4
3855696	99	28	26,2	cyclictest	64709	-21	kworker/u16:4+flush-8:0	10:22:29	4
3855696	99	28	26,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	07:32:28	4
3855705	99	27	26,0	cyclictest	0	-21	swapper/7	10:07:34	7
3855705	99	27	26,0	cyclictest	0	-21	swapper/7	09:27:33	7
3855701	99	27	25,2	cyclictest	262477	-21	kworker/u16:0+events_unbound	11:37:36	6
3855696	99	27	25,2	cyclictest	4022981	-21	kworker/u16:2+events_unbound	09:12:29	4
3855696	99	27	25,2	cyclictest	3870171	-21	kworker/u16:1+flush-8:0	08:17:25	4
3855696	99	27	25,2	cyclictest	128093	-21	kworker/u16:1+flush-8:0	11:12:29	4
3855701	99	26	24,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	09:02:27	6
3855701	99	26	24,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	09:02:27	6
3855701	99	26	24,2	cyclictest	236575	-21	kworker/u16:1+events_unbound	12:22:13	6
3855699	99	26	24,2	cyclictest	3801129	-21	kworker/u16:0+events_unbound	07:19:14	5
4022744	2	25	0,0	sleep4	0	-21	swapper/4	08:32:14	4
3855705	99	25	20,4	cyclictest	347270	-21	latency_hist	12:12:14	7
3855701	99	25	23,2	cyclictest	392779	-21	kworker/u16:3+events_unbound	12:27:45	6
3855701	99	25	22,2	cyclictest	4125761	-21	kworker/u16:0+flush-8:0	09:27:35	6
3855696	99	25	23,2	cyclictest	55006	-21	kworker/u16:2+events_unbound	10:37:28	4
3855696	99	25	23,2	cyclictest	4022981	-21	kworker/u16:2+flush-8:0	09:32:32	4
3855696	99	25	23,2	cyclictest	4022981	-21	kworker/u16:2+flush-8:0	08:32:28	4
3855696	99	25	23,2	cyclictest	236575	-21	kworker/u16:1+events_unbound	11:27:28	4
3855705	99	24	23,0	cyclictest	0	-21	swapper/7	07:52:28	7
3855705	99	24	19,4	cyclictest	274225	-21	latency_hist	11:42:13	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17