OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri Mar 06, 2026 12:43:35)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3206637	2	105	0,1	sleep7	0	-21	swapper/7	07:29:42	7
3792865	2	71	0,1	sleep7	3792863	-21	ssh	11:52:59	7
3148356	2	64	53,7	sleep6	0	-21	swapper/6	07:02:26	6
3148356	2	64	53,7	sleep6	0	-21	swapper/6	07:02:26	6
3148781	99	62	56,5	cyclictest	3467785	-21	latency_hist	09:39:22	7
3148781	99	48	43,4	cyclictest	3733312	-21	latency_hist	11:29:22	7
3148772	99	47	45,2	cyclictest	3784064	-21	kworker/u16:1+events_unbound	12:31:16	4
3148778	99	46	43,2	cyclictest	3540197	-21	kworker/u16:4+flush-8:0	10:19:22	6
3148778	99	46	43,2	cyclictest	3540197	-21	kworker/u16:4+flush-8:0	10:19:22	6
6265	89	45	0,38	rtkit-daemon	0	-21	swapper/4	07:01:05	4
6265	89	45	0,38	rtkit-daemon	0	-21	swapper/4	07:01:05	4
3148778	99	45	43,2	cyclictest	3540197	-21	kworker/u16:4+flush-8:0	11:09:22	6
3148772	99	45	43,2	cyclictest	3722951	-21	kworker/u16:1+events_unbound	11:32:27	4
3148772	99	45	43,2	cyclictest	3722951	-21	kworker/u16:1+events_unbound	11:32:27	4
3148772	99	44	42,2	cyclictest	3443742	-21	kworker/u16:3+events_unbound	09:37:32	4
3148772	99	44	42,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	09:45:56	4
3148781	99	43	0,41	cyclictest	3672888	-21	latency_hist	11:04:22	7
3148772	99	43	41,2	cyclictest	3767899	-21	kworker/u16:0+events_unbound	12:01:28	4
3148772	99	43	41,2	cyclictest	3354879	-21	kworker/u16:1+flush-8:0	09:59:22	4
3148772	99	43	40,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	10:29:31	4
3148261	2	43	31,8	sleep5	0	-21	swapper/5	07:01:05	5
3148261	2	43	31,8	sleep5	0	-21	swapper/5	07:01:05	5
3148772	99	42	40,2	cyclictest	3369413	-21	kworker/u16:0+events_unbound	09:27:32	4
3148772	99	42	40,2	cyclictest	3369413	-21	kworker/u16:0+events_unbound	09:20:40	4
3148772	99	42	40,2	cyclictest	3354879	-21	kworker/u16:1+events_unbound	10:28:28	4
3148776	99	41	38,2	cyclictest	3738567	-21	kworker/u16:3+flush-8:0	11:49:39	5
3148772	99	41	39,2	cyclictest	3443742	-21	kworker/u16:3+events_unbound	10:07:36	4
3148772	99	41	39,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	10:10:28	4
3148772	99	41	38,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	11:25:15	4
3139357	2	41	6,29	sleep7	0	-21	swapper/7	06:59:38	7
3139357	2	41	6,29	sleep7	0	-21	swapper/7	06:59:38	7
3148772	99	40	38,2	cyclictest	3784064	-21	kworker/u16:1+events_unbound	12:20:23	4
3148772	99	40	38,2	cyclictest	3784064	-21	kworker/u16:1+events_unbound	11:51:32	4
3148772	99	40	38,2	cyclictest	3648827	-21	kworker/u16:0+events_unbound	11:03:04	4
3148772	99	39	37,2	cyclictest	3784064	-21	kworker/u16:1+events_unbound	11:58:12	4
3148772	99	39	37,2	cyclictest	3648827	-21	kworker/u16:0+events_unbound	10:55:56	4
3148772	99	39	37,2	cyclictest	3648827	-21	kworker/u16:0+events_unbound	10:55:56	4
3148772	99	39	37,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	11:12:36	4
3148772	99	39	37,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	11:08:28	4
3148772	99	39	37,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	09:52:35	4
3148772	99	39	37,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	09:19:17	4
3148772	99	38	36,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	10:41:16	4
3148772	99	38	36,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	10:38:11	4
3148772	99	38	36,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	09:32:25	4
3148772	99	38	36,2	cyclictest	3354879	-21	kworker/u16:1+events_unbound	10:50:08	4
3148772	99	38	35,2	cyclictest	3709512	-21	kworker/u16:2+events_unbound	12:05:48	4
3148776	99	37	35,2	cyclictest	3709512	-21	kworker/u16:2+flush-8:0	12:04:43	5
3148772	99	37	35,2	cyclictest	3722951	-21	kworker/u16:1+events_unbound	11:40:51	4
3148772	99	37	35,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	10:21:11	4
3148772	99	37	35,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	10:15:44	4
3148772	99	37	35,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	10:15:44	4
3148772	99	37	35,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	09:10:32	4
3148772	99	37	35,2	cyclictest	3354879	-21	kworker/u16:1+events_unbound	10:03:47	4
3148772	99	37	35,2	cyclictest	3354879	-21	kworker/u16:1+events_unbound	09:43:20	4
3148772	99	37	35,2	cyclictest	3287338	-21	kworker/u16:4+events_unbound	09:08:05	4
3148772	99	37	34,2	cyclictest	3540197	-21	kworker/u16:4+flush-8:0	11:14:32	4
3148776	99	36	34,2	cyclictest	3354879	-21	kworker/u16:1+flush-8:0	09:05:00	5
3148772	99	36	34,2	cyclictest	3784064	-21	kworker/u16:1+events_unbound	12:16:19	4
3148772	99	36	34,2	cyclictest	3784064	-21	kworker/u16:1+events_unbound	12:16:19	4
3148772	99	36	34,2	cyclictest	3722951	-21	kworker/u16:1+events_unbound	11:37:16	4
3148772	99	36	34,2	cyclictest	3709512	-21	kworker/u16:2+events_unbound	12:13:19	4
3148772	99	36	34,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	11:20:59	4
3148772	99	36	34,2	cyclictest	3540197	-21	kworker/u16:4+events_unbound	10:46:13	4
3148781	99	35	30,4	cyclictest	3407471	-21	latency_hist	09:14:23	7
3148778	99	34	31,3	cyclictest	3685414	-21	kworker/u16:3+events_unbound	11:14:41	6
3148772	99	34	32,2	cyclictest	3767899	-21	kworker/u16:0+events_unbound	12:26:40	4
3148772	99	34	32,2	cyclictest	3738567	-21	kworker/u16:3+events_unbound	11:49:15	4
3148781	99	33	28,4	cyclictest	3540154	-21	latency_hist	10:09:22	7
3148778	99	33	31,2	cyclictest	3784064	-21	kworker/u16:1+flush-8:0	12:29:37	6
3148778	99	33	31,2	cyclictest	3404717	-21	kworker/u16:2+events_unbound	10:09:39	6
3148778	99	33	30,2	cyclictest	3404717	-21	kworker/u16:2+flush-8:0	09:59:28	6
3148778	99	33	30,2	cyclictest	3404717	-21	kworker/u16:2+flush-8:0	09:34:28	6
3148781	99	31	29,1	cyclictest	3794049	-21	ssh	11:54:28	7
3148778	99	30	27,2	cyclictest	3709512	-21	kworker/u16:2+flush-8:0	12:29:22	6
3148778	99	30	27,2	cyclictest	3709512	-21	kworker/u16:2+flush-8:0	11:34:27	6
3148776	99	30	28,2	cyclictest	3354879	-21	kworker/u16:1+flush-8:0	09:49:45	5
3148781	99	29	28,0	cyclictest	0	-21	swapper/7	10:09:53	7
3148781	99	29	24,4	cyclictest	3745412	-21	latency_hist	11:34:22	7
3148781	99	29	24,4	cyclictest	3745412	-21	latency_hist	11:34:22	7
3148778	99	29	27,2	cyclictest	3354879	-21	kworker/u16:1+events_unbound	09:44:42	6
3148776	99	29	25,3	cyclictest	3221609	-21	grep	07:39:40	5
3148772	99	29	27,2	cyclictest	3200211	-21	kworker/u16:0+events_unbound	07:54:46	4
3148772	99	29	26,2	cyclictest	3248698	-21	kworker/u16:2+events_unbound	08:39:25	4
3148781	99	28	0,26	cyclictest	0	-21	swapper/7	12:19:45	7
3148778	99	28	26,2	cyclictest	3287338	-21	kworker/u16:4+flush-8:0	08:19:17	6
3148776	99	28	26,2	cyclictest	3443742	-21	kworker/u16:3+events_unbound	09:54:44	5
3148776	99	28	25,2	cyclictest	3540197	-21	kworker/u16:4+flush-8:0	10:30:03	5
3148772	99	28	26,2	cyclictest	3248698	-21	kworker/u16:2+events_unbound	08:06:04	4
3148772	99	27	25,2	cyclictest	3200211	-21	kworker/u16:0+events_unbound	07:49:20	4
3148772	99	27	24,2	cyclictest	3248698	-21	kworker/u16:2+events_unbound	08:44:24	4
3148781	99	26	25,1	cyclictest	3383350	-21	diskmemload	12:00:14	7
3148781	99	26	25,0	cyclictest	0	-21	swapper/7	07:09:28	7
3148778	99	26	24,2	cyclictest	3404717	-21	kworker/u16:2+flush-8:0	11:00:24	6
3148772	99	26	24,2	cyclictest	3287338	-21	kworker/u16:4+events_unbound	08:25:29	4
3148772	99	26	24,2	cyclictest	3248698	-21	kworker/u16:2+events_unbound	08:35:29	4
3148772	99	26	24,2	cyclictest	3101881	-21	kworker/u16:4+events_unbound	07:09:20	4
3148778	99	25	23,2	cyclictest	3722951	-21	kworker/u16:1+events_unbound	11:39:50	6
3148778	99	25	23,2	cyclictest	3404717	-21	kworker/u16:2+flush-8:0	09:19:48	6
3148778	99	25	22,2	cyclictest	2413445	-21	kworker/u16:1+flush-8:0	07:04:44	6
3148778	99	24	22,2	cyclictest	3767899	-21	kworker/u16:0+flush-8:0	12:24:17	6
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17