OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Mon Mar 09, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3103238	2	169	0,1	sleep5	0	-21	swapper/5	08:44:33	5
3330741	2	98	0,1	sleep7	3330742	-21	grep	10:24:28	7
3495418	2	95	0,1	sleep7	3495417	-21	rm	11:30:57	7
3365277	2	94	0,0	sleep5	0	-21	swapper/5	10:39:25	5
2900905	99	67	64,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	09:59:25	6
2900411	2	57	47,7	sleep4	0	-21	swapper/4	07:01:12	4
2900411	2	57	47,7	sleep4	0	-21	swapper/4	07:01:12	4
2900486	2	55	44,8	sleep5	0	-21	swapper/5	07:02:18	5
2900486	2	55	44,8	sleep5	0	-21	swapper/5	07:02:18	5
2900905	99	52	50,2	cyclictest	3328241	-21	kworker/u16:2+flush-8:0	10:24:40	6
2900905	99	51	49,2	cyclictest	3124478	-21	kworker/u16:1+flush-8:0	10:04:26	6
2900478	2	51	41,7	sleep6	0	-21	swapper/6	07:02:11	6
2900478	2	51	41,7	sleep6	0	-21	swapper/6	07:02:11	6
2900905	99	50	48,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	09:34:45	6
2900900	99	50	48,2	cyclictest	3379763	-21	kworker/u16:3+events_unbound	11:24:41	5
2900905	99	49	47,2	cyclictest	3146253	-21	kworker/u16:0+flush-8:0	09:14:40	6
2900900	99	48	46,2	cyclictest	2977843	-21	kworker/u16:2+flush-8:0	08:54:25	5
2900905	99	47	45,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	09:09:35	6
2900905	99	46	44,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	11:54:27	6
2900905	99	46	44,2	cyclictest	3424614	-21	kworker/u16:2+flush-8:0	11:09:25	6
2900905	99	46	44,2	cyclictest	3397953	-21	kworker/u16:1+events_unbound	10:59:40	6
2900905	99	46	44,2	cyclictest	1465851	-21	kworker/u16:4+flush-8:0	07:49:36	6
2900905	99	45	43,2	cyclictest	3618334	-21	kworker/u16:3+events_unbound	12:27:39	6
2900905	99	45	43,2	cyclictest	3424614	-21	kworker/u16:2+flush-8:0	11:19:47	6
2900905	99	45	43,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	10:11:32	6
2900905	99	45	43,2	cyclictest	2873312	-21	kworker/u16:1+events_unbound	07:44:40	6
2900910	99	44	43,0	cyclictest	0	-21	swapper/7	09:34:36	7
2900905	99	44	41,2	cyclictest	3328241	-21	kworker/u16:2+events_unbound	10:29:11	6
2900900	99	44	42,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	12:11:04	5
2900900	99	44	42,2	cyclictest	3154899	-21	kworker/u16:3+flush-8:0	09:09:36	5
2900900	99	43	41,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	09:29:42	5
2900905	99	42	40,2	cyclictest	3497259	-21	kworker/u16:1+events_unbound	12:02:24	6
2900905	99	42	40,2	cyclictest	2977843	-21	kworker/u16:2+flush-8:0	09:06:32	6
2900900	99	42	40,2	cyclictest	3379763	-21	kworker/u16:3+flush-8:0	11:23:04	5
2900905	99	41	39,2	cyclictest	3473241	-21	kworker/u16:0+events_unbound	11:28:51	6
2900905	99	41	39,2	cyclictest	3424614	-21	kworker/u16:2+events_unbound	11:45:49	6
2900905	99	41	39,2	cyclictest	3362212	-21	kworker/u16:2+events_unbound	10:46:05	6
2900905	99	41	39,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	09:56:17	6
2900905	99	41	39,2	cyclictest	2873312	-21	kworker/u16:1+flush-8:0	07:59:33	6
2900900	99	41	39,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	10:02:56	5
2900535	2	41	5,32	sleep7	0	-21	swapper/7	07:03:00	7
2900535	2	41	5,32	sleep7	0	-21	swapper/7	07:03:00	7
2900910	99	40	37,2	cyclictest	3457655	-21	ntpq	11:14:37	7
2900905	99	40	38,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	11:41:47	6
2900900	99	40	38,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	10:09:38	5
2900900	99	40	38,2	cyclictest	2977843	-21	kworker/u16:2+events_unbound	07:59:41	5
2900905	99	39	37,2	cyclictest	3567512	-21	kworker/u16:0+events_unbound	12:33:32	6
2900905	99	39	37,2	cyclictest	3497259	-21	kworker/u16:1+events_unbound	12:09:03	6
2900905	99	39	37,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	11:32:03	6
2900905	99	39	37,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	09:31:40	6
2900900	99	39	37,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	12:16:36	5
2900900	99	39	37,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	12:16:36	5
2900900	99	39	37,2	cyclictest	3349561	-21	kworker/u16:0+flush-8:0	10:37:20	5
2900900	99	39	37,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	09:52:23	5
2900910	99	38	37,0	cyclictest	0	-21	swapper/7	10:37:29	7
2900910	99	38	37,0	cyclictest	0	-21	swapper/7	09:44:36	7
2900910	99	38	33,4	cyclictest	3521362	-21	latency_hist	11:44:10	7
2900905	99	38	35,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	10:39:38	6
2900900	99	38	36,2	cyclictest	3618334	-21	kworker/u16:3+flush-8:0	12:26:43	5
2900900	99	38	36,2	cyclictest	3483502	-21	kworker/u16:4+flush-8:0	11:49:41	5
2900900	99	38	36,2	cyclictest	3057610	-21	kworker/u16:1+events_unbound	08:24:40	5
2900900	99	38	36,2	cyclictest	3047082	-21	kworker/u16:4+events_unbound	08:19:41	5
2900910	99	37	35,1	cyclictest	0	-21	swapper/7	11:26:15	7
2900905	99	37	35,2	cyclictest	3591651	-21	kworker/u16:2+events_unbound	12:13:11	6
2900905	99	37	35,2	cyclictest	3154899	-21	kworker/u16:3+flush-8:0	09:29:04	6
2900905	99	37	35,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	10:20:35	6
2900900	99	37	35,2	cyclictest	3362212	-21	kworker/u16:2+events_unbound	10:45:30	5
2900910	99	36	35,1	cyclictest	3134026	-21	diskmemload	10:18:03	7
2900910	99	36	35,1	cyclictest	3134026	-21	diskmemload	10:18:03	7
2900910	99	36	34,1	cyclictest	3386535	-21	ssh	10:46:09	7
2900910	99	36	33,2	cyclictest	2934432	-21	iostat	07:19:26	7
2900905	99	36	34,2	cyclictest	3533519	-21	kworker/u16:3+flush-8:0	11:52:35	6
2900905	99	36	34,2	cyclictest	3379763	-21	kworker/u16:3+events_unbound	11:14:37	6
2900905	99	36	34,2	cyclictest	3349561	-21	kworker/u16:0+events_unbound	10:51:45	6
2900905	99	36	34,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	10:15:44	6
2900905	99	36	34,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	10:15:44	6
2900900	99	36	34,2	cyclictest	3424614	-21	kworker/u16:2+flush-8:0	11:14:25	5
2900900	99	36	34,2	cyclictest	3424614	-21	kworker/u16:2+events_unbound	11:39:01	5
2900900	99	36	34,2	cyclictest	3379763	-21	kworker/u16:3+events_unbound	11:14:00	5
2900900	99	36	33,2	cyclictest	3497259	-21	kworker/u16:1+flush-8:0	11:44:20	5
2900905	99	35	33,2	cyclictest	3379763	-21	kworker/u16:3+flush-8:0	11:07:12	6
2900905	99	35	33,2	cyclictest	3154899	-21	kworker/u16:3+events_unbound	09:46:13	6
2900905	99	35	33,2	cyclictest	3096167	-21	kworker/u16:0+flush-8:0	08:44:38	6
2900900	99	35	33,2	cyclictest	3497259	-21	kworker/u16:1+events_unbound	12:08:41	5
2900900	99	35	33,2	cyclictest	3349561	-21	kworker/u16:0+events_unbound	10:32:17	5
2900900	99	35	33,2	cyclictest	1465851	-21	kworker/u16:4+flush-8:0	07:24:39	5
2900900	99	35	33,2	cyclictest	1465851	-21	kworker/u16:4+flush-8:0	07:24:39	5
2900910	99	34	33,0	cyclictest	0	-21	swapper/7	11:51:41	7
2900910	99	34	33,0	cyclictest	0	-21	swapper/7	10:23:41	7
2900905	99	34	32,2	cyclictest	3591651	-21	kworker/u16:2+events_unbound	12:15:02	6
2900905	99	34	32,2	cyclictest	3591651	-21	kworker/u16:2+events_unbound	12:15:02	6
2900905	99	34	32,2	cyclictest	3124478	-21	kworker/u16:1+events_unbound	09:41:26	6
2900905	99	34	32,2	cyclictest	3057610	-21	kworker/u16:1+flush-8:0	08:39:26	6
2900905	99	34	32,2	cyclictest	2977843	-21	kworker/u16:2+flush-8:0	08:29:41	6
2900905	99	34	32,2	cyclictest	2873312	-21	kworker/u16:1+events_unbound	08:04:42	6
2900900	99	34	32,2	cyclictest	3483502	-21	kworker/u16:4+events_unbound	12:24:02	5
2900900	99	34	32,2	cyclictest	3379763	-21	kworker/u16:3+flush-8:0	10:54:40	5
2900900	99	34	32,2	cyclictest	3047082	-21	kworker/u16:4+events_unbound	09:14:45	5
2900900	99	34	32,2	cyclictest	2977843	-21	kworker/u16:2+flush-8:0	07:39:38	5
2900900	99	34	31,2	cyclictest	1465851	-21	kworker/u16:4+flush-8:0	07:44:11	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17