OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri Dec 12, 2025 00:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
83035	2	71	61,7	sleep5	0	-21	swapper/5	03:20:04	5
83501	99	66	63,2	cyclictest	581198	-21	kworker/u16:3+flush-8:0	07:42:13	6
83497	99	57	54,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	05:27:13	4
83500	99	50	48,2	cyclictest	369563	-21	kworker/u16:2+flush-8:0	06:47:42	5
83007	2	48	38,7	sleep4	0	-21	swapper/4	03:19:39	4
83506	99	47	42,4	cyclictest	399752	-21	latency_hist	05:57:13	7
83506	99	45	40,4	cyclictest	198183	-21	latency_hist	04:17:14	7
83497	99	45	43,2	cyclictest	470741	-21	kworker/u16:3+events_unbound	07:02:13	4
83506	99	44	39,4	cyclictest	490555	-21	latency_hist	06:42:13	7
83500	99	42	40,2	cyclictest	521055	-21	kworker/u16:4+events_unbound	07:20:29	5
83500	99	42	40,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	06:15:24	5
83497	99	42	39,2	cyclictest	26655	-21	kworker/u16:2+flush-8:0	03:32:14	4
82967	2	42	32,7	sleep6	0	-21	swapper/6	03:19:09	6
83506	99	41	36,4	cyclictest	581394	-21	latency_hist	07:27:14	7
83500	99	41	39,2	cyclictest	258395	-21	kworker/u16:2+events_unbound	05:02:41	5
83497	99	41	37,3	cyclictest	712878	-21	kworker/u16:2+flush-8:0	08:37:28	4
73576	2	40	3,31	sleep7	0	-21	swapper/7	03:17:17	7
83501	99	38	35,2	cyclictest	521055	-21	kworker/u16:4+flush-8:0	07:42:33	6
83501	99	38	35,2	cyclictest	521055	-21	kworker/u16:4+flush-8:0	07:42:33	6
83500	99	38	36,2	cyclictest	661141	-21	kworker/u16:5+events_unbound	08:27:43	5
83500	99	38	36,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	05:07:39	5
83501	99	37	34,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	06:47:32	6
83500	99	37	35,2	cyclictest	359372	-21	kworker/u16:1+flush-8:0	05:45:25	5
83506	99	36	31,4	cyclictest	188118	-21	latency_hist	04:12:14	7
83501	99	36	33,2	cyclictest	470741	-21	kworker/u16:3+flush-8:0	06:57:31	6
83500	99	36	33,2	cyclictest	651629	-21	kworker/u16:3+events_unbound	08:07:16	5
83497	99	36	33,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	05:52:33	4
83500	99	35	33,2	cyclictest	470741	-21	kworker/u16:3+events_unbound	07:09:31	5
83500	99	35	33,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	05:38:25	5
83500	99	35	33,2	cyclictest	278535	-21	kworker/u16:0+events_unbound	05:14:09	5
83500	99	35	33,2	cyclictest	104189	-21	kworker/u16:3+events_unbound	03:31:25	5
83500	99	35	32,2	cyclictest	460156	-21	kworker/u16:1+flush-8:0	06:32:14	5
83500	99	34	32,2	cyclictest	460156	-21	kworker/u16:1+events_unbound	06:32:13	5
83500	99	34	32,2	cyclictest	308936	-21	kworker/u16:3+flush-8:0	06:02:48	5
83500	99	34	32,2	cyclictest	278535	-21	kworker/u16:0+events_unbound	04:59:00	5
83500	99	34	32,2	cyclictest	167650	-21	kworker/u16:1+events_unbound	04:13:49	5
83500	99	34	32,2	cyclictest	148288	-21	kworker/u16:2+events_unbound	03:55:10	5
83500	99	34	31,2	cyclictest	560989	-21	kworker/u16:1+events_unbound	07:22:23	5
83497	99	34	32,2	cyclictest	521055	-21	kworker/u16:4+flush-8:0	07:07:13	4
83506	99	33	27,5	cyclictest	4184977	-17	auditd	05:50:04	7
83501	99	33	30,2	cyclictest	470741	-21	kworker/u16:3+flush-8:0	07:07:27	6
83500	99	33	31,2	cyclictest	721791	-21	kworker/u16:3+events_unbound	08:43:13	5
83500	99	33	31,2	cyclictest	601972	-21	kworker/u16:1+flush-8:0	07:37:57	5
83500	99	33	31,2	cyclictest	52105	-21	kworker/u16:4+events_unbound	03:48:29	5
83500	99	33	31,2	cyclictest	52105	-21	kworker/u16:4+events_unbound	03:38:29	5
83500	99	33	31,2	cyclictest	470741	-21	kworker/u16:3+events_unbound	06:53:49	5
83500	99	33	31,2	cyclictest	167650	-21	kworker/u16:1+events_unbound	04:40:26	5
83497	99	33	31,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	06:12:37	4
83497	99	33	31,2	cyclictest	128946	-21	kworker/u16:0+flush-8:0	04:32:37	4
83501	99	32	30,2	cyclictest	278535	-21	kworker/u16:0+events_unbound	05:27:32	6
83497	99	32	30,2	cyclictest	128946	-21	kworker/u16:0+flush-8:0	04:42:37	4
83497	99	32	29,2	cyclictest	4155589	-21	kworker/u16:0+events_unbound	03:22:31	4
83497	99	32	29,2	cyclictest	148288	-21	kworker/u16:2+events_unbound	03:52:33	4
83501	99	31	29,2	cyclictest	278535	-21	kworker/u16:0+events_unbound	05:02:31	6
83501	99	31	29,2	cyclictest	167650	-21	kworker/u16:1+events_unbound	04:22:27	6
83501	99	31	29,2	cyclictest	148288	-21	kworker/u16:2+flush-8:0	03:57:27	6
83500	99	31	29,2	cyclictest	653104	-21	kworker/u16:4+events_unbound	08:23:29	5
83500	99	31	29,2	cyclictest	653104	-21	kworker/u16:4+events_unbound	08:23:29	5
83500	99	31	29,2	cyclictest	167650	-21	kworker/u16:1+events_unbound	04:46:49	5
83500	99	31	28,2	cyclictest	653104	-21	kworker/u16:4+events_unbound	08:17:16	5
83497	99	31	28,2	cyclictest	52105	-21	kworker/u16:4+flush-8:0	03:42:26	4
83501	99	30	29,1	cyclictest	308936	-21	kworker/u16:3+events_unbound	05:52:37	6
83500	99	30	28,2	cyclictest	712878	-21	kworker/u16:2+events_unbound	08:34:24	5
83500	99	30	28,2	cyclictest	653104	-21	kworker/u16:4+events_unbound	08:17:08	5
83500	99	30	28,2	cyclictest	521054	-21	kworker/u16:0+events_unbound	07:49:31	5
83500	99	30	28,2	cyclictest	521054	-21	kworker/u16:0+events_unbound	07:43:17	5
83500	99	30	28,2	cyclictest	521054	-21	kworker/u16:0+events_unbound	07:43:17	5
83500	99	30	28,2	cyclictest	460156	-21	kworker/u16:1+events_unbound	06:41:40	5
83500	99	30	28,2	cyclictest	369563	-21	kworker/u16:2+flush-8:0	06:23:32	5
83500	99	30	28,2	cyclictest	369563	-21	kworker/u16:2+events_unbound	06:08:23	5
83500	99	30	28,2	cyclictest	308936	-21	kworker/u16:3+flush-8:0	05:59:28	5
83500	99	30	28,2	cyclictest	288735	-21	kworker/u16:1+events_unbound	05:20:32	5
83500	99	30	28,2	cyclictest	26655	-21	kworker/u16:2+events_unbound	03:23:22	5
83500	99	30	28,2	cyclictest	258395	-21	kworker/u16:2+events_unbound	04:50:20	5
83500	99	30	28,2	cyclictest	167650	-21	kworker/u16:1+flush-8:0	04:35:20	5
83500	99	30	28,2	cyclictest	167650	-21	kworker/u16:1+flush-8:0	04:22:53	5
83500	99	30	28,2	cyclictest	128946	-21	kworker/u16:0+events_unbound	04:06:16	5
83497	99	30	28,2	cyclictest	521055	-21	kworker/u16:4+events_unbound	07:27:26	4
83506	99	29	24,4	cyclictest	551080	-21	latency_hist	07:12:13	7
83500	99	29	27,2	cyclictest	470741	-21	kworker/u16:3+events_unbound	07:03:12	5
83500	99	29	27,2	cyclictest	167650	-21	kworker/u16:1+flush-8:0	04:19:49	5
83500	99	29	27,2	cyclictest	128946	-21	kworker/u16:0+events_unbound	04:28:23	5
83501	99	28	26,2	cyclictest	52105	-21	kworker/u16:4+flush-8:0	03:27:33	6
83501	99	28	26,2	cyclictest	167650	-21	kworker/u16:1+events_unbound	04:37:48	6
83500	99	28	26,2	cyclictest	712878	-21	kworker/u16:2+events_unbound	08:51:12	5
83500	99	28	26,2	cyclictest	651629	-21	kworker/u16:3+events_unbound	08:04:12	5
83500	99	28	26,2	cyclictest	560989	-21	kworker/u16:1+events_unbound	07:30:00	5
83500	99	28	26,2	cyclictest	278535	-21	kworker/u16:0+flush-8:0	06:46:04	5
83506	99	27	22,4	cyclictest	561193	-21	latency_hist	07:17:13	7
83501	99	27	24,2	cyclictest	571210	-21	kworker/u16:2+flush-8:0	07:22:30	6
83500	99	27	25,2	cyclictest	721791	-21	kworker/u16:3+flush-8:0	08:41:36	5
83500	99	27	25,2	cyclictest	521054	-21	kworker/u16:0+events_unbound	07:34:36	5
83500	99	27	25,2	cyclictest	52105	-21	kworker/u16:4+events_unbound	03:43:24	5
83500	99	27	25,2	cyclictest	439137	-21	kworker/u16:4+events_unbound	06:18:40	5
83500	99	27	25,2	cyclictest	328883	-21	kworker/u16:2+events_unbound	05:23:32	5
83500	99	27	25,2	cyclictest	308936	-21	kworker/u16:3+events_unbound	05:28:48	5
83500	99	27	25,2	cyclictest	278535	-21	kworker/u16:0+events_unbound	05:33:14	5
83500	99	27	25,2	cyclictest	167650	-21	kworker/u16:1+events_unbound	04:11:05	5
83497	99	27	23,3	cyclictest	692892	-21	kworker/u16:0+events_unbound	08:22:32	4
83497	99	27	23,3	cyclictest	692892	-21	kworker/u16:0+events_unbound	08:22:32	4
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17