OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri Mar 13, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
10600	2	105	0,1	sleep7	10602	-21	cut	09:04:15	7
260193	2	95	0,1	sleep7	260194	-21	grep	10:49:14	7
286599	2	65	0,1	sleep7	286602	-21	cut	10:59:16	7
450486	2	64	0,1	sleep5	0	-21	swapper/5	12:09:10	5
3962080	2	64	35,8	sleep4	0	-21	swapper/4	07:00:34	4
3962610	99	59	57,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	10:59:24	5
209430	2	57	0,0	sleep7	209432	-21	sed	10:29:10	7
3962610	99	50	47,2	cyclictest	268317	-21	kworker/u16:3+events_unbound	11:39:12	5
3962100	2	49	39,7	sleep6	0	-21	swapper/6	07:00:52	6
3962610	99	47	45,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	12:14:24	5
3962618	99	45	40,4	cyclictest	26517	-21	latency_hist	09:13:54	7
3962610	99	44	42,2	cyclictest	99142	-21	kworker/u16:4+events_unbound	10:24:11	5
3962610	99	44	42,2	cyclictest	253728	-21	kworker/u16:0+events_unbound	10:54:19	5
3962610	99	43	41,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	12:00:44	5
3962618	99	42	37,4	cyclictest	4186273	-21	latency_hist	08:58:55	7
3962614	99	42	40,2	cyclictest	268317	-21	kworker/u16:3+flush-8:0	11:48:54	6
3962610	99	42	39,2	cyclictest	388863	-21	kworker/u16:2+events_unbound	11:54:09	5
3962610	99	42	39,2	cyclictest	388863	-21	kworker/u16:2+events_unbound	11:54:09	5
3962610	99	41	39,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	11:09:47	5
3962610	99	41	39,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	10:36:45	5
3962610	99	41	38,2	cyclictest	268317	-21	kworker/u16:3+events_unbound	11:04:09	5
3962618	99	40	35,4	cyclictest	436986	-21	latency_hist	12:03:54	7
3962610	99	40	38,2	cyclictest	99142	-21	kworker/u16:4+events_unbound	10:48:00	5
3962610	99	40	38,2	cyclictest	96601	-21	kworker/u16:3+events_unbound	10:02:20	5
3962610	99	40	38,2	cyclictest	96601	-21	kworker/u16:3+events_unbound	10:02:20	5
3962610	99	40	38,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	07:14:18	5
3962610	99	40	38,2	cyclictest	145132	-21	kworker/u16:0+events_unbound	10:05:16	5
3953123	2	40	3,31	sleep7	0	-21	swapper/7	06:59:00	7
3962610	99	39	37,2	cyclictest	48976	-21	kworker/u16:2+events_unbound	09:42:04	5
3962610	99	39	37,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	12:21:36	5
3962610	99	39	37,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	11:49:27	5
3962610	99	39	37,2	cyclictest	145132	-21	kworker/u16:0+events_unbound	10:20:15	5
3962610	99	38	36,2	cyclictest	48976	-21	kworker/u16:2+events_unbound	09:28:33	5
3962610	99	38	36,2	cyclictest	473139	-21	kworker/u16:3+events_unbound	12:30:12	5
3962610	99	38	36,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	09:38:43	5
3962610	99	38	36,2	cyclictest	304453	-21	kworker/u16:4+events_unbound	11:32:00	5
3962610	99	38	36,2	cyclictest	268317	-21	kworker/u16:3+events_unbound	11:14:09	5
3962610	99	38	36,2	cyclictest	253728	-21	kworker/u16:0+events_unbound	10:51:15	5
3962618	99	37	32,4	cyclictest	3995075	-21	latency_hist	07:18:55	7
3962610	99	37	35,2	cyclictest	4119643	-21	kworker/u16:1+events_unbound	09:17:56	5
3962610	99	37	35,2	cyclictest	4119643	-21	kworker/u16:1+events_unbound	09:06:11	5
3962610	99	37	35,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	07:04:18	5
3962610	99	37	35,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	12:05:52	5
3962610	99	37	35,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	12:05:52	5
3962610	99	37	35,2	cyclictest	2033	-21	kworker/u16:4+events_unbound	09:12:04	5
3962610	99	37	35,2	cyclictest	145132	-21	kworker/u16:0+events_unbound	10:18:00	5
3962610	99	36	34,2	cyclictest	96601	-21	kworker/u16:3+events_unbound	09:52:16	5
3962610	99	36	34,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	09:20:40	5
3962610	99	36	34,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	09:20:40	5
3962610	99	36	34,2	cyclictest	304453	-21	kworker/u16:4+events_unbound	11:20:07	5
3962610	99	36	34,2	cyclictest	229630	-21	kworker/u16:1+flush-8:0	10:41:04	5
3962610	99	36	34,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	12:25:31	5
3962610	99	36	34,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	11:38:39	5
3962610	99	36	34,2	cyclictest	145132	-21	kworker/u16:0+events_unbound	10:09:04	5
3962608	99	36	33,2	cyclictest	145132	-21	kworker/u16:0+flush-8:0	10:23:54	4
3962610	99	35	33,2	cyclictest	48976	-21	kworker/u16:2+events_unbound	09:48:43	5
3962610	99	35	33,2	cyclictest	48976	-21	kworker/u16:2+events_unbound	09:29:04	5
3962610	99	35	33,2	cyclictest	304453	-21	kworker/u16:4+events_unbound	11:47:07	5
3962610	99	34	32,2	cyclictest	96601	-21	kworker/u16:3+events_unbound	09:56:53	5
3962610	99	34	32,2	cyclictest	147938	-21	kworker/u16:2+events_unbound	10:31:01	5
3962155	2	34	24,7	sleep5	0	-21	swapper/5	07:01:39	5
446085	2	33	0,0	sleep7	446086	-21	grep	12:04:24	7
446085	2	33	0,0	sleep7	446086	-21	grep	12:04:24	7
3962614	99	31	29,2	cyclictest	4023448	-21	kworker/u16:2+events_unbound	08:19:24	6
3962610	99	31	29,2	cyclictest	268317	-21	kworker/u16:3+events_unbound	11:28:27	5
3962610	99	30	28,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	07:38:16	5
3962608	99	29	26,3	cyclictest	48976	-21	kworker/u16:2+flush-8:0	09:34:31	4
3962608	99	29	25,3	cyclictest	4023448	-21	kworker/u16:2+flush-8:0	08:19:21	4
3962614	99	28	26,2	cyclictest	4119643	-21	kworker/u16:1+events_unbound	09:29:08	6
3962614	99	28	26,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	08:39:07	6
3962614	99	28	25,3	cyclictest	99142	-21	kworker/u16:4+flush-8:0	10:14:15	6
3962614	99	28	23,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	11:13:57	6
3962608	99	28	26,2	cyclictest	229630	-21	kworker/u16:1+flush-8:0	11:54:25	4
3962608	99	28	26,2	cyclictest	229630	-21	kworker/u16:1+flush-8:0	11:54:25	4
3962614	99	27	25,2	cyclictest	145132	-21	kworker/u16:0+events_unbound	10:06:12	6
3962614	99	26	24,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	08:44:09	6
3962610	99	26	24,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	08:11:16	5
3962608	99	26	24,2	cyclictest	48976	-21	kworker/u16:2+events_unbound	09:39:18	4
316150	2	26	0,1	chrt	316148	-21	ssh	11:13:18	7
3962614	99	25	23,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	07:48:49	6
3962614	99	25	23,2	cyclictest	388863	-21	kworker/u16:2+events_unbound	12:23:16	6
3962614	99	25	23,2	cyclictest	145132	-21	kworker/u16:0+events_unbound	10:09:06	6
3962610	99	25	23,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	08:36:15	5
3962610	99	25	23,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	07:55:04	5
3962610	99	25	23,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	07:55:04	5
3962610	99	25	23,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	08:18:57	5
3962610	99	25	23,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	07:10:05	5
3962610	99	25	23,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	07:10:05	5
3962610	99	25	23,2	cyclictest	2433275	-21	kworker/u16:1+events_unbound	07:27:17	5
3962608	99	25	23,2	cyclictest	99142	-21	kworker/u16:4+flush-8:0	10:04:24	4
3962608	99	25	23,2	cyclictest	3995268	-21	kworker/u16:2+flush-8:0	07:19:25	4
3962614	99	24	22,2	cyclictest	2433275	-21	kworker/u16:1+flush-8:0	07:18:50	6
3962614	99	24	22,2	cyclictest	229630	-21	kworker/u16:1+events_unbound	11:05:51	6
3962614	99	23	21,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	09:09:16	6
3962614	99	23	21,2	cyclictest	3962811	-21	kworker/u16:0+events_unbound	08:54:17	6
3962614	99	23	21,2	cyclictest	388863	-21	kworker/u16:2+events_unbound	11:55:15	6
3962614	99	23	21,2	cyclictest	388863	-21	kworker/u16:2+events_unbound	11:55:15	6
3962614	99	23	21,2	cyclictest	147938	-21	kworker/u16:2+events_unbound	10:21:27	6
3962610	99	23	21,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	08:58:16	5
3962610	99	23	21,2	cyclictest	3964362	-21	kworker/u16:3+events_unbound	08:50:17	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17