OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #3, slot #4

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance

Characteristics of the 100 highest latencies:

System rack3slot4.osadl.org (updated Wed Nov 26, 2025 00:46:19)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3306634	99	132	100,24	cyclictest	3511628	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	23:23:05	0
3306640	99	131	121,6	cyclictest	210	-21	kswapd0	21:20:01	1
3306634	99	127	6,80	cyclictest	3513756	-21	kworker/0:2+events@ igb_watchdog_task	23:25:37	0
3306640	99	121	110,7	cyclictest	3371983	-21	apt-get	21:25:15	1
3306640	99	121	110,7	cyclictest	3371983	-21	apt-get	21:25:15	1
3306634	99	118	2,76	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:15:41	0
3306634	99	118	2,76	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:15:41	0
3306634	99	115	5,73	cyclictest	3565431	-21	kworker/0:1+events@ igb_watchdog_task	00:35:19	0
3306634	99	107	3,62	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	20:20:02	0
3306634	99	105	79,19	cyclictest	3328583	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	20:02:37	0
3306634	99	105	49,16	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:46:43	0
3306634	99	104	65,15	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:10:35	0
3306634	99	104	5,78	cyclictest	13	-21	ksoftirqd/0	22:30:24	0
3306634	99	104	1,59	cyclictest	3374398	-21	kworker/0:2+events@ igb_watchdog_task	21:28:16	0
3306634	99	104	1,59	cyclictest	3374398	-21	kworker/0:2+events@ igb_watchdog_task	21:28:16	0
3306634	99	103	2,52	cyclictest	3358013	-21	kworker/0:0+events@ igb_watchdog_task	21:15:42	0
3306634	99	100	45,15	cyclictest	3565431	-21	kworker/0:1+events@ igb_watchdog_task	00:32:26	0
3306634	99	100	39,24	cyclictest	3302245	-21	kworker/0:1+events@ igb_watchdog_task	19:11:24	0
3306634	99	100	2,67	cyclictest	3318520	-21	kworker/0:2+events@ igb_watchdog_task	19:40:02	0
3306657	99	99	93,3	cyclictest	0	-21	swapper/5	23:23:05	5
3306648	99	99	86,7	cyclictest	3470116	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:03:03	3
3306634	99	99	40,36	cyclictest	3394325	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	21:57:53	0
3306634	99	99	3,69	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:00:34	0
3306634	99	99	3,69	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:00:33	0
3306648	99	98	84,7	cyclictest	3523478	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	00:20:11	3
3306648	99	98	84,7	cyclictest	3523478	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	00:20:10	3
3306634	99	98	55,35	cyclictest	3314318	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	19:27:31	0
3306634	99	98	55,35	cyclictest	3314318	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	19:27:31	0
3306634	99	98	52,13	cyclictest	3358013	-21	kworker/0:0+events@ igb_watchdog_task	21:20:18	0
3306634	99	97	1,56	cyclictest	3533274	-21	kworker/0:1+events@ igb_watchdog_task	23:50:14	0
3306634	99	96	57,18	cyclictest	3310263	-21	kworker/0:2+events@ igb_watchdog_task	19:20:27	0
3306634	99	95	51,27	cyclictest	3344737	-21	kworker/0:1+events@ igb_watchdog_task	21:11:48	0
3306634	99	95	46,28	cyclictest	3565431	-21	kworker/0:1+events@ igb_watchdog_task	00:25:14	0
3306634	99	95	2,56	cyclictest	3565431	-21	kworker/0:1+events@ igb_watchdog_task	00:20:42	0
3306634	99	95	2,56	cyclictest	3565431	-21	kworker/0:1+events@ igb_watchdog_task	00:20:41	0
3306634	99	94	80,7	cyclictest	3361000	-21	kworker/u12:4+events_unbound@ flush_memcg_stats_dwork	21:35:26	0
3306634	99	94	47,28	cyclictest	3302245	-21	kworker/0:1+events@ igb_watchdog_task	19:15:10	0
3306634	99	92	55,29	cyclictest	3340771	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	20:42:43	0
3306634	99	92	55,29	cyclictest	3340771	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	20:42:43	0
3306634	99	90	66,18	cyclictest	3505606	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	23:18:04	0
3306634	99	89	69,5	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:55:30	0
3306634	99	89	3,63	cyclictest	15	-21	rcuc/0	23:00:13	0
3306634	99	89	33,39	cyclictest	662	50	irq/530-eth0-Tx	23:30:40	0
3306634	99	89	33,39	cyclictest	662	50	irq/530-eth0-Tx	23:30:39	0
3306634	99	88	76,6	cyclictest	3564189	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	00:18:13	0
3306634	99	88	63,18	cyclictest	3328583	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	20:07:37	0
3306634	99	88	63,18	cyclictest	3328583	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	20:07:37	0
3306634	99	88	57,14	cyclictest	3341459	-21	kworker/0:0+events@ igb_watchdog_task	20:35:36	0
3306634	99	88	55,19	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	20:25:22	0
3306634	99	88	53,16	cyclictest	3548461	-21	kworker/0:0+events@ igb_watchdog_task	23:55:24	0
3306640	99	87	68,7	cyclictest	3564189	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	00:16:02	1
3306634	99	87	63,18	cyclictest	3312271	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	19:31:02	0
3306634	99	87	51,24	cyclictest	3548461	-21	kworker/0:0+events@ igb_watchdog_task	00:03:17	0
3306634	99	87	50,17	cyclictest	3374398	-21	kworker/0:2+events@ igb_watchdog_task	21:34:15	0
3306634	99	87	31,34	cyclictest	3433321	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	22:22:38	0
3306634	99	86	50,17	cyclictest	3344737	-21	kworker/0:1+events@ igb_watchdog_task	20:50:32	0
3306634	99	86	50,16	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:53:29	0
3306634	99	86	29,34	cyclictest	3554651	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	00:12:58	0
3306634	99	85	37,30	cyclictest	3344737	-21	kworker/0:1+events@ igb_watchdog_task	20:55:22	0
3498923	2	83	0,5	chrt	0	-21	swapper/2	23:11:59	2
3498923	2	83	0,5	chrt	0	-21	swapper/2	23:11:59	2
3306634	99	83	52,13	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	19:50:20	0
3306634	99	83	45,16	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:40:01	0
3306648	99	82	69,7	cyclictest	3523478	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	23:53:09	3
3306634	99	82	1,63	cyclictest	0	-21	swapper/0	22:28:02	0
3306648	99	81	69,6	cyclictest	3344879	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	21:47:51	3
3306634	99	81	53,14	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	21:54:59	0
3306634	99	81	41,24	cyclictest	3513756	-21	kworker/0:2+events@ igb_watchdog_task	23:39:35	0
3306634	99	81	28,35	cyclictest	3340771	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	20:46:29	0
3537544	2	80	0,5	chrt	0	-21	swapper/2	23:45:17	2
3306634	99	80	36,36	cyclictest	3318545	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	19:45:44	0
3306648	99	79	65,7	cyclictest	3463099	-21	kworker/u12:3+events_unbound@ flush_memcg_stats_dwork	22:42:59	3
3306634	99	79	45,21	cyclictest	3513756	-21	kworker/0:2+events@ igb_watchdog_task	23:40:13	0
3306634	99	79	2,58	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	20:21:24	0
3567404	2	78	0,5	sleep1	0	-21	swapper/1	00:10:11	1
3306634	99	78	34,22	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	22:40:49	0
3306634	99	78	33,38	cyclictest	3485403	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	23:05:11	0
3306657	99	77	72,2	cyclictest	0	-21	swapper/5	20:02:36	5
3306640	99	77	20,24	cyclictest	3564189	-21	kworker/u12:0+rpciod@ rpc_async_schedule	00:31:42	1
3306634	99	77	57,6	cyclictest	3344737	-21	kworker/0:1+events@ igb_watchdog_task	21:00:34	0
3306634	99	77	54,15	cyclictest	3426803	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:36:53	0
3306634	99	77	42,24	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	19:55:28	0
3306634	99	76	45,18	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	20:10:48	0
3306634	99	76	31,24	cyclictest	3344737	-21	kworker/0:1+events@ igb_watchdog_task	21:05:32	0
3306657	99	75	70,2	cyclictest	0	-21	swapper/5	22:10:35	5
3306648	99	74	59,8	cyclictest	3564189	-21	kworker/u12:0+events_unbound@ flush_memcg_stats_dwork	00:35:36	3
3306640	99	74	65,6	cyclictest	210	-21	kswapd0	22:10:30	1
3306634	99	74	54,5	cyclictest	3321118	-21	kworker/0:1+events@ igb_watchdog_task	20:30:24	0
3306634	99	74	29,9	cyclictest	3500067	-21	rm	23:13:16	0
3306634	99	74	29,9	cyclictest	3500067	-21	rm	23:13:16	0
3306634	99	74	26,18	cyclictest	13	-21	ksoftirqd/0	23:45:22	0
3306648	99	73	60,6	cyclictest	3316330	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	19:32:32	3
3306643	99	73	50,8	cyclictest	3531279	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	23:43:08	2
3306634	99	73	40,16	cyclictest	3390487	-21	kworker/0:1+events@ igb_watchdog_task	21:41:01	0
3306634	99	73	1,58	cyclictest	0	-21	swapper/0	21:49:11	0
3306648	99	72	59,7	cyclictest	3523478	-21	kworker/u12:1+events_unbound@ flush_memcg_stats_dwork	00:25:28	3
3306643	99	72	17,27	cyclictest	3507445	-21	latency_hist	23:20:00	2
3306652	99	71	65,2	cyclictest	0	-21	swapper/4	00:35:46	4
3306648	99	71	57,7	cyclictest	3426803	-21	kworker/u12:2+events_unbound@ flush_memcg_stats_dwork	22:21:02	3
3306657	99	70	64,3	cyclictest	0	-21	swapper/5	23:00:29	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17