OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Mon Dec 01, 2025 12:43:32)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2611967	99	63	57,5	cyclictest	589235	-21	systemd-journal	17:29:37	7
2611586	2	54	20,30	sleep7	0	-21	swapper/7	15:22:31	7
2611967	99	49	43,5	cyclictest	2898545	-21	latency_hist	17:42:55	7
2611967	99	49	43,5	cyclictest	2898545	-21	latency_hist	17:42:55	7
2611615	2	48	38,7	sleep4	0	-21	swapper/4	15:22:53	4
2611967	99	47	46,0	cyclictest	0	-21	swapper/7	19:33:07	7
2611967	99	46	41,4	cyclictest	2767182	-21	latency_hist	16:37:56	7
2611433	2	46	36,7	sleep6	0	-21	swapper/6	15:20:22	6
2611967	99	45	39,4	cyclictest	2868107	-21	latency_hist	17:27:55	7
2611966	99	43	41,2	cyclictest	2878036	-21	kworker/u16:3+events_unbound	18:41:51	6
2611966	99	43	41,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	16:08:28	6
2611966	99	43	41,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	16:08:28	6
2611957	99	41	39,2	cyclictest	2878036	-21	kworker/u16:3+flush-8:0	17:34:48	4
2611967	99	40	35,4	cyclictest	3151759	-21	latency_hist	19:47:55	7
2611966	99	39	37,2	cyclictest	3160963	-21	kworker/u16:3+flush-8:0	20:01:08	6
2611966	99	39	37,2	cyclictest	3121334	-21	kworker/u16:4+flush-8:0	19:48:20	6
2611966	99	39	37,2	cyclictest	2716619	-21	kworker/u16:3+events_unbound	16:25:36	6
2611957	99	39	37,2	cyclictest	2878036	-21	kworker/u16:3+flush-8:0	17:51:23	4
2611480	2	39	29,7	sleep5	0	-21	swapper/5	15:21:01	5
2611967	99	38	0,36	cyclictest	3106458	-21	grep	19:23:15	7
2611966	99	38	36,2	cyclictest	2918981	-21	kworker/u16:7+flush-8:0	19:23:20	6
2611966	99	38	36,2	cyclictest	2878089	-21	kworker/u16:6+flush-8:0	18:15:55	6
2611966	99	38	36,2	cyclictest	2571339	-21	kworker/u16:6+flush-8:0	15:40:34	6
2611966	99	37	35,2	cyclictest	2726410	-21	kworker/u16:4+flush-8:0	16:54:44	6
2611957	99	37	35,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	16:43:17	4
2611966	99	36	34,2	cyclictest	3202231	-21	kworker/u16:2+events_unbound	20:21:52	6
2611962	99	36	35,1	cyclictest	2878036	-21	kworker/u16:3+flush-8:0	17:51:47	5
2611966	99	35	33,2	cyclictest	2918981	-21	kworker/u16:7+events_unbound	17:58:55	6
2611966	99	35	33,2	cyclictest	2809239	-21	kworker/u16:2+flush-8:0	16:59:48	6
2611966	99	35	33,2	cyclictest	2767407	-21	kworker/u16:2+events_unbound	16:46:05	6
2611966	99	35	33,2	cyclictest	2766938	-21	kworker/u16:0+events_unbound	17:22:46	6
2611962	99	35	33,2	cyclictest	2666231	-21	kworker/u16:1+flush-8:0	17:29:33	5
2611966	99	34	32,2	cyclictest	3233693	-21	kworker/u16:0+flush-8:0	20:32:54	6
2611966	99	34	32,2	cyclictest	2878036	-21	kworker/u16:3+events_unbound	17:36:46	6
2611966	99	34	32,2	cyclictest	2571339	-21	kworker/u16:6+events_unbound	16:06:56	6
2611966	99	34	32,2	cyclictest	2571339	-21	kworker/u16:6+events_unbound	15:46:56	6
2611966	99	34	31,2	cyclictest	3040814	-21	kworker/u16:2+flush-8:0	18:57:55	6
2611966	99	33	31,2	cyclictest	3024746	-21	kworker/u16:1+events_unbound	18:45:57	6
2611962	99	33	31,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	16:42:36	5
2611966	99	32	30,2	cyclictest	3121334	-21	kworker/u16:4+events_unbound	20:15:55	6
2611966	99	32	30,2	cyclictest	2809239	-21	kworker/u16:2+flush-8:0	17:05:46	6
2611966	99	32	29,2	cyclictest	3121334	-21	kworker/u16:4+events_unbound	19:43:07	6
2611962	99	32	30,2	cyclictest	2918981	-21	kworker/u16:7+events_unbound	19:35:20	5
2611967	99	31	30,0	cyclictest	0	-21	swapper/7	15:58:24	7
2611966	99	31	29,2	cyclictest	3233693	-21	kworker/u16:0+events_unbound	20:48:47	6
2611966	99	31	29,2	cyclictest	3010182	-21	kworker/u16:0+events_unbound	18:48:59	6
2611966	99	31	29,2	cyclictest	2726410	-21	kworker/u16:4+flush-8:0	16:52:05	6
2611966	99	31	28,2	cyclictest	3081084	-21	kworker/u16:0+flush-8:0	19:12:55	6
2611962	99	31	29,2	cyclictest	2766938	-21	kworker/u16:0+events_unbound	16:46:44	5
2611962	99	31	28,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	17:37:55	5
2611962	99	31	28,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	17:37:55	5
2611967	99	30	29,0	cyclictest	0	-21	swapper/7	15:53:08	7
2611966	99	30	28,2	cyclictest	2918979	-21	kworker/u16:4+events_unbound	18:25:53	6
2611966	99	30	28,2	cyclictest	2918979	-21	kworker/u16:4+events_unbound	18:25:53	6
2611966	99	30	28,2	cyclictest	2878089	-21	kworker/u16:6+events_unbound	18:32:48	6
2611966	99	30	28,2	cyclictest	2878036	-21	kworker/u16:3+events_unbound	18:11:49	6
2611957	99	30	27,2	cyclictest	2918981	-21	kworker/u16:7+flush-8:0	20:28:20	4
2611967	99	29	28,0	cyclictest	0	-21	swapper/7	20:18:25	7
2611967	99	29	0,27	cyclictest	589235	-21	systemd-journal	15:25:32	7
2611967	99	29	0,27	cyclictest	589235	-21	systemd-journal	15:25:32	7
2611966	99	29	27,2	cyclictest	3172037	-21	kworker/u16:0+events_unbound	20:09:04	6
2611966	99	29	27,2	cyclictest	2918981	-21	kworker/u16:7+flush-8:0	18:53:24	6
2611966	99	29	27,2	cyclictest	2918981	-21	kworker/u16:7+events_unbound	18:05:20	6
2611966	99	29	27,2	cyclictest	2666231	-21	kworker/u16:1+flush-8:0	17:38:40	6
2611966	99	29	27,2	cyclictest	2666231	-21	kworker/u16:1+flush-8:0	17:38:40	6
2611966	99	29	27,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	17:14:55	6
2611966	99	29	27,2	cyclictest	2621816	-21	kworker/u16:0+events_unbound	16:01:36	6
2611962	99	29	27,2	cyclictest	2918981	-21	kworker/u16:7+flush-8:0	19:53:21	5
2611962	99	29	27,2	cyclictest	2918981	-21	kworker/u16:7+flush-8:0	19:53:21	5
2611957	99	29	27,2	cyclictest	3233693	-21	kworker/u16:0+events_unbound	20:37:07	4
2611957	99	29	27,2	cyclictest	3040814	-21	kworker/u16:2+events_unbound	19:31:08	4
2611957	99	29	27,2	cyclictest	3024746	-21	kworker/u16:1+flush-8:0	19:52:57	4
2611957	99	29	27,2	cyclictest	3024746	-21	kworker/u16:1+flush-8:0	19:52:57	4
2611967	99	28	0,26	cyclictest	2950019	-21	cat	18:07:56	7
2611966	99	28	26,2	cyclictest	3252623	-21	kworker/u16:2+events_unbound	20:47:36	6
2611966	99	28	26,2	cyclictest	2918981	-21	kworker/u16:7+events_unbound	20:03:36	6
2611966	99	28	26,2	cyclictest	2918980	-21	kworker/u16:5+events_unbound	17:57:17	6
2611966	99	28	26,2	cyclictest	2666231	-21	kworker/u16:1+flush-8:0	16:34:49	6
2611966	99	28	26,2	cyclictest	2621816	-21	kworker/u16:0+flush-8:0	15:30:41	6
2611966	99	28	25,2	cyclictest	3024746	-21	kworker/u16:1+events_unbound	19:07:58	6
2611966	99	28	25,2	cyclictest	3024746	-21	kworker/u16:1+events_unbound	19:07:58	6
2611966	99	28	25,2	cyclictest	2546875	-21	kworker/u16:2+events_unbound	15:23:08	6
2611966	99	28	25,2	cyclictest	2546875	-21	kworker/u16:2+events_unbound	15:23:08	6
2611962	99	28	26,2	cyclictest	3191841	-21	kworker/u16:1+events_unbound	20:44:56	5
2611962	99	28	26,2	cyclictest	3024746	-21	kworker/u16:1+events_unbound	19:03:32	5
2611962	99	28	26,2	cyclictest	2666231	-21	kworker/u16:1+flush-8:0	16:17:52	5
2611962	99	28	26,2	cyclictest	2546875	-21	kworker/u16:2+events_unbound	15:32:52	5
2611962	99	28	26,2	cyclictest	2341549	-21	kworker/u16:3+events_unbound	15:36:33	5
2611957	99	28	26,2	cyclictest	2666231	-21	kworker/u16:1+flush-8:0	16:28:16	4
2611967	99	27	0,25	cyclictest	2918712	-21	dnf	17:51:29	7
2611966	99	27	25,2	cyclictest	3024746	-21	kworker/u16:1+events_unbound	19:34:13	6
2611966	99	27	25,2	cyclictest	2918981	-21	kworker/u16:7+events_unbound	17:52:52	6
2611962	99	27	25,2	cyclictest	3160963	-21	kworker/u16:3+events_unbound	19:59:22	5
2611962	99	27	25,2	cyclictest	3121334	-21	kworker/u16:4+flush-8:0	19:51:35	5
2611962	99	27	25,2	cyclictest	3121334	-21	kworker/u16:4+events_unbound	19:31:43	5
2611962	99	27	25,2	cyclictest	2918981	-21	kworker/u16:7+flush-8:0	18:00:15	5
2611962	99	27	25,2	cyclictest	2666231	-21	kworker/u16:1+events_unbound	17:15:07	5
2611962	99	27	25,2	cyclictest	2621816	-21	kworker/u16:0+flush-8:0	15:49:17	5
2611962	99	27	25,2	cyclictest	2621816	-21	kworker/u16:0+events_unbound	15:59:48	5
2611962	99	27	25,2	cyclictest	2571339	-21	kworker/u16:6+events_unbound	16:11:20	5
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17