OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Wed Apr 29, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2247578	2	95	0,1	sleep7	2247579	-21	grep	07:24:24	7
2936164	2	74	0,1	sleep6	0	-21	swapper/6	12:29:24	6
2802845	2	56	0,0	sleep5	0	-21	swapper/5	11:34:25	5
2209743	99	50	48,2	cyclictest	2577441	-21	kworker/u16:5+events_unbound	10:49:19	4
2204390	2	50	2,40	sleep7	0	-21	swapper/7	07:04:29	7
2209368	2	48	38,7	sleep4	0	-21	swapper/4	07:07:55	4
2209743	99	46	44,2	cyclictest	2687163	-21	kworker/u16:4+flush-8:0	10:59:19	4
2209743	99	46	44,2	cyclictest	2687163	-21	kworker/u16:4+events_unbound	10:54:16	4
2209743	99	46	44,2	cyclictest	2687163	-21	kworker/u16:4+events_unbound	10:54:16	4
2209426	2	46	35,8	sleep5	0	-21	swapper/5	07:08:44	5
2209743	99	45	43,2	cyclictest	2820760	-21	kworker/u16:3+flush-8:0	11:54:18	4
2209743	99	42	39,2	cyclictest	2601767	-21	kworker/u16:2+flush-8:0	12:14:22	4
2209743	99	41	39,2	cyclictest	2601767	-21	kworker/u16:2+events_unbound	12:08:00	4
2209743	99	40	38,2	cyclictest	2601767	-21	kworker/u16:2+events_unbound	10:24:35	4
2209743	99	40	38,2	cyclictest	2378082	-21	kworker/u16:0+events_unbound	10:44:33	4
2209743	99	40	37,2	cyclictest	2723380	-21	kworker/u16:1+events_unbound	11:29:19	4
2209756	99	39	38,0	cyclictest	0	-21	swapper/7	12:29:16	7
2209743	99	39	36,2	cyclictest	2820760	-21	kworker/u16:3+flush-8:0	12:09:17	4
2209743	99	39	36,2	cyclictest	2171679	-21	kworker/u16:4+events_unbound	08:14:16	4
2209743	99	38	36,2	cyclictest	2538519	-21	kworker/u16:2+flush-8:0	09:54:23	4
2209743	99	38	36,2	cyclictest	2405546	-21	kworker/u16:4+events_unbound	09:19:17	4
2209743	99	38	35,2	cyclictest	2908915	-21	kworker/u16:0+events_unbound	12:32:12	4
2209743	99	38	35,2	cyclictest	2820760	-21	kworker/u16:3+events_unbound	11:59:03	4
2209743	99	38	35,2	cyclictest	2577441	-21	kworker/u16:5+events_unbound	10:19:11	4
2209743	99	38	35,2	cyclictest	2378082	-21	kworker/u16:0+flush-8:0	09:09:15	4
2209743	99	37	36,1	cyclictest	2443796	-21	kworker/u16:1+events_unbound	09:18:36	4
2209743	99	37	34,2	cyclictest	2820761	-21	kworker/u16:4+events_unbound	11:49:03	4
2209743	99	37	34,2	cyclictest	2711204	-21	kworker/u16:0+events_unbound	11:19:19	4
2209743	99	37	34,2	cyclictest	2476736	-21	kworker/u16:2+flush-8:0	09:24:16	4
2209756	99	36	35,0	cyclictest	0	-21	swapper/7	12:34:55	7
2209743	99	36	34,2	cyclictest	2820761	-21	kworker/u16:4+events_unbound	11:46:44	4
2209743	99	36	34,2	cyclictest	2723380	-21	kworker/u16:1+events_unbound	11:15:12	4
2209743	99	35	34,1	cyclictest	2929420	-21	kworker/u16:3+events_unbound	12:37:11	4
2209743	99	35	33,2	cyclictest	2723380	-21	kworker/u16:1+events_unbound	11:27:43	4
2928660	2	34	0,1	chrt	2928658	-21	sh	12:27:32	7
2209756	99	34	31,2	cyclictest	2760237	-21	/usr/sbin/munin	11:19:15	7
2209743	99	34	32,2	cyclictest	2711204	-21	kworker/u16:0+events_unbound	11:39:24	4
2209743	99	34	32,2	cyclictest	2708011	-21	kworker/u16:6+events_unbound	11:38:12	4
2209743	99	34	32,2	cyclictest	2577441	-21	kworker/u16:5+events_unbound	10:14:35	4
2209743	99	34	32,2	cyclictest	2577441	-21	kworker/u16:5+events_unbound	10:14:35	4
2209743	99	34	32,2	cyclictest	2550874	-21	kworker/u16:3+events_unbound	09:52:48	4
2209743	99	34	32,2	cyclictest	2117756	-21	kworker/u16:1+events_unbound	07:14:17	4
2209743	99	34	31,2	cyclictest	2443796	-21	kworker/u16:1+events_unbound	09:35:44	4
2209743	99	34	31,2	cyclictest	2443796	-21	kworker/u16:1+events_unbound	09:35:44	4
2209756	99	33	31,1	cyclictest	2793241	-21	ssh	11:29:32	7
2209743	99	33	31,2	cyclictest	2708011	-21	kworker/u16:6+events_unbound	11:06:44	4
2209743	99	33	31,2	cyclictest	2601767	-21	kworker/u16:2+events_unbound	12:23:11	4
2209743	99	33	31,2	cyclictest	2378082	-21	kworker/u16:0+flush-8:0	10:34:19	4
2209743	99	33	31,2	cyclictest	2378082	-21	kworker/u16:0+events_unbound	10:33:47	4
2209743	99	33	31,2	cyclictest	2378082	-21	kworker/u16:0+events_unbound	10:04:16	4
2209743	99	33	31,2	cyclictest	2290285	-21	kworker/u16:2+flush-8:0	07:49:20	4
2209743	99	33	30,2	cyclictest	2290285	-21	kworker/u16:2+events_unbound	08:39:15	4
2209415	2	33	23,7	sleep6	0	-21	swapper/6	07:08:37	6
2209756	99	32	28,3	cyclictest	2608763	-21	cut	10:14:24	7
2209756	99	32	28,3	cyclictest	2608763	-21	cut	10:14:24	7
2209743	99	32	30,2	cyclictest	2601767	-21	kworker/u16:2+events_unbound	11:11:20	4
2209743	99	32	30,2	cyclictest	2405546	-21	kworker/u16:4+events_unbound	08:49:27	4
2209756	99	31	29,1	cyclictest	0	-21	swapper/7	11:01:12	7
2209756	99	31	28,2	cyclictest	2882256	-21	perl	12:09:15	7
2209743	99	31	29,2	cyclictest	2723380	-21	kworker/u16:1+events_unbound	12:28:47	4
2209743	99	31	29,2	cyclictest	2538519	-21	kworker/u16:2+flush-8:0	09:59:51	4
2209743	99	31	28,2	cyclictest	2171679	-21	kworker/u16:4+flush-8:0	08:19:12	4
2209756	99	30	29,0	cyclictest	0	-21	swapper/7	09:49:03	7
2209756	99	30	0,28	cyclictest	2583508	-21	taskset	10:04:24	7
2209752	99	30	27,3	cyclictest	2723380	-21	kworker/u16:1+flush-8:0	12:19:31	6
2209756	99	29	28,0	cyclictest	0	-21	swapper/7	11:50:23	7
2209756	99	29	28,0	cyclictest	0	-21	swapper/7	07:59:17	7
2209756	99	29	27,1	cyclictest	0	-21	swapper/7	11:09:32	7
2209756	99	29	26,1	cyclictest	2910659	-21	ssh	12:19:24	7
2209743	99	29	27,2	cyclictest	2577441	-21	kworker/u16:5+events_unbound	10:39:34	4
2209743	99	29	27,2	cyclictest	2443796	-21	kworker/u16:1+events_unbound	09:29:11	4
2209743	99	29	27,2	cyclictest	2290285	-21	kworker/u16:2+events_unbound	08:33:57	4
2209756	99	28	26,1	cyclictest	0	-21	swapper/7	12:04:31	7
2209743	99	28	26,2	cyclictest	2538519	-21	kworker/u16:2+events_unbound	09:47:08	4
2209743	99	28	26,2	cyclictest	2117756	-21	kworker/u16:1+events_unbound	08:04:24	4
2209756	99	27	0,25	cyclictest	2556542	-21	cut	09:54:22	7
2209743	99	27	25,2	cyclictest	2443796	-21	kworker/u16:1+events_unbound	09:39:52	4
2209743	99	27	25,2	cyclictest	2117756	-21	kworker/u16:1+flush-8:0	07:39:12	4
2209756	99	26	22,3	cyclictest	2398388	-21	cut	08:44:19	7
2209756	99	26	0,24	cyclictest	635	-21	systemd-journal	08:29:02	7
2209752	99	26	24,2	cyclictest	2443796	-21	kworker/u16:1+flush-8:0	09:49:16	6
2209743	99	26	24,2	cyclictest	2378082	-21	kworker/u16:0+events_unbound	10:11:24	4
2209752	99	25	23,2	cyclictest	2601767	-21	kworker/u16:2+flush-8:0	10:54:24	6
2209752	99	25	23,2	cyclictest	2601767	-21	kworker/u16:2+flush-8:0	10:54:24	6
2209756	99	24	23,0	cyclictest	0	-21	swapper/7	09:33:43	7
2209756	99	24	23,0	cyclictest	0	-21	swapper/7	09:25:11	7
2209756	99	24	21,2	cyclictest	2901212	-21	cut	12:14:25	7
2209756	99	24	0,22	cyclictest	2659818	-21	timerwakeupswit	10:34:33	7
2209756	99	23	22,0	cyclictest	0	-21	swapper/7	11:27:10	7
2209756	99	23	21,1	cyclictest	0	-21	swapper/7	10:27:12	7
2209752	99	23	21,2	cyclictest	2577441	-21	kworker/u16:5+flush-8:0	10:29:29	6
2209756	99	22	16,5	cyclictest	2395723	-21	latency_hist	08:44:01	7
2209743	99	22	19,2	cyclictest	2405546	-21	kworker/u16:4+events_unbound	08:54:20	4
2209743	99	22	19,2	cyclictest	2405546	-21	kworker/u16:4+events_unbound	08:54:20	4
2209756	99	21	20,0	cyclictest	0	-21	swapper/7	10:19:03	7
2209756	99	21	16,4	cyclictest	2468097	-21	latency_hist	09:19:02	7
2209752	99	21	19,2	cyclictest	2290285	-21	kworker/u16:2+flush-8:0	08:44:28	6
2209743	99	21	19,2	cyclictest	2290285	-21	kworker/u16:2+flush-8:0	07:54:27	4
2209743	99	21	19,2	cyclictest	2250791	-21	kworker/u16:0+flush-8:0	07:28:57	4
2209756	99	20	19,0	cyclictest	0	-21	swapper/7	11:57:39	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17