OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Fri May 15, 2026 00:43:33)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
4073803	2	96	0,1	sleep4	0	-21	swapper/4	00:13:00	4
3391233	2	59	49,7	sleep4	0	-21	swapper/4	19:06:34	4
3391025	2	57	46,7	sleep5	0	-21	swapper/5	19:03:38	5
3391647	99	52	49,2	cyclictest	3629766	-21	kworker/u16:0+flush-8:0	21:08:27	5
3391647	99	42	40,2	cyclictest	3710773	-21	kworker/u16:0+events_unbound	21:48:29	5
3391161	2	41	6,30	sleep7	0	-21	swapper/7	19:05:32	7
3391647	99	40	37,2	cyclictest	3481816	-21	kworker/u16:2+events_unbound	20:13:14	5
3391654	99	39	33,5	cyclictest	3433706	-21	latency_hist	19:28:01	7
3391647	99	37	35,2	cyclictest	3855841	-21	kworker/u16:2+events_unbound	22:51:46	5
3391647	99	37	35,2	cyclictest	3650200	-21	kworker/u16:3+events_unbound	22:04:50	5
3391647	99	36	34,2	cyclictest	3928451	-21	kworker/u16:4+events_unbound	23:36:21	5
3391647	99	36	34,2	cyclictest	3855841	-21	kworker/u16:2+events_unbound	00:11:53	5
3391647	99	36	34,2	cyclictest	3820702	-21	kworker/u16:0+events_unbound	23:49:21	5
3391647	99	36	34,2	cyclictest	3820702	-21	kworker/u16:0+events_unbound	22:34:42	5
3391647	99	36	34,2	cyclictest	3820702	-21	kworker/u16:0+events_unbound	22:34:42	5
3391647	99	36	34,2	cyclictest	3646893	-21	kworker/u16:2+events_unbound	21:20:26	5
3391647	99	36	34,2	cyclictest	3629766	-21	kworker/u16:0+flush-8:0	21:16:18	5
3391647	99	36	34,2	cyclictest	3521774	-21	kworker/u16:4+flush-8:0	21:32:41	5
3391647	99	35	33,2	cyclictest	3820702	-21	kworker/u16:0+events_unbound	23:32:21	5
3391647	99	35	33,2	cyclictest	3806515	-21	kworker/u16:1+flush-8:0	22:22:21	5
3391647	99	35	33,2	cyclictest	3806515	-21	kworker/u16:1+events_unbound	22:56:41	5
3391647	99	35	33,2	cyclictest	3650200	-21	kworker/u16:3+events_unbound	21:42:54	5
3391647	99	34	32,2	cyclictest	4025013	-21	kworker/u16:1+events_unbound	00:06:50	5
3391647	99	34	32,2	cyclictest	3855841	-21	kworker/u16:2+events_unbound	23:16:49	5
3391647	99	34	32,2	cyclictest	3820702	-21	kworker/u16:0+events_unbound	23:40:49	5
3391647	99	34	32,2	cyclictest	3710773	-21	kworker/u16:0+events_unbound	21:53:49	5
3391647	99	34	32,2	cyclictest	3710773	-21	kworker/u16:0+events_unbound	21:53:49	5
3391647	99	34	32,2	cyclictest	3521774	-21	kworker/u16:4+events_unbound	22:01:46	5
3391647	99	34	32,2	cyclictest	3344322	-21	kworker/u16:4+flush-8:0	19:33:54	5
3391647	99	34	31,2	cyclictest	3820702	-21	kworker/u16:0+flush-8:0	22:28:50	5
3391647	99	33	31,2	cyclictest	4025013	-21	kworker/u16:1+events_unbound	00:15:17	5
3391647	99	33	31,2	cyclictest	3855841	-21	kworker/u16:2+events_unbound	23:02:29	5
3391647	99	33	31,2	cyclictest	3855841	-21	kworker/u16:2+events_unbound	00:01:18	5
3391647	99	33	31,2	cyclictest	3820702	-21	kworker/u16:0+flush-8:0	22:39:45	5
3391647	99	33	31,2	cyclictest	3820702	-21	kworker/u16:0+events_unbound	23:09:45	5
3391647	99	33	31,2	cyclictest	3806515	-21	kworker/u16:1+events_unbound	23:04:18	5
3391647	99	33	31,2	cyclictest	3650200	-21	kworker/u16:3+events_unbound	22:24:46	5
3391647	99	33	31,2	cyclictest	3646893	-21	kworker/u16:2+events_unbound	21:34:17	5
3391647	99	33	31,2	cyclictest	3521774	-21	kworker/u16:4+events_unbound	21:27:18	5
3391647	99	32	30,2	cyclictest	3928451	-21	kworker/u16:4+events_unbound	23:45:13	5
3391647	99	32	30,2	cyclictest	3928451	-21	kworker/u16:4+events_unbound	23:25:13	5
3391647	99	32	30,2	cyclictest	3855842	-21	kworker/u16:4+events_unbound	22:45:38	5
3391647	99	32	30,2	cyclictest	3855841	-21	kworker/u16:2+flush-8:0	00:31:17	5
3391647	99	32	30,2	cyclictest	3710773	-21	kworker/u16:0+events_unbound	22:15:25	5
3391647	99	32	30,2	cyclictest	3521774	-21	kworker/u16:4+events_unbound	22:09:41	5
3391647	99	31	29,2	cyclictest	3928451	-21	kworker/u16:4+events_unbound	23:55:26	5
3391647	99	31	29,2	cyclictest	3650200	-21	kworker/u16:3+events_unbound	21:44:22	5
3391647	99	31	29,2	cyclictest	3521774	-21	kworker/u16:4+events_unbound	20:57:09	5
3391647	99	31	29,2	cyclictest	3401246	-21	kworker/u16:2+events_unbound	19:32:10	5
3391649	99	30	28,2	cyclictest	3127692	-21	kworker/u16:1+flush-8:0	19:33:26	6
3391649	99	30	26,3	cyclictest	3855841	-21	kworker/u16:2+flush-8:0	22:48:05	6
3391649	99	30	26,3	cyclictest	3710773	-21	kworker/u16:0+flush-8:0	22:03:00	6
3391647	99	30	28,2	cyclictest	4097643	-21	kworker/u16:0+events_unbound	00:25:01	5
3391647	99	30	28,2	cyclictest	3928451	-21	kworker/u16:4+events_unbound	23:21:18	5
3391647	99	30	28,2	cyclictest	3855841	-21	kworker/u16:2+events_unbound	00:37:49	5
3391647	99	30	28,2	cyclictest	3471180	-21	kworker/u16:3+events_unbound	20:06:55	5
3391654	99	29	0,27	cyclictest	1234	-21	snmpd	20:33:15	7
3391647	99	29	27,2	cyclictest	4061442	-21	kworker/u16:3+events_unbound	00:20:29	5
3391647	99	29	27,2	cyclictest	3453179	-21	kworker/u16:0+flush-8:0	19:55:35	5
3391654	99	28	24,3	cyclictest	124	-21	kswapd0	00:12:59	7
3391647	99	28	26,2	cyclictest	3471180	-21	kworker/u16:3+events_unbound	20:00:54	5
3391643	99	28	26,2	cyclictest	4025013	-21	kworker/u16:1+events_unbound	00:13:15	4
3391654	99	27	22,4	cyclictest	3637918	-21	latency_hist	21:13:00	7
3391647	99	27	25,2	cyclictest	3521774	-21	kworker/u16:4+events_unbound	21:04:54	5
3391647	99	27	25,2	cyclictest	3127692	-21	kworker/u16:1+events_unbound	19:27:54	5
3391647	99	27	24,2	cyclictest	3481816	-21	kworker/u16:2+events_unbound	20:23:10	5
4133447	2	26	0,1	chrt	4133445	-21	ssh	00:37:15	7
3391649	99	26	24,2	cyclictest	3453179	-21	kworker/u16:0+flush-8:0	19:53:15	6
3391647	99	26	24,2	cyclictest	3401246	-21	kworker/u16:2+events_unbound	19:19:10	5
3391654	99	25	24,0	cyclictest	0	-21	swapper/7	22:53:20	7
3391654	99	25	0,23	cyclictest	827	-21	dbus-daemon	19:23:01	7
3391647	99	25	23,2	cyclictest	3521774	-21	kworker/u16:4+flush-8:0	20:33:23	5
3391092	2	25	15,7	sleep6	0	-21	swapper/6	19:04:35	6
3391654	99	24	23,0	cyclictest	0	-21	swapper/7	23:08:21	7
3391654	99	24	22,1	cyclictest	0	-21	swapper/7	21:53:17	7
3391654	99	24	22,1	cyclictest	0	-21	swapper/7	21:53:17	7
3391654	99	24	18,5	cyclictest	3452950	-21	latency_hist	19:38:00	7
3391643	99	24	21,2	cyclictest	3481816	-21	kworker/u16:2+flush-8:0	20:28:25	4
3391643	99	24	21,2	cyclictest	3481816	-21	kworker/u16:2+flush-8:0	20:28:25	4
3391654	99	23	18,3	cyclictest	3860500	-21	kthreadcore	22:43:22	7
3391654	99	23	0,21	cyclictest	3854594	-21	taskset	22:40:57	7
3391647	99	23	21,2	cyclictest	3521774	-21	kworker/u16:4+flush-8:0	20:22:53	5
3391654	99	22	17,4	cyclictest	3819466	-21	latency_hist	22:27:59	7
3391654	99	22	17,4	cyclictest	3686382	-21	latency_hist	21:33:00	7
3391654	99	22	1,20	cyclictest	857	-21	udisksd	22:17:38	7
3391649	99	22	20,2	cyclictest	3928451	-21	kworker/u16:4+flush-8:0	23:18:25	6
3391649	99	22	20,2	cyclictest	3855841	-21	kworker/u16:2+flush-8:0	00:03:30	6
3391649	99	22	20,2	cyclictest	3521774	-21	kworker/u16:4+flush-8:0	21:58:00	6
3391649	99	22	20,2	cyclictest	3521774	-21	kworker/u16:4+flush-8:0	21:58:00	6
3391647	99	22	20,2	cyclictest	3453179	-21	kworker/u16:0+events_unbound	20:12:53	5
3391654	99	21	19,1	cyclictest	3781032	-21	ssh	22:09:02	7
3391654	99	21	19,1	cyclictest	0	-21	swapper/7	21:03:16	7
3391654	99	21	18,2	cyclictest	3832861	-21	iostat	22:33:16	7
3391654	99	21	18,2	cyclictest	3832861	-21	iostat	22:33:16	7
3391654	99	21	15,5	cyclictest	635	-21	systemd-journal	20:03:00	7
3391647	99	21	19,2	cyclictest	3471180	-21	kworker/u16:3+flush-8:0	20:39:57	5
3391654	99	20	0,18	cyclictest	3513799	-21	cut	20:08:17	7
3391647	99	20	18,2	cyclictest	3481816	-21	kworker/u16:2+flush-8:0	20:32:57	5
3391647	99	20	18,2	cyclictest	3481816	-21	kworker/u16:2+flush-8:0	20:32:57	5
3391654	99	19	16,2	cyclictest	0	-21	swapper/7	20:53:01	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17