OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Sun May 31, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3109848	2	58	48,7	sleep4	0	-21	swapper/4	19:05:42	4
3109848	2	58	48,7	sleep4	0	-21	swapper/4	19:05:42	4
3110234	99	44	40,4	cyclictest	3257480	-21	kworker/u16:2+flush-8:0	22:26:56	5
3102639	2	43	4,33	sleep7	0	-21	swapper/7	19:02:16	7
3102639	2	43	4,33	sleep7	0	-21	swapper/7	19:02:16	7
3110242	99	36	33,2	cyclictest	3429982	-21	ssh	21:42:11	7
3110234	99	36	34,2	cyclictest	3597579	-21	kworker/u16:4+flush-8:0	00:01:51	5
3110242	99	34	29,4	cyclictest	3597387	-21	latency_hist	22:51:56	7
3110234	99	34	32,2	cyclictest	3718146	-21	kworker/u16:0+events_unbound	00:26:50	5
3110234	99	34	32,2	cyclictest	3718146	-21	kworker/u16:0+events_unbound	00:26:50	5
3110234	99	34	32,2	cyclictest	3654854	-21	kworker/u16:2+events_unbound	23:36:42	5
3390514	2	33	0,0	chrt	3390512	-21	ssh	21:23:09	7
3110242	99	33	0,32	cyclictest	124	-21	kswapd0	00:27:09	7
3110242	99	33	0,32	cyclictest	124	-21	kswapd0	00:27:09	7
3110234	99	33	31,2	cyclictest	3802331	-21	kworker/u16:4+events_unbound	00:16:42	5
3110234	99	33	31,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	21:41:51	5
3110234	99	33	31,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	21:41:51	5
3110234	99	33	31,2	cyclictest	3047314	-21	kworker/u16:1+events_unbound	19:22:19	5
3109790	2	33	22,8	sleep5	0	-21	swapper/5	19:04:55	5
3109790	2	33	22,8	sleep5	0	-21	swapper/5	19:04:55	5
3633392	2	32	0,1	chrt	3633394	-21	ssh	23:06:16	7
3633392	2	32	0,1	chrt	3633394	-21	ssh	23:06:16	7
3110234	99	32	30,2	cyclictest	3325173	-21	kworker/u16:0+events_unbound	21:17:18	5
3110234	99	32	27,4	cyclictest	3047314	-21	kworker/u16:1+flush-8:0	19:41:56	5
3110242	99	30	25,4	cyclictest	635	-21	systemd-journal	22:56:51	7
3110234	99	30	26,3	cyclictest	3152628	-21	kworker/u16:3+flush-8:0	20:12:16	5
3110234	99	30	26,3	cyclictest	3152628	-21	kworker/u16:3+flush-8:0	20:12:16	5
3109906	2	28	18,7	sleep6	0	-21	swapper/6	19:06:32	6
3109906	2	28	18,7	sleep6	0	-21	swapper/6	19:06:32	6
3110242	99	27	24,2	cyclictest	3763304	-21	sh	23:57:19	7
3110242	99	27	22,4	cyclictest	3428741	-21	latency_hist	21:41:56	7
3110242	99	27	22,4	cyclictest	3428741	-21	latency_hist	21:41:56	7
3110238	99	27	25,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	20:27:10	6
3110234	99	27	24,2	cyclictest	3152628	-21	kworker/u16:3+flush-8:0	20:22:14	5
3110234	99	27	22,5	cyclictest	3440788	-21	kworker/u16:0+flush-8:0	22:36:56	5
3110242	99	26	25,0	cyclictest	0	-21	swapper/7	19:22:11	7
3110242	99	26	20,5	cyclictest	635	-21	systemd-journal	23:21:56	7
3110242	99	26	0,1	cyclictest	0	-21	swapper/7	20:07:20	7
3110234	99	25	23,2	cyclictest	3740532	-21	kworker/u16:1+events_unbound	00:05:10	5
3110234	99	25	23,2	cyclictest	3654854	-21	kworker/u16:2+flush-8:0	23:27:02	5
3110234	99	25	23,2	cyclictest	3277165	-21	kworker/u16:1+events_unbound	21:42:15	5
3110234	99	25	23,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	21:07:27	5
3110242	99	24	19,4	cyclictest	635	-21	systemd-journal	22:26:56	7
3110234	99	24	22,2	cyclictest	3740532	-21	kworker/u16:1+events_unbound	00:36:50	5
3110234	99	24	22,2	cyclictest	3654854	-21	kworker/u16:2+events_unbound	23:46:42	5
3110234	99	24	22,2	cyclictest	3654854	-21	kworker/u16:2+events_unbound	23:46:42	5
3110234	99	24	22,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	22:12:28	5
3110234	99	24	22,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	22:04:06	5
3110234	99	23	21,2	cyclictest	3777047	-21	kworker/u16:2+events_unbound	00:09:34	5
3110234	99	23	21,2	cyclictest	3257480	-21	kworker/u16:2+flush-8:0	22:47:24	5
3110234	99	23	21,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	22:07:06	5
3110234	99	22	20,2	cyclictest	3152628	-21	kworker/u16:3+events_unbound	20:32:13	5
3110234	99	21	18,2	cyclictest	3718146	-21	kworker/u16:0+flush-8:0	00:17:06	5
3110234	99	21	17,3	cyclictest	3257480	-21	kworker/u16:2+flush-8:0	21:22:16	5
3110242	99	20	17,2	cyclictest	124	-21	kswapd0	00:32:09	7
3110234	99	20	18,2	cyclictest	3597579	-21	kworker/u16:4+flush-8:0	23:17:22	5
3110234	99	20	18,2	cyclictest	3597579	-21	kworker/u16:4+events_unbound	22:54:06	5
3110234	99	20	18,2	cyclictest	3277165	-21	kworker/u16:1+events_unbound	21:32:28	5
3110234	99	20	18,2	cyclictest	3047314	-21	kworker/u16:1+events_unbound	19:27:16	5
3110234	99	20	18,2	cyclictest	3047314	-21	kworker/u16:1+events_unbound	19:27:16	5
3110231	99	20	16,3	cyclictest	3718146	-21	kworker/u16:0+flush-8:0	00:07:23	4
3110242	99	19	2,15	cyclictest	1234	-21	snmpd	21:50:46	7
3110242	99	19	16,1	cyclictest	3502765	-21	ssh	22:12:13	7
3110242	99	19	0,17	cyclictest	3706044	-21	latency_hist	23:36:55	7
3110234	99	19	17,2	cyclictest	3440788	-21	kworker/u16:0+flush-8:0	22:37:16	5
3110234	99	19	17,2	cyclictest	3257480	-21	kworker/u16:2+events_unbound	21:56:40	5
3110242	99	18	9,8	cyclictest	6101	-21	gmain	23:12:14	7
3110242	99	18	17,0	cyclictest	0	-21	swapper/7	19:42:27	7
3110242	99	18	14,3	cyclictest	124	-21	kswapd0	00:06:56	7
3110242	99	18	0,16	cyclictest	0	-21	swapper/7	20:27:18	7
3110234	99	18	16,2	cyclictest	3718146	-21	kworker/u16:0+events_unbound	23:55:35	5
3110234	99	18	16,2	cyclictest	3654854	-21	kworker/u16:2+events_unbound	23:37:21	5
3110234	99	18	16,2	cyclictest	3546837	-21	kworker/u16:1+events_unbound	22:43:54	5
3110234	99	18	15,2	cyclictest	3597579	-21	kworker/u16:4+flush-8:0	22:57:14	5
3110242	99	17	12,4	cyclictest	3501047	-21	latency_hist	22:11:56	7
3110242	99	17	11,5	cyclictest	3228965	-21	latency_hist	20:06:56	7
3110242	99	17	11,5	cyclictest	3200213	-21	latency_hist	19:51:57	7
3110234	99	17	15,2	cyclictest	3718146	-21	kworker/u16:0+events_unbound	23:47:54	5
3110234	99	17	15,2	cyclictest	3718146	-21	kworker/u16:0+events_unbound	00:30:34	5
3110234	99	17	15,2	cyclictest	3718146	-21	kworker/u16:0+events_unbound	00:30:34	5
3110234	99	17	15,2	cyclictest	3597427	-21	kworker/u16:1+events_unbound	23:04:27	5
3110234	99	17	15,2	cyclictest	3597427	-21	kworker/u16:1+events_unbound	23:04:27	5
3110234	99	17	15,2	cyclictest	3380494	-21	kworker/u16:3+events_unbound	22:20:31	5
3110234	99	17	15,2	cyclictest	3380494	-21	kworker/u16:3+events_unbound	22:20:31	5
3110234	99	17	15,2	cyclictest	3152628	-21	kworker/u16:3+flush-8:0	20:07:14	5
3110234	99	17	15,2	cyclictest	3152628	-21	kworker/u16:3+flush-8:0	19:32:20	5
3110231	99	17	13,3	cyclictest	3802269	-40	logrotate	00:16:28	4
3110231	99	17	13,3	cyclictest	3243125	-21	kthreadcore	20:12:16	4
3110231	99	17	13,3	cyclictest	3243125	-21	kthreadcore	20:12:16	4
3110231	99	17	0,3	cyclictest	3570236	-21	ntpq	22:37:19	4
3110231	99	17	0,1	cyclictest	0	-21	swapper/4	22:50:43	4
3110231	99	17	0,1	cyclictest	0	-21	swapper/4	21:18:48	4
3110231	99	17	0,14	cyclictest	0	-21	swapper/4	21:47:39	4
3728659	2	16	0,1	chrt	3728658	-21	ssh	23:44:06	7
3728659	2	16	0,1	chrt	3728658	-21	ssh	23:44:06	7
3110242	99	16	11,4	cyclictest	3694036	-21	latency_hist	23:31:56	7
3110238	99	16	13,2	cyclictest	3152628	-21	kworker/u16:3+flush-8:0	19:37:12	6
3110234	99	16	14,2	cyclictest	3125340	-21	kworker/u16:2+flush-8:0	19:17:24	5
3110231	99	16	0,15	cyclictest	3352939	-21	latency	21:07:18	4
3110242	99	15	12,2	cyclictest	0	-21	swapper/7	00:11:56	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17