OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Jul 28, 2026 00:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
3108979	2	104	0,1	sleep5	0	-21	swapper/5	20:24:34	5
3542077	2	97	0,0	sleep5	0	-21	swapper/5	23:34:19	5
3423000	2	97	0,0	sleep4	0	-21	swapper/4	22:44:25	4
3558172	2	95	0,1	sleep7	3558173	-21	grep	23:39:25	7
2950346	2	57	22,30	sleep7	0	-21	swapper/7	19:05:44	7
2950897	99	56	51,4	cyclictest	125	-21	kswapd0	23:24:04	7
2950354	2	55	44,8	sleep4	0	-21	swapper/4	19:05:49	4
2950897	99	51	49,1	cyclictest	3489551	-21	ssh	23:09:32	7
2950897	99	48	47,0	cyclictest	0	-21	swapper/7	00:09:20	7
2950897	99	48	47,0	cyclictest	0	-21	swapper/7	00:09:20	7
2950897	99	48	45,2	cyclictest	3187454	-21	diskmemload	21:14:32	7
2950888	99	47	45,2	cyclictest	3024432	-21	kworker/u16:4+flush-8:0	19:54:24	4
2950888	99	45	42,2	cyclictest	3248543	-21	kworker/u16:0+events_unbound	21:49:19	4
2950888	99	42	40,2	cyclictest	3589706	-21	kworker/u16:2+events_unbound	00:34:31	4
2950888	99	42	40,2	cyclictest	3148457	-21	kworker/u16:3+events_unbound	21:04:28	4
2950897	99	41	40,0	cyclictest	0	-21	swapper/7	23:49:20	7
2950888	99	41	39,2	cyclictest	3456539	-21	kworker/u16:1+flush-8:0	23:34:27	4
2950281	2	41	30,8	sleep5	0	-21	swapper/5	19:04:46	5
2950897	99	40	38,1	cyclictest	0	-21	swapper/7	21:54:24	7
2950897	99	40	36,3	cyclictest	3012371	-21	timerwakeupswit	19:34:36	7
2950888	99	40	38,2	cyclictest	3602344	-21	kworker/u16:4+flush-8:0	00:04:20	4
2950888	99	40	38,2	cyclictest	3456540	-21	kworker/u16:3+events_unbound	23:29:38	4
2950897	99	39	37,1	cyclictest	3319681	-21	ssh	21:59:39	7
2950888	99	39	37,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	22:04:33	4
2950888	99	39	37,2	cyclictest	3195157	-21	kworker/u16:1+events_unbound	21:19:40	4
2950888	99	39	37,2	cyclictest	3167997	-21	kworker/u16:0+events_unbound	21:09:35	4
2950888	99	39	37,2	cyclictest	3024432	-21	kworker/u16:4+events_unbound	19:44:36	4
2950888	99	39	37,2	cyclictest	2921253	-21	kworker/u16:2+events_unbound	19:24:20	4
3418877	2	38	0,1	chrt	3418878	-21	sh	22:42:35	7
2950897	99	38	35,2	cyclictest	3380118	-21	ntpq	22:24:33	7
2950888	99	38	36,2	cyclictest	3589706	-21	kworker/u16:2+events_unbound	23:54:33	4
2950888	99	38	36,2	cyclictest	3195157	-21	kworker/u16:1+flush-8:0	21:44:18	4
2950888	99	38	35,2	cyclictest	3346654	-21	kworker/u16:0+events_unbound	22:39:32	4
3679223	2	37	0,1	chrt	0	-21	swapper/6	00:29:25	6
2950897	99	37	35,1	cyclictest	3060539	-21	smartctl	19:59:36	7
2950888	99	37	35,2	cyclictest	3456540	-21	kworker/u16:3+flush-8:0	23:06:43	4
2950888	99	37	35,2	cyclictest	3024432	-21	kworker/u16:4+flush-8:0	19:59:24	4
2950888	99	37	34,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	21:24:22	4
2950897	99	36	34,1	cyclictest	3502970	-21	ssh	23:15:40	7
2950897	99	36	33,2	cyclictest	3064568	-21	grep	20:04:24	7
2950897	99	36	33,2	cyclictest	3064568	-21	grep	20:04:24	7
2950897	99	36	32,3	cyclictest	3239887	-21	grep	21:29:24	7
2950897	99	36	32,3	cyclictest	3239887	-21	grep	21:29:24	7
2950888	99	36	34,2	cyclictest	3384431	-21	kworker/u16:2+events_unbound	22:59:40	4
2950888	99	36	34,2	cyclictest	3199817	-21	kworker/u16:4+flush-8:0	23:24:36	4
2950888	99	36	34,2	cyclictest	3199817	-21	kworker/u16:4+flush-8:0	22:19:56	4
2950888	99	36	34,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	21:29:36	4
2950888	99	36	34,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	21:29:36	4
2950888	99	36	34,2	cyclictest	3148457	-21	kworker/u16:3+events_unbound	22:02:01	4
2950888	99	36	34,2	cyclictest	3148457	-21	kworker/u16:3+events_unbound	21:16:00	4
2950491	2	36	25,8	sleep6	0	-21	swapper/6	19:07:44	6
2950897	99	35	34,0	cyclictest	0	-21	swapper/7	23:31:37	7
2950897	99	35	33,1	cyclictest	3222390	-21	ssh	21:20:49	7
2950897	99	35	32,2	cyclictest	3073569	-21	kthreadcore	20:09:25	7
2950897	99	35	31,2	cyclictest	3518279	-21	iostat_ios	23:24:23	7
2950888	99	35	33,2	cyclictest	3589706	-21	kworker/u16:2+events_unbound	00:03:01	4
2950888	99	35	33,2	cyclictest	3456539	-21	kworker/u16:1+events_unbound	23:13:56	4
2950888	99	35	33,2	cyclictest	3195157	-21	kworker/u16:1+flush-8:0	22:15:11	4
2950888	99	35	32,2	cyclictest	3602344	-21	kworker/u16:4+events_unbound	00:28:33	4
2950888	99	35	32,2	cyclictest	3148457	-21	kworker/u16:3+events_unbound	21:39:36	4
2950897	99	34	33,0	cyclictest	0	-21	swapper/7	22:16:52	7
2950897	99	34	32,1	cyclictest	3697468	-21	ssh	00:35:00	7
2950897	99	34	32,1	cyclictest	3674734	-21	ssh	00:28:00	7
2950897	99	34	30,3	cyclictest	3359807	-21	df	22:19:17	7
2950888	99	34	32,2	cyclictest	3602344	-21	kworker/u16:4+events_unbound	00:21:17	4
2950888	99	34	32,2	cyclictest	3456539	-21	kworker/u16:1+flush-8:0	23:19:30	4
2950888	99	34	32,2	cyclictest	3384431	-21	kworker/u16:2+events_unbound	22:31:53	4
2950888	99	34	32,2	cyclictest	3199817	-21	kworker/u16:4+flush-8:0	22:09:30	4
2950888	99	34	32,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	22:56:52	4
2950888	99	34	32,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	21:54:39	4
2950888	99	34	32,2	cyclictest	3099321	-21	kworker/u16:0+flush-8:0	20:24:35	4
3339826	2	33	0,1	chrt	3339827	-21	kthreadcore	22:09:25	7
2950897	99	33	32,0	cyclictest	0	-21	swapper/7	23:55:45	7
2950888	99	33	31,2	cyclictest	3602344	-21	kworker/u16:4+events_unbound	00:14:31	4
2950888	99	33	31,2	cyclictest	3517405	-21	kworker/u16:0+events_unbound	23:40:56	4
2950888	99	33	31,2	cyclictest	3456540	-21	kworker/u16:3+events_unbound	23:51:08	4
2950888	99	33	31,2	cyclictest	3456539	-21	kworker/u16:1+events_unbound	23:48:55	4
2950888	99	33	31,2	cyclictest	3399022	-21	kworker/u16:1+flush-8:0	22:38:44	4
2950888	99	33	31,2	cyclictest	3346654	-21	kworker/u16:0+flush-8:0	22:52:52	4
2950888	99	33	31,2	cyclictest	3346654	-21	kworker/u16:0+flush-8:0	22:52:52	4
2950888	99	33	30,2	cyclictest	3167997	-21	kworker/u16:0+events_unbound	20:59:23	4
2950888	99	33	30,2	cyclictest	2877686	-21	kworker/u16:3+events_unbound	20:04:22	4
2950888	99	33	30,2	cyclictest	2877686	-21	kworker/u16:3+events_unbound	20:04:22	4
2950897	99	32	31,0	cyclictest	0	-21	swapper/7	00:07:59	7
2950897	99	32	30,1	cyclictest	3648668	-21	ssh	00:14:43	7
2950897	99	32	30,1	cyclictest	0	-21	swapper/7	21:24:48	7
2950888	99	32	30,2	cyclictest	3602344	-21	kworker/u16:4+events_unbound	00:12:24	4
2950888	99	32	30,2	cyclictest	3602344	-21	kworker/u16:4+events_unbound	00:12:24	4
2950888	99	32	30,2	cyclictest	3248543	-21	kworker/u16:0+events_unbound	21:37:33	4
2950888	99	32	30,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	23:18:45	4
2950897	99	31	28,2	cyclictest	3575850	-21	ssh	23:45:16	7
2950888	99	31	29,2	cyclictest	3638836	-21	kworker/u16:0+events_unbound	00:33:24	4
2950888	99	31	29,2	cyclictest	3051493	-21	kworker/u16:0+events_unbound	20:09:37	4
2950897	99	30	29,0	cyclictest	0	-21	swapper/7	21:04:22	7
2950897	99	30	27,2	cyclictest	3453660	-21	ssh	22:54:39	7
2950897	99	30	25,3	cyclictest	3399021	-21	cat	22:34:25	7
2950888	99	30	28,2	cyclictest	3199817	-21	kworker/u16:4+events_unbound	22:27:51	4
2950897	99	28	26,1	cyclictest	3393907	-21	ssh	22:31:28	7
2950897	99	28	23,3	cyclictest	3442512	-21	ssh	22:51:16	7
2950897	99	28	23,3	cyclictest	3442512	-21	ssh	22:51:16	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17