OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of system in rack #0, slot #2

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack0slot2.osadl.org (updated Tue Jul 07, 2026 12:43:34)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1121912	2	95	0,1	sleep7	1121913	-21	rm	10:35:54	7
651882	2	49	15,29	sleep7	0	-21	swapper/7	07:08:17	7
652488	99	48	43,4	cyclictest	1321508	-21	latency_hist	11:59:57	7
652488	99	44	42,1	cyclictest	0	-21	swapper/7	12:06:31	7
652488	99	38	34,2	cyclictest	760593	-21	munin-run	07:59:57	7
6278	89	38	0,31	rtkit-daemon	0	-21	swapper/5	07:05:02	5
652484	99	36	34,2	cyclictest	719217	-21	kworker/u16:1+events_unbound	08:09:57	6
652480	99	35	33,2	cyclictest	961544	-21	kworker/u16:1+flush-8:0	10:24:56	5
652480	99	35	33,2	cyclictest	961544	-21	kworker/u16:1+events_unbound	09:58:13	5
652484	99	34	32,2	cyclictest	961544	-21	kworker/u16:1+flush-8:0	10:09:52	6
652480	99	34	29,2	cyclictest	1112176	-21	kworker/u16:0+events_unbound	10:38:09	5
652480	99	33	31,2	cyclictest	939597	-21	kworker/u16:0+events_unbound	09:46:28	5
652480	99	33	31,2	cyclictest	1062939	-21	kworker/u16:3+events_unbound	10:43:08	5
652488	99	32	28,3	cyclictest	914456	-21	latency_hist	09:14:57	7
652488	99	32	27,4	cyclictest	1296792	-21	latency_hist	11:49:57	7
652488	99	32	27,4	cyclictest	1296792	-21	latency_hist	11:49:57	7
652484	99	32	30,2	cyclictest	782690	-21	kworker/u16:3+flush-8:0	08:54:57	6
652480	99	32	30,2	cyclictest	1371636	-21	kworker/u16:2+events_unbound	12:28:48	5
652480	99	32	30,2	cyclictest	1371636	-21	kworker/u16:2+events_unbound	12:28:48	5
652480	99	32	30,2	cyclictest	1343958	-21	kworker/u16:4+flush-8:0	12:18:05	5
652480	99	32	30,2	cyclictest	1343958	-21	kworker/u16:4+events_unbound	12:14:48	5
652480	99	32	30,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	11:21:17	5
652488	99	31	30,0	cyclictest	0	-21	swapper/7	10:26:24	7
652488	99	31	28,2	cyclictest	1383251	-21	ssh	12:21:44	7
652488	99	31	28,2	cyclictest	1210886	-21	ssh	11:15:01	7
652484	99	31	27,3	cyclictest	781670	-21	kworker/u16:2+flush-8:0	08:10:12	6
652480	99	31	29,2	cyclictest	939597	-21	kworker/u16:0+events_unbound	09:52:33	5
652480	99	31	29,2	cyclictest	939597	-21	kworker/u16:0+events_unbound	09:40:44	5
652480	99	31	29,2	cyclictest	939597	-21	kworker/u16:0+events_unbound	09:40:44	5
652480	99	31	29,2	cyclictest	924374	-21	kworker/u16:4+events_unbound	09:18:33	5
652480	99	31	29,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	11:40:41	5
652480	99	31	29,2	cyclictest	1133961	-21	kworker/u16:1+flush-8:0	10:54:57	5
652480	99	31	28,2	cyclictest	1013168	-21	kworker/u16:5+events_unbound	10:05:00	5
652488	99	30	29,0	cyclictest	0	-21	swapper/7	10:09:59	7
652488	99	30	29,0	cyclictest	0	-21	swapper/7	09:44:09	7
652488	99	30	29,0	cyclictest	0	-21	swapper/7	09:44:09	7
652488	99	30	29,0	cyclictest	0	-21	swapper/7	09:27:12	7
652488	99	30	28,1	cyclictest	1369994	-21	ssh	12:16:08	7
652488	99	30	27,2	cyclictest	1011702	-21	ssh	09:52:28	7
652480	99	30	29,1	cyclictest	1133961	-21	kworker/u16:1+events_unbound	11:19:25	5
652480	99	30	28,2	cyclictest	924374	-21	kworker/u16:4+events_unbound	09:28:33	5
652480	99	30	28,2	cyclictest	1397860	-21	kworker/u16:3+events_unbound	12:36:45	5
652480	99	30	28,2	cyclictest	1397860	-21	kworker/u16:3+events_unbound	12:32:19	5
652480	99	30	28,2	cyclictest	1333793	-21	kworker/u16:1+events_unbound	12:22:09	5
652480	99	30	28,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	11:36:04	5
652480	99	30	28,2	cyclictest	1133961	-21	kworker/u16:1+flush-8:0	10:55:37	5
652480	99	30	28,2	cyclictest	1133961	-21	kworker/u16:1+events_unbound	11:48:48	5
652480	99	30	28,2	cyclictest	1133961	-21	kworker/u16:1+events_unbound	11:48:48	5
652480	99	30	28,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	11:53:04	5
652480	99	30	28,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	10:46:41	5
652478	99	30	28,2	cyclictest	1133961	-21	kworker/u16:1+flush-8:0	10:50:28	4
652059	2	30	20,7	sleep6	0	-21	swapper/6	07:09:13	6
652488	99	29	28,0	cyclictest	0	-21	swapper/7	12:12:05	7
652488	99	29	28,0	cyclictest	0	-21	swapper/7	11:35:37	7
652488	99	29	28,0	cyclictest	0	-21	swapper/7	10:21:17	7
652488	99	29	27,1	cyclictest	0	-21	swapper/7	09:38:33	7
652488	99	29	26,2	cyclictest	1233543	-21	ssh	11:22:04	7
652480	99	29	27,2	cyclictest	961544	-21	kworker/u16:1+events_unbound	09:32:04	5
652480	99	29	27,2	cyclictest	782690	-21	kworker/u16:3+events_unbound	09:37:30	5
652480	99	29	27,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	12:02:52	5
652480	99	29	27,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	11:04:09	5
652480	99	29	27,2	cyclictest	1062939	-21	kworker/u16:3+events_unbound	10:27:22	5
652480	99	29	27,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	11:06:16	5
652480	99	29	27,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	11:06:16	5
652480	99	29	27,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	10:17:24	5
652480	99	29	27,2	cyclictest	1013168	-21	kworker/u16:5+events_unbound	10:12:29	5
652488	99	28	27,0	cyclictest	0	-21	swapper/7	12:04:48	7
652488	99	28	27,0	cyclictest	0	-21	swapper/7	10:51:05	7
652488	99	28	26,1	cyclictest	0	-21	swapper/7	11:29:55	7
652488	99	28	25,2	cyclictest	1408781	-21	ssh	12:31:56	7
652488	99	28	23,4	cyclictest	1397995	-21	latency_hist	12:29:56	7
652488	99	28	23,4	cyclictest	1397995	-21	latency_hist	12:29:56	7
652480	99	28	26,2	cyclictest	719217	-21	kworker/u16:1+flush-8:0	08:20:26	5
652480	99	28	26,2	cyclictest	600455	-21	kworker/u16:3+flush-8:0	07:19:57	5
652480	99	28	26,2	cyclictest	1062939	-21	kworker/u16:3+events_unbound	10:31:17	5
652480	99	28	26,2	cyclictest	1013168	-21	kworker/u16:5+events_unbound	10:04:01	5
652478	99	28	26,2	cyclictest	1060244	-21	kworker/u16:2+flush-8:0	12:06:04	4
652478	99	28	26,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	11:50:12	4
652488	99	27	25,1	cyclictest	0	-21	swapper/7	09:31:56	7
652488	99	27	22,3	cyclictest	925147	-21	sh	09:17:05	7
652484	99	27	25,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	11:17:32	6
652480	99	27	25,2	cyclictest	892852	-21	kworker/u16:2+events_unbound	09:14:38	5
652480	99	27	25,2	cyclictest	697769	-21	kworker/u16:0+flush-8:0	08:29:57	5
652480	99	27	25,2	cyclictest	1161570	-21	kworker/u16:0+events_unbound	11:58:23	5
652480	99	27	25,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	11:12:00	5
652023	2	27	16,7	sleep4	0	-21	swapper/4	07:09:03	4
1291497	2	27	0,1	chrt	84	1	rcuc/6	11:45:17	6
1291497	2	27	0,1	chrt	84	1	rcuc/6	11:45:17	6
652488	99	26	25,0	cyclictest	0	-21	swapper/7	10:40:53	7
652488	99	26	25,0	cyclictest	0	-21	swapper/7	10:18:05	7
652488	99	26	24,1	cyclictest	936727	-21	ssh	09:20:40	7
652488	99	26	24,1	cyclictest	1050157	-21	ssh	10:09:51	7
652488	99	26	0,1	cyclictest	0	-21	swapper/7	07:10:21	7
652484	99	26	24,2	cyclictest	782690	-21	kworker/u16:3+events_unbound	09:40:40	6
652484	99	26	24,2	cyclictest	782690	-21	kworker/u16:3+events_unbound	09:40:40	6
652484	99	26	24,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	10:40:16	6
652484	99	26	24,2	cyclictest	1060244	-21	kworker/u16:2+events_unbound	10:27:32	6
652480	99	26	24,2	cyclictest	782690	-21	kworker/u16:3+events_unbound	09:22:41	5
652478	99	26	22,3	cyclictest	1397861	-21	kworker/u16:5+flush-8:0	12:30:09	4
652488	99	25	24,0	cyclictest	0	-21	swapper/7	11:02:41	7
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17