OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rack8slot6s.osadl.org (updated Tue Aug 25, 2026 12:43:40)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1253	99	1052	1050,2	cyclictest	27230	-21	kworker/2:0+events	10:18:25	2
1249	99	1052	1050,2	cyclictest	29112	-21	kworker/1:1+events	09:36:52	1
1249	99	1051	1049,2	cyclictest	8969	-21	kworker/1:0+events	09:49:23	1
1249	99	1051	1049,2	cyclictest	8969	-21	kworker/1:0+events	09:49:22	1
1249	99	1051	1049,2	cyclictest	1654	-21	kworker/1:2+events	09:28:20	1
1249	99	1050	1048,2	cyclictest	14931	-21	kworker/1:2+events	09:44:46	1
1249	99	1050	1048,2	cyclictest	14931	-21	kworker/1:2+events	09:44:46	1
1249	99	1050	1047,3	cyclictest	885	-21	kworker/1:2+events	09:12:34	1
1253	99	1049	1047,2	cyclictest	27230	-21	kworker/2:0+events	10:07:11	2
1253	99	1049	1047,2	cyclictest	21182	-21	kworker/2:2+events	10:23:18	2
1261	99	1048	1046,2	cyclictest	7238	-21	kworker/3:1+events	07:38:40	3
1253	99	1048	1046,2	cyclictest	14314	-21	kworker/2:0+events	12:38:12	2
1253	99	1047	1045,2	cyclictest	27230	-21	kworker/2:0+events	10:03:20	2
1253	99	1045	1043,2	cyclictest	27230	-21	kworker/2:0+events	10:11:50	2
1261	99	1043	1040,3	cyclictest	31648	-21	turbostat	09:29:55	3
1253	99	1037	1035,2	cyclictest	27230	-21	kworker/2:0+events	10:37:23	2
1249	99	1035	1032,3	cyclictest	8969	-21	kworker/1:0+events	09:51:07	1
1261	99	1030	1029,1	cyclictest	3587	-21	kworker/3:2+events	12:10:55	3
1261	99	1030	1027,3	cyclictest	14503	-21	kworker/3:0+events	12:05:21	3
1261	99	1029	1028,1	cyclictest	4880	-21	kworker/3:3+events	11:50:43	3
1261	99	1029	1027,2	cyclictest	25771	-21	kworker/3:0+events	08:16:05	3
1261	99	1027	1025,2	cyclictest	8895	-21	kworker/3:0+events	12:32:23	3
1261	99	1027	1025,2	cyclictest	4324	-21	kworker/3:1+events	11:05:43	3
1261	99	1027	1025,2	cyclictest	4324	-21	kworker/3:1+events	10:46:06	3
1261	99	1027	1024,3	cyclictest	3217	-21	kworker/3:1+events	08:32:09	3
1261	99	1026	1024,2	cyclictest	30239	-21	kworker/3:2+events	11:36:37	3
1249	99	1026	1024,2	cyclictest	885	-21	kworker/1:2+events	09:15:43	1
1261	99	1025	1023,2	cyclictest	30239	-21	kworker/3:2+events	11:12:44	3
1261	99	1025	1021,2	cyclictest	4880	-21	kworker/3:3+events	11:55:20	3
1261	99	1024	1022,2	cyclictest	3217	-21	kworker/3:1+events	08:36:16	3
1261	99	1024	1022,2	cyclictest	13490	-21	kworker/3:2+events	09:02:12	3
1261	99	1023	1021,2	cyclictest	3587	-21	kworker/3:2+events	12:27:17	3
1261	99	1023	1021,2	cyclictest	25771	-21	kworker/3:0+events	08:12:59	3
1261	99	1022	1020,2	cyclictest	7238	-21	kworker/3:1+events	07:26:46	3
1261	99	1022	1020,2	cyclictest	25771	-21	kworker/3:0+events	08:27:09	3
1261	99	1021	1018,2	cyclictest	31480	-21	kworker/3:0+events	12:20:21	3
1261	99	1020	1018,2	cyclictest	26550	-21	kworker/3:1+events	12:15:21	3
1261	99	1020	1018,2	cyclictest	14503	-21	kworker/3:0+events	12:02:10	3
1261	99	1020	1017,3	cyclictest	15786	-21	kworker/3:0+events	11:03:52	3
1261	99	1020	1017,3	cyclictest	13490	-21	kworker/3:2+events	08:58:17	3
1261	99	1020	1016,3	cyclictest	9391	-21	kworker/3:1+events	08:49:19	3
1253	99	1020	1018,2	cyclictest	21182	-21	kworker/2:2+events	10:29:24	2
1261	99	1019	1017,2	cyclictest	25783	-21	kworker/3:1+events	11:40:22	3
1261	99	1019	1017,2	cyclictest	21652	-21	kworker/3:2+events	08:09:26	3
1261	99	1019	1016,3	cyclictest	4324	-21	kworker/3:1+events	10:58:29	3
1261	99	1019	1016,3	cyclictest	13490	-21	kworker/3:2+events	09:05:21	3
1261	99	1018	1016,2	cyclictest	5278	-21	kworker/3:2+events	08:42:21	3
1261	99	1018	1016,2	cyclictest	3799	-21	kworker/3:2+events	09:34:11	3
1261	99	1017	1015,2	cyclictest	3799	-21	kworker/3:2+events	10:41:49	3
1261	99	1017	1015,2	cyclictest	15500	-21	kworker/3:0+events	07:47:57	3
1261	99	1015	1013,2	cyclictest	21012	-21	kworker/3:0+events	11:47:32	3
1261	99	1015	1012,3	cyclictest	10911	-21	kworker/3:0+events	07:34:50	3
1249	99	1015	1013,2	cyclictest	21852	-21	turbostat	08:04:55	1
1261	99	1014	1012,2	cyclictest	9391	-21	kworker/3:1+events	08:53:19	3
1249	99	1010	1008,2	cyclictest	14380	-21	turbostat	12:39:55	1
1261	99	1009	1003,3	cyclictest	7238	-21	kworker/3:1+events	07:40:23	3
1261	99	1008	1006,2	cyclictest	19606	-21	kworker/3:1+events	07:55:52	3
1261	99	1008	1006,2	cyclictest	15500	-21	kworker/3:0+events	07:50:22	3
1261	99	1007	1005,2	cyclictest	15786	-21	kworker/3:0+events	10:52:54	3
1261	99	1007	1004,2	cyclictest	19606	-21	kworker/3:1+events	08:01:29	3
1261	99	1007	1003,2	cyclictest	29868	-21	kworker/3:1+events	08:20:22	3
1247	99	997	995,2	cyclictest	18508	-21	turbostat	11:34:54	0
1249	99	993	992,1	cyclictest	17701	-21	kworker/1:0+events	09:22:13	1
1247	99	993	992,1	cyclictest	3293	-21	kworker/0:2+events	11:21:04	0
1247	99	993	991,2	cyclictest	5233	-21	turbostat	07:24:55	0
1247	99	978	975,3	cyclictest	3859	-21	turbostat	09:34:55	0
1261	99	975	973,2	cyclictest	13900	-21	turbostat	11:29:54	3
1261	99	966	963,3	cyclictest	22712	-21	turbostat	09:54:55	3
1247	99	961	960,1	cyclictest	3293	-21	kworker/0:2+events	11:15:44	0
1249	99	960	957,3	cyclictest	17937	-21	turbostat	10:24:56	1
1247	99	957	956,1	cyclictest	3293	-21	kworker/0:2+events	11:29:34	0
1253	99	954	952,2	cyclictest	27230	-21	kworker/2:0+events	09:56:26	2
1247	99	954	952,2	cyclictest	14105	-21	turbostat	12:04:54	0
1249	99	950	947,3	cyclictest	32163	-21	turbostat	10:39:54	1
1247	99	947	946,1	cyclictest	27181	-21	kworker/0:1+events	07:12:35	0
1247	99	946	945,1	cyclictest	27181	-21	kworker/0:1+events	07:18:41	0
1249	99	937	935,2	cyclictest	20277	-21	kworker/1:1+events	11:32:33	1
1261	99	936	934,2	cyclictest	8448	-21	turbostat	09:39:55	3
1247	99	920	917,3	cyclictest	17818	-21	turbostat	07:54:55	0
1249	99	907	906,1	cyclictest	7641	-21	turbostat	08:49:55	1
1247	99	887	885,2	cyclictest	11672	-21	turbostat	08:59:55	0
1253	99	885	884,1	cyclictest	21182	-21	kworker/2:2+events	10:30:23	2
1249	99	866	863,3	cyclictest	19830	-21	turbostat	07:59:55	1
1249	99	666	662,3	cyclictest	3859	-21	turbostat	09:34:55	1
1261	99	647	644,3	cyclictest	32163	-21	turbostat	10:39:55	3
1247	99	632	630,2	cyclictest	14380	-21	turbostat	12:39:55	0
1261	99	626	624,2	cyclictest	18508	-21	turbostat	11:34:55	3
1261	99	623	620,3	cyclictest	5233	-21	turbostat	07:24:55	3
1249	99	589	585,4	cyclictest	13900	-21	turbostat	11:29:54	1
1253	99	586	583,3	cyclictest	22712	-21	turbostat	09:54:55	2
1249	99	585	584,1	cyclictest	15716	-21	turbostat	09:09:55	1
1247	99	570	567,3	cyclictest	32450	-21	turbostat	11:14:54	0
1253	99	524	523,1	cyclictest	9350	-21	turbostat	12:34:55	2
1247	99	26	26,0	cyclictest	30679	-21	fwupd	09:58:00	0
1249	99	19	18,1	cyclictest	677	50	irq/123-lan	07:20:20	1
1249	99	18	18,0	cyclictest	677	50	irq/123-lan	10:55:15	1
1249	99	18	17,1	cyclictest	677	50	irq/123-lan	12:25:18	1
1249	99	18	17,1	cyclictest	677	50	irq/123-lan	11:47:54	1
1249	99	18	17,1	cyclictest	677	50	irq/123-lan	10:14:19	1
1249	99	18	17,1	cyclictest	677	50	irq/123-lan	08:30:24	1
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17