OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails

Click here to display the system's profile data.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackfslot7s.osadl.org (updated Sun Dec 14, 2025 00:44:16)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
596046	99	3544	18,3518	cyclictest	686985	-21	kworker/3:1+events	00:10:17	3
596038	99	3068	15,3050	cyclictest	615389	-21	kworker/1:1+events	20:10:16	1
596046	99	3037	15,3018	cyclictest	638419	-21	kworker/3:2+events	21:30:14	3
596046	99	2858	12,2843	cyclictest	665815	-21	kworker/3:1+events	23:10:14	3
596042	99	2700	13,2684	cyclictest	662534	-21	kworker/2:1+events	22:35:01	2
596035	99	2690	13,2672	cyclictest	608755	-21	kworker/0:1+events	20:15:13	0
596046	99	2649	10,2635	cyclictest	638419	-21	kworker/3:2+events	21:25:16	3
596035	99	2601	9,2588	cyclictest	661138	-21	kworker/0:1+events	22:40:19	0
596038	99	2542	10,2529	cyclictest	660955	-21	kworker/1:0+events	22:42:35	1
596038	99	2493	12,2478	cyclictest	630608	-21	kworker/1:2+events	20:58:28	1
596042	99	2490	12,2475	cyclictest	644500	-21	kworker/2:0+events	21:46:54	2
596038	99	2488	12,2473	cyclictest	660955	-21	kworker/1:0+events	22:28:52	1
596038	99	2487	12,2472	cyclictest	649292	-21	kworker/1:0+events	21:55:58	1
596046	99	2486	11,2472	cyclictest	657470	-21	kworker/3:2+events	22:17:48	3
596046	99	2480	12,2464	cyclictest	652765	-21	kworker/3:1+events	22:10:14	3
596038	99	2462	11,2447	cyclictest	680452	-21	kworker/1:1+events	23:30:51	1
596035	99	2430	9,2417	cyclictest	604169	-21	kworker/0:0+events	19:35:18	0
596042	99	2428	12,2413	cyclictest	631982	-21	kworker/2:1+events	21:15:17	2
596035	99	2415	11,2400	cyclictest	663212	-21	kworker/0:2+events	23:05:13	0
596042	99	2411	12,2395	cyclictest	638175	-21	kworker/2:2+events	21:20:17	2
596046	99	2401	12,2385	cyclictest	689969	-21	kworker/3:3+events	00:05:18	3
596035	99	2397	11,2383	cyclictest	680180	-21	kworker/0:1+events	23:30:28	0
596035	99	2386	12,2371	cyclictest	633399	-21	kworker/0:0+events	21:13:03	0
596046	99	2361	12,2346	cyclictest	678551	-21	kworker/3:0+events	23:29:41	3
596046	99	2331	11,2318	cyclictest	685117	-21	kworker/3:2+events	23:50:17	3
596046	99	2321	12,2305	cyclictest	678551	-21	kworker/3:0+events	23:48:24	3
596046	99	2317	11,2302	cyclictest	657470	-21	kworker/3:2+events	22:22:23	3
596038	99	2311	11,2296	cyclictest	695946	-21	kworker/1:0+events	00:26:37	1
596038	99	2311	11,2296	cyclictest	695946	-21	kworker/1:0+events	00:26:37	1
596038	99	2311	10,2298	cyclictest	630608	-21	kworker/1:2+events	21:16:49	1
596042	99	2310	11,2296	cyclictest	675652	-21	kworker/2:0+events	23:32:45	2
596038	99	2310	11,2295	cyclictest	623179	-21	kworker/1:0+events	20:35:37	1
596038	99	2309	9,2296	cyclictest	630608	-21	kworker/1:2+events	21:01:11	1
596038	99	2306	9,2294	cyclictest	650977	-21	kworker/1:2+events	22:01:19	1
596038	99	2306	10,2293	cyclictest	683308	-21	kworker/1:2+events	23:41:35	1
596038	99	2305	11,2291	cyclictest	618646	-21	kworker/1:2+events	20:20:26	1
596038	99	2304	9,2292	cyclictest	638453	-21	kworker/1:0+events	21:32:55	1
596038	99	2304	10,2290	cyclictest	655579	-21	kworker/1:2+events	22:14:47	1
596046	99	2303	11,2289	cyclictest	597147	-21	kworker/3:2+events	19:30:46	3
596042	99	2303	12,2288	cyclictest	685027	-21	kworker/2:2+events	23:51:50	2
596038	99	2303	10,2290	cyclictest	657440	-21	kworker/1:0+events	22:20:28	1
596038	99	2302	9,2290	cyclictest	646355	-21	kworker/1:1+events	21:47:39	1
596038	99	2302	11,2288	cyclictest	605530	-21	kworker/1:2+events	19:52:01	1
596038	99	2302	11,2287	cyclictest	603900	-21	kworker/1:1+events	19:36:49	1
596038	99	2300	9,2289	cyclictest	695946	-21	kworker/1:0+events	00:33:07	1
596038	99	2300	11,2286	cyclictest	627071	-21	kworker/1:1+events	20:44:49	1
596038	99	2300	11,2286	cyclictest	618646	-21	kworker/1:2+events	20:29:37	1
596038	99	2300	10,2287	cyclictest	607433	-21	kworker/1:0+events	19:49:21	1
596038	99	2299	10,2286	cyclictest	692743	-21	kworker/1:2+events	00:06:05	1
596038	99	2299	10,2285	cyclictest	613526	-21	kworker/1:2+events	20:08:02	1
596038	99	2297	9,2285	cyclictest	662564	-21	kworker/1:2+events	22:35:45	1
596038	99	2296	11,2282	cyclictest	686656	-21	kworker/1:2+events	23:54:06	1
596038	99	2296	10,2283	cyclictest	677047	-21	kworker/1:0+events	23:27:43	1
596038	99	2296	10,2283	cyclictest	653938	-21	kworker/1:1+events	22:15:31	1
596038	99	2296	10,2282	cyclictest	673751	-21	kworker/1:1+events	23:12:54	1
596038	99	2295	11,2281	cyclictest	680452	-21	kworker/1:1+events	23:55:38	1
596035	99	2282	9,2270	cyclictest	620062	-21	kworker/0:0+events	20:44:27	0
596038	99	2280	10,2267	cyclictest	667177	-21	kworker/1:2+events	22:55:11	1
596035	99	2272	10,2259	cyclictest	644589	-21	kworker/0:1+events	21:51:26	0
596038	99	2270	10,2257	cyclictest	627071	-21	kworker/1:1+events	20:46:44	1
596035	99	2268	11,2254	cyclictest	647868	-21	kworker/0:2+events	22:04:01	0
596035	99	2267	9,2254	cyclictest	655609	-21	kworker/0:0+events	22:13:15	0
596035	99	2266	11,2253	cyclictest	655609	-21	kworker/0:0+events	22:20:07	0
596035	99	2266	10,2253	cyclictest	615478	-21	kworker/0:0+events	20:11:24	0
596035	99	2265	11,2251	cyclictest	644589	-21	kworker/0:1+events	21:41:35	0
596035	99	2265	11,2250	cyclictest	604169	-21	kworker/0:0+events	19:57:44	0
596042	99	2264	1,2259	cyclictest	631982	-21	kworker/2:1+events	21:06:33	2
596035	99	2263	11,2249	cyclictest	661138	-21	kworker/0:1+events	22:35:23	0
596035	99	2261	11,2248	cyclictest	683428	-21	kworker/0:0+events	23:40:02	0
596035	99	2261	11,2247	cyclictest	644589	-21	kworker/0:1+events	21:46:31	0
596035	99	2260	9,2249	cyclictest	685203	-21	kworker/0:1+events	23:46:28	0
596035	99	2256	10,2243	cyclictest	696219	-21	kworker/0:0+events	00:30:46	0
596042	99	2248	7,2238	cyclictest	694610	-21	kworker/2:0+events	00:19:00	2
596035	99	2248	11,2234	cyclictest	633399	-21	kworker/0:0+events	21:06:56	0
596035	99	2244	12,2229	cyclictest	638360	-21	kworker/0:1+events	21:27:14	0
596046	99	2242	11,2228	cyclictest	694405	-21	kworker/3:0+events	00:31:34	3
596035	99	2241	11,2228	cyclictest	655609	-21	kworker/0:0+events	22:15:10	0
596046	99	2240	9,2229	cyclictest	630236	-21	kworker/3:1+events	20:55:02	3
596035	99	2238	11,2224	cyclictest	678583	-21	kworker/0:0+events	23:24:42	0
596035	99	2238	10,2224	cyclictest	663212	-21	kworker/0:2+events	22:53:14	0
596035	99	2237	9,2224	cyclictest	696219	-21	kworker/0:0+events	00:28:10	0
596035	99	2237	9,2224	cyclictest	696219	-21	kworker/0:0+events	00:28:10	0
596035	99	2237	11,2223	cyclictest	629034	-21	kworker/0:1+events	20:55:27	0
596035	99	2237	10,2224	cyclictest	702331	-21	kworker/0:2+events	00:37:40	0
596035	99	2235	9,2223	cyclictest	604169	-21	kworker/0:0+events	19:47:28	0
596038	99	2234	10,2221	cyclictest	649292	-21	kworker/1:0+events	22:09:27	1
596035	99	2234	10,2221	cyclictest	657562	-21	kworker/0:2+events	22:28:07	0
596035	99	2232	10,2218	cyclictest	661138	-21	kworker/0:1+events	22:31:56	0
596046	99	2228	11,2214	cyclictest	657470	-21	kworker/3:2+events	22:46:45	3
596042	99	2226	9,2215	cyclictest	698624	-21	kworker/2:2+events	00:30:01	2
596046	99	2223	11,2209	cyclictest	654545	-21	kworker/3:0+events	22:28:29	3
596035	99	2220	11,2206	cyclictest	620062	-21	kworker/0:0+events	20:31:11	0
596038	99	2219	9,2207	cyclictest	603900	-21	kworker/1:1+events	19:33:49	1
596035	99	2215	10,2202	cyclictest	620062	-21	kworker/0:0+events	20:22:44	0
596035	99	2210	10,2196	cyclictest	688138	-21	kworker/0:0+events	23:53:43	0
596038	99	2208	10,2195	cyclictest	667177	-21	kworker/1:2+events	23:00:58	1
596038	99	2198	11,2184	cyclictest	628959	-21	kworker/1:0+events	20:50:33	1
596038	99	2194	10,2181	cyclictest	695946	-21	kworker/1:0+events	00:22:46	1
596046	99	2191	10,2178	cyclictest	633581	-21	kworker/3:1+events	21:03:29	3
596046	99	2191	10,2177	cyclictest	678551	-21	kworker/3:0+events	23:35:47	3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2024-07-29 - 23:00