OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails

Click here to display the system's profile data.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 100 highest latencies:

System rackeslot8s.osadl.org (updated Sat Jul 12, 2025 00:43:50)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
2101195	99	754	748,5	cyclictest	2402238	-21	kworker/1:2+events	23:59:13	1
2101195	99	753	747,5	cyclictest	2215492	-21	kworker/1:1+events	21:10:20	1
2101195	99	752	746,5	cyclictest	2215492	-21	kworker/1:1+events	21:28:42	1
2101195	99	750	747,2	cyclictest	2215492	-21	kworker/1:1+events	21:44:17	1
2101195	99	750	746,3	cyclictest	2215492	-21	kworker/1:1+events	21:18:49	1
2101195	99	750	745,4	cyclictest	2215492	-21	kworker/1:1+events	21:45:44	1
2101195	99	747	741,5	cyclictest	2364636	-21	turbostat	22:44:56	1
2101195	99	747	739,3	cyclictest	2215492	-21	kworker/1:1+events	21:50:23	1
2101195	99	747	739,3	cyclictest	2215492	-21	kworker/1:1+events	21:50:22	1
2101195	99	745	740,4	cyclictest	2402238	-21	kworker/1:2+events	23:34:50	1
2101195	99	745	740,4	cyclictest	2402238	-21	kworker/1:2+events	00:27:47	1
2101195	99	745	739,5	cyclictest	2402238	-21	kworker/1:2+events	23:44:44	1
2101195	99	744	740,3	cyclictest	2402238	-21	kworker/1:2+events	00:36:28	1
2101195	99	744	739,4	cyclictest	2402238	-21	kworker/1:2+events	23:48:50	1
2101195	99	743	739,3	cyclictest	2215492	-21	kworker/1:1+events	21:00:44	1
2101195	99	743	738,4	cyclictest	2215492	-21	kworker/1:1+events	21:34:49	1
2101201	99	742	738,3	cyclictest	2088346	-21	kworker/3:0+events	20:00:21	3
2101195	99	742	738,3	cyclictest	2402238	-21	kworker/1:2+events	00:34:54	1
2101195	99	742	737,4	cyclictest	4186258	-21	kworker/1:1+events	19:27:40	1
2101195	99	742	737,4	cyclictest	2184461	-21	kworker/1:2+events	20:30:21	1
2101195	99	742	731,6	cyclictest	4186258	-21	kworker/1:1+events	19:14:08	1
2101195	99	741	737,3	cyclictest	4186258	-21	kworker/1:1+events	20:15:45	1
2101195	99	741	737,3	cyclictest	4186258	-21	kworker/1:1+events	19:19:11	1
2101195	99	741	737,3	cyclictest	2215492	-21	kworker/1:1+events	20:52:57	1
2101195	99	741	736,4	cyclictest	4186258	-21	kworker/1:1+events	19:24:10	1
2101195	99	740	736,3	cyclictest	4186258	-21	kworker/1:1+events	19:32:55	1
2101195	99	740	732,4	cyclictest	2402238	-21	kworker/1:2+events	23:51:58	1
2101195	99	739	735,3	cyclictest	2402238	-21	kworker/1:2+events	23:35:22	1
2101195	99	739	735,3	cyclictest	2402238	-21	kworker/1:2+events	00:21:50	1
2101195	99	739	735,3	cyclictest	2402238	-21	kworker/1:2+events	00:16:06	1
2101195	99	739	735,3	cyclictest	2184461	-21	kworker/1:2+events	20:26:06	1
2101195	99	739	735,3	cyclictest	2184461	-21	kworker/1:2+events	20:21:28	1
2101195	99	739	734,4	cyclictest	2215492	-21	kworker/1:1+events	21:05:53	1
2101195	99	739	733,5	cyclictest	2215492	-21	kworker/1:1+events	22:48:16	1
2101201	99	738	730,6	cyclictest	2088346	-21	kworker/3:0+events	19:55:21	3
2101195	99	738	734,3	cyclictest	2215492	-21	kworker/1:1+events	22:10:22	1
2101195	99	738	734,3	cyclictest	2215492	-21	kworker/1:1+events	20:57:41	1
2101201	99	737	729,7	cyclictest	2171783	-21	kworker/3:1+events	20:40:21	3
2101201	99	736	733,2	cyclictest	2265061	-21	kworker/3:0+events	22:25:24	3
2101195	99	735	731,3	cyclictest	2215492	-21	kworker/1:1+events	21:37:28	1
2101201	99	731	725,5	cyclictest	2265061	-21	kworker/3:0+events	22:18:38	3
2101197	99	730	727,2	cyclictest	2314951	-21	kworker/2:2+events	00:11:04	2
2101201	99	729	724,4	cyclictest	2265061	-21	kworker/3:0+events	22:38:35	3
2101197	99	729	724,4	cyclictest	2147175	-21	kworker/2:0+events	21:20:22	2
2101201	99	726	722,3	cyclictest	2265061	-21	kworker/3:0+events	22:50:23	3
2101201	99	724	720,3	cyclictest	2265061	-21	kworker/3:0+events	22:41:07	3
2101201	99	724	720,3	cyclictest	2265061	-21	kworker/3:0+events	21:59:54	3
2101201	99	723	719,3	cyclictest	2265061	-21	kworker/3:0+events	22:58:46	3
2101201	99	721	714,3	cyclictest	2265061	-21	kworker/3:0+events	23:25:41	3
2101201	99	720	717,2	cyclictest	2265061	-21	kworker/3:0+events	23:24:22	3
2101201	99	720	716,3	cyclictest	2171783	-21	kworker/3:1+events	20:46:39	3
2101201	99	720	716,3	cyclictest	2088346	-21	kworker/3:0+events	19:51:26	3
2101201	99	720	716,3	cyclictest	2088346	-21	kworker/3:0+events	19:41:50	3
2101201	99	720	716,3	cyclictest	2088346	-21	kworker/3:0+events	19:41:49	3
2101201	99	720	714,5	cyclictest	2265061	-21	kworker/3:0+events	23:17:09	3
2101201	99	719	715,3	cyclictest	2088346	-21	kworker/3:0+events	19:46:25	3
2101195	99	719	716,2	cyclictest	2265025	-21	turbostat	21:24:55	1
2101201	99	718	714,3	cyclictest	2265061	-21	kworker/3:0+events	22:05:22	3
2101201	99	717	714,2	cyclictest	2265061	-21	kworker/3:0+events	22:21:46	3
2101201	99	717	713,3	cyclictest	2088346	-21	kworker/3:0+events	19:39:27	3
2101201	99	715	711,3	cyclictest	2265061	-21	kworker/3:0+events	23:02:29	3
2101201	99	715	711,3	cyclictest	2171783	-21	kworker/3:1+events	20:39:46	3
2101201	99	714	709,3	cyclictest	2265061	-21	kworker/3:0+events	23:11:58	3
2101201	99	713	710,2	cyclictest	2265061	-21	kworker/3:0+events	23:07:00	3
2101201	99	713	710,2	cyclictest	2265061	-21	kworker/3:0+events	22:31:26	3
2101197	99	712	709,2	cyclictest	2458153	-21	turbostat	23:59:55	2
2101197	99	711	705,5	cyclictest	2314951	-21	kworker/2:2+events	00:00:23	2
2101201	99	710	706,3	cyclictest	2265061	-21	kworker/3:0+events	22:02:12	3
2101201	99	709	704,4	cyclictest	2508400	-21	turbostat	00:39:55	3
2101197	99	708	701,5	cyclictest	2314951	-21	kworker/2:2+events	00:05:22	2
2101191	99	703	699,3	cyclictest	2165793	-21	turbostat	20:04:55	0
2101191	99	701	694,6	cyclictest	2165510	-21	kworker/0:1+events	20:10:20	0
2101195	99	700	696,3	cyclictest	2420859	-21	turbostat	23:29:54	1
2101195	99	683	678,4	cyclictest	2221669	-21	turbostat	20:49:56	1
2101201	99	677	674,2	cyclictest	2128542	-21	turbostat	19:34:56	3
2101201	99	677	673,3	cyclictest	2370853	-21	turbostat	22:49:55	3
2101191	99	674	671,2	cyclictest	2165510	-21	kworker/0:1+events	20:06:50	0
2101201	99	659	656,2	cyclictest	2327322	-21	turbostat	22:14:56	3
2101201	99	657	655,1	cyclictest	2203054	-21	turbostat	20:34:56	3
2101195	99	655	652,2	cyclictest	2178192	-21	turbostat	20:14:55	1
2101201	99	654	651,2	cyclictest	2302512	-21	turbostat	21:54:56	3
2101201	99	654	651,2	cyclictest	2302512	-21	turbostat	21:54:55	3
2101195	99	651	648,2	cyclictest	2476853	-21	turbostat	00:14:55	1
2101197	99	648	644,3	cyclictest	2258809	-21	turbostat	21:19:54	2
2101195	99	623	618,4	cyclictest	2321116	-21	turbostat	22:09:55	1
2101201	99	203	0,201	cyclictest	2246136	-21	taskset	21:01:17	3
2469784	2	74	0,0	sleep0	2469787	-21	grep	00:00:22	0
2101201	99	28	25,2	cyclictest	2240234	-21	sendmail	21:00:00	3
2101191	99	24	20,3	cyclictest	745	-21	cups-browsed	22:05:01	0
2101201	99	23	20,2	cyclictest	0	-21	swapper/3	00:00:00	3
2101197	99	22	3,7	cyclictest	2489846	-21	grep	00:20:11	2
2101197	99	22	3,7	cyclictest	2346448	-21	cat	22:25:14	2
2101197	99	22	2,8	cyclictest	2389904	-21	ls	23:00:00	2
2101197	99	21	5,5	cyclictest	2456701	-21	grep	23:50:20	2
2101197	99	21	5,5	cyclictest	2172915	-21	cut	20:05:19	2
2101197	99	21	4,5	cyclictest	2153581	-21	latency_hist	19:50:00	2
2101197	99	21	2,7	cyclictest	2382792	-21	processes	22:50:25	2
2101197	99	21	2,7	cyclictest	2239071	-21	cut	20:55:18	2
2101197	99	21	18,2	cyclictest	2497740	-21	kthreadcore	00:25:18	2
2101201	99	20	15,3	cyclictest	2107658	-21	latency_hist	19:15:00	3
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2022-03-28 - 20:20