OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack9slot4s (updated Tue Dec 09, 2025 12:59:03)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
9187	99	1145	1109,32	cyclictest	9542	-21	tr	07:25:15	0
9188	99	1122	1093,24	cyclictest	9386	-21	aten_r4power_en	07:25:15	1
9188	99	1118	1078,20	cyclictest	7186	-21	smartctl	08:40:28	1
8937	2	834	741,77	sleep0	0	-21	swapper/0	07:23:02	0
9041	2	561	517,29	sleep1	0	-21	swapper/1	07:24:08	1
9187	99	242	223,15	cyclictest	0	-21	swapper/0	08:40:27	0
9177	2	120	0,8	sleep0	9187	99	cyclictest	12:21:54	0
27887	2	96	0,4	sleep1	21	1	ktimersoftd/1	11:00:30	1
27981	2	80	0,8	sleep0	9187	99	cyclictest	10:27:23	0
27809	2	79	0,9	sleep0	9187	99	cyclictest	12:08:00	0
23598	2	65	0,8	sleep0	9187	99	cyclictest	12:36:53	0
15328	2	57	0,2	sleep1	15329	-21	sh	10:48:00	1
23435	2	56	0,2	sleep0	23436	-21	awk	09:15:27	0
8100	2	50	0,8	sleep0	9187	99	cyclictest	11:14:29	0
9188	99	49	23,4	cyclictest	22	-21	ksoftirqd/1	12:28:10	1
6144	2	49	0,7	sleep0	9187	99	cyclictest	12:53:06	0
9188	99	48	11,15	cyclictest	22	-21	ksoftirqd/1	12:21:24	1
9188	99	47	5,4	cyclictest	8	1	rcu_preempt	11:31:09	1
9187	99	47	2,15	cyclictest	0	-21	swapper/0	11:59:04	0
9188	99	46	1,7	cyclictest	22	-21	ksoftirqd/1	11:58:12	1
9188	99	46	1,6	cyclictest	22	-21	ksoftirqd/1	10:07:07	1
9188	99	45	1,7	cyclictest	22	-21	ksoftirqd/1	10:33:53	1
9188	99	45	1,6	cyclictest	22	-21	ksoftirqd/1	12:46:41	1
9188	99	45	1,6	cyclictest	22	-21	ksoftirqd/1	10:20:57	1
10694	2	45	0,4	sleep1	22	-21	ksoftirqd/1	10:10:04	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	12:38:04	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	12:33:52	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	12:03:33	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	10:03:15	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	09:51:29	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	09:30:35	1
9188	99	44	1,7	cyclictest	22	-21	ksoftirqd/1	09:27:27	1
9188	99	44	1,6	cyclictest	22	-21	ksoftirqd/1	11:50:47	1
9188	99	44	1,6	cyclictest	22	-21	ksoftirqd/1	11:23:16	1
9188	99	43	1,7	cyclictest	22	-21	ksoftirqd/1	12:16:55	1
9188	99	43	1,7	cyclictest	22	-21	ksoftirqd/1	11:40:11	1
9188	99	43	1,7	cyclictest	22	-21	ksoftirqd/1	11:38:08	1
9188	99	43	1,7	cyclictest	22	-21	ksoftirqd/1	10:43:19	1
9188	99	43	1,7	cyclictest	22	-21	ksoftirqd/1	09:13:53	1
9188	99	43	1,6	cyclictest	22	-21	ksoftirqd/1	12:06:45	1
9188	99	42	1,7	cyclictest	22	-21	ksoftirqd/1	12:11:34	1
9188	99	42	1,6	cyclictest	22	-21	ksoftirqd/1	10:26:35	1
9188	99	42	1,6	cyclictest	22	-21	ksoftirqd/1	09:21:32	1
1610	2	42	0,5	sleep0	9187	99	cyclictest	11:40:30	0
9188	99	41	1,7	cyclictest	22	-21	ksoftirqd/1	10:59:27	1
9188	99	41	1,7	cyclictest	22	-21	ksoftirqd/1	10:35:28	1
9188	99	41	1,7	cyclictest	22	-21	ksoftirqd/1	09:48:20	1
9188	99	41	1,7	cyclictest	22	-21	ksoftirqd/1	09:44:38	1
9188	99	41	1,6	cyclictest	22	-21	ksoftirqd/1	09:36:35	1
9188	99	41	1,4	cyclictest	22	-21	ksoftirqd/1	10:17:50	1
9188	99	41	1,4	cyclictest	22	-21	ksoftirqd/1	10:17:49	1
8925	2	41	0,5	sleep1	20	1	rcuc/1	11:15:14	1
22297	2	41	0,5	sleep0	9185	-21	cyclictest	07:55:23	0
9188	99	40	25,4	cyclictest	22	-21	ksoftirqd/1	09:18:31	1
9188	99	39	5,6	cyclictest	22	-21	ksoftirqd/1	08:37:56	1
9188	99	39	5,4	cyclictest	8	1	rcu_preempt	11:08:55	1
9188	99	39	5,4	cyclictest	8	1	rcu_preempt	07:40:17	1
9188	99	39	5,4	cyclictest	8	1	rcu_preempt	07:40:17	1
9188	99	39	4,4	cyclictest	8	1	rcu_preempt	11:25:26	1
9188	99	39	4,4	cyclictest	8	1	rcu_preempt	08:50:13	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	12:40:17	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	11:45:41	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	09:55:27	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	08:45:13	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	08:25:16	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	07:50:10	1
9188	99	39	1,7	cyclictest	22	-21	ksoftirqd/1	07:35:16	1
9188	99	39	11,9	cyclictest	22	-21	ksoftirqd/1	10:54:06	1
9187	99	39	1,15	cyclictest	11052	-21	cstates	10:10:16	0
9188	99	38	3,4	cyclictest	8	1	rcu_preempt	07:45:13	1
9188	99	38	1,7	cyclictest	22	-21	ksoftirqd/1	11:12:55	1
9188	99	38	1,7	cyclictest	22	-21	ksoftirqd/1	09:00:26	1
9188	99	38	1,6	cyclictest	22	-21	ksoftirqd/1	08:10:12	1
9188	99	37	2,4	cyclictest	8	1	rcu_preempt	08:05:11	1
9188	99	37	1,7	cyclictest	22	-21	ksoftirqd/1	12:53:10	1
9188	99	37	1,7	cyclictest	22	-21	ksoftirqd/1	08:55:15	1
9187	99	37	1,15	cyclictest	27458	-21	ls	11:00:25	0
9187	99	37	11,13	cyclictest	10963	-21	memory	11:50:24	0
9188	99	36	18,9	cyclictest	22	-21	ksoftirqd/1	07:57:27	1
9188	99	36	1,7	cyclictest	22	-21	ksoftirqd/1	08:15:13	1
9188	99	36	1,4	cyclictest	8	1	rcu_preempt	08:20:14	1
9187	99	36	1,16	cyclictest	0	-21	swapper/0	12:28:29	0
9188	99	35	12,4	cyclictest	8	1	rcu_preempt	08:00:24	1
9187	99	35	9,6	cyclictest	20139	-21	diskstats	12:00:16	0
11215	2	35	0,5	sleep0	11	1	rcuc/0	09:36:02	0
9188	99	34	20,4	cyclictest	22	-21	ksoftirqd/1	08:30:13	1
9187	99	34	7,4	cyclictest	8	1	rcu_preempt	10:18:42	0
9187	99	34	7,4	cyclictest	8	1	rcu_preempt	10:18:42	0
9187	99	34	3,4	cyclictest	8	1	rcu_preempt	09:23:18	0
9187	99	34	2,4	cyclictest	8	1	rcu_preempt	12:11:01	0
9187	99	34	2,20	cyclictest	17466	-21	ssh	09:10:01	0
9187	99	34	1,19	cyclictest	30887	-21	cat	08:20:01	0
21157	2	34	1,5	sleep0	0	-21	swapper/0	07:52:47	0
18090	2	34	1,5	sleep0	0	-21	swapper/0	12:30:53	0
15992	2	34	2,28	sleep0	0	-21	swapper/0	10:48:57	0
9187	99	33	4,8	cyclictest	1819	-21	memory	10:00:23	0
9187	99	33	4,4	cyclictest	8	1	rcu_preempt	11:29:40	0
9187	99	33	4,4	cyclictest	8	1	rcu_preempt	11:22:33	0
9187	99	33	4,4	cyclictest	8	1	rcu_preempt	10:08:40	0
9187	99	33	2,4	cyclictest	8	1	rcu_preempt	09:34:06	0
9187	99	33	23,6	cyclictest	14946	-21	sed	09:00:29	0
9187	99	33	2,15	cyclictest	21394	-21	ntp_states	10:20:27	0
9187	99	33	1,16	cyclictest	8039	-21	ssh	10:40:21	0
9187	99	33	1,14	cyclictest	831	-21	cat	10:00:00	0
794	2	33	0,11	sleep0	9187	99	cyclictest	11:06:01	0
31807	2	33	1,3	sleep0	31808	-21	ntp_states	08:20:28	0
19365	2	33	3,6	sleep0	19376	-21	ssh	10:51:52	0
1702	2	33	0,5	sleep0	9185	-21	cyclictest	09:26:23	0
9188	99	32	4,4	cyclictest	8	1	rcu_preempt	07:35:00	1
9187	99	32	7,4	cyclictest	8	1	rcu_preempt	09:45:32	0
9187	99	32	2,4	cyclictest	8	1	rcu_preempt	10:57:28	0
9187	99	32	1,4	cyclictest	8	1	rcu_preempt	11:35:32	0
9187	99	32	1,19	cyclictest	0	-21	swapper/0	12:43:26	0
9187	99	32	1,14	cyclictest	0	-21	swapper/0	07:48:39	0
9187	99	32	1,11	cyclictest	0	-21	swapper/0	09:51:03	0
5635	2	32	0,17	sleep0	11	1	rcuc/0	11:45:30	0
9188	99	31	2,15	cyclictest	15891	-21	df_inode	09:05:22	1
9187	99	31	3,4	cyclictest	8	1	rcu_preempt	10:34:23	0
9187	99	31	3,23	cyclictest	27793	-21	iostat_ios	08:10:22	0
9187	99	31	1,5	cyclictest	8	1	rcu_preempt	09:42:13	0
9187	99	31	15,6	cyclictest	3	-21	ksoftirqd/0	11:15:18	0
9187	99	31	1,4	cyclictest	8	1	rcu_preempt	07:42:07	0
9187	99	31	1,4	cyclictest	8	1	rcu_preempt	07:42:06	0
4080	2	31	1,5	sleep0	0	-21	swapper/0	08:34:49	0
16428	2	31	1,5	sleep0	0	-21	swapper/0	09:05:32	0
9187	99	30	6,4	cyclictest	8	1	rcu_preempt	11:30:23	0
9187	99	30	1,8	cyclictest	0	-21	swapper/0	07:35:30	0
9187	99	30	12,5	cyclictest	11446	-21	runrttasks	08:53:00	0
31717	2	30	1,4	sleep0	31723	-21	ntp_states	12:45:25	0
1846	2	30	1,5	sleep0	0	-21	swapper/0	08:27:18	0
9187	99	29	4,20	cyclictest	26578	-21	users	08:05:30	0
9187	99	29	4,13	cyclictest	8	1	rcu_preempt	08:55:15	0
9187	99	29	1,4	cyclictest	8	1	rcu_preempt	12:16:51	0
9187	99	29	10,6	cyclictest	3079	-21	/usr/sbin/munin	10:35:19	0
9187	99	28	2,13	cyclictest	5149	-21	ps	08:35:26	0
9187	99	28	11,5	cyclictest	12702	-21	/usr/sbin/munin	07:30:23	0
9187	99	28	1,13	cyclictest	24389	-21	fgrep	08:00:29	0
9187	99	28	10,6	cyclictest	8438	-21	fschecks_count	08:45:17	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17