OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the highest latencies:

System rack9slot4s (updated Tue Mar 31, 2026 00:59:04)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
19558	99	1160	1127,19	cyclictest	0	-21	swapper/0	19:27:01	0
19559	99	1092	1082,7	cyclictest	0	-21	swapper/1	19:27:24	1
19182	2	597	517,65	sleep0	0	-21	swapper/0	19:21:37	0
19438	2	464	419,30	sleep1	0	-21	swapper/1	19:24:19	1
23948	2	97	0,9	sleep0	19558	99	cyclictest	23:52:55	0
6314	2	91	0,8	sleep0	19558	99	cyclictest	21:54:57	0
29592	2	91	0,7	sleep1	19559	99	cyclictest	23:25:19	1
1303	2	86	0,2	sleep1	0	-21	swapper/1	23:30:03	1
514	2	82	0,5	sleep0	508	-21	ssh	22:55:18	0
24441	2	74	0,4	sleep1	9160	-21	kworker/1:0	23:20:10	1
26322	2	72	0,7	sleep1	19559	99	cyclictest	22:48:38	1
1027	2	58	0,7	sleep1	19559	99	cyclictest	22:22:11	1
19559	99	57	14,6	cyclictest	22	-21	ksoftirqd/1	22:09:02	1
19559	99	56	17,17	cyclictest	22	-21	ksoftirqd/1	21:24:37	1
19559	99	55	8,6	cyclictest	22	-21	ksoftirqd/1	23:11:45	1
19559	99	55	17,6	cyclictest	22	-21	ksoftirqd/1	00:40:26	1
19559	99	55	13,7	cyclictest	22	-21	ksoftirqd/1	23:58:26	1
10571	2	55	0,4	sleep1	19559	99	cyclictest	21:25:20	1
19559	99	54	17,6	cyclictest	22	-21	ksoftirqd/1	23:40:18	1
19559	99	53	15,5	cyclictest	22	-21	ksoftirqd/1	00:15:11	1
19559	99	53	14,6	cyclictest	22	-21	ksoftirqd/1	20:40:27	1
19559	99	52	28,4	cyclictest	22	-21	ksoftirqd/1	21:30:12	1
19559	99	52	14,6	cyclictest	22	-21	ksoftirqd/1	22:40:22	1
19559	99	52	14,6	cyclictest	22	-21	ksoftirqd/1	00:35:25	1
19559	99	51	6,6	cyclictest	22	-21	ksoftirqd/1	23:48:18	1
19559	99	51	6,5	cyclictest	22	-21	ksoftirqd/1	21:44:21	1
19559	99	51	4,6	cyclictest	22	-21	ksoftirqd/1	00:47:47	1
19559	99	51	17,6	cyclictest	22	-21	ksoftirqd/1	20:05:27	1
19559	99	51	16,5	cyclictest	22	-21	ksoftirqd/1	00:07:10	1
19559	99	51	14,6	cyclictest	22	-21	ksoftirqd/1	23:09:09	1
19559	99	51	14,6	cyclictest	22	-21	ksoftirqd/1	23:09:08	1
19559	99	51	14,5	cyclictest	22	-21	ksoftirqd/1	00:20:17	1
19559	99	51	13,6	cyclictest	22	-21	ksoftirqd/1	22:50:12	1
19559	99	50	4,6	cyclictest	22	-21	ksoftirqd/1	00:29:03	1
19559	99	50	17,5	cyclictest	22	-21	ksoftirqd/1	21:57:49	1
19559	99	50	17,4	cyclictest	22	-21	ksoftirqd/1	20:45:25	1
19559	99	50	14,6	cyclictest	22	-21	ksoftirqd/1	21:15:19	1
19558	99	50	1,16	cyclictest	1826	-21	ssh	00:03:51	0
29881	2	49	0,7	sleep0	0	-21	swapper/0	22:51:47	0
19559	99	49	4,6	cyclictest	22	-21	ksoftirqd/1	22:01:50	1
19559	99	49	13,6	cyclictest	22	-21	ksoftirqd/1	00:31:08	1
19559	99	49	12,6	cyclictest	22	-21	ksoftirqd/1	19:30:26	1
19559	99	49	12,5	cyclictest	22	-21	ksoftirqd/1	20:20:01	1
19559	99	48	3,6	cyclictest	22	-21	ksoftirqd/1	21:53:42	1
19559	99	48	19,5	cyclictest	22	-21	ksoftirqd/1	22:14:56	1
19559	99	48	19,4	cyclictest	22	-21	ksoftirqd/1	23:00:10	1
19559	99	48	16,4	cyclictest	22	-21	ksoftirqd/1	22:17:15	1
19559	99	48	15,4	cyclictest	22	-21	ksoftirqd/1	23:50:23	1
19559	99	48	14,6	cyclictest	22	-21	ksoftirqd/1	22:58:43	1
19559	99	48	13,6	cyclictest	22	-21	ksoftirqd/1	21:45:08	1
19559	99	48	12,6	cyclictest	22	-21	ksoftirqd/1	22:31:09	1
19559	99	48	12,6	cyclictest	22	-21	ksoftirqd/1	21:39:51	1
19559	99	48	10,9	cyclictest	22	-21	ksoftirqd/1	00:01:01	1
19559	99	47	19,5	cyclictest	22	-21	ksoftirqd/1	23:16:40	1
19559	99	47	16,5	cyclictest	22	-21	ksoftirqd/1	00:51:35	1
19559	99	47	14,4	cyclictest	22	-21	ksoftirqd/1	23:35:03	1
19559	99	47	14,4	cyclictest	22	-21	ksoftirqd/1	23:35:03	1
19559	99	47	11,6	cyclictest	22	-21	ksoftirqd/1	22:36:34	1
19559	99	47	11,6	cyclictest	22	-21	ksoftirqd/1	21:14:14	1
19559	99	47	11,5	cyclictest	22	-21	ksoftirqd/1	20:30:26	1
19559	99	47	10,6	cyclictest	22	-21	ksoftirqd/1	22:27:46	1
19559	99	47	10,6	cyclictest	22	-21	ksoftirqd/1	22:27:46	1
8592	2	46	0,3	sleep0	4	1	ktimersoftd/0	23:03:48	0
2990	2	46	0,4	sleep0	4	1	ktimersoftd/0	00:05:10	0
19559	99	46	11,15	cyclictest	22	-21	ksoftirqd/1	00:13:53	1
19559	99	45	25,4	cyclictest	22	-21	ksoftirqd/1	20:50:29	1
19559	99	45	25,4	cyclictest	22	-21	ksoftirqd/1	19:35:25	1
19559	99	45	11,5	cyclictest	22	-21	ksoftirqd/1	20:05:01	1
19559	99	45	10,6	cyclictest	22	-21	ksoftirqd/1	20:59:12	1
19559	99	44	10,5	cyclictest	22	-21	ksoftirqd/1	21:08:02	1
19559	99	43	9,4	cyclictest	22	-21	ksoftirqd/1	19:58:42	1
19559	99	42	6,6	cyclictest	22	-21	ksoftirqd/1	20:21:26	1
19559	99	42	26,4	cyclictest	22	-21	ksoftirqd/1	19:40:22	1
19559	99	42	24,4	cyclictest	22	-21	ksoftirqd/1	20:35:26	1
19559	99	42	21,4	cyclictest	22	-21	ksoftirqd/1	20:10:28	1
19559	99	42	21,4	cyclictest	22	-21	ksoftirqd/1	19:50:18	1
19559	99	41	25,4	cyclictest	22	-21	ksoftirqd/1	21:00:26	1
19558	99	40	1,18	cyclictest	19856	-21	cron	23:49:00	0
19558	99	40	1,18	cyclictest	0	-21	swapper/0	21:35:39	0
19558	99	39	9,18	cyclictest	3018	-21	date	20:05:01	0
19559	99	38	2,6	cyclictest	22	-21	ksoftirqd/1	20:25:14	1
19559	99	38	23,4	cyclictest	22	-21	ksoftirqd/1	19:45:15	1
19559	99	38	23,4	cyclictest	22	-21	ksoftirqd/1	19:45:15	1
19558	99	37	6,3	cyclictest	0	-21	swapper/0	22:31:36	0
19558	99	37	1,21	cyclictest	21047	-21	ssh	00:23:25	0
19558	99	36	5,27	cyclictest	19301	-21	cron	21:35:00	0
19558	99	36	4,26	cyclictest	30031	-21	diskstats	19:50:14	0
19558	99	36	1,5	cyclictest	6013	-21	ssh	23:34:56	0
19558	99	36	15,5	cyclictest	0	-21	swapper/0	23:35:45	0
19558	99	36	15,5	cyclictest	0	-21	swapper/0	23:35:44	0
19558	99	36	1,4	cyclictest	8	1	rcu_preempt	21:20:39	0
19558	99	36	1,22	cyclictest	14646	-21	ssh	22:35:44	0
30608	2	35	1,4	sleep0	27350	-21	diskmemload	00:33:17	0
23768	2	35	1,4	sleep0	23867	-21	grep	21:00:13	0
19558	99	35	9,13	cyclictest	3867	-21	sh	00:39:22	0
19558	99	35	3,16	cyclictest	20792	-21	users	23:15:30	0
19558	99	35	2,29	cyclictest	0	-21	swapper/0	21:16:56	0
19558	99	35	2,19	cyclictest	0	-21	swapper/0	00:26:28	0
19558	99	35	2,12	cyclictest	15072	-21	hddtemp_smartct	23:10:19	0
19558	99	35	15,5	cyclictest	12133	-21	ntp_states	23:40:25	0
19558	99	35	13,7	cyclictest	405	50	irq/92-i915	22:45:24	0
19558	99	35	10,12	cyclictest	3000	-21	wc	20:05:02	0
422	2	34	1,4	sleep0	27350	-21	diskmemload	23:29:01	0
19558	99	34	9,12	cyclictest	10953	-21	ssh	00:12:51	0
19558	99	34	5,4	cyclictest	8	1	rcu_preempt	22:41:45	0
19558	99	34	2,20	cyclictest	17283	-21	ssh	22:05:26	0
19558	99	34	18,4	cyclictest	3	-21	ksoftirqd/0	00:50:01	0
19558	99	34	17,14	cyclictest	3	-21	ksoftirqd/0	20:32:58	0
19558	99	34	1,20	cyclictest	28576	-21	ssh	23:57:36	0
19558	99	34	10,7	cyclictest	2206	-21	dbus-daemon	20:57:42	0
19558	99	34	10,12	cyclictest	2645	-21	snmpd	22:13:31	0
19558	99	33	8,5	cyclictest	30497	-21	gdbus	22:25:52	0
19558	99	33	8,5	cyclictest	30497	-21	gdbus	22:25:52	0
19558	99	33	7,12	cyclictest	0	-21	swapper/0	20:36:44	0
19558	99	33	6,14	cyclictest	2668	-21	sshd	19:30:25	0
19558	99	33	3,4	cyclictest	8	1	rcu_preempt	23:20:59	0
19558	99	33	3,4	cyclictest	8	1	rcu_preempt	21:41:09	0
19558	99	33	3,4	cyclictest	8	1	rcu_preempt	00:52:31	0
19558	99	33	3,4	cyclictest	8	1	rcu_preempt	00:16:20	0
19558	99	33	13,8	cyclictest	27812	-21	ssh	22:16:01	0
19558	99	33	1,30	cyclictest	0	-21	swapper/0	21:29:35	0
19558	99	33	1,20	cyclictest	29147	-21	ssh	21:10:36	0
19558	99	32	3,15	cyclictest	31523	-21	iostat	22:20:18	0
19558	99	32	1,4	cyclictest	8	1	rcu_preempt	00:40:23	0
19558	99	31	3,4	cyclictest	8	1	rcu_preempt	21:45:46	0
19558	99	31	2,4	cyclictest	8	1	rcu_preempt	23:05:02	0
19558	99	31	2,4	cyclictest	8	1	rcu_preempt	23:05:02	0
19558	99	31	2,14	cyclictest	864	-21	cron	20:00:00	0
27283	2	30	1,5	sleep0	0	-21	swapper/0	21:09:25	0
19558	99	30	4,4	cyclictest	8	1	rcu_preempt	19:45:12	0
19558	99	30	4,4	cyclictest	8	1	rcu_preempt	19:45:11	0
19558	99	30	3,15	cyclictest	7637	-21	ntp_states	20:15:24	0
19558	99	30	3,13	cyclictest	5760	-21	ntp_states	20:10:23	0
19558	99	30	21,6	cyclictest	0	-21	swapper/0	20:26:55	0
19558	99	30	1,4	cyclictest	8	1	rcu_preempt	22:01:45	0
19558	99	30	1,4	cyclictest	0	-21	swapper/0	19:42:55	0
19558	99	30	11,6	cyclictest	9497	-21	ssh	21:57:34	0
19558	99	30	1,12	cyclictest	0	-21	swapper/0	20:55:00	0
19558	99	30	10,5	cyclictest	16231	-21	/usr/sbin/munin	20:40:18	0
19558	99	29	11,6	cyclictest	9043	-21	cut	20:20:15	0
19558	99	29	11,6	cyclictest	25099	-21	mailstats	19:35:27	0
19472	2	29	1,5	sleep0	0	-21	swapper/0	20:46:27	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17