OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of the 376 highest latencies:

System rack9slot4s (updated Mon Jul 27, 2026 12:59:00)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
31016	99	1205	1162,26	cyclictest	16775	-21	smartctl	09:35:17	1
31015	99	1187	1148,21	cyclictest	0	-21	swapper/0	07:25:12	0
31016	99	1082	1069,9	cyclictest	0	-21	swapper/1	07:25:11	1
30689	2	468	423,30	sleep0	0	-21	swapper/0	07:22:09	0
29211	2	464	418,31	sleep1	0	-21	swapper/1	07:20:02	1
11976	2	118	0,2	sleep0	11977	-21	unixbench_multi	07:55:29	0
31015	99	108	85,20	cyclictest	0	-21	swapper/0	09:35:17	0
5152	2	86	0,5	sleep0	4992	-21	latency_hist	07:40:00	0
2189	2	76	0,5	sleep1	31016	99	cyclictest	07:30:27	1
32701	2	71	0,4	sleep1	20	1	rcuc/1	08:50:27	1
10947	2	58	0,4	sleep1	21	1	ktimersoftd/1	09:20:14	1
31016	99	56	3,19	cyclictest	0	-21	swapper/1	12:11:48	1
31016	99	45	7,6	cyclictest	22	-21	ksoftirqd/1	12:50:11	1
31016	99	44	8,5	cyclictest	22	-21	ksoftirqd/1	08:10:22	1
31016	99	44	8,5	cyclictest	22	-21	ksoftirqd/1	08:10:22	1
31016	99	44	2,10	cyclictest	0	-21	swapper/1	10:00:29	1
31016	99	43	9,6	cyclictest	22	-21	ksoftirqd/1	12:35:37	1
31016	99	43	9,6	cyclictest	22	-21	ksoftirqd/1	09:34:51	1
31016	99	43	3,4	cyclictest	0	-21	swapper/1	11:35:12	1
31016	99	43	15,6	cyclictest	22	-21	ksoftirqd/1	08:02:00	1
31016	99	43	14,6	cyclictest	22	-21	ksoftirqd/1	11:13:00	1
31016	99	42	9,6	cyclictest	22	-21	ksoftirqd/1	10:27:43	1
31016	99	42	9,6	cyclictest	22	-21	ksoftirqd/1	08:19:22	1
31016	99	42	8,6	cyclictest	22	-21	ksoftirqd/1	10:50:21	1
31016	99	42	8,6	cyclictest	22	-21	ksoftirqd/1	09:05:20	1
31016	99	42	8,6	cyclictest	22	-21	ksoftirqd/1	08:40:12	1
31016	99	42	18,4	cyclictest	22	-21	ksoftirqd/1	07:38:44	1
31016	99	41	9,6	cyclictest	22	-21	ksoftirqd/1	12:23:51	1
31016	99	41	9,6	cyclictest	22	-21	ksoftirqd/1	11:42:27	1
31016	99	41	9,4	cyclictest	8	1	rcu_preempt	08:35:12	1
31016	99	41	8,6	cyclictest	22	-21	ksoftirqd/1	10:20:47	1
31016	99	41	8,5	cyclictest	12533	-21	needreboot	12:15:25	1
31016	99	41	6,6	cyclictest	22	-21	ksoftirqd/1	09:10:25	1
31016	99	41	5,5	cyclictest	9756	-21	mailstats	09:15:28	1
31016	99	41	2,9	cyclictest	0	-21	swapper/1	11:20:19	1
31016	99	41	1,9	cyclictest	0	-21	swapper/1	12:25:12	1
31016	99	41	12,6	cyclictest	22	-21	ksoftirqd/1	07:57:49	1
31016	99	40	9,19	cyclictest	28924	-21	sh	11:34:59	1
31016	99	40	8,6	cyclictest	22	-21	ksoftirqd/1	11:48:48	1
31016	99	40	8,6	cyclictest	22	-21	ksoftirqd/1	10:35:00	1
31016	99	40	8,6	cyclictest	22	-21	ksoftirqd/1	07:52:15	1
31016	99	40	5,5	cyclictest	17842	-21	fschecks_time	12:30:16	1
31016	99	40	11,6	cyclictest	22	-21	ksoftirqd/1	09:00:03	1
31016	99	40	11,6	cyclictest	22	-21	ksoftirqd/1	08:57:34	1
31016	99	40	11,6	cyclictest	22	-21	ksoftirqd/1	07:48:00	1
31016	99	39	8,6	cyclictest	22	-21	ksoftirqd/1	11:55:18	1
31016	99	39	8,6	cyclictest	22	-21	ksoftirqd/1	10:05:12	1
31016	99	39	7,6	cyclictest	22	-21	ksoftirqd/1	12:46:39	1
31016	99	39	7,5	cyclictest	22	-21	ksoftirqd/1	11:00:27	1
31016	99	39	19,4	cyclictest	22	-21	ksoftirqd/1	10:31:34	1
31016	99	39	12,5	cyclictest	22	-21	ksoftirqd/1	09:48:28	1
31016	99	39	11,6	cyclictest	22	-21	ksoftirqd/1	10:46:36	1
12825	2	39	1,5	sleep1	12923	-21	fschecks_time	09:25:17	1
31016	99	38	8,6	cyclictest	24319	-21	cpuspeed_turbos	08:30:13	1
31016	99	38	7,6	cyclictest	22	-21	ksoftirqd/1	07:40:12	1
31016	99	38	5,6	cyclictest	23120	-21	awk	08:25:25	1
31016	99	38	5,5	cyclictest	24827	-21	switchtime	09:55:28	1
31016	99	38	2,6	cyclictest	22	-21	ksoftirqd/1	10:40:16	1
31016	99	38	11,6	cyclictest	22	-21	ksoftirqd/1	08:05:42	1
31016	99	38	11,5	cyclictest	22	-21	ksoftirqd/1	11:08:53	1
31016	99	38	11,5	cyclictest	22	-21	ksoftirqd/1	09:53:33	1
31016	99	37	9,6	cyclictest	22	-21	ksoftirqd/1	12:09:39	1
31016	99	37	9,6	cyclictest	22	-21	ksoftirqd/1	10:56:10	1
31016	99	37	8,5	cyclictest	3286	-21	runrttasks	08:24:48	1
31016	99	37	7,11	cyclictest	22	-21	ksoftirqd/1	09:40:19	1
31016	99	37	7,11	cyclictest	22	-21	ksoftirqd/1	09:40:18	1
31016	99	37	6,6	cyclictest	22	-21	ksoftirqd/1	10:15:16	1
31016	99	37	10,4	cyclictest	8	1	rcu_preempt	12:42:55	1
31016	99	36	8,6	cyclictest	22	-21	ksoftirqd/1	11:28:00	1
31016	99	36	8,4	cyclictest	8	1	rcu_preempt	08:45:34	1
31016	99	36	8,10	cyclictest	3967	-21	runrttasks	11:53:38	1
31016	99	36	4,9	cyclictest	23315	-21	sendmail-msp	11:20:00	1
31016	99	36	22,4	cyclictest	22	-21	ksoftirqd/1	12:00:20	1
31015	99	36	4,4	cyclictest	8	1	rcu_preempt	08:22:01	0
31015	99	36	1,21	cyclictest	12904	-21	grep	10:50:24	0
31016	99	35	6,9	cyclictest	0	-21	swapper/1	10:10:18	1
31015	99	35	1,21	cyclictest	935	-21	cron	08:54:59	0
31015	99	35	1,14	cyclictest	0	-21	swapper/0	11:57:59	0
12390	2	35	0,9	sleep0	31015	99	cyclictest	12:15:21	0
31015	99	34	9,13	cyclictest	8260	-21	chrt	09:11:39	0
31015	99	34	9,12	cyclictest	12971	-21	runrttasks	09:25:17	0
31015	99	34	1,14	cyclictest	18753	-21	sendmail_mailqu	11:05:27	0
31015	99	33	4,17	cyclictest	8407	-21	munin-node	10:40:11	0
31015	99	33	1,19	cyclictest	23930	-21	taskset	08:29:21	0
31015	99	33	1,11	cyclictest	8	1	rcu_preempt	09:31:16	0
19337	2	33	1,4	sleep0	19431	-21	cut	08:15:25	0
31015	99	32	8,12	cyclictest	30771	-21	taskset	11:39:51	0
31015	99	32	8,11	cyclictest	14772	-21	needreboot	10:55:24	0
31015	99	32	6,4	cyclictest	8	1	rcu_preempt	10:05:22	0
31015	99	32	3,4	cyclictest	8	1	rcu_preempt	10:15:11	0
31015	99	32	1,8	cyclictest	11296	-21	python	10:45:28	0
31015	99	32	1,4	cyclictest	8	1	rcu_preempt	08:55:27	0
31015	99	32	1,19	cyclictest	3717	-21	irqstats	09:00:22	0
24174	2	32	1,4	sleep0	24325	-21	awk	12:45:29	0
31015	99	31	4,14	cyclictest	28944	-21	python	08:40:29	0
31015	99	31	1,4	cyclictest	8	1	rcu_preempt	12:08:42	0
31015	99	31	1,4	cyclictest	8	1	rcu_preempt	11:02:54	0
31015	99	31	12,6	cyclictest	30274	-21	perl	08:45:18	0
31015	99	31	1,15	cyclictest	13483	-21	irqstats	08:00:20	0
31015	99	31	0,4	cyclictest	8	1	rcu_preempt	11:26:36	0
8170	2	30	1,5	sleep0	0	-21	swapper/0	10:39:31	0
31015	99	30	7,12	cyclictest	6307	-21	taskset	09:05:56	0
31015	99	30	3,4	cyclictest	8	1	rcu_preempt	10:20:21	0
31015	99	30	2,14	cyclictest	9044	-21	diskstats	09:15:15	0
31015	99	30	1,8	cyclictest	0	-21	swapper/0	11:20:20	0
31015	99	30	1,4	cyclictest	8	1	rcu_preempt	08:08:39	0
31015	99	30	1,4	cyclictest	8	1	rcu_preempt	07:45:01	0
31015	99	30	12,5	cyclictest	11565	-21	gdbus	11:45:57	0
31015	99	30	1,21	cyclictest	0	-21	swapper/0	12:53:20	0
31015	99	30	1,14	cyclictest	15533	-21	latency_hist	12:25:02	0
31015	99	30	1,14	cyclictest	11565	-21	gdbus	10:13:29	0
31015	99	30	10,16	cyclictest	22669	2	chrt	12:42:20	0
31015	99	29	7,8	cyclictest	2221	-21	cron	10:00:01	0
31015	99	29	2,3	cyclictest	8	1	rcu_preempt	11:43:13	0
31015	99	29	1,4	cyclictest	8	1	rcu_preempt	12:10:12	0
31015	99	29	1,4	cyclictest	8	1	rcu_preempt	09:44:39	0
31015	99	29	1,4	cyclictest	8	1	rcu_preempt	09:44:38	0
31015	99	29	11,5	cyclictest	25895	-21	cpuspeed_turbos	10:00:13	0
31015	99	29	11,5	cyclictest	22745	-21	ntp_states	09:50:21	0
31015	99	29	1,14	cyclictest	11565	-21	gdbus	07:45:17	0
31015	99	29	10,6	cyclictest	11022	-21	tune2fs	09:20:16	0
27511	2	29	1,5	sleep0	0	-21	swapper/0	08:37:36	0
21458	2	29	1,5	sleep0	0	-21	swapper/0	09:46:55	0
19321	2	29	1,5	sleep0	0	-21	swapper/0	12:35:11	0
18040	2	29	1,5	sleep0	0	-21	swapper/0	12:30:20	0
31015	99	28	6,8	cyclictest	24505	-21	fschecks_count	08:30:16	0
31015	99	28	4,16	cyclictest	0	-21	swapper/0	12:02:36	0
31015	99	28	2,13	cyclictest	2593	-21	df	11:50:14	0
31015	99	28	2,13	cyclictest	20698	-21	python	11:10:28	0
31015	99	28	1,14	cyclictest	3774	-21	python	10:25:29	0
31015	99	28	1,13	cyclictest	22517	-21	awk	11:15:29	0
10214	2	28	1,5	sleep0	0	-21	swapper/0	07:50:40	0
31015	99	27	2,3	cyclictest	8	1	rcu_preempt	11:31:44	0
31015	99	27	1,14	cyclictest	5898	-21	/usr/sbin/munin	10:30:30	0
31015	99	27	1,12	cyclictest	15626	-21	grep	12:25:11	0
31015	99	27	10,6	cyclictest	908	-21	diskstats	07:30:03	0
31015	99	26	2,19	cyclictest	16952	-21	sort	08:10:17	0
31015	99	26	2,19	cyclictest	16952	-21	sort	08:10:16	0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2019-05-28 - 00:17