OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system

Characteristics of all highest latencies:

System rackeslot6s.osadl.org (updated Wed Mar 11, 2026 12:44:48)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
749396	99	69	1,29	cyclictest	208	50	irq/208-8000000.ethernet	11:30:01	0
749398	99	67	16,10	cyclictest	756008	-21	/usr/sbin/munin	07:45:15	1
749398	99	57	45,7	cyclictest	0	-21	swapper/1	08:25:25	1
749399	99	54	38,12	cyclictest	765175	-21	sendmail	08:40:00	2
749399	99	51	40,5	cyclictest	3839428	-1	kworker/2:1H+kblockd	09:00:10	2
749398	99	50	37,8	cyclictest	0	-21	swapper/1	09:00:02	1
749398	99	50	32,13	cyclictest	23	1	ktimers/1	08:15:12	1
749396	99	50	44,4	cyclictest	773724	-21	munin-plugin-st	09:30:00	0
749398	99	49	12,4	cyclictest	778867	-21	apt-get	10:00:02	1
749396	99	49	44,2	cyclictest	0	-21	swapper/0	08:20:11	0
749396	99	49	42,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	11:10:00	0
749396	99	49	32,15	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	10:25:01	0
749398	99	48	42,3	cyclictest	760867	-21	munin-plugin-st	08:15:00	1
749398	99	48	1,10	cyclictest	0	-21	swapper/1	11:35:18	1
749398	99	47	32,8	cyclictest	0	-21	swapper/1	10:45:00	1
749398	99	47	30,10	cyclictest	0	-21	swapper/1	12:15:27	1
749399	99	46	9,30	cyclictest	0	-21	swapper/2	10:05:28	2
749399	99	46	34,7	cyclictest	796747	-21	grep	11:40:29	2
749399	99	46	34,7	cyclictest	796747	-21	grep	11:40:29	2
749399	99	46	12,27	cyclictest	0	-21	swapper/2	07:10:20	2
749398	99	46	38,4	cyclictest	757490	-21	munin-plugin-st	07:55:00	1
749398	99	46	34,9	cyclictest	778880	-21	sendmail	10:00:00	1
749396	99	46	36,6	cyclictest	805133	-21	grep	12:30:24	0
749396	99	46	2,23	cyclictest	750463	-21	munin-node	07:15:01	0
749399	99	45	7,9	cyclictest	14	-21	rcu_preempt	12:00:02	2
749399	99	45	36,7	cyclictest	792569	-21	munin-plugin-st	11:20:00	2
749399	99	45	32,10	cyclictest	1283	-21	cron	10:05:00	2
749396	99	45	38,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	11:30:12	0
749396	99	45	38,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	09:07:48	0
749396	99	45	28,5	cyclictest	0	-21	swapper/0	11:05:01	0
749396	99	45	1,8	cyclictest	0	-21	swapper/0	10:35:19	0
749396	99	45	1,8	cyclictest	0	-21	swapper/0	10:35:19	0
749399	99	44	26,14	cyclictest	14	-21	rcu_preempt	09:10:01	2
749399	99	44	16,18	cyclictest	31	1	ktimers/2	10:50:11	2
749398	99	44	35,5	cyclictest	0	-21	swapper/1	10:20:00	1
749398	99	44	32,8	cyclictest	0	-21	swapper/1	09:45:01	1
749398	99	44	32,6	cyclictest	774500	-21	grep	09:30:28	1
749396	99	44	37,4	cyclictest	206	50	irq/252-8000000.ethernet	11:35:23	0
749396	99	44	36,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	10:55:24	0
749396	99	44	35,4	cyclictest	0	-21	swapper/0	09:00:01	0
749396	99	44	31,8	cyclictest	0	-21	swapper/0	10:30:27	0
749396	99	44	1,3	cyclictest	0	-21	swapper/0	11:50:12	0
749396	99	44	10,4	cyclictest	787196	-21	tail	10:45:26	0
749399	99	43	26,11	cyclictest	0	-21	swapper/2	09:45:23	2
749398	99	43	27,8	cyclictest	0	-21	swapper/1	07:15:28	1
749396	99	43	35,5	cyclictest	208	50	irq/208-8000000.ethernet	08:05:27	0
749396	99	43	33,6	cyclictest	0	-21	swapper/0	09:40:29	0
749396	99	43	32,7	cyclictest	0	-21	swapper/0	11:55:27	0
749396	99	43	31,8	cyclictest	0	-21	swapper/0	10:10:28	0
749396	99	43	30,9	cyclictest	208	50	irq/208-8000000.ethernet	07:15:00	0
749396	99	43	30,8	cyclictest	0	-21	swapper/0	09:55:27	0
749396	99	43	26,12	cyclictest	0	-21	swapper/0	11:20:28	0
749396	99	43	24,16	cyclictest	801586	-21	ps	12:10:25	0
749396	99	43	2,39	cyclictest	0	-21	swapper/0	08:00:21	0
749396	99	43	2,39	cyclictest	0	-21	swapper/0	08:00:21	0
749396	99	43	20,21	cyclictest	0	-21	swapper/0	09:15:22	0
749399	99	42	30,8	cyclictest	0	-21	swapper/2	08:05:28	2
749398	99	42	25,11	cyclictest	0	-21	swapper/1	09:15:22	1
749398	99	42	1,9	cyclictest	22	-21	rcuc/1	11:50:00	1
749396	99	42	35,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	08:35:21	0
749396	99	42	31,7	cyclictest	0	-21	swapper/0	07:50:29	0
749396	99	42	1,4	cyclictest	806095	-21	head	12:35:28	0
749396	99	42	1,4	cyclictest	0	-21	swapper/0	11:15:14	0
749396	99	42	1,35	cyclictest	0	-21	swapper/0	12:10:01	0
749399	99	41	28,8	cyclictest	0	-21	swapper/2	08:00:01	2
749399	99	41	26,9	cyclictest	0	-21	swapper/2	08:20:26	2
749399	99	41	26,8	cyclictest	806175	-21	sendmail	12:40:00	2
749399	99	41	1,33	cyclictest	0	-21	swapper/2	09:55:26	2
749398	99	41	29,2	cyclictest	0	-21	swapper/1	11:05:02	1
749398	99	41	28,9	cyclictest	0	-21	swapper/1	08:20:25	1
749396	99	41	7,28	cyclictest	0	-21	swapper/0	09:05:01	0
749396	99	41	33,5	cyclictest	208	50	irq/208-8000000.ethernet	07:45:26	0
749396	99	41	31,6	cyclictest	0	-21	swapper/0	09:30:28	0
749396	99	41	30,7	cyclictest	0	-21	swapper/0	08:10:28	0
749396	99	41	28,9	cyclictest	0	-21	swapper/0	11:45:28	0
749396	99	41	1,4	cyclictest	783464	-21	diskstats	10:25:17	0
749396	99	41	1,38	cyclictest	0	-21	swapper/0	07:35:21	0
749399	99	40	9,28	cyclictest	0	-21	swapper/2	10:25:25	2
749399	99	40	27,6	cyclictest	0	-21	swapper/2	11:25:28	2
749399	99	40	23,11	cyclictest	766872	-21	munin-plugin-st	08:50:00	2
749398	99	40	29,6	cyclictest	0	-21	swapper/1	10:35:29	1
749398	99	40	29,6	cyclictest	0	-21	swapper/1	10:35:29	1
749398	99	40	28,7	cyclictest	0	-21	swapper/1	10:50:27	1
749398	99	40	26,7	cyclictest	0	-21	swapper/1	12:20:27	1
749398	99	40	26,7	cyclictest	0	-21	swapper/1	12:20:27	1
749396	99	40	34,3	cyclictest	753253	-21	latency_hist	07:30:01	0
749396	99	40	28,8	cyclictest	0	-21	swapper/0	07:20:28	0
749396	99	40	27,8	cyclictest	0	-21	swapper/0	09:10:28	0
749396	99	40	1,4	cyclictest	772420	-21	perl	09:20:19	0
749396	99	40	1,4	cyclictest	772420	-21	perl	09:20:19	0
749396	99	40	1,36	cyclictest	0	-21	swapper/0	07:45:01	0
749396	99	40	1,34	cyclictest	0	-21	swapper/0	10:05:01	0
749396	99	40	0,28	cyclictest	0	-21	swapper/0	07:35:01	0
749399	99	39	30,4	cyclictest	0	-21	swapper/2	10:15:19	2
749399	99	39	27,6	cyclictest	0	-21	swapper/2	10:35:00	2
749399	99	39	21,12	cyclictest	31	1	ktimers/2	11:50:13	2
749398	99	39	28,6	cyclictest	795038	-21	grep	11:30:27	1
749398	99	39	28,5	cyclictest	0	-21	swapper/1	10:05:28	1
749398	99	39	25,9	cyclictest	0	-21	swapper/1	09:35:27	1
749398	99	39	21,14	cyclictest	23	1	ktimers/1	07:10:13	1
749396	99	39	4,12	cyclictest	15	-21	rcuc/0	12:30:01	0
749396	99	39	33,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	08:40:18	0
749396	99	39	33,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	08:40:18	0
749396	99	39	33,3	cyclictest	802250	-21	dropbear	12:15:20	0
749396	99	39	33,3	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	08:35:00	0
749396	99	39	32,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	08:45:26	0
749396	99	39	31,4	cyclictest	208	50	irq/208-8000000.ethernet	12:20:10	0
749396	99	39	31,4	cyclictest	208	50	irq/208-8000000.ethernet	12:20:10	0
749396	99	39	31,4	cyclictest	1673	50	irq/118-8000000.ethernet-tx0	08:15:29	0
749396	99	39	25,9	cyclictest	0	-21	swapper/0	09:45:29	0
749396	99	39	22,14	cyclictest	0	-21	swapper/0	10:40:21	0
749399	99	38	9,26	cyclictest	0	-21	swapper/2	09:25:27	2
749399	99	38	30,5	cyclictest	0	-21	swapper/2	12:05:27	2
749399	99	38	30,4	cyclictest	0	-21	swapper/2	08:25:27	2
749399	99	38	24,9	cyclictest	0	-21	swapper/2	09:15:21	2
749399	99	38	24,10	cyclictest	0	-21	swapper/2	12:30:23	2
749399	99	38	16,15	cyclictest	31	1	ktimers/2	09:50:15	2
749399	99	38	15,14	cyclictest	785071	-21	/usr/sbin/munin	10:35:19	2
749399	99	38	15,14	cyclictest	785071	-21	/usr/sbin/munin	10:35:19	2
749398	99	38	29,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	10:10:22	1
749398	99	38	28,7	cyclictest	0	-21	swapper/1	08:00:28	1
749398	99	38	28,7	cyclictest	0	-21	swapper/1	08:00:28	1
749398	99	38	27,7	cyclictest	0	-21	swapper/1	12:05:27	1
749398	99	38	26,8	cyclictest	0	-21	swapper/1	11:55:27	1
749398	99	38	26,7	cyclictest	791663	-21	grep	11:10:29	1
749398	99	38	20,5	cyclictest	800153	-21	kworker/u8:0-flush-179:96	12:10:21	1
749396	99	38	32,3	cyclictest	0	-21	swapper/0	07:55:19	0
749396	99	38	23,13	cyclictest	0	-21	swapper/0	09:38:01	0
749399	99	37	27,5	cyclictest	787341	-21	grep	10:45:29	2
749399	99	37	27,5	cyclictest	0	-21	swapper/2	08:55:16	2
749399	99	37	25,8	cyclictest	0	-21	swapper/2	08:40:28	2
749399	99	37	25,8	cyclictest	0	-21	swapper/2	08:40:28	2
749399	99	37	2,32	cyclictest	0	-21	swapper/2	10:20:25	2
749399	99	37	1,13	cyclictest	0	-21	swapper/2	07:45:00	2
749398	99	37	27,5	cyclictest	0	-21	swapper/1	11:05:01	1
749398	99	37	27,5	cyclictest	0	-21	swapper/1	10:25:27	1
749398	99	37	25,8	cyclictest	771094	-21	grep	09:10:27	1
749398	99	37	24,8	cyclictest	0	-21	swapper/1	09:25:28	1
749398	99	37	1,29	cyclictest	0	-21	swapper/1	10:30:27	1
749396	99	37	32,2	cyclictest	0	-21	swapper/0	10:55:01	0
749396	99	37	1,34	cyclictest	0	-21	swapper/0	11:40:23	0
749396	99	37	1,34	cyclictest	0	-21	swapper/0	11:40:23	0
749399	99	36	3,29	cyclictest	0	-21	swapper/2	10:55:27	2
749399	99	36	29,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	08:30:25	2
749399	99	36	27,6	cyclictest	754041	-21	grep	07:30:27	2
749399	99	36	25,7	cyclictest	761648	-21	grep	08:15:27	2
749398	99	36	27,6	cyclictest	0	-21	swapper/1	11:40:28	1
749398	99	36	27,6	cyclictest	0	-21	swapper/1	11:40:28	1
749398	99	36	22,7	cyclictest	0	-21	swapper/1	09:45:29	1
749396	99	36	27,5	cyclictest	13	1	ktimers/0	08:50:27	0
749396	99	36	27,5	cyclictest	0	-21	swapper/0	09:50:28	0
749399	99	35	26,5	cyclictest	0	-21	swapper/2	11:35:19	2
749398	99	35	29,4	cyclictest	0	-21	swapper/1	09:05:25	1
749398	99	35	27,5	cyclictest	0	-21	swapper/1	10:20:24	1
749398	99	35	26,5	cyclictest	0	-21	swapper/1	07:30:28	1
749398	99	35	24,6	cyclictest	0	-21	swapper/1	11:50:29	1
749398	99	35	17,12	cyclictest	758338	-21	sendmail	08:00:00	1
749398	99	35	1,6	cyclictest	0	-21	swapper/1	08:40:00	1
749396	99	35	29,4	cyclictest	208	50	irq/208-8000000.ethernet	12:00:22	0
749396	99	35	27,5	cyclictest	197	50	irq/27-20000000.i2c	10:05:24	0
749396	99	35	1,32	cyclictest	0	-21	swapper/0	11:11:09	0
749396	99	35	1,32	cyclictest	0	-21	swapper/0	10:15:25	0
749399	99	34	28,3	cyclictest	752817	-21	kernelversion	07:25:26	2
749399	99	34	25,5	cyclictest	804326	-21	grep	12:25:28	2
749399	99	34	2,3	cyclictest	774454	-21	unixbench_multi	09:30:28	2
749399	99	34	21,8	cyclictest	0	-21	swapper/2	09:45:00	2
749398	99	34	26,5	cyclictest	766229	-21	/usr/sbin/munin	08:45:23	1
749398	99	34	24,5	cyclictest	0	-21	swapper/1	09:50:27	1
749399	99	33	27,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	12:20:26	2
749399	99	33	27,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	12:20:26	2
749399	99	33	27,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	07:17:48	2
749399	99	33	26,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	11:05:17	2
749399	99	33	25,4	cyclictest	0	-21	swapper/2	11:55:27	2
749398	99	33	27,4	cyclictest	23	1	ktimers/1	07:35:27	1
749398	99	33	24,5	cyclictest	0	-21	swapper/1	10:45:29	1
749398	99	33	22,7	cyclictest	0	-21	swapper/1	11:20:28	1
749398	99	33	1,4	cyclictest	0	-21	swapper/1	11:30:01	1
749399	99	32	24,5	cyclictest	0	-21	swapper/2	09:10:26	2
749399	99	32	23,4	cyclictest	0	-21	swapper/2	10:40:18	2
749399	99	32	22,8	cyclictest	772859	-21	sshd	09:21:18	2
749399	99	32	22,8	cyclictest	772859	-21	sshd	09:21:18	2
749399	99	32	22,6	cyclictest	0	-21	swapper/2	07:45:18	2
749399	99	32	20,7	cyclictest	0	-21	swapper/2	09:35:26	2
749399	99	32	15,12	cyclictest	31	1	ktimers/2	08:00:12	2
749399	99	32	15,12	cyclictest	31	1	ktimers/2	08:00:12	2
749398	99	32	21,6	cyclictest	0	-21	swapper/1	12:40:01	1
749398	99	32	20,7	cyclictest	0	-21	swapper/1	07:40:30	1
749398	99	32	1,7	cyclictest	0	-21	swapper/1	12:30:01	1
749398	99	32	1,25	cyclictest	0	-21	swapper/1	08:40:19	1
749398	99	32	1,25	cyclictest	0	-21	swapper/1	08:40:19	1
749396	99	32	2,4	cyclictest	762911	-21	lsblk	08:25:19	0
749399	99	31	24,4	cyclictest	1675	50	irq/134-8000000.ethernet-tx2	11:00:22	2
749399	99	31	23,5	cyclictest	0	-21	swapper/2	07:50:29	2
749398	99	31	23,5	cyclictest	772776	-21	tail	09:20:27	1
749398	99	31	23,5	cyclictest	772776	-21	tail	09:20:27	1
749398	99	31	23,4	cyclictest	1674	50	irq/126-8000000.ethernet-tx1	12:00:26	1
749398	99	31	23,4	cyclictest	0	-21	swapper/1	08:35:00	1
749398	99	31	21,6	cyclictest	0	-21	swapper/1	08:05:28	1
749398	99	31	20,4	cyclictest	0	-21	swapper/1	07:20:29	1
749399	99	30	15,7	cyclictest	0	-21	swapper/2	07:10:02	2
749398	99	30	2,4	cyclictest	753183	-21	munin-run	07:30:00	1
749398	99	30	21,4	cyclictest	0	-21	swapper/1	10:55:22	1
749398	99	30	20,6	cyclictest	0	-21	swapper/1	11:15:30	1
749399	99	29	19,6	cyclictest	795055	-21	grep	11:30:27	2
749398	99	29	16,9	cyclictest	23	1	ktimers/1	09:00:12	1
749399	99	28	19,6	cyclictest	791668	-21	grep	11:10:29	2
749399	99	28	18,7	cyclictest	0	-21	swapper/2	07:20:18	2
749399	99	28	18,6	cyclictest	0	-21	swapper/2	07:35:28	2
749399	99	28	18,3	cyclictest	0	-21	swapper/2	12:20:01	2
749399	99	28	17,7	cyclictest	0	-21	swapper/2	12:15:01	2
749399	99	28	16,8	cyclictest	0	-21	swapper/2	11:25:01	2
749399	99	28	14,8	cyclictest	780855	-21	/usr/sbin/munin	10:10:18	2
749398	99	28	18,6	cyclictest	0	-21	swapper/1	12:30:23	1
749399	99	27	19,4	cyclictest	0	-21	swapper/2	11:45:27	2
749399	99	27	15,8	cyclictest	0	-21	swapper/2	08:15:01	2
749398	99	27	19,5	cyclictest	0	-21	swapper/1	07:10:01	1
749398	99	27	19,3	cyclictest	0	-21	swapper/1	08:50:20	1
749396	99	27	15,7	cyclictest	749332	-21	latency_hist	07:10:02	0
749399	99	26	17,5	cyclictest	767612	-21	grep	08:50:27	2
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2020-10-04 - 12:21