OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8
Special All - All RT - Optimization - Ethernet - Thumbnails - Next

Click here to display the system's profile data or here to proceed to next system.

Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system

Characteristics of all highest latencies:

System rackeslot8s.osadl.org (updated Sun Sep 27, 2026 12:43:53)
Delayed (victim)					Switcher (culprit)			Timestamp	CPU
PID	Prio	Total latency (µs)	T^(,W^*) latency (µs)	Cmd	PID	Prio	Cmd	Timestamp	CPU
1316084	2	349	0,2	sleep1	0	-21	swapper/1		1
1316084	2	349	0,2	sleep1	0	-21	swapper/1		1
1249485	2	36	0,3	sleep1	1152126	-21	kworker/u16:2-kvfree_rcu_reclaim		1
1145970	99	25	22,1	cyclictest	0	-21	swapper/3		3
1596228	2	23	0,1	sleep2	41	1	irq_work/2		2
1442677	2	23	0,2	sleep2	45	-21	ksoftirqd/2		2
1145970	99	22	18,1	cyclictest	0	-21	swapper/3		3
1145970	99	20	16,1	cyclictest	0	-21	swapper/3		3
1574593	2	19	0,1	sleep0	22	1	rcuc/0		0
1166315	2	19	0,9	sleep2	1166444	-21	cat		2
1145970	99	19	8,7	cyclictest	1338599	-21	turbostat		3
1145970	99	19	17,0	cyclictest	0	-21	swapper/3		3
1145970	99	19	16,1	cyclictest	0	-21	swapper/3		3
1145970	99	19	16,1	cyclictest	0	-21	swapper/3		3
1145970	99	19	15,2	cyclictest	0	-21	swapper/3		3
1588563	2	18	0,7	sleep2	1590245	-21	cut		2
1145970	99	18	16,1	cyclictest	0	-21	swapper/3		3
1145970	99	18	16,1	cyclictest	0	-21	swapper/3		3
1145970	99	18	15,1	cyclictest	0	-21	swapper/3		3
1145970	99	18	14,1	cyclictest	0	-21	swapper/3		3
1145970	99	18	1,12	cyclictest	1382425	-21	taskset		3
1384830	2	17	2,1	sleep3	0	-21	swapper/3		3
1145970	99	17	15,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	15,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	14,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	14,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	14,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	14,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	14,1	cyclictest	0	-21	swapper/3		3
1145970	99	17	1,11	cyclictest	1522216	-21	taskset		3
1145970	99	17	1,11	cyclictest	1468343	-21	taskset		3
1145970	99	17	1,11	cyclictest	1345250	-21	taskset		3
1145970	99	17	1,11	cyclictest	1177680	-21	taskset		3
1145970	99	17	1,11	cyclictest	1146130	-21	taskset		3
1145970	99	16	2,9	cyclictest	1243569	-21	taskset		3
1145970	99	16	2,10	cyclictest	1583974	-21	taskset		3
1145970	99	16	2,10	cyclictest	1528763	-21	taskset		3
1145970	99	16	2,10	cyclictest	1492244	-21	taskset		3
1145970	99	16	2,10	cyclictest	1257650	-21	taskset		3
1145970	99	16	13,1	cyclictest	0	-21	swapper/3		3
1145970	99	16	13,1	cyclictest	0	-21	swapper/3		3
1145970	99	16	13,1	cyclictest	0	-21	swapper/3		3
1145970	99	16	1,11	cyclictest	1601497	-21	chrt		3
1145970	99	16	1,11	cyclictest	1601164	-21	sleep		3
1145970	99	16	1,11	cyclictest	1562902	-21	chrt		3
1145970	99	16	1,11	cyclictest	1561502	-21	taskset		3
1145970	99	16	1,11	cyclictest	1436279	-21	taskset		3
1145970	99	16	1,11	cyclictest	1406960	-21	taskset		3
1145970	99	16	1,11	cyclictest	1398912	-21	taskset		3
1145970	99	16	1,11	cyclictest	1200157	-21	taskset		3
1145970	99	16	1,11	cyclictest	1192234	-21	taskset		3
1145970	99	16	1,10	cyclictest	1255799	-21	sleep		3
1145970	99	16	1,10	cyclictest	1218095	-21	taskset		3
1145970	99	16	1,10	cyclictest	1167297	-21	taskset		3
1145970	99	16	1,10	cyclictest	1159064	-21	taskset		3
1484965	2	15	0,5	sleep2	1485541	-21	/usr/sbin/munin		2
1145970	99	15	2,10	cyclictest	0	-21	swapper/3		3
1145970	99	15	2,10	cyclictest	0	-21	swapper/3		3
1145970	99	15	1,9	cyclictest	1546005	-21	taskset		3
1145970	99	15	1,9	cyclictest	1292999	-21	taskset		3
1145970	99	15	1,9	cyclictest	1237134	2	sleep3		3
1145970	99	15	1,9	cyclictest	1217469	-21	taskset		3
1145970	99	15	13,1	cyclictest	0	-21	swapper/3		3
1145970	99	15	13,1	cyclictest	0	-21	swapper/3		3
1145970	99	15	12,1	cyclictest	0	-21	swapper/3		3
1145970	99	15	1,10	cyclictest	1609239	-21	taskset		3
1145970	99	15	1,10	cyclictest	1591899	-21	taskset		3
1145970	99	15	1,10	cyclictest	1485542	-21	taskset		3
1145970	99	15	1,10	cyclictest	1413013	-21	taskset		3
1145970	99	15	1,10	cyclictest	1313426	-21	chrt		3
1145970	99	15	1,10	cyclictest	1313426	-21	chrt		3
1145970	99	15	1,10	cyclictest	1301509	-21	taskset		3
1145970	99	15	1,10	cyclictest	1299334	-21	taskset		3
1145970	99	15	1,10	cyclictest	1274599	-21	taskset		3
1145970	99	15	1,10	cyclictest	1268271	-21	taskset		3
1145970	99	15	1,10	cyclictest	1224435	-21	chrt		3
1305719	2	14	0,10	sleep0	1305766	-21	ntpq		0
1273989	2	14	0,6	sleep0	1274025	-21	ntpq		0
1145970	99	14	11,1	cyclictest	0	-21	swapper/3		3
1608266	2	12	0,9	sleep1	1608304	-21	9		1
1584970	2	12	0,9	sleep1	1300186	-21	diskmemload		1
1417094	2	12	0,8	sleep2	1418720	-21	cut		2
1250715	2	12	0,4	sleep2	1251474	-21	cut		2
1145768	2	12	0,2	sleep3	0	-21	swapper/3		3
1145768	2	12	0,2	sleep3	0	-21	swapper/3		3
1538742	2	11	0,7	sleep1	1539312	-21	sed		1
1461220	2	11	1,7	sleep0	1461279	-21	test		0
1421731	2	11	0,8	sleep1	1421735	-21	ssh		1
1405684	2	11	0,8	sleep2	1405928	-21	users		2
1379254	2	11	0,1	sleep1	31	1	irq_work/1		1
1345889	2	11	0,8	sleep2	1300186	-21	diskmemload		2
1314213	2	11	0,8	sleep0	1314215	-21	ssh		0
1314213	2	11	0,8	sleep0	1314215	-21	ssh		0
1139820	2	11	1,4	sleep2	1139853	-21	apt-get		2
1139820	2	11	1,4	sleep2	1139853	-21	apt-get		2
1587355	2	10	1,6	sleep1	0	-21	swapper/1		1
1585581	2	10	0,2	sleep2	0	-21	swapper/2		2
1530842	2	10	0,7	sleep2	1531001	-21	gltestperf		2
1522565	2	10	0,7	sleep1	1522572	-21	ssh		1
1482115	2	10	0,6	sleep1	1482162	-21	ntpq		1
1475800	2	10	0,7	sleep0	1300186	-21	diskmemload		0
1474520	2	10	1,5	sleep2	1474567	-21	ntpq		2
1414679	2	10	0,7	sleep2	1300186	-21	diskmemload		2
1368005	2	10	0,8	sleep0	0	-21	swapper/0		0
1315968	2	10	0,7	sleep0	1316083	-21	/usr/sbin/munin		0
1315968	2	10	0,7	sleep0	1316083	-21	/usr/sbin/munin		0
1250134	2	10	0,7	sleep1	1250713	-21	cut		1
1600718	2	9	1,5	sleep1	1300186	-21	diskmemload		1
1583282	2	9	3,3	sleep0	1583327	-21	ntpq		0
1522469	2	9	0,5	sleep2	1300186	-21	diskmemload		2
1415862	2	9	0,2	sleep0	1415942	-21	ssh		0
1375798	2	9	0,7	sleep2	1375804	-21	ssh		2
1361879	2	9	1,3	sleep1	1361881	-21	apt-get		1
1360824	2	9	0,6	sleep1	1300186	-21	diskmemload		1
1346585	2	9	1,3	sleep2	1346607	-21	apt-get		2
1345145	2	9	1,2	sleep0	0	-21	swapper/0		0
1331213	2	9	1,4	sleep1	1331250	-21	apt-get		1
1324314	2	9	0,6	sleep1	1325066	-21	cut		1
1211752	2	9	1,4	sleep2	1211909	-21	diskstats		2
1186769	2	9	0,6	sleep2	1187667	-21	cut		2
1141255	2	9	1,1	sleep0	17	1	rcu_preempt		0
1141255	2	9	1,1	sleep0	17	1	rcu_preempt		0
1586394	2	8	1,4	sleep0	1587354	-21	irqstats		0
1576933	2	8	0,2	sleep2	0	-21	swapper/2		2
1564261	2	8	0,5	sleep2	1565871	-21	cut		2
1560126	2	8	0,4	sleep0	1560171	-21	ntpq		0
1554897	2	8	1,3	sleep2	1554934	-21	apt-get		2
1548008	2	8	2,3	sleep1	1549124	-21	ssh		1
1523010	2	8	1,3	sleep1	1483293	-21	kworker/u16:0+rpciod		1
1507124	2	8	0,5	sleep1	1300186	-21	diskmemload		1
1466856	2	8	1,4	sleep2	1466894	-21	ntp_states		2
1463437	2	8	0,5	sleep1	1465161	-21	cut		1
1452833	2	8	1,5	sleep2	1452837	-21	ssh		2
1437307	2	8	0,5	sleep0	1300186	-21	diskmemload		0
1431333	2	8	0,4	sleep2	1432294	-21	irqstats		2
1429370	2	8	0,5	sleep0	1429376	-21	ssh		0
1399379	2	8	0,2	sleep2	1399385	-21	ssh		2
1377096	2	8	1,3	sleep2	1377135	-21	apt-get		2
1370448	2	8	0,5	sleep1	1371229	-21	cut		1
1369468	2	8	1,3	sleep0	0	-21	swapper/0		0
1336295	2	8	0,4	sleep2	1336421	-21	ntpq		2
1330404	2	8	0,5	sleep2	1330408	-21	ssh		2
1315871	2	8	1,3	sleep2	1315903	-21	apt-get		2
1315871	2	8	1,3	sleep2	1315903	-21	apt-get		2
1275546	2	8	0,5	sleep1	1275686	-21	/usr/sbin/munin		1
1254904	2	8	0,4	sleep0	1254932	-21	ntpq		0
1188721	2	8	0,5	sleep0	1190563	-21	kthreadcore		0
1185116	2	8	0,4	sleep2	1185157	-21	ntpq		2
1159634	2	8	1,3	sleep0	1145933	-21	kworker/0:2+kacpid		0
1144946	2	8	1,3	sleep1	0	-21	swapper/1		1
1144946	2	8	1,3	sleep1	0	-21	swapper/1		1
1606702	2	7	0,3	sleep0	1606748	-21	ntpq		0
1601468	2	7	1,3	sleep2	0	-21	swapper/2		2
1593709	2	7	0,4	sleep0	0	-21	swapper/0		0
1569836	2	7	0,1	sleep1	0	-21	swapper/1		1
1561717	2	7	0,4	sleep1	1300186	-21	diskmemload		1
1553205	2	7	1,3	sleep0	1553240	-21	quilt		0
1544166	2	7	0,1	sleep2	0	-21	swapper/2		2
1528286	2	7	1,3	sleep2	1528405	-21	python3		2
1507657	2	7	0,4	sleep0	1507819	-21	diskstats		0
1507608	2	7	0,3	sleep2	1507649	-21	ssh		2
1506920	2	7	1,3	sleep2	1300186	-21	diskmemload		2
1500141	2	7	0,3	sleep0	1500213	-21	/usr/sbin/munin		0
1495311	2	7	0,1	sleep1	17	1	rcu_preempt		1
1486260	2	7	0,4	sleep0	1487905	-21	cut		0
1476939	2	7	1,3	sleep2	0	-21	swapper/2		2
1475603	2	7	1,3	sleep1	1300186	-21	diskmemload		1
1459053	2	7	0,3	sleep2	1459099	-21	ssh		2
1451562	2	7	0,3	sleep0	1451691	-21	python3		0
1447811	2	7	0,3	sleep1	1449476	-21	grep		1
1430834	2	7	0,3	sleep1	1430871	-21	apt-get		1
1424085	2	7	1,3	sleep2	0	-21	swapper/2		2
1423167	2	7	1,2	sleep1	0	-21	swapper/1		1
1407806	2	7	0,3	sleep0	1407838	-21	apt-get		0
1398356	2	7	0,1	sleep1	0	-21	swapper/1		1
1390168	2	7	0,4	sleep0	1390258	-21	ssh		0
1384880	2	7	0,3	sleep1	1385038	-21	df_inode		1
1342798	2	7	1,3	sleep1	1343909	-21	memory		1
1323964	2	7	1,4	sleep0	1324312	-21	cut		0
1314368	2	7	0,1	sleep1	0	-21	swapper/1		1
1314368	2	7	0,1	sleep1	0	-21	swapper/1		1
1314156	2	7	0,1	sleep2	0	-21	swapper/2		2
1314156	2	7	0,1	sleep2	0	-21	swapper/2		2
1307751	2	7	0,1	sleep1	1307757	-21	ssh		1
1300542	2	7	0,3	sleep2	1300545	-21	apt-get		2
1295093	2	7	0,4	sleep0	1295715	-21	tr		0
1280326	2	7	0,4	sleep0	1280370	-21	ntpq		0
1268833	2	7	0,3	sleep2	1268866	-21	apt-get		2
1261164	2	7	0,4	sleep2	1261201	-21	ntpq		2
1235853	2	7	0,3	sleep1	1235890	-21	ntpq		1
1229614	2	7	0,4	sleep1	1229641	-21	ntpq		1
1217028	2	7	0,4	sleep1	1211715	-21	/usr/sbin/munin		1
1210580	2	7	0,4	sleep1	1210618	-21	ntp_states		1
1201150	2	7	0,5	sleep0	1202947	-21	grep		0
1194195	2	7	0,5	sleep1	1195905	-21	cut		1
1192863	2	7	0,4	sleep2	1193062	-21	hddtemp_smartct		2
1178920	2	7	0,1	sleep1	17	1	rcu_preempt		1
1624220	2	6	0,1	sleep2	0	-21	swapper/2		2
1619530	2	6	0,3	sleep0	1621231	-21	cut		0
1615290	2	6	0,3	sleep2	1615296	-21	ssh		2
1614628	2	6	0,3	sleep0	1614676	-21	ntp_states		0
1609307	2	6	0,3	sleep1	1609318	-21	ssh		1
1570423	2	6	0,2	sleep1	0	-21	swapper/1		1
1562688	2	6	0,2	sleep0	1562722	-21	apt-get		0
1546025	2	6	0,2	sleep2	0	-21	swapper/2		2
1543561	2	6	0,3	sleep0	1543612	-21	ntp_states		0
1537304	2	6	0,3	sleep1	1300186	-21	diskmemload		1
1531133	2	6	0,3	sleep0	1532093	-21	irqstats		0
1528945	2	6	0,3	sleep0	1528951	-21	ssh		0
1522430	2	6	0,1	sleep0	0	-21	swapper/0		0
1492396	2	6	0,3	sleep0	1492424	-21	awk		0
1484509	2	6	0,2	sleep1	0	-21	swapper/1		1
1477103	2	6	0,3	sleep0	0	-21	swapper/0		0
1461603	2	6	1,2	sleep0	0	-21	swapper/0		0
1453820	2	6	0,1	sleep1	0	-21	swapper/1		1
1445824	2	6	0,1	sleep1	0	-21	swapper/1		1
1438547	2	6	1,2	sleep0	0	-21	swapper/0		0
1407417	2	6	1,2	sleep1	0	-21	swapper/1		1
1406266	2	6	0,1	sleep0	223	-21	plymouthd		0
1382285	2	6	1,2	sleep0	1382332	-21	ntpq		0
1362933	2	6	0,3	sleep2	1363828	-21	cat		2
1360219	2	6	0,2	sleep0	0	-21	swapper/0		0
1354191	2	6	0,2	sleep2	1354226	-21	apt-get		2
1351747	2	6	0,3	sleep0	1351796	-21	ntpq		0
1349719	2	6	0,3	sleep1	1351504	-21	ssh		1
1336783	2	6	0,3	sleep0	1337033	-21	users		0
1299291	2	6	0,3	sleep2	1299333	-21	ntp_states		2
1288266	2	6	0,3	sleep0	0	-21	swapper/0		0
1287394	2	6	0,4	sleep0	1249526	-21	kworker/0:2+kacpid		0
1286607	2	6	0,3	sleep1	1286656	-21	ntpq		1
1281723	2	6	0,4	sleep2	1281911	-21	idleruntime		2
1281215	2	6	0,1	sleep2	0	-21	swapper/2		2
1264103	2	6	0,3	sleep0	1265742	-21	kthreadcore		0
1259417	2	6	1,2	sleep0	0	-21	swapper/0		0
1249448	2	6	0,3	sleep0	1249482	-21	awk		0
1248508	2	6	0,4	sleep2	1248550	-21	ntpq		2
1242744	2	6	0,1	sleep2	0	-21	swapper/2		2
1230226	2	6	0,1	sleep0	0	-21	swapper/0		0
1219492	2	6	0,3	sleep0	1220881	-21	cat		0
1204230	2	6	0,3	sleep1	1204261	-21	ntpq		1
1199439	2	6	0,3	sleep2	1200156	-21	sed		2
1192651	2	6	0,4	sleep0	0	-21	swapper/0		0
1185809	2	6	0,1	sleep1	0	-21	swapper/1		1
1178739	2	6	1,2	sleep2	0	-21	swapper/2		2
1174041	2	6	0,3	sleep0	1174578	-21	irqrtprio		0
1169441	2	6	1,2	sleep2	0	-21	swapper/2		2
1168415	2	6	0,3	sleep1	926	-21	sshd		1
1166701	2	6	0,1	sleep1	0	-21	swapper/1		1
1160441	2	6	0,1	sleep1	0	-21	swapper/1		1
1159770	2	6	0,3	sleep2	1159815	-21	ntpq		2
1617150	2	5	0,2	sleep1	0	-21	swapper/1		1
1510442	2	5	0,2	sleep1	1512175	-21	grep		1
1492181	2	5	0,2	sleep2	1492187	-21	ssh		2
1408677	2	5	0,2	sleep1	1409262	-21	ssh		1
1392500	2	5	0,1	sleep0	1392534	-21	apt-get		0
1390670	2	5	0,3	sleep2	0	-21	swapper/2		2
1300137	2	5	0,1	sleep1	0	-21	swapper/1		1
1293558	2	5	0,3	sleep2	44	1	ktimers/2		2
1289178	2	5	0,1	sleep1	0	-21	swapper/1		1
1274027	2	5	0,2	sleep1	1274076	-21	ntpq		1
1262536	2	5	0,2	sleep2	0	-21	swapper/2		2
1262193	2	5	0,2	sleep1	0	-21	swapper/1		1
1238415	2	5	0,2	sleep0	1239168	-21	kthreadcore		0
1232571	2	5	0,2	sleep2	0	-21	swapper/2		2
1230741	2	5	0,2	sleep0	0	-21	swapper/0		0
1224097	2	5	1,3	sleep2	44	1	ktimers/2		2
1211714	2	5	0,2	sleep0	0	-21	swapper/0		0
1206813	2	5	1,2	sleep2	0	-21	swapper/2		2
1205396	2	5	0,2	sleep0	0	-21	swapper/0		0
1185198	2	5	0,3	sleep0	1185321	-21	python3		0
1166942	2	5	0,1	sleep0	0	-21	swapper/0		0
1152247	2	5	0,3	sleep0	1152288	-21	systemd-run		0
1146178	2	5	0,2	sleep1	0	-21	swapper/1		1
1239172	2	4	0,2	sleep1	1240982	-21	cut		1
1224842	2	4	0,2	sleep2	0	-21	swapper/2		2
1220882	2	4	0,2	sleep1	1222754	-21	kthreadcore		1
1186312	2	4	0,1	sleep1	0	-21	swapper/1		1
1147570	2	4	0,2	sleep2	1148516	-21	kthreadcore		2
1262500	2	3	0,1	sleep1	0	-21	swapper/1		1
1167331	2	3	0,1	sleep0	0	-21	swapper/0		0
^Timer ^*Wakeup (latency=timer+wakeup+contextswitch)


Access: public		Short link to this page
This page was last modified on 2022-03-28 - 20:20