You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 20:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Mon Dec 01, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10600259912190,1210cyclictest1496912-21turbostat3
10600259911061,1100cyclictest1432186-21sleep3
122123621210,119sleep01220908-21kworker/u16:1+efi_rts_wq0
1060025992212,0cyclictest0-21swapper/33
1060025992118,1cyclictest0-21swapper/33
1060025992112,7cyclictest0-21swapper/33
1060025992011,0cyclictest0-21swapper/33
106002599187,8cyclictest1175883-21turbostat3
1060025991816,1cyclictest0-21swapper/33
1060025991811,0cyclictest0-21swapper/33
1060025991714,1cyclictest0-21swapper/33
11584232160,6sleep014-21ksoftirqd/00
11584232160,6sleep014-21ksoftirqd/00
106002599167,7cyclictest1262351-21turbostat3
106002599167,7cyclictest11950612sleep33
106002599161,11cyclictest1344053-21taskset3
106002599161,11cyclictest1094854-21chrt3
106002599161,11cyclictest1081382-21taskset3
106002599154,8cyclictest1335156-21turbostat3
106002599152,9cyclictest1408036-21taskset3
106002599151,9cyclictest13677812sleep33
1060025991513,1cyclictest0-21swapper/33
1060025991513,1cyclictest0-21swapper/33
1060025991513,1cyclictest0-21swapper/33
1060025991513,1cyclictest0-21swapper/33
1060025991512,2cyclictest0-21swapper/33
1060025991512,2cyclictest0-21swapper/33
1060025991512,1cyclictest0-21swapper/33
1060025991512,1cyclictest0-21swapper/33
106002599151,11cyclictest1399662-21turbostat3
106002599151,11cyclictest1303172-21chrt3
106002599145,6cyclictest14486572sleep33
106002599144,7cyclictest1365260-21taskset3
106002599141,9cyclictest15133922sleep33
1060025991413,0cyclictest0-21swapper/33
1060025991412,1cyclictest0-21swapper/33
1060025991412,1cyclictest0-21swapper/33
1060025991412,1cyclictest0-21swapper/33
1060025991411,1cyclictest0-21swapper/33
1060025991411,1cyclictest0-21swapper/33
1060025991411,1cyclictest0-21swapper/33
1060025991411,1cyclictest0-21swapper/33
106002599141,10cyclictest1533394-21taskset3
106002599141,10cyclictest1456897-21taskset3
106002599141,10cyclictest1446552-21taskset3
106002599141,10cyclictest1378144-21taskset3
106002599141,10cyclictest1324851-21taskset3
106002599141,10cyclictest1285586-21taskset3
106002599141,10cyclictest1207563-21taskset3
106002599141,10cyclictest1181534-21taskset3
106002599141,10cyclictest1162800-21taskset3
106002599141,10cyclictest1162800-21taskset3
106002599141,10cyclictest1150221-21taskset3
106002599141,10cyclictest1101756-21taskset3
106002599140,5cyclictest0-21swapper/33
190898130,8rtkit-daemon1237824-21ssh0
106002599134,7cyclictest1087605-21taskset3
106002599134,6cyclictest12300892sleep33
106002599131,9cyclictest1537556-21chrt3
106002599131,9cyclictest1486907-21chrt3
106002599131,9cyclictest1252709-21chrt3
106002599131,9cyclictest1113082-21taskset3
1060025991311,1cyclictest0-21swapper/33
1060025991311,1cyclictest0-21swapper/33
106002599131,10cyclictest1358753-21chrt3
106002599131,10cyclictest1358753-21chrt3
1060025991310,1cyclictest0-21swapper/33
106002599130,10cyclictest1511429-21chrt3
106002599130,10cyclictest1301407-21taskset3
106002599130,10cyclictest1270705-21taskset3
190898120,6rtkit-daemon1251810-21ntp_states0
14407722121,8sleep11440575-21/usr/sbin/munin1
14161222120,6sleep20-21swapper/22
12648632120,1sleep00-21swapper/00
106002599121,9cyclictest1503828-21taskset3
106002599120,10cyclictest1120630-21taskset3
10594462120,3sleep30-21swapper/33
15345972111,7sleep01534740-21/usr/sbin/munin0
14483282110,8sleep01448336-21ssh0
13704812110,5sleep10-21swapper/11
190898100,7rtkit-daemon473-21gmain0
190898100,6rtkit-daemon697-21runrttasks0
190898100,5rtkit-daemon438-21wpa_supplicant0
190898100,5rtkit-daemon1448908-21diskstats0
190898100,5rtkit-daemon1136972-21ntp_states2
190898100,3rtkit-daemon473-21gmain0
190898100,3rtkit-daemon1350301-21ssh1
190898100,3rtkit-daemon1066555-21munin-node2
190898100,3rtkit-daemon1066555-21munin-node0
14977272100,7sleep01498722-21irqstats0
14960462106,2sleep10-21swapper/11
14730722102,4sleep21473107-21apt-get2
14639082100,1sleep00-21swapper/00
14220112100,1sleep01422019-21ssh0
14066712100,7sleep21406679-21ssh2
13935742100,7sleep21395536-21cut2
13274272100,1sleep01327587-21diskstats0
12524142100,7sleep21252422-21ssh2
12439812101,6sleep01244237-21unixbench_singl0
19089890,6rtkit-daemon1066555-21munin-node2
19089890,5rtkit-daemon319-21systemd-udevd2
19089890,5rtkit-daemon1522641-21irqrtprio0
19089890,5rtkit-daemon1490269-21sed2
19089890,5rtkit-daemon1413418-21python30
19089890,5rtkit-daemon1183346-21irqcore2
19089890,5rtkit-daemon1143166-21needreboot2
19089890,4rtkit-daemon1284977-21ssh0
19089890,4rtkit-daemon1276510-21ssh0
19089890,4rtkit-daemon1198284-21cut2
1551155290,6sleep21551400-21awk2
1536510290,6sleep11536518-21ssh1
1482551291,5sleep11483732-21kthreadcore1
1397077291,6sleep11397141-21ntp_states1
1382250290,6sleep01382259-21sshd0
1309653291,5sleep21309651-21ssh2
1213620291,2sleep00-21swapper/00
1211551291,5sleep11212586-21memory1
1150380290,6sleep01150465-21latency_hist0
1150239291,2sleep10-21swapper/11
1125693291,5sleep01125892-21hddtemp_smartct0
19089880,5rtkit-daemon1928-21sendmail-mta2
19089880,4rtkit-daemon667-21snmpd0
19089880,4rtkit-daemon667-21snmpd0
19089880,4rtkit-daemon667-21snmpd0
19089880,4rtkit-daemon438-21wpa_supplicant2
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon438-21wpa_supplicant0
19089880,4rtkit-daemon438-21wpa_supplicant0
19089880,4rtkit-daemon1511171-21ssh2
19089880,4rtkit-daemon1109979-21cut2
19089880,4rtkit-daemon1106364-21munin-run2
19089880,4rtkit-daemon1074351-21sed2
19089880,4rtkit-daemon1066555-21munin-node2
19089880,4rtkit-daemon1066555-21munin-node2
19089880,3rtkit-daemon1075207-21df2
1551528281,4sleep01551536-21ssh0
1551044280,1sleep10-21swapper/11
1497433280,5sleep21497556-21fw_packets2
1425928280,5sleep01427862-21cut0
1424389281,4sleep21424398-21/usr/sbin/munin2
1422301280,6sleep11422307-21ssh1
1358404280,5sleep01358412-21ssh0
1358404280,5sleep01358412-21ssh0
1352116281,4sleep11353111-21irqstats1
1352116281,4sleep11353111-21irqstats1
1317587280,1sleep20-21swapper/22
1300860281,4sleep11300862-21ssh1
1294527281,4sleep21294556-21munin-run2
1263118281,4sleep21263787-21cut2
1261687281,5sleep11261693-21ssh1
1228332280,2sleep21228340-21ssh2
1087613281,3sleep11087646-21apt-get1
19089870,4rtkit-daemon667-21snmpd2
19089870,4rtkit-daemon667-21snmpd2
19089870,4rtkit-daemon421-21avahi-daemon2
19089870,4rtkit-daemon1342905-21pool-udisksd0
19089870,4rtkit-daemon1066555-21munin-node2
19089870,4rtkit-daemon1066555-21munin-node0
19089870,3rtkit-daemon674-21lldpd2
19089870,3rtkit-daemon671-21ntpd1
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd2
19089870,3rtkit-daemon667-21snmpd0
19089870,3rtkit-daemon667-21snmpd0
19089870,3rtkit-daemon473-21gmain2
19089870,3rtkit-daemon473-21gmain1
19089870,3rtkit-daemon1928-21sendmail-mta2
19089870,3rtkit-daemon1928-21sendmail-mta2
19089870,3rtkit-daemon1455160-21ssh2
19089870,3rtkit-daemon1066555-21munin-node1
1537649271,3sleep01537787-21expr0
1519345270,5sleep01519350-21sh0
1504031271,4sleep11504039-21ssh1
1487591270,1sleep00-21swapper/00
1456855271,3sleep20-21swapper/22
1446421271,4sleep20-21swapper/22
1415752270,4sleep11415802-21munin-run1
1413169271,3sleep21413252-21ntp_states2
1406569271,3sleep11406575-21ssh1
1366737270,1sleep00-21swapper/00
1338040270,5sleep11340103-21cut1
1326085270,1sleep10-21swapper/11
1317637270,1sleep00-21swapper/00
1280214270,4sleep11281450-21grep1
1277180270,4sleep11277188-21ssh1
1249032270,4sleep11251165-21kthreadcore1
1240347270,4sleep21242354-21tr2
1195219271,4sleep01195419-21hddtemp_smartct0
1150659270,3sleep11150692-21apt-get1
1138069271,4sleep01138100-21apt0
1130640270,4sleep21130696-21ntp_states2
1124532271,3sleep11124748-21unixbench-2d1
1081787270,4sleep21082672-21sed2
1081089271,3sleep11081287-21sed1
19089860,3rtkit-daemon81364-21packagekitd2
19089860,3rtkit-daemon667-21snmpd2
1522844260,4sleep11524010-21kthreadcore1
1507604261,2sleep00-21swapper/00
1490227261,2sleep00-21swapper/00
1483733260,3sleep21485823-21grep2
1472638260,1sleep20-21swapper/22
1472562260,1sleep00-21swapper/00
1384221261,2sleep10-21swapper/11
1366012260,1sleep10-21swapper/11
1341699260,4sleep21341707-21ssh2
1327794261,2sleep2421rcuc/22
1325236260,3sleep21325244-21ssh2
1317431260,1sleep10-21swapper/11
1301751261,3sleep21183850-21kworker/u16:2+nfsiod2
1293912260,3sleep11293920-21ssh1
1284524260,1sleep20-21swapper/22
1262007260,1sleep20-21swapper/22
1236399260,4sleep11236410-21ssh1
1220316261,2sleep10-21swapper/11
1201380260,3sleep11201443-21ls1
1201344261,2sleep01201377-21apt-get0
1194559260,1sleep10-21swapper/11
1181892260,1sleep00-21swapper/00
1177827260,3sleep11179877-21grep1
1169800260,2sleep01169833-21apt-get0
1138924261,2sleep10-21swapper/11
1118918261,3sleep01118967-21latency_hist0
1112644260,3sleep11112876-21date1
1108060261,2sleep00-21swapper/00
1100590260,3sleep11100759-21if_err_eno11
1095996260,3sleep01098090-21kthreadcore0
1093315260,1sleep00-21swapper/00
1068858260,3sleep01068905-21fschecks_count0
1060801260,3sleep11061786-21kthreadcore1
1057470260,2sleep10-21swapper/11
19089850,1rtkit-daemon0-21swapper/11
1528549250,3sleep21528557-21ssh2
1432440250,3sleep10-21swapper/11
1431790250,1sleep10-21swapper/11
1389026250,3sleep21389082-21ntp_states2
1376966251,2sleep20-21swapper/22
1375757250,2sleep10-21swapper/11
1374994250,3sleep20-21swapper/22
1343909250,3sleep21344043-21ps2
1333571250,1sleep10-21swapper/11
1310825250,1sleep10-21swapper/11
1277402250,3sleep21277410-21ssh2
1268050251,2sleep13055290-21kworker/u16:0+rpciod1
1245898251,2sleep10-21swapper/11
1229319250,3sleep10-21swapper/11
1188788250,2sleep00-21swapper/00
1175587250,1sleep10-21swapper/11
1165861250,3sleep01168002-21kthreadcore0
1162681250,1sleep10-21swapper/11
1162681250,1sleep10-21swapper/11
1137083251,2sleep10-21swapper/11
1136980251,2sleep00-21swapper/00
1125493251,1sleep11125526-21apt-get1
1121826251,2sleep00-21swapper/00
1118908250,1sleep10-21swapper/11
1105538250,2sleep01105655-21python30
1083788250,2sleep00-21swapper/00
1081000250,1sleep00-21swapper/00
1074427250,1sleep10-21swapper/11
1060663250,3sleep01060799-21cut0
1055428250,2sleep00-21swapper/00
1438478240,2sleep21438484-21ssh2
1236651240,3sleep20-21swapper/22
1195420240,2sleep11195734-21cut1
1188329240,3sleep10-21swapper/11
1168005240,1sleep10-21swapper/11
1505232230,1sleep10-21swapper/11
1182421230,1sleep00-21swapper/00
1155629230,1sleep00-21swapper/00
1098095230,1sleep10-21swapper/11
1085924230,1sleep10-21swapper/11
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional