You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 13:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Sep 18, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158840421910,2sleep10-21swapper/123:00:081
1296107214797,37sleep20-21swapper/219:09:262
12960032145100,16sleep10-21swapper/119:08:041
1295866214599,16sleep00-21swapper/019:06:230
1296000214299,34sleep30-21swapper/319:08:023
166895021230,2sleep1321rcuc/123:50:151
1296275994236,4cyclictest1475135-21tail21:45:243
1296275993530,3cyclictest1660226-21/usr/sbin/munin23:45:003
1296275993423,7cyclictest1624932-21grep23:20:283
1296275992924,3cyclictest0-21swapper/300:00:003
1296275992920,6cyclictest1723431-21cut00:25:163
1296275992920,6cyclictest1613009-21cut23:15:143
1296275992622,2cyclictest1353549-21dump-pmu-power20:04:593
1296275992621,3cyclictest1487351-21sed21:55:223
1296275992616,7cyclictest1449489-21cut21:30:193
1296275992519,3cyclictest1446257-21cron21:30:003
1296275992516,6cyclictest1583840-21cut22:55:203
1296275992420,2cyclictest0-21swapper/320:05:233
1296275992417,5cyclictest1328210-21cat19:40:003
1296275992415,6cyclictest1540701-21cut22:30:163
129627599241,17cyclictest943-21runrttasks00:05:143
17194852230,2sleep0221rcuc/000:23:480
129627599238,10cyclictest703-21dbus-daemon22:25:013
129627599234,16cyclictest1-21systemd22:50:013
129627599232,8cyclictest1450312-21kworker/u16:2+rpciod21:36:293
1296275992319,2cyclictest1482344-21latency_hist21:55:013
1296275992319,2cyclictest0-21swapper/321:10:193
1296275992318,2cyclictest1676329-21latency_hist23:55:013
1296275992317,4cyclictest0-21swapper/300:20:003
1296275992316,5cyclictest1503587-21grep22:05:273
1296275992311,7cyclictest1492145-21tr22:00:163
1296275992219,2cyclictest1727560-21cpuspeed_turbos00:30:153
1296275992218,2cyclictest1378535-21grep20:30:013
1296275992217,3cyclictest943-21runrttasks23:05:173
1296275992214,6cyclictest1382522-21grep20:30:193
1296275992214,6cyclictest1317665-21grep19:25:223
129627599221,17cyclictest1522836-21sh22:19:393
14194182210,2sleep20-21swapper/221:10:532
1296275992118,2cyclictest703-21dbus-daemon00:37:133
129627599211,7cyclictest1338165-21runrttasks19:49:433
1296275992117,2cyclictest0-21swapper/322:40:013
1296275992117,2cyclictest0-21swapper/319:20:133
129627599211,6cyclictest1560014-21sh22:40:213
1296275992116,3cyclictest1705629-21apt-key00:15:013
1296275992116,3cyclictest0-21swapper/300:00:223
1296275992113,5cyclictest1434758-21find21:20:223
1296275992113,5cyclictest1434758-21find21:20:213
129627599211,16cyclictest1538376-21sh22:28:413
1296275992110,7cyclictest1605321-21tr23:10:173
129627599209,7cyclictest1647842-21tr23:35:193
129627599204,6cyclictest1633943-21sh23:26:513
129627599204,6cyclictest1633943-21sh23:26:503
1296275992017,2cyclictest1593024-21head23:00:283
1296275992016,2cyclictest0-21swapper/322:55:003
1296275992016,2cyclictest0-21swapper/322:10:133
1296275992016,2cyclictest0-21swapper/321:15:123
1296275992015,3cyclictest1661115-21kthreadcore23:45:183
1296275992014,3cyclictest0-21swapper/300:22:013
1296275992013,5cyclictest1402734-21uname20:50:263
129627599201,2cyclictest1450312-21kworker/u16:2+rpciod21:44:253
1296275992010,7cyclictest1343775-21apt-get19:55:113
129627599197,8cyclictest1636336-21seq23:30:013
129627599192,14cyclictest1403784-21apt-get20:55:113
1296275991914,3cyclictest726-21in:imuxsock19:41:433
1296275991911,6cyclictest1307753-21grep19:15:283
129627599189,6cyclictest1363131-21sshd20:10:423
129627599189,6cyclictest1343497-21cron19:54:593
129627599181,6cyclictest1363741-21/usr/sbin/munin20:15:123
129627599181,5cyclictest1414413-21sed21:10:003
129627599181,5cyclictest1368843-21/usr/sbin/munin20:20:123
1296275991813,3cyclictest945-21snmpd20:40:093
129627599181,15cyclictest1393732-21runrttasks20:45:103
16899042171,12sleep11414218-21diskmemload00:02:031
16759092170,2sleep00-21swapper/023:54:440
15199942171,12sleep21520005-21sh22:15:312
13002772171,4sleep11300322-21ntpq19:10:221
129627599178,6cyclictest1300340-21ntpq19:10:223
129627599176,7cyclictest1387658-21/usr/sbin/munin20:35:213
17312192161,10sleep21731260-21ntpq00:30:222
16448082160,2sleep11644965-21ssh23:35:161
13171112161,10sleep11317159-21ntpq19:25:171
129627599168,5cyclictest945-21snmpd21:01:233
129627599161,2cyclictest1322693-21/usr/sbin/munin19:30:273
16185682150,1sleep20-21swapper/223:17:452
16004732152,7sleep01600524-21ntpq23:05:200
15614132151,4sleep01414218-21diskmemload22:41:350
15396442158,3sleep0171ktimers/022:30:130
15148602151,8sleep20-21swapper/222:14:522
14375272151,9sleep20-21swapper/221:23:572
14375272151,9sleep20-21swapper/221:23:562
16525812140,2sleep10-21swapper/123:40:141
16272642141,3sleep01414218-21diskmemload23:23:490
16200962141,9sleep01-21systemd23:19:590
15947922141,8sleep20-21swapper/223:02:552
15358042141,8sleep11535976-21ssh22:25:261
14583342142,6sleep01458471-21ntpq21:35:210
16831332131,8sleep01414218-21diskmemload23:58:520
16744762131,8sleep21674486-21ssh23:52:212
16363542131,3sleep21636362-21ssh23:30:042
16105942133,2sleep2421ktimers/223:12:412
14812452130,5sleep10-21swapper/121:53:471
14748932131,8sleep11475086-21ssh21:45:241
14689442134,2sleep0191rcu_preempt21:43:020
17127042121,7sleep21712711-21ssh00:19:342
16443692120,2sleep00-21swapper/023:35:020
15966712121,7sleep11596821-21tr23:05:131
15871772121,4sleep0191rcu_preempt22:58:470
15768982121,6sleep01414218-21diskmemload22:50:380
15635432120,1sleep20-21swapper/222:44:422
15530972121,8sleep01553111-21ssh22:36:150
15307902121,4sleep00-21swapper/022:24:270
15274262121,8sleep11527490-21ntpq22:20:231
15111752121,5sleep11511238-21ntpq22:10:211
15043312121,7sleep21414218-21diskmemload22:06:212
14539332120,2sleep00-21swapper/021:34:330
14368432121,7sleep01436857-21ssh21:22:550
14368432121,7sleep01436857-21ssh21:22:540
14267972120,8sleep21427029-21tail21:15:272
14206482121,6sleep01414218-21diskmemload21:12:440
13591702121,8sleep00-21swapper/020:10:170
17339662111,3sleep10-21swapper/100:34:571
17270262111,4sleep21414218-21diskmemload00:29:562
17040462115,1sleep10-21swapper/100:12:091
16984992111,4sleep01698517-21apt-get00:10:150
16902772111,3sleep20-21swapper/200:03:002
16669732111,3sleep10-21swapper/123:48:071
16357472110,7sleep01414218-21diskmemload23:29:340
16357472110,7sleep01414218-21diskmemload23:29:330
15861952111,3sleep10-21swapper/122:57:241
15771492111,3sleep10-21swapper/122:50:581
15702362111,5sleep11570244-21sshd22:47:421
15556302111,3sleep20-21swapper/222:39:542
15454992111,3sleep20-21swapper/222:32:232
15435912111,5sleep11543655-21ntpq22:30:221
15218232111,3sleep00-21swapper/022:18:110
14760072111,3sleep20-21swapper/221:46:332
14667612111,3sleep20-21swapper/221:40:252
14443392111,3sleep20-21swapper/221:27:092
14441362111,3sleep00-21swapper/021:26:510
14145432111,5sleep11414574-21ssh21:10:081
13033962111,4sleep11303414-21apt-get19:15:141
17392252101,5sleep11704495-21kworker/u16:2+rpciod00:36:221
17251662101,4sleep01414218-21diskmemload00:25:550
17193192100,1sleep20-21swapper/200:23:322
17054022101,4sleep21414218-21diskmemload00:14:542
16977962101,4sleep11414218-21diskmemload00:08:271
16972782101,4sleep21414218-21diskmemload00:07:202
16967312101,4sleep01414218-21diskmemload00:06:130
16896382101,3sleep00-21swapper/000:01:370
16826372100,1sleep20-21swapper/223:58:022
16663602101,3sleep20-21swapper/223:47:152
16605822101,4sleep01660601-21apt-get23:45:130
16571802101,2sleep20-21swapper/223:40:312
16439902101,3sleep00-21swapper/023:34:520
16334442100,2sleep1191rcu_preempt23:26:071
16334442100,2sleep1191rcu_preempt23:26:061
16255312101,4sleep21414218-21diskmemload23:21:112
16201652101,4sleep11414218-21diskmemload23:20:011
16181042101,3sleep10-21swapper/123:17:041
16103082101,4sleep01414218-21diskmemload23:12:150
16018442101,3sleep20-21swapper/223:06:222
15884312101,2sleep00-21swapper/023:00:120
15878492101,3sleep20-21swapper/222:59:462
15699512101,4sleep01414218-21diskmemload22:47:170
15691012101,2sleep20-21swapper/222:46:032
15611042100,3sleep10-21swapper/122:41:081
15557122101,4sleep11555758-21munin-run22:40:011
15387782100,1sleep20-21swapper/222:29:192
15370772101,4sleep01414218-21diskmemload22:26:490
15303422100,3sleep20-21swapper/222:23:492
15203192101,2sleep10-21swapper/122:15:591
14979632101,3sleep11414218-21diskmemload22:03:531
14905002101,2sleep10-21swapper/121:59:511
14886512101,4sleep01414218-21diskmemload21:57:140
14813542100,3sleep20-21swapper/221:53:562
14504692101,5sleep21450525-21ntpq21:30:232
14466262101,2sleep10-21swapper/121:30:141
14444322101,3sleep10-21swapper/121:27:161
14135612101,5sleep11413577-21munin-run21:04:591
14079332101,4sleep11408150-21users20:55:251
13683442100,1sleep00-21swapper/020:18:050
13542532100,6sleep21355502-21kthreadcore20:05:182
13371062101,3sleep01337179-21ntpq19:45:200
1740480290,3sleep20-21swapper/200:39:312
1733622291,3sleep01414218-21diskmemload00:34:110
1725977291,3sleep11414218-21diskmemload00:27:401
1719592290,6sleep10-21swapper/100:24:031
1710963291,2sleep10-21swapper/100:16:041
1681806291,2sleep10-21swapper/123:56:361
1657444290,2sleep00-21swapper/023:40:530
1649688291,3sleep21414218-21diskmemload23:36:252
1641236290,1sleep10-21swapper/123:30:441
1634420291,3sleep21414218-21diskmemload23:27:362
1634420291,3sleep21414218-21diskmemload23:27:352
1611021290,2sleep10-21swapper/123:13:201
1576028291,3sleep21576080-21ntpq22:50:192
1513417291,3sleep01414218-21diskmemload22:12:450
1505032291,3sleep11505042-21ssh22:07:221
1504074291,4sleep01414218-21diskmemload22:06:000
1496006291,3sleep01414218-21diskmemload22:01:020
1491932291,4sleep20-21swapper/222:00:172
1490274290,2sleep20-21swapper/221:59:322
1479639291,3sleep01414218-21diskmemload21:51:280
1476268291,3sleep01414218-21diskmemload21:46:560
1467690291,2sleep10-21swapper/121:41:121
1459774291,2sleep10-21swapper/121:36:221
1430545291,3sleep11414218-21diskmemload21:20:071
1430545291,3sleep11414218-21diskmemload21:20:061
1427376291,2sleep10-21swapper/121:15:501
1422500291,2sleep00-21swapper/021:15:090
1408978291,4sleep01409081-21perl21:00:130
1407583291,4sleep01407636-21ntp_states20:55:210
1397486291,4sleep11397535-21ntpq20:45:191
1395348291,4sleep21397135-21cut20:45:162
1372911291,4sleep01373158-21/usr/sbin/munin20:20:270
1362492291,4sleep11362542-21ntpq20:10:211
1332188291,4sleep21332238-21ntpq19:40:202
1322091293,2sleep0221rcuc/019:30:200
1740539280,2sleep00-21swapper/000:39:380
1710984280,2sleep00-21swapper/000:16:090
1461234280,2sleep20-21swapper/221:38:312
1387393281,3sleep21387471-21ld20:35:182
1382508281,3sleep01382557-21ntpq20:30:200
1368777281,2sleep20-21swapper/220:20:112
1367918281,3sleep21368120-21users20:15:282
1365621281,4sleep11367313-21latency20:15:181
1350476284,1sleep10-21swapper/120:00:151
1348314280,1sleep20-21swapper/219:57:122
1339127281,3sleep11339221-21kthreadcore19:50:171
1333431280,3sleep10-21swapper/119:45:131
1317569281,3sleep01317786-21users19:25:240
1313190281,3sleep11313208-21munin-run19:25:001
1308436280,3sleep21308454-21apt-get19:20:132
1296449281,3sleep00-21swapper/019:10:120
1413966270,1sleep10-21swapper/121:07:011
1408806270,2sleep20-21swapper/221:00:122
1403349270,1sleep20-21swapper/220:53:032
1373793270,2sleep10-21swapper/120:25:111
1363162270,1sleep20-21swapper/220:11:062
1358247270,1sleep10-21swapper/120:06:491
1348819270,2sleep00-21swapper/020:00:100
1327912270,1sleep10-21swapper/119:37:151
1323365270,1sleep00-21swapper/019:35:060
1318953271,3sleep11320275-21kthreadcore19:30:171
1313448271,2sleep20-21swapper/219:25:092
1307459271,3sleep21307513-21switchtime19:15:272
1301162270,1sleep20-21swapper/219:14:082
1414007260,1sleep20-21swapper/221:07:292
1413912260,1sleep00-21swapper/021:06:190
1399228261,2sleep00-21swapper/020:50:190
1388146260,2sleep10-21swapper/120:35:501
1383765260,2sleep00-21swapper/020:35:120
1374934260,2sleep20-21swapper/220:25:172
1357292260,2sleep01357450-21strings20:05:190
1348997261,2sleep20-21swapper/220:00:122
1343929261,2sleep00-21swapper/019:55:140
1339333261,1sleep20-21swapper/219:50:182
1338726260,1sleep00-21swapper/019:50:120
1334778261,2sleep20-21swapper/219:45:182
1327897260,1sleep20-21swapper/219:37:042
1312838260,1sleep00-21swapper/019:20:340
1307868260,1sleep00-21swapper/019:16:300
1408523250,1sleep20-21swapper/220:59:592
1403378250,1sleep10-21swapper/120:53:261
1397441250,1sleep00-21swapper/020:45:180
1392448250,1sleep10-21swapper/120:40:191
1389253251,1sleep20-21swapper/220:40:162
1388943250,1sleep00-21swapper/020:40:140
1383263251,2sleep20-21swapper/220:32:152
1378934250,1sleep10-21swapper/120:30:131
1372756250,1sleep10-21swapper/120:20:241
1347436250,1sleep10-21swapper/119:55:191
1332419251,2sleep10-21swapper/119:40:231
1332023250,2sleep00-21swapper/019:40:180
1323015250,3sleep20-21swapper/219:33:462
1377389240,1sleep00-21swapper/020:25:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional