You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-19 - 15:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sun Oct 19, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
196580021890,1sleep10-21swapper/112:35:001
15374172148103,15sleep20-21swapper/207:06:572
1537385214397,15sleep30-21swapper/307:06:343
15375972132117,6sleep00-21swapper/007:09:150
15374062132108,18sleep10-21swapper/107:06:481
1537791993928,7cyclictest1839560-21grep11:10:273
1537791993826,8cyclictest1966423-21cut12:35:143
17527362320,1sleep065-21kcompactd010:11:400
1537791993227,3cyclictest1668068-21sh09:15:003
1537791993126,3cyclictest703-21dbus-daemon12:10:003
1537791993024,4cyclictest1568943-40gdbus07:35:243
1537791993024,4cyclictest1568943-40gdbus07:35:243
1537791992923,3cyclictest1949811-21sort12:25:143
1537791992823,3cyclictest1732707-21turbostat10:00:003
1537791992823,3cyclictest0-21swapper/308:20:003
1537791992822,4cyclictest726-21in:imuxsock10:50:273
1537791992715,8cyclictest736-21gdbus09:40:233
1537791992521,3cyclictest726-21in:imuxsock09:35:003
1537791992519,3cyclictest0-21swapper/310:15:213
153779199247,5cyclictest1867687-21/usr/sbin/munin11:30:183
153779199245,5cyclictest1703712-21sh09:39:303
1537791992419,3cyclictest1660452-21idleruntime-cro09:10:003
153779199241,19cyclictest1894066-21taskset11:50:153
153779199235,6cyclictest1745125-21sh10:05:503
1537791992318,3cyclictest0-21swapper/309:45:163
153779199227,13cyclictest1884512-21sh11:42:513
1537791992219,2cyclictest1635375-21cut08:45:003
1537791992218,3cyclictest1559940-21/usr/sbin/munin07:30:233
1537791992217,3cyclictest1558846-21pmu-power07:25:223
1537791992214,5cyclictest1928808-21grep12:10:263
1537791992214,5cyclictest1823898-21cut11:00:193
1537791992213,2cyclictest0-21swapper/310:45:273
153779199221,18cyclictest1862206-21runrttasks11:27:193
153779199221,18cyclictest1553756-21ntpq07:20:223
1537791992211,7cyclictest1689277-21sh09:28:543
1537791992210,8cyclictest736-21gdbus10:05:003
1537791992210,7cyclictest1913080-21sshd12:00:413
153779199219,8cyclictest1889888-21sed11:45:203
153779199219,8cyclictest1660423-21diskmemload11:17:363
1537791992117,2cyclictest1589962-21/usr/sbin/munin08:00:013
1537791992117,2cyclictest0-21swapper/308:15:013
1537791992116,3cyclictest1775953-21cron10:30:003
1537791992116,3cyclictest1754122-21munin-plugin-st10:15:003
1537791992115,4cyclictest511ktimers/310:37:263
1537791992115,4cyclictest1962399-21unixbench_multi12:30:293
1537791992113,6cyclictest1722673-21cut09:50:253
1537791992113,6cyclictest1722673-21cut09:50:253
1537791992112,6cyclictest1907019-21runrttasks11:58:063
153779199211,17cyclictest1948107-21sh12:23:393
153779199209,8cyclictest1849711-21apt-get11:20:123
153779199206,8cyclictest1541846-21ntpq07:10:243
1537791992016,2cyclictest0-21swapper/312:15:143
1537791992014,4cyclictest511ktimers/310:20:153
1537791992012,5cyclictest1640337-21sshd08:49:423
1537791992012,5cyclictest1640337-21sshd08:49:423
1537791992011,7cyclictest1660423-21diskmemload10:41:363
1537791992011,7cyclictest0-21swapper/311:35:593
1537791992010,8cyclictest0-21swapper/310:30:513
1537791992010,6cyclictest1583733-21runrttasks07:50:173
153779199200,17cyclictest1832452-21runrttasks11:05:373
153779199200,17cyclictest1681195-21sh09:22:133
19548382191,5sleep11660423-21diskmemload12:26:261
153779199197,9cyclictest1574891-21awk07:45:003
1537791991915,2cyclictest1615312-21sh08:25:003
1537791991911,5cyclictest1674538-21sh09:18:253
153779199189,7cyclictest1545079-21df_inode07:15:123
1537791991814,2cyclictest1813697-21gltestperf10:55:153
1537791991814,2cyclictest1813697-21gltestperf10:55:153
1537791991813,3cyclictest1619462-21pmu-power08:25:233
153779199180,16cyclictest1654680-21/usr/sbin/munin09:00:253
153779199180,15cyclictest1629885-21sed08:35:273
18487062171,2sleep20-21swapper/211:19:252
17875102171,5sleep01787568-21ntpq10:35:210
153779199179,6cyclictest1650329-21chrt08:59:393
1537791991710,5cyclictest1640643-21/usr/sbin/munin08:50:113
19325992161,2sleep10-21swapper/112:15:041
18618292161,10sleep11861834-21ssh11:26:411
17904102161,3sleep11790426-21ssh10:39:551
17466552161,3sleep00-21swapper/010:09:000
15904322161,2sleep10-21swapper/108:00:141
1537791991611,3cyclictest1590321-21cpuspeed_turbos08:00:123
19573382151,3sleep20-21swapper/212:29:482
19217592151,3sleep00-21swapper/012:06:570
17271232151,10sleep21728892-21kthreadcore09:55:192
153779199151,9cyclictest1595227-21munin-node08:05:123
153779199151,2cyclictest1621009-21/usr/sbin/munin08:30:163
1537791991511,2cyclictest0-21swapper/307:45:133
18243282141,7sleep01824384-21ntpq11:00:230
17081582141,6sleep21708219-21ntpq09:40:192
19215912131,8sleep11660423-21diskmemload12:06:421
19123312131,8sleep21912408-21sed12:00:222
17680472131,2sleep20-21swapper/210:23:452
17413442131,8sleep11743000-21cut10:05:201
17318882131,7sleep01660423-21diskmemload09:58:410
16814352131,8sleep11681452-21ssh09:22:431
15645712130,1sleep10-21swapper/107:33:371
15379832131,8sleep01538105-21tr07:10:160
19623712120,8sleep01962493-21ssh12:30:300
19387352121,6sleep21660423-21diskmemload12:17:252
18707852121,7sleep21870791-21ssh11:35:002
17600812120,8sleep11760093-21ssh10:17:221
16592722121,7sleep21655660-21/usr/sbin/munin09:05:202
19483902110,7sleep21948400-21ssh12:24:032
19471722111,3sleep00-21swapper/012:22:240
18843382111,3sleep10-21swapper/111:42:331
18340252111,6sleep11834035-21sh11:08:331
18113282110,7sleep21811338-21ssh10:52:072
18043212110,1sleep00-21swapper/010:48:280
17908412111,5sleep11790863-21apt-get10:40:111
17378492110,1sleep00-21swapper/010:00:580
17189432111,6sleep11719826-21kthreadcore09:50:171
17189432111,6sleep11719826-21kthreadcore09:50:171
16962332111,7sleep01660423-21diskmemload09:34:000
15704952111,7sleep01570707-21grep07:40:170
19714312101,3sleep20-21swapper/212:36:172
19382252101,3sleep00-21swapper/012:16:430
19320632101,3sleep20-21swapper/212:14:412
19297722100,1sleep10-21swapper/112:11:351
19235142100,3sleep20-21swapper/212:09:232
19084122101,4sleep11908571-21df_inode12:00:121
19061852101,3sleep20-21swapper/211:56:452
19047022101,3sleep10-21swapper/111:55:191
18992602100,3sleep20-21swapper/211:52:242
18924122101,2sleep20-21swapper/211:48:402
18912512101,4sleep11660423-21diskmemload11:46:301
18785142100,3sleep20-21swapper/211:40:052
18757662101,2sleep20-21swapper/211:35:402
18711022101,3sleep10-21swapper/111:35:071
18688022101,4sleep01660423-21diskmemload11:31:290
18617562100,1sleep20-21swapper/211:26:342
18544912100,3sleep20-21swapper/211:21:012
18481272101,3sleep10-21swapper/111:18:151
18409722101,3sleep10-21swapper/111:12:581
18329272100,3sleep20-21swapper/211:06:302
18196102101,3sleep20-21swapper/210:59:122
18196102101,3sleep20-21swapper/210:59:122
18042082101,4sleep11660423-21diskmemload10:48:171
18032152101,3sleep20-21swapper/210:46:142
17907902101,3sleep00-21swapper/010:40:060
17883342101,3sleep20-21swapper/210:35:392
17813512101,3sleep20-21swapper/210:31:142
17811342101,3sleep10-21swapper/110:30:531
17756192101,2sleep00-21swapper/010:29:280
17669372101,3sleep10-21swapper/110:21:351
17545092101,4sleep01754676-21diskstats10:15:130
17535012100,1sleep10-21swapper/110:13:581
17528602101,2sleep20-21swapper/210:11:542
17392942101,4sleep11660423-21diskmemload10:03:391
17330492101,4sleep21733068-21apt-get10:00:122
17118722100,6sleep21713294-21kthreadcore09:45:162
17044822101,6sleep01704615-21gltestperf09:40:130
17030192101,3sleep00-21swapper/009:38:170
17021882101,3sleep20-21swapper/209:36:272
16899082101,3sleep10-21swapper/109:30:121
16887172100,2sleep10-21swapper/109:27:481
16885212101,3sleep00-21swapper/009:27:270
16814532101,3sleep20-21swapper/209:22:442
16809162101,4sleep01660423-21diskmemload09:21:410
16741682100,1sleep00-21swapper/009:17:450
16722242101,4sleep21672282-21ntpq09:15:222
16530002101,4sleep21654251-21memory09:00:192
16506482101,4sleep01650667-21apt-get09:00:130
16356192101,4sleep21635638-21apt-get08:45:132
16356192101,4sleep21635638-21apt-get08:45:122
16105362101,4sleep11610555-21apt-get08:20:141
16048282101,3sleep00-21swapper/008:13:240
15654282105,2sleep21566933-21kthreadcore07:35:182
15654282105,2sleep21566933-21kthreadcore07:35:182
15549262101,3sleep20-21swapper/207:25:122
1971566291,2sleep10-21swapper/112:36:281
1966182290,3sleep00-21swapper/012:35:110
1963926291,3sleep21660423-21diskmemload12:32:292
1954528291,5sleep01660423-21diskmemload12:26:010
1946835290,2sleep10-21swapper/112:21:571
1929169291,3sleep01660423-21diskmemload12:10:470
1908301290,2sleep00-21swapper/012:00:030
1899875291,2sleep00-21swapper/011:53:300
1899737291,2sleep10-21swapper/111:53:151
1891866290,6sleep00-21swapper/011:47:410
1883315291,4sleep01660423-21diskmemload11:40:430
1869631290,5sleep134-21ksoftirqd/111:32:591
1861606291,2sleep00-21swapper/011:26:160
1848114290,3sleep00-21swapper/011:18:150
1839601290,2sleep20-21swapper/211:10:312
1839589291,4sleep01839598-21ssh11:10:290
1827026290,3sleep20-21swapper/211:04:132
1825386291,3sleep11660423-21diskmemload11:01:121
1819738291,2sleep00-21swapper/010:59:260
1819738291,2sleep00-21swapper/010:59:260
1813256291,3sleep10-21swapper/110:55:071
1813256291,3sleep10-21swapper/110:55:071
1811816291,3sleep11660423-21diskmemload10:52:581
1781874291,3sleep01660423-21diskmemload10:32:170
1775242290,2sleep10-21swapper/110:28:491
1773978291,3sleep21660423-21diskmemload10:26:052
1768290291,3sleep01660423-21diskmemload10:24:150
1754427291,2sleep20-21swapper/210:15:072
1723385290,2sleep20-21swapper/209:51:062
1723385290,2sleep20-21swapper/209:51:062
1718783293,3sleep01718942-21ssh09:50:160
1718783293,3sleep01718942-21ssh09:50:160
1695151291,3sleep21660423-21diskmemload09:31:422
1687765291,2sleep20-21swapper/209:26:002
1672150291,4sleep11672221-21ntpq09:15:211
1667260290,1sleep00-21swapper/009:13:320
1665926291,2sleep20-21swapper/209:11:012
1664999291,4sleep11665060-21ntp_states09:10:241
1659444291,4sleep01659485-21ntpq09:05:220
1644270291,4sleep21644343-21ntpq08:50:172
1640364291,4sleep11640383-21munin-run08:50:001
1640364291,4sleep11640383-21munin-run08:50:001
1584155290,1sleep10-21swapper/107:50:241
1580130291,3sleep01580149-21apt-get07:50:110
1564067290,6sleep00-21swapper/007:30:270
1546604291,4sleep01548410-21cut07:15:160
1901091281,3sleep01901307-21hddtemp_smartct11:55:140
1876292280,2sleep00-21swapper/011:36:340
1855243281,3sleep01660423-21diskmemload11:23:120
1850148281,3sleep11850517-21cat11:20:161
1832380280,3sleep00-21swapper/011:05:290
1811267281,3sleep01660423-21diskmemload10:52:010
1795434281,3sleep21795449-21ssh10:40:362
1729501281,3sleep11729545-21ssh09:55:221
1716272281,2sleep10-21swapper/109:46:161
1708995281,3sleep11709009-21ssh09:40:511
1647896280,2sleep10-21swapper/108:55:161
1646237281,3sleep01647894-21cut08:55:160
1625580280,3sleep11625598-21apt-get08:35:131
1615269281,4sleep21615494-21sed08:25:002
1605223280,3sleep11605242-21apt-get08:15:131
1599759280,1sleep10-21swapper/108:07:201
1595650281,3sleep21596020-21kthreadcore08:05:162
1579535280,1sleep10-21swapper/107:45:471
1575120280,2sleep20-21swapper/207:45:122
1573875281,3sleep11573921-21ntpq07:40:221
1569729280,1sleep10-21swapper/107:39:061
1569729280,1sleep10-21swapper/107:39:061
1569708280,1sleep00-21swapper/007:38:490
1569708280,1sleep00-21swapper/007:38:490
1563688281,2sleep20-21swapper/207:30:212
1548411280,2sleep11548577-21strings07:15:181
1744989271,3sleep21744998-21ssh10:05:332
1716567271,2sleep01716577-21ssh09:46:470
1700936271,3sleep11700987-21ssh09:35:211
1639676270,3sleep01639747-21find08:45:260
1639676270,3sleep01639747-21find08:45:260
1633773271,3sleep01634280-21grep08:40:170
1630609270,2sleep20-21swapper/208:40:112
1625248270,5sleep00-21swapper/008:34:120
1620569270,2sleep20-21swapper/208:30:132
1620558270,1sleep10-21swapper/108:30:041
1610488270,3sleep01609987-21apt-get08:20:010
1609845270,1sleep20-21swapper/208:18:272
1600237270,2sleep20-21swapper/208:10:132
1589842270,1sleep00-21swapper/007:58:530
1585136270,2sleep10-21swapper/107:55:091
1554534270,1sleep20-21swapper/207:23:242
1548583270,2sleep20-21swapper/207:15:192
1659724260,3sleep11659878-21sed09:05:271
1641917261,2sleep00-21swapper/008:50:150
1630910260,3sleep11630610-21/usr/sbin/munin08:40:141
1620302261,2sleep10-21swapper/108:30:001
1620008260,1sleep20-21swapper/208:26:162
1619614260,3sleep01619749-21timerwakeupswit08:25:260
1594694260,1sleep00-21swapper/008:01:380
1590216260,1sleep20-21swapper/208:00:112
1586373261,2sleep20-21swapper/207:55:142
1559662260,1sleep00-21swapper/007:29:510
1559626260,1sleep10-21swapper/107:29:281
1554600260,1sleep00-21swapper/007:24:140
1651329250,2sleep11652997-21kthreadcore09:00:181
1625578250,2sleep00-21swapper/008:35:120
1609413250,1sleep00-21swapper/008:15:280
1603916250,1sleep10-21swapper/108:10:201
1597726250,1sleep00-21swapper/008:05:180
1584186250,2sleep21584358-21ls07:50:252
1574832250,1sleep20-21swapper/207:44:442
1645604240,1sleep20-21swapper/208:55:062
1640799240,1sleep10-21swapper/108:50:121
1625601240,1sleep20-21swapper/208:35:142
1575293240,1sleep00-21swapper/007:45:140
1549991240,1sleep10-21swapper/107:20:141
1540276240,1sleep10-21swapper/107:10:211
1537663240,1sleep20-21swapper/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional