You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-13 - 22:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sat Dec 13, 2025 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1547600211955,59sleep30-21swapper/307:07:173
154289228469,9sleep20-21swapper/207:05:032
154772627963,10sleep10-21swapper/107:08:551
154289527756,14sleep00-21swapper/007:05:050
18098052660,3sleep2421rcuc/211:15:172
1547955992516,6cyclictest67150irq/132-enp1s0-TxRx-309:35:133
154795599227,9cyclictest1609592-21taskset08:05:153
154795599227,12cyclictest0-21swapper/309:55:183
154795599222,13cyclictest1586994-21taskset07:45:113
1547955992219,1cyclictest0-21swapper/308:25:003
154795599212,13cyclictest1781653-21taskset10:50:123
1547955992118,1cyclictest0-21swapper/309:35:003
154795599202,13cyclictest17381952sleep310:05:203
154795599192,11cyclictest1776462-21taskset10:45:113
1547955991916,1cyclictest0-21swapper/310:20:163
1547955991916,1cyclictest0-21swapper/307:25:143
1547955991916,1cyclictest0-21swapper/307:15:083
154795599191,15cyclictest0-21swapper/311:55:003
154795599191,15cyclictest0-21swapper/311:15:123
154795599191,13cyclictest1797373-21chrt11:05:123
154795599190,16cyclictest0-21swapper/311:20:153
18033112181,7sleep21804987-21kthreadcore11:10:162
1547955991816,1cyclictest0-21swapper/312:30:123
1547955991815,1cyclictest0-21swapper/307:55:143
1547955991814,2cyclictest0-21swapper/307:35:553
154795599181,13cyclictest0-21swapper/311:00:103
154795599181,13cyclictest0-21swapper/309:15:003
154795599181,12cyclictest0-21swapper/308:10:113
154795599181,11cyclictest1748150-21taskset10:15:173
154795599180,14cyclictest0-21swapper/312:20:113
154795599179,4cyclictest18340082sleep311:40:113
1547955991714,1cyclictest0-21swapper/311:45:173
1547955991714,1cyclictest0-21swapper/309:19:593
1547955991714,1cyclictest0-21swapper/307:30:183
154795599171,13cyclictest1765317-21taskset10:30:333
154795599171,13cyclictest0-21swapper/309:50:113
154795599171,12cyclictest1807564-21turbostat11:14:593
154795599171,12cyclictest1655325-21chrt08:50:123
154795599171,12cyclictest1655078-21chrt08:50:003
154795599170,14cyclictest0-21swapper/309:25:233
154795599170,13cyclictest0-21swapper/312:25:203
154795599170,13cyclictest0-21swapper/308:55:113
154795599170,12cyclictest0-21swapper/312:05:133
154795599170,10cyclictest1786826-21taskset10:55:013
18031012161,10sleep11803310-21tr11:10:151
17554312162,9sleep10-21swapper/110:25:021
16319422161,4sleep11632658-21memory08:25:191
16319422161,4sleep11632658-21memory08:25:181
154795599167,6cyclictest1744364-21taskset10:13:453
154795599167,6cyclictest1670670-21taskset09:03:553
154795599162,9cyclictest1713352-21taskset09:45:113
154795599161,9cyclictest1892262-21taskset12:35:043
154795599161,9cyclictest1641773-21taskset08:35:173
154795599161,9cyclictest1618363-21taskset08:15:053
154795599161,3cyclictest0-21swapper/310:00:113
1547955991613,1cyclictest0-21swapper/312:00:183
1547955991613,1cyclictest0-21swapper/311:35:133
1547955991613,1cyclictest0-21swapper/311:35:133
154795599161,12cyclictest1770962-21sleep10:40:003
154795599161,12cyclictest0-21swapper/309:05:183
154795599161,11cyclictest1548102-21chrt07:10:053
154795599161,10cyclictest1712152-21taskset09:40:223
154795599161,10cyclictest1649778-21taskset08:44:443
154795599160,13cyclictest0-21swapper/312:10:143
154795599160,13cyclictest0-21swapper/311:30:193
154795599160,13cyclictest0-21swapper/307:50:223
154795599160,12cyclictest0-21swapper/311:25:173
154795599151,9cyclictest1775495-21taskset10:40:263
1547955991512,1cyclictest0-21swapper/310:25:193
1547955991512,1cyclictest0-21swapper/308:30:133
154795599151,10cyclictest1875266-21chrt12:15:253
154795599150,13cyclictest0-21swapper/311:55:263
154795599150,12cyclictest0-21swapper/309:20:183
154795599150,12cyclictest0-21swapper/308:00:103
154795599150,12cyclictest0-21swapper/307:40:203
154795599150,12cyclictest0-21swapper/307:25:003
154795599150,12cyclictest0-21swapper/307:25:003
18872692144,5sleep11887405-21cut12:30:151
18168862141,8sleep21817027-21netstat11:20:182
16815682141,8sleep11681578-21latency_hist09:15:001
16645132140,1sleep10-21swapper/108:55:211
16346442141,8sleep21635600-21cut08:30:172
16338122141,9sleep21633825-21latency_hist08:30:012
16338122141,9sleep21633825-21latency_hist08:30:002
1547955991411,1cyclictest0-21swapper/308:25:153
1547955991411,1cyclictest0-21swapper/308:25:153
17978632130,8sleep21797960-21ps11:05:162
17659502131,7sleep21765957-21cpu10:35:132
17256022131,7sleep01727404-21cut09:55:190
18620382121,7sleep11863880-21cut12:05:181
18566172121,6sleep11858407-21kthreadcore12:00:171
17996082121,7sleep01797352-21/usr/sbin/munin11:05:170
17190952121,7sleep11719190-21kthreadcore09:50:161
16992172121,3sleep21700988-21kthreadcore09:30:162
16880842120,8sleep01689802-21cut09:20:160
16187852121,7sleep01618940-21/usr/sbin/munin08:15:160
15811042121,2sleep10-21swapper/107:35:521
15553452121,7sleep11555328-21/usr/sbin/munin07:15:081
18694102111,5sleep21869461-21ntpq12:10:202
18484092111,5sleep21848458-21ntpq11:50:202
17641302111,7sleep11764584-21needreboot10:30:191
17611302112,4sleep21761246-21irqstats10:30:162
16651502110,1sleep00-21swapper/008:55:320
16525702111,5sleep11653884-21memory08:45:181
16398822110,1sleep00-21swapper/008:35:150
15971772111,5sleep11597202-21munin-run07:55:001
15869722112,4sleep21586991-21apt-get07:45:112
15752042114,3sleep01575327-21find07:30:220
15607092111,5sleep01560925-21diskstats07:20:130
15607092111,5sleep01560925-21diskstats07:20:120
18807492101,6sleep1800-21snmpd12:21:541
18747082101,4sleep01874754-21ntpq12:15:190
18734652101,4sleep21874564-21memory12:15:172
18554872101,2sleep00-21swapper/012:00:170
18293292101,5sleep20-21swapper/211:35:182
18293292101,5sleep20-21swapper/211:35:172
18132652101,5sleep11813371-21/usr/sbin/munin11:20:131
17959382101,4sleep21796079-21ntpq11:00:182
17948582101,4sleep11795937-21memory11:00:171
17874122102,4sleep01788700-21grep10:55:150
17871272101,4sleep21787339-21hddtemp_smartct10:55:142
17500352101,4sleep11750251-21diskstats10:20:131
17437822101,5sleep11743838-21sed10:10:311
17329652101,4sleep21733022-21ntpq10:00:272
17281582101,4sleep11728375-21users09:55:281
17277022101,4sleep21727750-21ntpq09:55:212
17186052101,5sleep21718788-21expr09:50:122
17136382101,4sleep11713762-21perl09:45:141
17068172101,4sleep21706866-21ntpq09:35:212
17028142100,6sleep11702815-21/usr/sbin/munin09:35:121
16922742101,2sleep20-21swapper/209:25:122
16488012101,4sleep11648851-21ntpq08:40:211
16487652101,5sleep01648800-21ld08:40:200
16241582101,5sleep20-21swapper/208:20:142
16171562101,4sleep21617201-21ntpq08:10:202
15811002101,4sleep21580551-31find07:35:492
15750952101,4sleep21575141-21ntpq07:30:202
15750422101,4sleep11575093-21ntpq07:30:201
15558272101,4sleep01555898-21irqstats07:15:110
15554002101,4sleep21555615-21diskstats07:15:092
1892293291,4sleep21892557-21df_inode12:35:122
1884620291,4sleep21885129-21needreboot12:25:182
1882831291,4sleep11884619-21cut12:25:171
1881827291,4sleep01882830-21cut12:25:160
1865913291,4sleep11865984-21irqstats12:10:161
1864565291,3sleep21864782-21users12:05:272
1859226292,3sleep21859463-21/usr/sbin/munin12:00:262
1849666291,3sleep11849685-21apt-get11:55:121
1843301290,2sleep10-21swapper/111:45:231
1828493291,4sleep11828514-21munin-run11:35:011
1793136291,4sleep01794857-21grep11:00:160
1771750291,4sleep21772161-21cut10:40:182
1762327291,5sleep00-21swapper/010:30:180
1755450291,2sleep00-21swapper/010:25:140
1703106291,4sleep01703229-21gltestperf09:35:150
1701497291,3sleep01701551-21ntpq09:30:180
1691333291,4sleep11691555-21unixbench_multi09:20:251
1686060291,4sleep01-21systemd09:15:230
1678640291,4sleep21680290-21cut09:10:202
1655039291,4sleep21655076-21idleruntime-cro08:50:002
1640192291,4sleep21641772-21cut08:35:172
1627424291,3sleep11627565-21ntpq08:20:171
1607347290,1sleep20-21swapper/208:00:582
1601795291,4sleep01602039-21users07:55:230
1581762291,3sleep01582018-21df07:40:120
1569765291,3sleep21569878-21ntpq07:25:192
1566202291,4sleep11566305-21perl07:25:141
1896285281,3sleep11896331-21ntpq12:35:191
1876028281,4sleep00-21swapper/012:20:120
1848523281,4sleep01848610-21/usr/sbin/munin11:50:220
1833141280,4sleep01833358-21users11:35:280
1833141280,4sleep01833358-21users11:35:270
1781629280,3sleep20-21swapper/210:50:122
1754811280,1sleep20-21swapper/210:23:462
1746383281,3sleep21748147-21kthreadcore10:15:172
1743582281,3sleep21743666-21ps10:10:292
1734551281,3sleep21734258-21/usr/sbin/munin10:05:152
1698106281,3sleep11698141-21kthreadcore09:30:151
1675684280,1sleep00-21swapper/009:05:290
1671041281,3sleep21671202-21cstates09:05:112
1662424281,3sleep21664219-21kthreadcore08:55:172
1655301281,2sleep20-21swapper/208:50:122
1613185280,3sleep10-21swapper/108:10:111
1608500281,3sleep21608535-21kthreadcore08:05:152
1597074280,1sleep20-21swapper/207:53:452
1591021281,3sleep01591072-21/usr/sbin/munin07:45:210
1581098280,3sleep01580551-31find07:35:470
1548673281,3sleep11549689-21kthreadcore07:10:111
1896204270,3sleep00-21swapper/012:35:180
1874756270,4sleep10-21swapper/112:15:201
1869517271,3sleep01869633-21grep12:10:230
1860716271,2sleep01862036-21kthreadcore12:05:170
1850785271,3sleep21852483-21kthreadcore11:55:182
1843966271,3sleep21843957-21gdbus11:47:162
1838668270,1sleep10-21swapper/111:41:171
1834031270,3sleep01834287-21df11:40:120
1828303270,1sleep00-21swapper/011:32:350
1823255270,1sleep10-21swapper/111:29:591
1822903270,1sleep20-21swapper/211:25:312
1818882270,3sleep01820068-21kthreadcore11:25:170
1813682271,3sleep01815299-21kthreadcore11:20:170
1788702271,3sleep11790516-21cut10:55:171
1776953271,2sleep20-21swapper/210:45:162
1760246270,1sleep20-21swapper/210:27:552
1750534271,2sleep01751740-21kthreadcore10:20:160
1749677271,3sleep00-21swapper/010:20:000
1745296271,2sleep10-21swapper/110:15:161
1744195270,1sleep00-21swapper/010:11:340
1739055270,1sleep10-21swapper/110:07:461
1717378271,2sleep00-21swapper/009:45:210
1695660271,3sleep10-21swapper/109:25:181
1671019270,2sleep10-21swapper/109:05:111
1670425270,1sleep20-21swapper/209:00:462
1666069271,3sleep11666124-21fschecks_count09:00:131
1649669270,1sleep20-21swapper/208:43:192
1639048271,3sleep11639099-21gpgv08:35:001
1638913270,1sleep00-21swapper/008:33:190
1630195271,2sleep00-21swapper/008:25:170
1630195271,2sleep00-21swapper/008:25:170
1628511271,2sleep01628512-21sshd08:24:130
1622325271,2sleep10-21swapper/108:15:211
1591621270,1sleep10-21swapper/107:45:541
1584637270,1sleep10-21swapper/107:40:171
1560926271,3sleep11561133-21grep07:20:131
1560926271,3sleep11561133-21grep07:20:131
1549692271,2sleep20-21swapper/207:10:122
1886779261,2sleep20-21swapper/212:30:122
1853478260,2sleep00-21swapper/011:55:190
1849067260,1sleep10-21swapper/111:50:431
1838142260,3sleep20-21swapper/211:40:222
1828753260,2sleep10-21swapper/111:35:131
1828753260,2sleep10-21swapper/111:35:121
1827451260,2sleep20-21swapper/211:30:202
1812541260,4sleep00-21swapper/011:16:510
1811802260,2sleep10-21swapper/111:15:201
1806687261,2sleep00-21swapper/011:10:220
1801767262,1sleep10-21swapper/111:05:261
1781908261,2sleep00-21swapper/010:50:140
1781213260,1sleep10-21swapper/110:48:031
1771236260,2sleep10-21swapper/110:40:131
1771211260,2sleep00-21swapper/010:40:120
1770927261,2sleep0248-21systemd-journal10:40:010
1738249260,2sleep00-21swapper/010:05:210
1729603260,4sleep10-21swapper/110:00:181
1713349260,3sleep20-21swapper/209:45:112
1712830260,1sleep10-21swapper/109:41:411
1711442261,2sleep00-21swapper/009:40:210
1708669261,2sleep20-21swapper/209:40:182
1693885261,2sleep00-21swapper/009:25:170
1682694261,2sleep10-21swapper/109:15:161
1681524260,1sleep00-21swapper/009:14:270
1665784260,4sleep00-21swapper/009:00:080
1659616261,2sleep00-21swapper/008:50:260
1639585260,2sleep10-21swapper/108:35:121
1612632260,4sleep00-21swapper/008:06:180
1586563260,4sleep20-21swapper/207:43:122
1567122261,2sleep00-21swapper/007:25:170
1565452260,1sleep20-21swapper/207:22:272
1565452260,1sleep20-21swapper/207:22:272
1886727250,1sleep00-21swapper/012:30:010
1879640250,1sleep20-21swapper/212:20:182
1785424251,1sleep10-21swapper/110:50:191
1770661250,1sleep10-21swapper/110:36:371
1730943250,2sleep00-21swapper/010:00:240
1722666250,2sleep01722802-21mailstats09:50:230
1691688250,3sleep20-21swapper/209:20:502
1659170250,1sleep10-21swapper/108:50:191
1622381250,1sleep20-21swapper/208:15:212
1616973250,2sleep00-21swapper/008:10:180
1611882250,1sleep10-21swapper/108:05:181
1604777250,2sleep00-21swapper/008:00:160
1597887250,1sleep10-21swapper/107:55:121
1597216250,1sleep20-21swapper/207:55:012
1595957250,1sleep01596130-21strings07:50:180
1838959240,1sleep00-21swapper/011:45:000
1776183240,1sleep00-21swapper/010:45:020
1681792240,1sleep20-21swapper/209:15:022
1650092240,1sleep00-21swapper/008:45:130
1602414240,1sleep10-21swapper/108:00:011
1548104240,1sleep00-21swapper/007:10:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional