You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 22:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Jan 01, 2026 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1546269212165,52sleep30-21swapper/307:05:333
154628328553,9sleep20-21swapper/207:05:442
154657627661,9sleep10-21swapper/107:09:291
154640127354,14sleep00-21swapper/007:07:150
16902052400,2sleep20-21swapper/209:20:182
1546735992522,1cyclictest0-21swapper/310:20:003
1546735992420,2cyclictest0-21swapper/311:40:013
17632092230,2sleep30-21swapper/310:30:193
1546735992320,1cyclictest0-21swapper/312:10:003
154673599225,11cyclictest1828066-21taskset11:34:563
1546735992219,1cyclictest0-21swapper/308:45:113
1546735992117,2cyclictest0-21swapper/310:25:003
16814872201,13sleep01681554-21irqstats09:15:130
1546735992017,1cyclictest0-21swapper/309:35:243
17333292191,4sleep01733533-21cat10:05:110
15629872191,5sleep01563491-21memory07:20:170
154673599192,11cyclictest1554444-21taskset07:15:103
1546735991917,1cyclictest0-21swapper/310:30:013
1546735991916,1cyclictest0-21swapper/310:59:593
1546735991916,1cyclictest0-21swapper/310:00:283
1546735991916,1cyclictest0-21swapper/307:40:223
1546735991916,1cyclictest0-21swapper/307:15:013
154673599191,14cyclictest0-21swapper/307:50:113
18801422181,12sleep21880175-21ntpq12:20:212
16544552180,2sleep00-21swapper/008:50:120
154673599188,6cyclictest1574791-21taskset07:31:263
154673599187,8cyclictest1738320-21turbostat10:14:543
154673599187,7cyclictest1717111-21taskset09:46:423
154673599182,11cyclictest1849348-21taskset11:55:123
1546735991816,1cyclictest0-21swapper/310:05:163
1546735991816,1cyclictest0-21swapper/308:35:173
1546735991815,1cyclictest0-21swapper/311:10:123
1546735991815,1cyclictest0-21swapper/309:45:003
1546735991815,1cyclictest0-21swapper/308:00:163
1546735991814,2cyclictest0-21swapper/309:15:153
154673599181,14cyclictest0-21swapper/309:20:233
154673599181,13cyclictest17704372sleep310:40:143
154673599181,12cyclictest0-21swapper/309:10:113
154673599181,12cyclictest0-21swapper/309:10:113
154673599181,11cyclictest1808687-21taskset11:15:153
154673599180,13cyclictest0-21swapper/308:25:133
154673599177,8cyclictest1559429-21turbostat07:24:553
154673599177,6cyclictest1796888-21taskset11:05:133
1546735991714,1cyclictest0-21swapper/311:45:283
1546735991714,1cyclictest0-21swapper/311:20:183
1546735991714,1cyclictest0-21swapper/308:45:003
1546735991711,3cyclictest0-21swapper/309:05:113
154673599171,12cyclictest1622477-21turbostat08:20:003
154673599171,10cyclictest1897090-21taskset12:40:013
154673599171,10cyclictest1720907-21taskset09:50:183
154673599170,12cyclictest0-21swapper/307:45:113
18581682161,10sleep11858698-21memory12:00:201
18533722161,10sleep01853426-21ntpq11:55:220
18231282161,10sleep01823386-21df_inode11:30:120
17481312166,2sleep1171rcu_preempt10:15:261
16372492161,9sleep01637296-21open_inodes08:30:240
15704112161,10sleep11570531-21tune2fs07:30:161
154673599165,8cyclictest1580496-21sleep07:40:003
154673599161,3cyclictest0-21swapper/310:35:113
1546735991613,1cyclictest0-21swapper/311:40:163
1546735991613,1cyclictest0-21swapper/308:10:163
154673599161,2cyclictest0-21swapper/308:50:113
154673599161,12cyclictest1696447-21turbostat09:30:013
154673599161,12cyclictest1638183-21turbostat08:35:003
154673599161,11cyclictest1886432-21turbostat12:30:003
154673599161,11cyclictest1849093-21sleep11:55:003
154673599161,10cyclictest1891547-21taskset12:34:043
154673599161,10cyclictest1664735-21taskset09:00:003
154673599160,13cyclictest0-21swapper/311:00:253
154673599160,13cyclictest0-21swapper/309:55:123
154673599160,13cyclictest0-21swapper/307:55:183
16747992151,9sleep21674875-21awk09:05:252
16705482151,9sleep11670761-21/usr/sbin/munin09:05:111
154673599151,8cyclictest1775601-21taskset10:45:013
1546735991512,1cyclictest0-21swapper/312:20:153
1546735991512,1cyclictest0-21swapper/312:10:213
1546735991512,1cyclictest0-21swapper/309:05:003
1546735991512,1cyclictest0-21swapper/308:20:253
154673599151,12cyclictest0-21swapper/310:50:233
154673599151,10cyclictest1872175-21taskset12:15:163
154673599151,10cyclictest1872175-21taskset12:15:153
154673599150,12cyclictest0-21swapper/312:00:203
154673599150,12cyclictest0-21swapper/311:25:213
18959132141,9sleep11895967-21ld12:35:231
18693492141,8sleep21869387-21ntpq12:10:232
18437342141,8sleep0800-21snmpd11:48:570
18113612141,8sleep01807038-21/usr/sbin/munin11:15:260
154673599141,9cyclictest1566658-21chrt07:25:163
1546735991412,1cyclictest0-21swapper/309:30:233
17228622131,7sleep21722881-21apt-get09:55:122
16972692131,8sleep21698586-21cut09:30:172
16182672131,8sleep01619886-21cut08:15:170
15867662131,8sleep11588362-21grep07:45:161
15698602131,8sleep11569967-21cut07:30:011
15472032131,4sleep21547360-21gltestperf07:10:142
154673599131,8cyclictest1607394-21taskset08:05:133
18351882121,6sleep01836921-21kthreadcore11:40:190
16486792121,7sleep21648963-21latency_hist08:45:002
16014392121,6sleep21601460-21munin-run08:00:002
15636292121,5sleep21563661-21ntpq07:20:202
15544222122,5sleep21554441-21apt-get07:15:102
15483112121,7sleep11550008-21grep07:10:171
18427912111,5sleep11842829-21ntpq11:45:211
18056142111,6sleep21805694-21grep11:10:192
17941602111,4sleep11795145-21memory11:00:201
17814452115,3sleep11782847-21kthreadcore10:50:181
17808632111,4sleep01780882-21apt-get10:50:130
17638432111,5sleep21763890-21ntpq10:30:232
17592922112,4sleep01759322-21munin-run10:30:000
16954452111,5sleep21695491-21ntpq09:25:212
16177652111,6sleep11617823-21fschecks_count08:15:141
16109962111,4sleep21611044-21ntpq08:05:212
18641042102,3sleep11864144-21ntpq12:05:211
18606652101,5sleep00-21swapper/012:05:150
18483872101,5sleep11848585-21grep11:50:291
18376362101,4sleep11837687-21ntp_states11:40:231
18126132101,4sleep11812632-21apt-get11:20:111
18075372101,4sleep11807606-21irqstats11:15:151
17965852101,4sleep11796841-21df11:05:131
17743282101,4sleep21774366-21ntpq10:40:212
17604022101,2sleep10-21swapper/110:30:181
17320662101,4sleep11732112-21ntpq10:00:241
17268852101,5sleep11726932-21cut09:55:231
17176542101,5sleep11717745-21grep09:50:141
17126892101,4sleep21712897-21hddtemp_smartct09:45:132
17021932101,5sleep11702328-21grep09:35:141
16865382101,4sleep01686747-21hddtemp_smartct09:20:130
16859632101,5sleep21685972-21apt09:20:002
16495112101,5sleep21649720-21smartctl08:45:132
16437052101,3sleep10-21swapper/108:40:111
16331922101,4sleep21633450-21df_inode08:30:132
16279312101,4sleep11627950-21apt-get08:25:121
16267002101,4sleep21626749-21ntpq08:20:192
16070512101,4sleep11607254-21diskstats08:05:121
16022262101,4sleep11602296-21irqstats08:00:141
16020162101,4sleep01602225-21hddtemp_smartct08:00:130
15970022101,5sleep01597102-21cut07:55:130
15756552101,5sleep21575852-21ls07:35:152
1885410291,4sleep11885462-21ld12:25:211
1859811290,1sleep00-21swapper/012:04:280
1855399291,4sleep21855469-21irqstats12:00:162
1849835291,4sleep21849913-21kernelversion11:55:162
1828610291,4sleep11828643-21cut11:35:151
1827964294,3sleep1331ktimers/111:33:381
1779719294,2sleep11779808-21/usr/sbin/munin10:45:241
1770082291,4sleep2248-21systemd-journal10:40:002
1759832290,3sleep01759851-21apt-get10:30:130
1754930291,4sleep11755739-21grep10:25:161
1753128291,3sleep21753169-21ntpq10:20:252
1743822291,3sleep20-21swapper/210:15:112
1717632291,3sleep01717651-21apt-get09:50:120
1716342291,3sleep01716380-21ntpq09:45:200
1707745291,4sleep21709199-21grep09:40:152
1703238291,4sleep01704918-21cut09:35:170
1684942291,3sleep11684988-21ntpq09:15:181
1623313291,4sleep11624441-21cat08:20:141
1621504291,4sleep21621556-21modprobe08:15:232
1596507291,4sleep11596760-21df07:55:101
1595874290,1sleep00-21swapper/007:50:270
1575612291,4sleep11575653-21fschecks_count07:35:141
1565199291,3sleep01565408-21hddtemp_smartct07:25:150
1895968280,4sleep20-21swapper/212:35:242
1887249281,3sleep01888271-21kthreadcore12:30:160
1876475281,3sleep01876686-21smartctl12:20:140
1876178281,2sleep11876197-21apt-get12:20:111
1821861280,1sleep20-21swapper/211:25:222
1818429281,3sleep11818481-21kthreadcore11:25:171
1817384280,1sleep00-21swapper/011:22:020
1813186281,3sleep21814573-21kthreadcore11:20:172
1801316280,1sleep20-21swapper/211:07:172
1796563280,2sleep00-21swapper/011:05:120
1796096280,1sleep20-21swapper/211:02:252
1786086280,2sleep20-21swapper/210:55:112
1781104280,1sleep20-21swapper/210:50:152
1775636281,3sleep21775744-21cpuspeed_turbos10:45:132
1765091281,2sleep10-21swapper/110:35:101
1754104281,3sleep11754151-21systemd-run10:25:011
1753229280,1sleep00-21swapper/010:20:270
1728682281,3sleep01730123-21kthreadcore10:00:190
1719177281,3sleep20-21swapper/209:50:182
1707459281,4sleep01707671-21smartctl09:40:140
1707163280,2sleep10-21swapper/109:40:101
1700727281,3sleep01700773-21sshd09:30:240
1690354281,3sleep11690471-21/usr/sbin/munin09:20:221
1678491281,3sleep11679567-21memory09:10:171
1678491281,3sleep11679567-21memory09:10:161
1675490281,3sleep01675534-21systemd-run09:09:590
1616537281,3sleep21616695-21sed08:10:252
1586046281,2sleep00-21swapper/007:45:110
1581934281,3sleep20-21swapper/207:40:152
1581298281,3sleep11581933-21kthreadcore07:40:141
1551896281,3sleep01552290-21apt-config07:15:000
1896743270,1sleep00-21swapper/012:38:150
1881952270,3sleep01882353-21kthreadcore12:25:160
1874963270,1sleep00-21swapper/012:15:220
1874963270,1sleep00-21swapper/012:15:210
1871449271,2sleep20-21swapper/212:15:162
1871449271,2sleep20-21swapper/212:15:162
1870279270,1sleep00-21swapper/012:14:060
1844075270,1sleep00-21swapper/011:50:090
1833582270,2sleep21833601-21apt-get11:40:122
1826910270,1sleep20-21swapper/211:30:182
1806460270,1sleep00-21swapper/011:10:560
1790095270,2sleep00-21swapper/010:55:220
1780277270,1sleep00-21swapper/010:45:520
1770025270,1sleep00-21swapper/010:39:170
1758307271,3sleep20-21swapper/210:25:212
1739674271,2sleep01741333-21cut10:10:170
1738952271,3sleep21739118-21if_err_enp1s010:10:152
1734735271,2sleep10-21swapper/110:05:161
1727782270,2sleep00-21swapper/009:59:530
1701904271,2sleep21701923-21apt-get09:35:112
1696696270,2sleep10-21swapper/109:30:121
1696327270,1sleep00-21swapper/009:28:400
1669941271,3sleep10-21swapper/109:05:001
1669892270,4sleep20-21swapper/209:04:252
1660848271,2sleep00-21swapper/008:55:160
1659687270,2sleep20-21swapper/208:55:112
1654477271,2sleep10-21swapper/108:50:131
1653470271,3sleep00-21swapper/008:45:220
1644001271,3sleep00-21swapper/008:40:140
1637919270,1sleep10-21swapper/108:31:541
1631998270,4sleep00-21swapper/008:25:220
1622741270,2sleep00-21swapper/008:20:100
1612248271,2sleep00-21swapper/008:10:120
1602297270,3sleep21603521-21cut08:00:152
1590743270,1sleep20-21swapper/207:46:292
1580214270,2sleep00-21swapper/007:37:010
1575040270,1sleep20-21swapper/207:34:392
1571379271,3sleep00-21swapper/007:30:190
1559698270,2sleep11559717-21apt-get07:20:111
1556480270,4sleep00-21swapper/007:15:170
1552294270,3sleep11551905-21apt-get07:15:021
1888272260,2sleep10-21swapper/112:30:171
1875640260,1sleep10-21swapper/112:16:471
1875640260,1sleep10-21swapper/112:16:471
1865928261,2sleep10-21swapper/112:10:181
1864934260,1sleep20-21swapper/212:07:512
1849320260,1sleep10-21swapper/111:55:071
1822690260,1sleep00-21swapper/011:28:040
1812054261,2sleep20-21swapper/211:20:012
1806681260,4sleep10-21swapper/111:13:471
1792482261,2sleep00-21swapper/011:00:180
1774673260,3sleep11774888-21/usr/sbin/munin10:40:281
1747859260,2sleep00-21swapper/010:15:210
1742614261,2sleep11742670-21ldconfig10:10:221
1738145260,1sleep20-21swapper/210:07:582
1732714260,1sleep20-21swapper/210:00:452
1692036261,2sleep10-21swapper/109:25:161
1680531260,1sleep20-21swapper/209:12:312
1680531260,1sleep20-21swapper/209:12:302
1676829261,2sleep00-21swapper/009:10:160
1676829261,2sleep00-21swapper/009:10:150
1666412261,2sleep00-21swapper/009:00:180
1663976260,2sleep10-21swapper/108:55:251
1653236261,2sleep10-21swapper/108:45:201
1642737261,2sleep20-21swapper/208:35:272
1639495261,2sleep10-21swapper/108:35:181
1638982261,2sleep00-21swapper/008:35:160
1627953260,2sleep20-21swapper/208:25:132
1614105261,2sleep10-21swapper/108:10:161
1596038260,4sleep20-21swapper/207:52:342
1585648260,1sleep00-21swapper/007:43:180
1565480261,2sleep20-21swapper/207:25:172
1848950250,1sleep20-21swapper/211:53:312
1842831250,1sleep20-21swapper/211:45:222
1828972250,2sleep00-21swapper/011:35:180
1791040250,3sleep10-21swapper/110:59:521
1774412250,1sleep00-21swapper/010:40:230
1716645250,1sleep10-21swapper/109:45:251
1656379250,1sleep20-21swapper/208:50:182
1607531251,1sleep00-21swapper/008:05:150
1594344250,1sleep10-21swapper/107:50:161
1886969240,1sleep20-21swapper/212:30:142
1881687240,1sleep20-21swapper/212:25:142
1832132240,1sleep20-21swapper/211:35:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional