You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 23:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Wed Nov 26, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16959559911700,1166cyclictest17558112sleep307:55:013
1695955993640,360cyclictest2166032-21turbostat12:29:563
1695955992118,1cyclictest0-21swapper/311:54:593
1695955992018,1cyclictest0-21swapper/310:50:223
169595599196,10cyclictest1850101-21turbostat09:14:543
190898180,7rtkit-daemon1962834-21ssh10:18:200
1695955991814,2cyclictest0-21swapper/308:20:253
169595599176,9cyclictest2133426-21turbostat12:09:553
1695955991714,1cyclictest0-21swapper/308:35:133
169595599171,13cyclictest2157804-21taskset12:20:003
169595599171,12cyclictest1696148-21taskset07:10:133
169595599166,7cyclictest1931168-21turbostat10:00:003
169595599164,8cyclictest21175762sleep311:55:123
1695955991613,1cyclictest0-21swapper/307:28:033
169595599161,11cyclictest2129233-21taskset12:00:173
169595599161,11cyclictest2093384-21taskset11:40:133
169595599161,11cyclictest1768137-21turbostat08:05:013
18304062150,3sleep01830485-21find08:50:200
169595599156,6cyclictest20461312sleep311:10:163
169595599156,6cyclictest18666682sleep309:20:133
1695955991513,0cyclictest0-21swapper/307:21:573
1695955991512,1cyclictest0-21swapper/312:10:153
1695955991512,1cyclictest0-21swapper/311:18:263
1695955991512,1cyclictest0-21swapper/310:20:203
1695955991512,1cyclictest0-21swapper/309:54:143
1695955991512,1cyclictest0-21swapper/309:47:293
169595599151,11cyclictest19629752chrt10:18:393
169595599151,11cyclictest1863942-21taskset09:15:233
169595599151,10cyclictest1850101-21turbostat09:10:003
169595599145,6cyclictest19783392sleep310:26:453
1695955991412,1cyclictest0-21swapper/312:36:003
1695955991412,1cyclictest0-21swapper/311:21:523
1695955991412,1cyclictest0-21swapper/310:38:553
1695955991412,1cyclictest0-21swapper/308:15:523
1695955991412,0cyclictest0-21swapper/310:45:543
1695955991412,0cyclictest0-21swapper/307:35:433
1695955991412,0cyclictest0-21swapper/307:31:193
1695955991412,0cyclictest0-21swapper/307:31:183
1695955991411,1cyclictest0-21swapper/308:47:413
169595599141,10cyclictest2001536-21taskset10:40:233
169595599141,10cyclictest1898592-21turbostat09:40:013
169595599141,10cyclictest1883041-21chrt09:30:133
169595599141,10cyclictest1837398-21taskset08:58:293
169595599141,10cyclictest1755262-21taskset07:51:053
169595599141,10cyclictest1749005-21chrt07:46:203
169595599140,11cyclictest2041831-21chrt11:05:193
169595599140,11cyclictest1742397-21taskset07:40:233
169595599140,11cyclictest1742397-21taskset07:40:223
169595599134,7cyclictest1906626-21taskset09:44:243
169595599132,8cyclictest1704844-21taskset07:15:153
169595599131,9cyclictest1980464-21taskset10:30:163
169595599131,9cyclictest1939633-21chrt10:05:123
169595599131,9cyclictest1880012-21taskset09:25:243
1695955991312,0cyclictest0-21swapper/309:01:553
1695955991311,1cyclictest0-21swapper/312:30:423
1695955991311,1cyclictest0-21swapper/311:00:283
169595599131,10cyclictest1937418-21taskset10:00:523
169595599131,10cyclictest1799535-21taskset08:26:233
1695955991311,0cyclictest0-21swapper/312:20:063
1695955991310,1cyclictest0-21swapper/311:45:243
1695955991310,1cyclictest0-21swapper/311:30:163
169595599130,10cyclictest2086473-21taskset11:35:163
169595599130,10cyclictest2074820-21taskset11:25:343
169595599130,10cyclictest1805808-21taskset08:30:593
169595599130,10cyclictest1780771-21chrt08:12:403
169595599130,10cyclictest1768626-21taskset08:05:033
19314782122,6sleep21931515-21apt-get10:00:122
18893462120,4sleep21889354-21ssh09:32:182
169595599124,6cyclictest20264272sleep310:55:583
169595599121,9cyclictest1947663-21chrt10:10:123
169595599120,9cyclictest1830928-21taskset08:50:543
169595599120,9cyclictest1812701-21chrt08:40:143
19544662110,5sleep11954473-21sh10:12:481
190898110,8rtkit-daemon1818762-21kworker/0:0+kacpid10:33:200
190898110,7rtkit-daemon667-21snmpd11:14:470
190898110,7rtkit-daemon1947042-21ssh10:09:210
190898110,4rtkit-daemon667-21snmpd12:33:230
190898110,4rtkit-daemon2099539-21ssh11:41:260
190898110,4rtkit-daemon2036052-21ssh11:04:320
190898110,3rtkit-daemon1928-21sendmail-mta08:55:230
190898110,3rtkit-daemon1702340-21munin-node12:04:410
17099272113,6sleep21709998-21cat07:15:222
16954702110,3sleep30-21swapper/307:05:473
20675642100,7sleep22067571-21sh11:22:262
20613092100,1sleep00-21swapper/011:20:160
19476192101,5sleep21947658-21apt-get10:10:122
190898100,5rtkit-daemon2141856-21apt-get12:10:120
190898100,5rtkit-daemon1882437-21munin-run09:30:002
190898100,5rtkit-daemon1711577-21df07:20:150
190898100,3rtkit-daemon667-21snmpd09:59:560
2166759290,2sleep02167307-21irqcore12:25:150
2100205291,6sleep12100213-21ssh11:43:001
1934269290,1sleep01936424-21kthreadcore10:00:170
1920681291,5sleep01920796-21find09:50:240
19089890,6rtkit-daemon667-21snmpd11:17:250
19089890,5rtkit-daemon1977754-21users10:25:230
19089890,5rtkit-daemon1897095-21ssh09:36:352
19089890,5rtkit-daemon1843129-21/usr/sbin/munin09:00:210
19089890,4rtkit-daemon2075882-21ssh11:27:580
19089890,4rtkit-daemon2004499-21fschecks_count10:45:140
19089890,4rtkit-daemon1953650-21ssh10:10:550
19089890,4rtkit-daemon1809735-21cut08:35:160
19089890,3rtkit-daemon81364-21packagekitd07:38:390
19089890,2rtkit-daemon473-21gmain07:51:020
19089890,1rtkit-daemon0-21swapper/009:14:040
1888738291,6sleep11888744-21ssh09:30:521
1866591290,3sleep21866599-21ssh09:20:082
1849202290,5sleep11849231-21ntp_states09:05:201
1798896291,5sleep01798973-21ntp_states08:25:200
1762123290,7sleep01762156-21apt-get08:00:120
1744457290,2sleep00-21swapper/007:45:170
2179924281,5sleep22180112-21timerwakeupswit12:30:222
2114780280,2sleep12114833-21ssh11:50:231
2093352281,3sleep22093375-21apt-get11:40:132
2077071281,4sleep22077150-21cut11:30:112
2075911281,4sleep12075919-21ssh11:28:011
2020362281,4sleep12020399-21apt-get10:55:131
2017721281,4sleep02017826-21switchtime10:50:220
2010386280,2sleep12010394-21ssh10:46:061
1963923280,3sleep01963958-21apt-get10:20:120
1929833280,1sleep20-21swapper/209:56:582
1912383281,4sleep01912466-21ntp_states09:45:220
19089880,5rtkit-daemon667-21snmpd08:47:360
19089880,5rtkit-daemon1928-21sendmail-mta10:57:230
19089880,5rtkit-daemon1928-21sendmail-mta07:41:380
19089880,5rtkit-daemon1928-21sendmail-mta07:41:370
19089880,4rtkit-daemon674-21lldpd10:37:460
19089880,4rtkit-daemon667-21snmpd10:40:540
19089880,4rtkit-daemon438-21wpa_supplicant08:08:560
19089880,4rtkit-daemon438-21wpa_supplicant07:26:530
19089880,4rtkit-daemon421-21avahi-daemon11:35:130
19089880,4rtkit-daemon421-21avahi-daemon09:44:060
19089880,4rtkit-daemon3054984-21kworker/u16:2+flush-8:007:15:050
19089880,4rtkit-daemon2140565-21ssh12:07:520
19089880,4rtkit-daemon2044626-21ssh11:10:082
19089880,4rtkit-daemon1864894-21ssh09:16:490
19089880,4rtkit-daemon1802089-21cat08:30:180
19089880,4rtkit-daemon1702340-21munin-node11:50:510
19089880,4rtkit-daemon1702340-21munin-node11:06:070
19089880,4rtkit-daemon1702340-21munin-node08:15:180
19089880,4rtkit-daemon1657392-21munin-node07:09:510
19089880,3rtkit-daemon2107370-21ssh11:45:480
19089880,3rtkit-daemon14-21ksoftirqd/011:30:170
1906871281,4sleep21906915-21/usr/sbin/munin09:45:002
1814160281,4sleep11815775-21/usr/sbin/munin08:40:201
1773829280,5sleep11773881-21ps08:05:231
1712137280,6sleep11713000-21cut07:20:191
1696573280,4sleep21697565-21irqstats07:10:172
2157662271,3sleep22157800-21apt-key12:20:012
2143350270,4sleep12145185-21kthreadcore12:10:171
2131556270,5sleep22131564-21ssh12:00:412
2117782270,4sleep12117906-21sh11:55:141
2117620270,4sleep02117781-21diskstats11:55:130
2085482270,4sleep22086471-21cut11:35:172
2077447270,4sleep12078032-21cut11:30:141
2036589270,4sleep22036626-21apt-get11:05:112
2035632270,5sleep22035636-21ssh11:03:342
2026862271,3sleep22026866-21ssh10:57:022
2001741271,3sleep12001995-21unixbench_singl10:40:261
1996255271,3sleep21996277-21apt-get10:40:152
1980263270,4sleep21980463-21hddtemp_smartct10:30:172
1977909270,1sleep20-21swapper/210:25:442
1962348270,1sleep20-21swapper/210:17:112
19089870,4rtkit-daemon674-21lldpd09:06:450
19089870,3rtkit-daemon671-21ntpd07:10:050
19089870,3rtkit-daemon667-21snmpd08:10:430
19089870,3rtkit-daemon473-21gmain08:20:060
19089870,3rtkit-daemon421-21avahi-daemon09:39:221
19089870,3rtkit-daemon1928-21sendmail-mta12:15:030
19089870,3rtkit-daemon1761226-21find07:55:220
19089870,3rtkit-daemon1702340-21munin-node12:37:320
19089870,3rtkit-daemon1702340-21munin-node08:40:590
19089870,3rtkit-daemon0-21swapper/007:30:580
19089870,3rtkit-daemon0-21swapper/007:30:570
1873404270,5sleep11873412-21ssh09:22:441
1856911271,3sleep1667-21snmpd09:12:061
1838797270,4sleep11839930-21grep09:00:151
1812423271,3sleep21812511-21latency_hist08:40:002
1806830270,1sleep10-21swapper/108:35:151
1782412270,5sleep21783569-21cut08:15:172
1736528271,4sleep1667-21snmpd07:37:451
1717170270,4sleep21717171-21sshd07:21:482
1689856271,3sleep20-21swapper/207:05:142
2164739260,4sleep20-21swapper/212:21:562
2163303260,4sleep12163470-21grep12:20:191
2156968260,1sleep10-21swapper/112:18:241
2148601260,4sleep20-21swapper/212:12:442
2104279260,4sleep22104500-21kthreadcore11:45:192
2070486261,2sleep20-21swapper/211:25:182
2059819260,1sleep10-21swapper/111:18:241
2043068260,1sleep10-21swapper/111:07:021
2017828260,4sleep12018106-21sed10:50:231
1994067260,4sleep21986959-21kworker/u16:2+nfsiod10:35:492
1988135260,3sleep11988149-21grep10:35:121
1986598260,3sleep11986606-21ssh10:32:161
1947292260,3sleep21947300-21ssh10:09:522
1939520260,3sleep11939563-21ssh10:05:111
1930655260,1sleep10-21swapper/109:58:531
1921127261,3sleep21921133-21ssh09:50:342
1914111260,4sleep20-21swapper/209:48:092
19089860,3rtkit-daemon81364-21packagekitd12:20:190
19089860,2rtkit-daemon2018904-21ssh10:52:192
1888107262,2sleep01888226-21mailstats09:30:310
1874693261,2sleep10-21swapper/109:25:161
1874203260,1sleep00-21swapper/009:24:340
1865791260,1sleep21865799-21ssh09:18:512
1859916261,2sleep10-21swapper/109:15:171
1849232261,2sleep20-21swapper/209:05:212
1837289261,3sleep1667-21snmpd08:57:031
1820427260,4sleep21821595-21kthreadcore08:45:172
1802241261,2sleep10-21swapper/108:30:191
1795245261,2sleep10-21swapper/108:25:171
1787537261,2sleep20-21swapper/208:20:132
1781623261,3sleep10-21swapper/108:15:161
1780056260,3sleep21780115-21ntp_states08:10:202
1775330261,2sleep10-21swapper/108:10:141
1768824260,4sleep21768939-21perl08:05:152
1760998260,3sleep21761073-21ps07:55:212
1754769260,3sleep11754894-21python307:50:361
1749560261,2sleep20-21swapper/207:50:132
1745619261,2sleep10-21swapper/107:45:181
1689820260,2sleep11689853-21apt-get07:05:131
2188668250,1sleep10-21swapper/112:36:101
2182542250,2sleep20-21swapper/212:35:112
2181241250,2sleep12181249-21ssh12:32:411
2172729250,3sleep20-21swapper/212:26:502
2167309251,2sleep12166337-21/usr/sbin/munin12:25:171
2139822251,2sleep11963906-21kworker/u16:1+rpciod12:06:041
2135268251,2sleep20-21swapper/212:05:172
2126367250,2sleep12127220-21kthreadcore12:00:161
2115661250,3sleep20-21swapper/211:51:192
2102851250,2sleep10-21swapper/111:45:181
2090769251,2sleep12090777-21ssh11:35:321
2061844250,3sleep12062744-21kthreadcore11:20:171
2051401250,3sleep10-21swapper/111:12:321
2029522250,3sleep12030626-21cut11:00:161
2004186250,3sleep22004194-21ssh10:45:032
1965385250,2sleep10-21swapper/110:20:171
1963371250,3sleep11946933-21kworker/u16:2+rpciod10:19:381
1938090250,3sleep11812410-21kworker/u16:2+rpciod10:02:271
1905259250,3sleep10-21swapper/109:41:101
1850398250,3sleep21850406-21ssh09:10:112
1830910250,1sleep20-21swapper/208:50:422
1828351250,3sleep11830197-21sed08:50:171
1824853250,1sleep10-21swapper/108:48:341
1805852250,1sleep20-21swapper/208:31:362
1768083250,1sleep10-21swapper/108:04:341
1763982251,2sleep20-21swapper/208:00:192
1755824250,2sleep10-21swapper/107:55:121
1749071250,1sleep20-21swapper/207:47:122
1742696250,1sleep10-21swapper/107:40:471
1742696250,1sleep10-21swapper/107:40:461
1725432251,2sleep20-21swapper/207:30:182
1725432251,2sleep20-21swapper/207:30:182
1710806250,1sleep10-21swapper/107:19:261
1696405250,2sleep10-21swapper/107:10:161
2123404240,3sleep20-21swapper/211:55:312
2053631240,2sleep20-21swapper/211:15:162
1972190240,2sleep11972397-21sh10:25:141
1969985240,2sleep21969993-21ssh10:21:062
1921118240,2sleep10-21swapper/109:50:331
1913071240,2sleep11913079-21ssh09:45:391
1890863240,2sleep01891018-21df_inode09:35:130
1839931240,2sleep20-21swapper/209:00:162
1817948240,2sleep21695949-21cyclictest08:40:262
1742271240,2sleep20-21swapper/207:40:232
1742271240,2sleep20-21swapper/207:40:222
1729864241,2sleep1331ktimers/107:33:371
1729864241,2sleep1331ktimers/107:33:361
1719176240,2sleep11721184-21kthreadcore07:25:151
1879689230,1sleep00-21swapper/009:25:230
1837228230,1sleep20-21swapper/208:56:162
1793979230,1sleep20-21swapper/208:25:142
1792333230,1sleep10-21swapper/108:20:201
1735724230,1sleep20-21swapper/207:35:222
1717868230,1sleep20-21swapper/207:25:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional