You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Sat Nov 22, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17467849912320,1228cyclictest18184642sleep320:05:013
17467849911600,1156cyclictest17867812sleep319:40:013
17467849911401,1133cyclictest1932373-21sleep21:30:013
17467849911200,1116cyclictest17530032sleep319:14:573
1746784992700,267cyclictest2232392-21turbostat00:39:553
17865502710,3sleep21223-21cron19:40:002
1746784991916,1cyclictest0-21swapper/300:05:143
1746784991815,1cyclictest0-21swapper/323:45:013
1746784991815,1cyclictest0-21swapper/319:40:003
1746784991815,1cyclictest0-21swapper/300:15:133
174678499176,7cyclictest1817969-21turbostat20:05:003
1746784991714,1cyclictest0-21swapper/321:20:013
174678499171,12cyclictest1767978-21taskset19:25:153
19571622161,11sleep30-21swapper/321:45:103
174678499166,8cyclictest2086484-21turbostat23:09:553
174678499165,8cyclictest1908183-21taskset21:15:013
19757892150,7sleep21977766-21tr21:55:202
174678499156,6cyclictest20372782sleep322:33:203
174678499154,8cyclictest1849615-21turbostat20:30:003
174678499154,8cyclictest1849615-21turbostat20:30:003
1746784991513,1cyclictest0-21swapper/321:30:193
1746784991513,0cyclictest0-21swapper/322:52:173
1746784991513,0cyclictest0-21swapper/322:20:163
1746784991513,0cyclictest0-21swapper/300:25:153
1746784991512,1cyclictest0-21swapper/323:50:213
1746784991512,1cyclictest0-21swapper/323:03:153
1746784991512,1cyclictest0-21swapper/322:35:243
1746784991512,1cyclictest0-21swapper/322:05:243
1746784991512,1cyclictest0-21swapper/320:55:323
1746784991512,1cyclictest0-21swapper/319:45:593
1746784991512,1cyclictest0-21swapper/319:33:483
174678499151,11cyclictest2111030-21taskset23:20:123
1746784991511,1cyclictest0-21swapper/322:05:003
174678499151,10cyclictest18940942sleep321:05:113
174678499151,10cyclictest1824763-21taskset20:10:093
174678499145,6cyclictest2158928-21turbostat23:50:013
174678499145,6cyclictest1875002-21taskset20:50:093
174678499144,7cyclictest1923246-21taskset21:22:413
174678499141,9cyclictest21028822sleep323:15:123
1746784991413,0cyclictest0-21swapper/300:30:563
1746784991412,1cyclictest0-21swapper/321:55:023
1746784991411,1cyclictest0-21swapper/323:35:173
1746784991411,1cyclictest0-21swapper/322:45:583
1746784991411,1cyclictest0-21swapper/322:45:583
1746784991411,1cyclictest0-21swapper/321:42:593
1746784991411,1cyclictest0-21swapper/320:24:163
174678499141,10cyclictest2197105-21taskset00:10:213
174678499141,10cyclictest2167222-21taskset23:55:103
174678499141,10cyclictest2134406-21chrt23:34:103
174678499141,10cyclictest2012455-21taskset22:17:163
174678499141,10cyclictest1947595-21taskset21:37:213
174678499140,11cyclictest1970317-21taskset21:50:193
22378072131,9sleep22237976-21ntpq00:35:212
20467712130,3sleep22046921-21ps22:40:152
174678499133,7cyclictest0-21swapper/319:55:173
174678499131,9cyclictest2208392-21taskset00:20:143
174678499131,9cyclictest2175530-21taskset00:00:033
174678499131,9cyclictest2076802-21chrt22:56:193
174678499131,9cyclictest20464112sleep322:40:133
174678499131,9cyclictest2025482-21taskset22:25:173
174678499131,9cyclictest1862170-21sleep20:40:003
1746784991312,0cyclictest0-21swapper/319:18:593
1746784991311,1cyclictest0-21swapper/322:11:023
174678499131,10cyclictest1837061-21turbostat20:20:003
174678499131,10cyclictest1805408-21taskset19:55:003
1746784991310,1cyclictest0-21swapper/320:46:383
1746784991310,1cyclictest0-21swapper/320:30:153
1746784991310,1cyclictest0-21swapper/320:30:153
174678499130,10cyclictest2094869-21taskset23:10:103
174678499130,10cyclictest1761651-21taskset19:20:023
190898120,8rtkit-daemon1753179-21munin-node20:44:262
19080432120,8sleep21908181-21ls21:15:012
174678499129,1cyclictest0-21swapper/323:25:163
174678499129,1cyclictest0-21swapper/320:40:093
174678499121,9cyclictest1893802-21taskset21:04:313
20936752110,7sleep22093683-21ssh23:08:012
190898110,7rtkit-daemon667-21snmpd00:13:280
190898110,4rtkit-daemon2061535-21ssh22:48:270
190898110,4rtkit-daemon2061535-21ssh22:48:270
190898110,3rtkit-daemon2086470-21cron23:04:592
18377212111,7sleep11838334-21sed20:20:191
18375192110,1sleep00-21swapper/020:20:170
22323602100,7sleep12232385-21munin-run00:35:001
20680322101,7sleep22068247-21sed22:50:262
20355182101,2sleep10-21swapper/122:30:221
20150782100,8sleep12016247-21grep22:20:161
19816602101,5sleep21981699-21apt-get22:00:122
190898100,6rtkit-daemon421-21avahi-daemon23:06:541
190898100,6rtkit-daemon2167384-21diskstats23:55:110
190898100,5rtkit-daemon2111023-21apt-get23:20:122
190898100,5rtkit-daemon1798765-21users19:45:222
190898100,3rtkit-daemon667-21snmpd22:22:490
18747512100,6sleep21874817-21sed20:50:012
18486832100,7sleep11848754-21ntp_states20:25:201
18486832100,7sleep11848754-21ntp_states20:25:191
17997912101,6sleep01800352-21irqcore19:50:170
174678499108,1cyclictest0-21swapper/319:10:013
2224666291,4sleep22224705-21apt-get00:30:122
2198437290,2sleep20-21swapper/200:12:252
2162010290,6sleep22164097-21cut23:50:172
2149377290,6sleep22149381-21ssh23:41:262
2141328290,7sleep22141330-21ssh23:36:302
2135057291,4sleep02135094-21apt-get23:35:120
2085977290,1sleep02085985-21ssh23:03:510
1990280290,6sleep01991265-21/usr/sbin/munin22:05:160
1957332291,5sleep11957538-21hddtemp_smartct21:45:131
1954238291,5sleep01954300-21ntp_states21:40:200
1947269291,6sleep11947275-21ssh21:36:311
1938798290,1sleep20-21swapper/221:30:482
1931419290,1sleep11900110-21diskmemload21:27:581
1916944291,5sleep11917580-21cut21:20:171
19089890,6rtkit-daemon81364-21packagekitd21:33:211
19089890,6rtkit-daemon319-21systemd-udevd22:19:360
19089890,6rtkit-daemon1928-21sendmail-mta22:05:401
19089890,5rtkit-daemon2143405-21df23:40:160
19089890,5rtkit-daemon2125944-21ssh23:28:152
19089890,5rtkit-daemon2053579-21sh22:43:380
19089890,5rtkit-daemon1900110-21diskmemload21:51:331
19089890,4rtkit-daemon667-21snmpd22:25:520
19089890,4rtkit-daemon2224085-21apt-get00:30:010
19089890,4rtkit-daemon1957165-21apt-get21:45:112
19089890,4rtkit-daemon1889132-21cut21:00:162
19089890,3rtkit-daemon2189865-21ssh00:06:120
19089890,3rtkit-daemon2133092-21ssh23:31:072
19089890,2rtkit-daemon171rcu_preempt00:02:530
19089890,1rtkit-daemon1907-21rtkit-daemon00:27:380
1887857291,5sleep11888006-21df_inode21:00:131
2199891281,4sleep22199934-21ssh00:15:132
2174798280,1sleep10-21swapper/123:59:371
2127036281,3sleep12127069-21apt-get23:30:121
2102565281,4sleep02102615-21munin-run23:15:010
2060435280,1sleep20-21swapper/222:45:532
2060435280,1sleep20-21swapper/222:45:532
2027310280,5sleep12027375-21ntp_states22:25:211
2016248281,5sleep20-21swapper/222:20:172
2011760281,4sleep12011768-21ssh22:15:341
1933433283,2sleep01934430-21irqstats21:30:180
19089880,5rtkit-daemon319-21systemd-udevd19:25:382
19089880,5rtkit-daemon1753179-21munin-node22:58:291
19089880,4rtkit-daemon674-21lldpd19:15:150
19089880,4rtkit-daemon667-21snmpd20:50:152
19089880,4rtkit-daemon667-21snmpd00:20:180
19089880,4rtkit-daemon2151358-21df_inode23:45:140
19089880,4rtkit-daemon2069418-21ssh22:52:540
19089880,4rtkit-daemon1806062-21hddtemp_smartct19:55:142
19089880,4rtkit-daemon1774243-21/usr/sbin/munin19:30:252
19089880,4rtkit-daemon1753179-21munin-node22:03:280
19089880,3rtkit-daemon2036296-21ssh22:30:590
19089880,3rtkit-daemon1800620-21sed19:50:172
19089880,1rtkit-daemon0-21swapper/220:08:432
19089880,1rtkit-daemon0-21swapper/123:01:291
19089880,1rtkit-daemon0-21swapper/023:50:460
1906936281,4sleep01906944-21ssh21:12:220
1900647281,4sleep11900846-21hddtemp_smartct21:10:151
1746932281,3sleep01746965-21apt-get19:10:100
1745591281,3sleep11745753-21awk19:05:181
2192210271,3sleep12192484-21irqcore00:10:161
2182219270,5sleep22182217-21ssh00:02:112
2156724270,4sleep22156917-21users23:45:242
2102856270,1sleep20-21swapper/223:15:072
2094921271,3sleep12094958-21apt-get23:10:171
2076878273,2sleep00-21swapper/022:56:330
2056006270,4sleep12057798-21cut22:45:201
2056006270,4sleep12057798-21cut22:45:201
2046359270,1sleep10-21swapper/122:40:061
2045082271,4sleep22045089-21sh22:37:442
2012322271,1sleep20-21swapper/222:16:562
2003118271,3sleep02003195-21/usr/sbin/munin22:10:210
1993041271,3sleep21994834-21cat22:05:182
1971527270,4sleep01971535-21ssh21:51:370
1948453270,1sleep20-21swapper/221:39:172
1914238271,4sleep01900110-21diskmemload21:15:350
19089870,4rtkit-daemon671-21ntpd22:35:060
19089870,4rtkit-daemon473-21gmain23:18:061
19089870,4rtkit-daemon473-21gmain21:55:111
19089870,4rtkit-daemon473-21gmain19:05:310
19089870,4rtkit-daemon421-21avahi-daemon19:24:122
19089870,3rtkit-daemon81364-21packagekitd20:36:512
19089870,3rtkit-daemon674-21lldpd00:15:560
19089870,3rtkit-daemon671-21ntpd20:20:172
19089870,3rtkit-daemon667-21snmpd21:20:042
19089870,3rtkit-daemon1954881-21ssh21:40:321
19089870,3rtkit-daemon1908386-21munin-node21:15:142
19089870,3rtkit-daemon1886694-21timerandwakeup20:55:222
19089870,3rtkit-daemon1792614-21sshd19:42:082
19089870,3rtkit-daemon1753179-21munin-node21:59:240
19089870,3rtkit-daemon1753179-21munin-node21:41:582
19089870,3rtkit-daemon1753179-21munin-node21:07:072
19089870,3rtkit-daemon1753179-21munin-node20:25:422
19089870,3rtkit-daemon1753179-21munin-node20:25:412
19089870,2rtkit-daemon438-21wpa_supplicant22:13:542
19089870,2rtkit-daemon1836658-21users20:15:242
19089870,1rtkit-daemon0-21swapper/220:32:112
19089870,1rtkit-daemon0-21swapper/220:32:112
1849870270,4sleep11849892-21apt-get20:30:121
1849870270,4sleep11849892-21apt-get20:30:111
1786516271,4sleep1440-21in:imuxsock19:40:001
1774665270,4sleep01775456-21sed19:30:170
1774278271,4sleep11774421-21df19:30:141
2238920260,1sleep10-21swapper/100:36:281
2238587260,1sleep00-21swapper/000:35:340
2223543260,1sleep20-21swapper/200:28:442
2209547261,2sleep10-21swapper/100:20:161
2206151260,4sleep12206159-21ssh00:16:331
2181621260,1sleep10-21swapper/100:00:501
2167602261,2sleep20-21swapper/223:55:142
2166471260,1sleep10-21swapper/123:54:051
2133246260,3sleep02133254-21ssh23:31:270
2126575260,3sleep12126583-21ssh23:29:441
2110952260,1sleep10-21swapper/123:20:071
2110042260,1sleep00-21swapper/023:18:370
2044526260,4sleep12044534-21ssh22:36:271
2035631260,2sleep22035747-21find22:30:232
1972471260,1sleep20-21swapper/221:53:482
1963708260,1sleep00-21swapper/021:47:210
1940986261,2sleep00-21swapper/021:35:150
1932309260,3sleep01932317-21ssh21:29:580
19089860,3rtkit-daemon671-21ntpd23:35:041
19089860,3rtkit-daemon473-21gmain20:00:072
19089860,3rtkit-daemon1928-21sendmail-mta20:11:052
19089860,3rtkit-daemon1928-21sendmail-mta19:19:502
1908488261,3sleep11908386-21/usr/sbin/munin21:15:161
1887821261,2sleep00-21swapper/021:00:120
1868744261,3sleep01868881-21cpuspeed_turbos20:45:140
1858407261,3sleep00-21swapper/020:35:200
1848922261,2sleep01849181-21unixbench_singl20:25:230
1848922261,2sleep01849181-21unixbench_singl20:25:230
1818878260,4sleep11819451-21sed20:05:161
1813396260,4sleep11814599-21kthreadcore20:00:181
1805847260,3sleep01805935-21tune2fs19:55:140
1792820260,1sleep00-21swapper/019:44:480
1780798260,3sleep01780961-21/usr/sbin/munin19:35:140
1768136260,4sleep01768336-21hddtemp_smartct19:25:170
1753007261,3sleep21753021-21idleruntime-cro19:15:012
1741663261,2sleep20-21swapper/219:05:142
2216221250,2sleep12215738-21apt-get00:25:021
2183679250,3sleep20-21swapper/200:05:042
2157844250,3sleep12157852-21ssh23:47:311
2150325250,3sleep12150327-21ssh23:43:471
2125827250,1sleep00-21swapper/023:27:550
2110704250,3sleep00-21swapper/023:20:020
2096390251,2sleep20-21swapper/223:10:222
2092775250,3sleep00-21swapper/023:05:510
2065374250,3sleep12067496-21kthreadcore22:50:211
2028677250,3sleep22028681-21ssh22:27:222
1988285250,3sleep11900110-21diskmemload22:02:211
1921911250,3sleep01922071-21cat21:20:240
1900084250,1sleep00-21swapper/021:09:540
1881258250,2sleep10-21swapper/120:55:121
1880848250,1sleep10-21swapper/120:53:001
1875416251,2sleep00-21swapper/020:50:150
1870807251,2sleep10-21swapper/120:45:191
1861444251,2sleep10-21swapper/120:35:251
1855512250,1sleep00-21swapper/020:30:360
1855512250,1sleep00-21swapper/020:30:350
1832474250,2sleep00-21swapper/020:15:170
1830036250,3sleep11830245-21sed20:10:211
1817812250,1sleep00-21swapper/020:03:060
1805206250,1sleep10-21swapper/119:52:221
1794979250,3sleep11797034-21kthreadcore19:45:161
1794023251,2sleep00-21swapper/019:45:160
1786799250,2sleep10-21swapper/119:40:161
1763605250,2sleep01765689-21kthreadcore19:20:190
1747942250,2sleep10-21swapper/119:10:151
2211456240,2sleep20-21swapper/200:20:172
2186413240,2sleep10-21swapper/100:05:191
2076023240,2sleep20-21swapper/222:55:232
2003801240,2sleep12003809-21ssh22:10:461
1931139240,2sleep21931147-21ssh21:27:152
1868269241,2sleep0151ktimers/020:42:380
1862408240,2sleep10-21swapper/120:40:101
1824907240,2sleep01824938-21fschecks_count20:10:120
1818678240,2sleep00-21swapper/020:05:150
1768337240,2sleep10-21swapper/119:25:181
1886205230,1sleep00-21swapper/020:55:200
1831175230,1sleep10-21swapper/120:15:141
1805938230,1sleep10-21swapper/119:55:151
1761865230,1sleep10-21swapper/119:20:161
1755536230,1sleep10-21swapper/119:15:161
1894082220,1sleep10-21swapper/121:05:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional