You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-09 - 09:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Fri Jan 09, 2026 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1719401212754,68sleep30-21swapper/319:07:353
17146782118104,9sleep20-21swapper/219:05:102
171957128855,10sleep10-21swapper/119:09:461
171942327758,13sleep00-21swapper/019:07:520
1719710992421,1cyclictest0-21swapper/323:55:123
171971099241,10cyclictest2049685-21chrt00:17:043
1719710992320,1cyclictest0-21swapper/322:30:553
1719710992320,1cyclictest0-21swapper/322:30:553
20178782221,9sleep02018213-21latency_hist23:50:010
1719710992219,1cyclictest0-21swapper/321:00:213
1719710992110,8cyclictest1774466-21chrt19:57:123
1719710992017,1cyclictest0-21swapper/321:55:233
1719710992017,1cyclictest0-21swapper/320:10:003
1719710992017,1cyclictest0-21swapper/300:05:213
1719710992016,2cyclictest0-21swapper/322:45:003
20661752191,5sleep12066190-21cpu00:35:111
20661752191,5sleep12066190-21cpu00:35:111
19655112191,2sleep00-21swapper/023:00:110
1719710991916,1cyclictest0-21swapper/321:20:213
1719710991916,1cyclictest0-21swapper/321:00:013
171971099191,5cyclictest2001918-21turbostat23:35:003
1719710991915,2cyclictest0-21swapper/323:50:013
1719710991915,2cyclictest0-21swapper/321:45:123
171971099191,15cyclictest2012364-21turbostat23:44:593
171971099182,11cyclictest1943080-21taskset22:35:203
1719710991815,1cyclictest0-21swapper/321:05:173
1719710991815,1cyclictest0-21swapper/320:10:253
1719710991814,2cyclictest0-21swapper/321:15:153
171971099181,13cyclictest0-21swapper/320:30:113
171971099181,13cyclictest0-21swapper/319:15:123
171971099181,13cyclictest0-21swapper/300:00:113
171971099181,10cyclictest1826000-21taskset20:45:183
17951532171,11sleep11795262-21processes20:15:231
171971099177,6cyclictest1985847-21taskset23:16:043
171971099177,6cyclictest1822283-21taskset20:44:053
171971099172,10cyclictest2018430-21taskset23:50:113
171971099172,10cyclictest1881838-21taskset21:40:133
1719710991714,1cyclictest0-21swapper/323:10:133
1719710991714,1cyclictest0-21swapper/300:35:253
1719710991714,1cyclictest0-21swapper/300:35:243
1719710991713,2cyclictest0-21swapper/323:30:003
1719710991713,2cyclictest0-21swapper/322:00:163
171971099171,13cyclictest0-21swapper/319:25:203
171971099171,12cyclictest2044670-21turbostat00:15:013
171971099171,12cyclictest1876105-21chrt21:35:123
171971099170,14cyclictest0-21swapper/323:20:243
17373012161,2sleep10-21swapper/119:25:011
171971099168,5cyclictest1747804-21turbostat19:34:593
171971099162,9cyclictest2002169-21taskset23:35:123
171971099162,9cyclictest2002169-21taskset23:35:123
171971099161,9cyclictest1896601-21taskset21:52:513
171971099161,9cyclictest1719882-21taskset19:10:063
1719710991613,1cyclictest0-21swapper/323:05:273
1719710991613,1cyclictest0-21swapper/321:32:083
1719710991612,2cyclictest0-21swapper/322:20:173
1719710991612,2cyclictest0-21swapper/319:45:143
1719710991612,2cyclictest0-21swapper/300:20:103
171971099161,12cyclictest1954797-21turbostat22:50:003
171971099161,12cyclictest0-21swapper/322:05:113
171971099161,12cyclictest0-21swapper/320:35:113
171971099160,13cyclictest0-21swapper/319:40:203
171971099160,12cyclictest0-21swapper/320:00:093
171971099160,10cyclictest2065849-21taskset00:34:163
171971099160,10cyclictest1801021-21taskset20:21:013
171971099160,10cyclictest1752625-21taskset19:35:243
171971099157,1cyclictest0-21swapper/323:00:173
171971099151,9cyclictest2055429-21taskset00:25:063
171971099151,9cyclictest1964062-21taskset22:55:163
171971099151,9cyclictest1917396-21taskset22:10:303
1719710991513,1cyclictest0-21swapper/321:10:023
171971099151,11cyclictest1796084-21sleep20:19:593
171971099151,10cyclictest1918034-21taskset22:15:123
171971099151,10cyclictest1828162-21chrt20:50:113
171971099150,13cyclictest0-21swapper/322:50:173
171971099150,12cyclictest0-21swapper/322:25:213
19380672140,2sleep10-21swapper/122:30:261
19380672140,2sleep10-21swapper/122:30:251
18711242141,9sleep11870836-21/usr/sbin/munin21:30:131
171971099141,9cyclictest1732724-21chrt19:20:123
171971099141,8cyclictest1764399-21taskset19:50:073
1719710991411,1cyclictest0-21swapper/321:25:123
171971099140,9cyclictest18015962sleep320:25:113
171971099140,9cyclictest18015962sleep320:25:113
20594642131,7sleep12055435-21/usr/sbin/munin00:25:211
20123532131,8sleep12012765-21latency_hist23:45:011
19290542131,2sleep00-21swapper/022:25:150
19182872131,8sleep01918311-21awk22:15:140
18074212131,8sleep01808391-21cut20:30:170
17364062131,8sleep21736495-21proc_pri19:20:212
19707212121,6sleep01970739-21apt-get23:05:110
19608062121,7sleep01960943-21kthreadcore22:55:140
19013592121,7sleep01901604-21/usr/sbin/munin21:55:260
18447812121,7sleep01845320-21cut21:05:160
18392052121,7sleep01839272-21kernelversion21:00:150
17523172121,7sleep21752372-21ld19:35:202
20502852111,5sleep12050488-21diskstats00:20:121
20499182111,4sleep12049949-21munin-run00:19:591
20405702111,6sleep12042233-21kthreadcore00:10:181
20346432111,5sleep12034767-21perl00:05:131
20131632111,5sleep12013208-21grep23:45:131
20021472110,5sleep20-21swapper/223:35:122
20021472110,5sleep20-21swapper/223:35:122
20012692111,4sleep12001513-21users23:30:261
19759542112,4sleep21975972-21apt-get23:10:112
19597142110,1sleep20-21swapper/222:50:552
19482822111,4sleep11948331-21ntpq22:40:201
19080922111,6sleep01909207-21kthreadcore22:05:160
19076062112,2sleep1171rcu_preempt22:05:131
18765912111,2sleep20-21swapper/221:35:162
18708352111,6sleep21870853-21/usr/sbin/munin21:30:112
18602702111,4sleep11860289-21apt-get21:20:121
18470302111,5sleep21848078-21memory21:05:182
18322032111,5sleep01832254-21ntpq20:50:210
18160762111,4sleep21816123-21ntpq20:35:202
18083922111,6sleep11810130-21kthreadcore20:30:181
17594222111,5sleep21759546-21perl19:45:142
20675822101,6sleep22069320-21kthreadcore00:35:162
20675822101,6sleep22069320-21kthreadcore00:35:162
20489312101,4sleep02048976-21ntpq00:15:220
20399412101,5sleep22039985-21tune2fs00:10:152
20184082101,3sleep22018427-21apt-get23:50:112
20182142101,4sleep12017894-21apt-get23:50:021
20048112101,4sleep02005961-21memory23:35:180
20048112101,4sleep02005961-21memory23:35:180
19904912101,5sleep11990614-21find23:20:201
19747212101,4sleep21974753-21ntpq23:05:222
19652282101,4sleep21965300-21munin-run23:00:012
19430312102,3sleep21943072-21ntpq22:35:202
19376292101,4sleep21937742-21ntpq22:30:192
19376292101,4sleep21937742-21ntpq22:30:192
19049242100,1sleep20-21swapper/222:00:172
18915662103,3sleep01891616-21systemd21:50:010
18905922101,5sleep11890641-21ntpq21:45:211
18709202101,4sleep01871123-21diskstats21:30:120
18482702101,4sleep11848324-21ntpq21:05:211
18436972101,5sleep11843724-21perf21:05:011
18384152101,4sleep21838439-21munin-run21:00:002
18265092101,4sleep01826597-21ntpq20:45:200
18109172101,4sleep21810974-21ntp_states20:30:232
17860692101,4sleep01786172-21perl20:10:130
17746872101,4sleep21774707-21munin-run20:00:002
17736672101,4sleep11773714-21ntpq19:55:181
17467892101,4sleep21746836-21ntpq19:30:192
17362992101,5sleep01736348-21ntpq19:20:190
17314422101,4sleep11731659-21users19:15:271
2064896291,3sleep02064942-21ntpq00:30:210
2061223291,4sleep12061270-21fschecks_count00:30:141
2035763291,4sleep02037502-21tr00:05:160
2029057291,3sleep02029076-21apt-get00:00:110
2028681291,4sleep22028842-21cups-browsed00:00:012
2012766291,3sleep22012366-21apt-get23:45:022
1996890290,5sleep20-21swapper/223:30:112
1990617290,1sleep20-21swapper/223:20:212
1988492295,1sleep00-21swapper/023:20:150
1966793291,5sleep10-21swapper/123:00:161
1965793291,4sleep21965917-21perl23:00:132
1955333290,6sleep11955491-21gltestperf22:50:141
1953769291,4sleep11953820-21ld22:45:211
1949019290,1sleep00-21swapper/022:41:430
1928221290,1sleep00-21swapper/022:25:000
1918528291,4sleep21918768-21cut22:15:162
1891909291,4sleep11892112-21diskstats21:50:141
1885386291,4sleep11885443-21ntp_states21:40:191
1864302291,3sleep21864349-21ntpq21:20:212
1855027291,3sleep11855046-21apt-get21:15:111
1849526291,4sleep21849664-21grep21:10:112
1822356291,4sleep11822377-21/usr/sbin/munin20:45:011
1815986291,3sleep11816075-21ntpq20:35:191
1796326291,3sleep01796345-21apt-get20:20:110
1794993291,4sleep21795046-21ld20:15:202
1785758291,4sleep11785262-21apt-get20:10:021
1774967291,3sleep21774986-21apt-get20:00:092
1757586291,4sleep01757634-21ntpq19:40:190
1753638291,3sleep11753890-21df19:40:121
2056038281,3sleep0768-21ntpd00:25:160
2055727281,4sleep22055938-21hddtemp_smartct00:25:132
2054955280,1sleep00-21swapper/000:22:150
2007975281,3sleep22009148-21kthreadcore23:40:172
1992236281,3sleep11993744-21kthreadcore23:25:161
1981485281,3sleep00-21swapper/023:15:130
1980780280,1sleep10-21swapper/123:12:541
1972087281,3sleep11973821-21kthreadcore23:05:171
1960320281,3sleep11960511-21expr22:55:111
1939019280,2sleep10-21swapper/122:35:111
1923821281,3sleep11925176-21kthreadcore22:20:161
1903237281,3sleep11904922-21kthreadcore22:00:161
1886869281,3sleep2800-21snmpd21:45:142
1881937281,3sleep01883481-21kthreadcore21:40:150
1867283280,5sleep10-21swapper/121:25:171
1866109281,3sleep00-21swapper/021:25:160
1824225281,3sleep21825997-21kthreadcore20:45:182
1785788281,2sleep21785807-21apt-get20:10:112
1784279281,2sleep01784327-21ntpq20:05:200
1784091281,3sleep21784231-21/usr/sbin/munin20:05:172
1766373282,3sleep10-21swapper/119:50:181
1753329281,3sleep03207559-21gdbus19:39:580
1742795281,3sleep11742814-21apt-get19:30:101
1719886281,2sleep20-21swapper/219:10:102
2070783270,1sleep00-21swapper/000:35:300
2070783270,1sleep00-21swapper/000:35:300
2061276270,3sleep20-21swapper/200:30:152
2049894270,1sleep20-21swapper/200:19:452
2039666270,2sleep00-21swapper/000:10:130
2029688270,3sleep12031324-21kthreadcore00:00:161
2028516270,1sleep10-21swapper/123:57:551
2023057271,2sleep01881049-21kworker/u16:3+ext4-rsv-conversion23:50:450
1991633270,2sleep01991652-21apt-get23:25:120
1981209270,2sleep20-21swapper/223:15:122
1980812270,1sleep00-21swapper/023:13:170
1956633271,3sleep01958412-21kthreadcore22:50:170
1954584271,3sleep0800-21snmpd22:47:280
1951111271,2sleep20-21swapper/222:45:182
1948943270,1sleep20-21swapper/222:40:442
1935014271,3sleep00-21swapper/022:30:170
1935014271,3sleep00-21swapper/022:30:170
1917631270,1sleep10-21swapper/122:13:301
1912779270,2sleep20-21swapper/222:10:112
1902813271,2sleep00-21swapper/022:00:150
1901258270,1sleep20-21swapper/221:55:242
1892396271,2sleep00-21swapper/021:50:170
1849504270,2sleep10-21swapper/121:10:101
1843347270,1sleep20-21swapper/221:00:372
1837730272,2sleep01837958-21unixbench_multi20:55:260
1833425270,2sleep10-21swapper/120:55:111
1816990270,1sleep00-21swapper/020:38:490
1801877271,3sleep10-21swapper/120:25:131
1801877271,3sleep10-21swapper/120:25:131
1797652271,2sleep10-21swapper/120:20:161
1785022270,1sleep10-21swapper/120:07:051
1769356270,1sleep00-21swapper/019:54:180
1768531271,2sleep20-21swapper/219:50:232
1759700271,2sleep00-21swapper/019:45:150
1731769270,1sleep00-21swapper/019:16:420
1727069271,1sleep21727088-21apt-get19:15:122
1997044260,2sleep0151ktimers/023:30:130
1996280260,4sleep20-21swapper/223:25:382
1932204260,2sleep21932448-21strings22:25:182
1930393261,2sleep10-21swapper/122:25:171
1922620260,4sleep10-21swapper/122:15:431
1917547260,1sleep00-21swapper/022:12:250
1901976260,3sleep10-21swapper/121:59:561
1885991260,1sleep20-21swapper/221:40:452
1881048261,2sleep10-21swapper/121:40:001
1864981260,1sleep00-21swapper/021:21:140
1859105260,2sleep00-21swapper/021:15:220
1855638261,2sleep20-21swapper/221:15:162
1833006260,1sleep20-21swapper/220:52:592
1832256260,2sleep10-21swapper/120:50:221
1819171261,2sleep20-21swapper/220:40:172
1805453261,2sleep20-21swapper/220:25:182
1805453261,2sleep20-21swapper/220:25:182
1802878261,2sleep00-21swapper/020:25:160
1802878261,2sleep00-21swapper/020:25:150
1801293260,4sleep20-21swapper/220:24:302
1779996261,2sleep00-21swapper/020:04:560
1775262260,2sleep11775474-21smartctl20:00:121
1770057261,2sleep00-21swapper/019:55:140
1758383260,1sleep20-21swapper/219:42:132
1746893261,2sleep00-21swapper/019:30:210
1741546261,2sleep00-21swapper/019:25:200
2055137250,1sleep20-21swapper/200:24:342
2038286250,2sleep20-21swapper/200:05:182
2033978250,3sleep20-21swapper/200:03:292
2027703250,1sleep00-21swapper/023:55:210
2006903251,2sleep11719702-21cyclictest23:37:151
2006903251,2sleep11719702-21cyclictest23:37:151
1943134250,2sleep00-21swapper/022:35:220
1923402250,1sleep20-21swapper/222:20:132
1876076250,2sleep00-21swapper/021:35:060
1870219250,3sleep20-21swapper/221:25:342
1817867250,1sleep00-21swapper/020:40:150
1748771250,1sleep10-21swapper/119:35:151
1737849250,1sleep20-21swapper/219:25:142
1737737250,2sleep10-21swapper/119:25:131
1722877250,1sleep00-21swapper/019:10:170
2007729240,1sleep00-21swapper/023:40:150
1984997240,1sleep10-21swapper/123:15:191
1907306240,1sleep20-21swapper/222:05:022
1892113240,1sleep20-21swapper/221:50:142
1849239240,1sleep00-21swapper/021:10:010
1822635240,1sleep10-21swapper/120:45:121
1791266240,1sleep00-21swapper/020:15:140
1759161240,1sleep10-21swapper/119:45:121
1720271240,1sleep10-21swapper/119:10:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional