You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-25 - 02:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Dec 25, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1800390211757,55sleep30-21swapper/319:06:233
180059828857,10sleep20-21swapper/219:08:582
180040628352,9sleep10-21swapper/119:06:321
180061628061,13sleep00-21swapper/019:09:110
20859602350,4sleep134-21ksoftirqd/123:35:191
20177542260,6sleep244-21ksoftirqd/222:30:212
20808802242,11sleep22080937-21/usr/sbin/munin23:30:242
1800830992320,1cyclictest0-21swapper/319:15:223
1800830992319,1cyclictest0-21swapper/321:05:163
180083099222,13cyclictest1896639-21taskset20:35:193
1800830992218,2cyclictest0-21swapper/323:00:003
1800830992117,2cyclictest0-21swapper/321:05:013
1800830992117,2cyclictest0-21swapper/319:59:553
21179102201,10sleep22118121-21grep00:05:272
1800830992017,1cyclictest0-21swapper/321:50:253
1800830992017,1cyclictest0-21swapper/300:30:163
1800830992017,1cyclictest0-21swapper/300:10:283
20071422191,12sleep02007255-21ntpq22:20:190
180083099198,8cyclictest1881994-21turbostat20:29:553
180083099192,11cyclictest2081962-21taskset23:35:003
1800830991916,1cyclictest0-21swapper/322:20:163
180083099191,14cyclictest0-21swapper/320:20:123
180083099191,13cyclictest2019025-21chrt22:35:123
180083099190,15cyclictest1960051-21chrt21:35:273
180083099189,6cyclictest2150035-21turbostat00:40:003
180083099188,6cyclictest2107882-21taskset23:59:123
1800830991815,1cyclictest0-21swapper/323:50:183
1800830991815,1cyclictest0-21swapper/322:30:193
1800830991815,1cyclictest0-21swapper/322:05:163
1800830991815,1cyclictest0-21swapper/320:50:183
1800830991815,1cyclictest0-21swapper/320:50:173
1800830991814,2cyclictest0-21swapper/321:50:013
180083099181,13cyclictest0-21swapper/320:30:133
180083099181,13cyclictest0-21swapper/319:10:103
180083099181,10cyclictest2066373-21taskset23:20:013
20344372171,11sleep22029214-21cron22:49:542
19019212171,11sleep11901977-21ntpq20:40:191
180083099179,5cyclictest2118619-21turbostat00:10:003
180083099178,6cyclictest1818497-21taskset19:21:403
180083099177,6cyclictest1918210-21taskset20:55:433
180083099171,9cyclictest2108247-21taskset00:00:063
1800830991714,1cyclictest0-21swapper/322:25:153
1800830991712,3cyclictest0-21swapper/322:15:113
180083099171,12cyclictest1834484-21sleep19:39:593
180083099171,12cyclictest0-21swapper/320:40:173
180083099171,12cyclictest0-21swapper/300:25:123
180083099171,10cyclictest2087328-21chrt23:40:023
180083099171,10cyclictest1933053-21taskset21:10:193
180083099171,10cyclictest1844870-21taskset19:48:333
21067872161,10sleep12106928-21ntpq23:55:201
19514852161,10sleep11953169-21tr21:30:171
18458002161,10sleep11845905-21grep19:50:141
18286102161,10sleep11828811-21users19:30:261
180083099168,6cyclictest1939494-21turbostat21:20:003
180083099164,9cyclictest2071682-21chrt23:25:133
1800830991614,1cyclictest0-21swapper/322:40:163
180083099161,3cyclictest0-21swapper/322:50:123
180083099161,13cyclictest0-21swapper/323:00:133
180083099161,13cyclictest0-21swapper/320:45:153
180083099161,12cyclictest0-21swapper/323:35:163
180083099161,12cyclictest0-21swapper/320:00:163
1800830991611,2cyclictest0-21swapper/300:15:113
180083099161,11cyclictest0-21swapper/321:25:103
180083099161,10cyclictest1992871-21chrt22:10:103
180083099161,10cyclictest1845506-21taskset19:50:033
1800830991610,4cyclictest67150irq/132-enp1s0-TxRx-319:25:193
180083099160,14cyclictest0-21swapper/323:10:233
180083099160,14cyclictest0-21swapper/300:20:233
180083099160,13cyclictest0-21swapper/323:05:193
180083099160,13cyclictest0-21swapper/321:55:253
21404302151,10sleep02141886-21kthreadcore00:30:180
20416682151,9sleep02043410-21cut22:55:170
20135062151,4sleep02013558-21unin-run22:30:000
19659472151,9sleep21965966-21apt-get21:45:102
19246122155,6sleep11925825-21kthreadcore21:05:171
18315292151,3sleep01833225-21wc19:35:170
180083099159,4cyclictest0-21swapper/321:30:133
180083099151,9cyclictest2029465-21taskset22:45:063
180083099151,9cyclictest1861267-21chrt20:05:013
180083099151,9cyclictest1828459-21taskset19:30:223
1800830991512,1cyclictest0-21swapper/320:15:123
180083099151,10cyclictest2060896-21chrt23:15:113
180083099151,10cyclictest1960868-21taskset21:40:123
180083099150,12cyclictest0-21swapper/322:00:183
180083099150,12cyclictest0-21swapper/321:20:203
21450622141,9sleep02145183-21grep00:35:100
21296402141,8sleep12129736-21fschecks_time00:20:121
20596642141,2sleep00-21swapper/023:10:210
180083099144,8cyclictest1866242-21turbostat20:14:563
180083099141,10cyclictest1834750-21chrt19:40:133
180083099140,11cyclictest0-21swapper/323:45:183
20178612131,8sleep02013772-21/usr/sbin/munin22:30:250
18508132131,8sleep11851072-21df_inode19:55:141
21032532121,7sleep02103327-21fschecks_time23:55:150
20627352125,3sleep02064501-21grep23:15:170
18694182121,7sleep21870313-21missed_timers20:10:192
18576592121,7sleep11859395-21kthreadcore20:00:201
21468482111,6sleep12148618-21kthreadcore00:35:161
21401212111,4sleep22140331-21hddtemp_smartct00:30:152
21304862110,7sleep02132153-21kthreadcore00:20:150
20664322115,2sleep10-21swapper/123:20:121
20292102111,5sleep02029231-21munin-run22:45:000
20242622112,4sleep12024280-21apt-get22:40:121
19849952111,5sleep01986221-21memory22:00:180
19829162111,6sleep21983334-21kthreadcore22:00:162
19660332111,4sleep01966236-21diskstats21:45:120
19559842111,7sleep20-21swapper/221:35:162
19397762111,5sleep11940029-21df21:20:111
19227362111,5sleep11922793-21ntpq21:00:201
19031342112,4sleep11903153-21apt-get20:45:131
18455412110,1sleep00-21swapper/019:50:130
18404832111,5sleep01840553-21irqstats19:45:150
18297402111,4sleep11829950-21hddtemp_smartct19:35:141
18242322111,5sleep21824251-21apt-get19:30:112
21253752101,2sleep20-21swapper/200:15:172
21231372101,5sleep02123149-21run-parts00:10:250
21110672101,4sleep22112141-21memory00:00:212
20608742101,3sleep22060893-21apt-get23:15:112
20239772101,4sleep12024008-21munin-run22:39:591
20030152101,4sleep22003086-21/usr/sbin/munin22:20:012
19974382100,1sleep20-21swapper/222:10:262
19920602101,5sleep21992224-21vmstat22:05:262
19397542101,3sleep00-21swapper/021:20:100
19320172101,4sleep21933052-21memory21:10:192
18880602101,5sleep11889627-21tr20:30:171
18810002101,5sleep21881025-21gcc20:20:202
18770082100,5sleep10-21swapper/120:20:111
18677302100,6sleep11869417-21cut20:10:171
18630622100,6sleep01864807-21kthreadcore20:05:200
18600222100,2sleep00-21swapper/020:00:220
18557402101,5sleep21855798-21lockfile-create20:00:002
18547272102,4sleep01854772-21ntpq19:55:210
18439812101,4sleep11844029-21ntpq19:45:201
18050352101,4sleep01805089-21ntpq19:10:210
2113298291,4sleep02113316-21munin-run00:05:000
2108282291,3sleep12108540-21df_inode00:00:161
2075605291,4sleep22075652-21ntpq23:25:222
2071330292,3sleep00-21swapper/023:24:280
2068469291,4sleep22070177-21sed23:20:172
2055975291,4sleep12056174-21cut23:10:141
2046097291,4sleep22047737-21cut23:00:172
2045197291,2sleep10-21swapper/123:00:121
2033499291,4sleep12033549-21ntpq22:45:201
2033444291,4sleep02033498-21ld22:45:190
2024283291,4sleep22024261-21/usr/sbin/munin22:40:132
2012377291,3sleep22012494-21ntpq22:25:182
2003859291,4sleep12004677-21grep22:20:151
1987144290,1sleep10-21swapper/122:02:041
1981852290,1sleep20-21swapper/221:57:052
1971787291,4sleep11971834-21fschecks_count21:50:131
1960716291,4sleep21960867-21cat21:40:122
1959466291,3sleep01959522-21ntp_states21:35:220
1945628291,4sleep01947246-21cut21:25:160
1942298291,3sleep21943590-21memory21:20:162
1939471291,3sleep21939501-21munin-run21:20:002
1930377291,4sleep11932016-21grep21:10:171
1929239291,2sleep00-21swapper/021:10:120
1928024291,3sleep21928081-21ntp_states21:05:212
1924040291,3sleep01924059-21apt-get21:05:120
1914120291,4sleep21914568-21cat20:55:152
1913853291,4sleep01913919-21tune2fs20:55:140
1887456291,3sleep00-21swapper/020:30:120
1861781291,4sleep21861852-21irqstats20:05:172
1849399291,4sleep21849520-21ntpq19:50:192
1839136290,1sleep10-21swapper/119:40:281
1822991291,4sleep01823038-21ntpq19:25:190
1817885291,4sleep11818005-21grep19:20:241
1812935291,4sleep11813152-21users19:15:261
1812483291,3sleep21812534-21ntpq19:15:192
2124507282,3sleep00-21swapper/000:15:150
2119154281,3sleep22119367-21hddtemp_smartct00:10:152
2114126281,3sleep02115069-21kthreadcore00:05:170
2102972281,2sleep20-21swapper/223:55:132
2096504281,2sleep02096553-21ntpq23:45:210
2092827281,3sleep12092874-21fschecks_count23:45:141
2087341281,2sleep10-21swapper/123:40:121
2051024281,3sleep02052262-21kthreadcore23:05:150
2019004281,2sleep20-21swapper/222:35:122
2008203280,1sleep20-21swapper/222:24:092
1977072281,3sleep11977091-21apt-get21:55:111
1965372280,1sleep10-21swapper/121:40:591
1965036282,3sleep0221rcuc/021:40:260
1950249281,3sleep01950268-21apt-get21:30:120
1923165281,3sleep01923409-21users21:00:260
1919404281,3sleep21920813-21kthreadcore21:00:172
1908960281,1sleep20-21swapper/220:50:192
1908960281,1sleep20-21swapper/220:50:182
1901872281,2sleep01901913-21ntpq20:40:180
1896544281,3sleep21896632-21ntpq20:35:202
1887093281,2sleep00-21swapper/020:28:440
1866500281,2sleep00-21swapper/020:10:120
1834265280,6sleep20-21swapper/219:37:432
2149903270,4sleep20-21swapper/200:38:332
2141888271,3sleep12143577-21cat00:30:181
2139318270,1sleep00-21swapper/000:26:530
2124122270,2sleep10-21swapper/100:15:121
2115070271,2sleep10-21swapper/100:05:181
2102697270,1sleep00-21swapper/023:54:500
2098021271,3sleep12098091-21grep23:50:141
2097177271,2sleep20-21swapper/223:46:142
2091319270,2sleep20-21swapper/223:40:222
2077441271,2sleep00-21swapper/023:30:170
2076191270,1sleep10-21swapper/123:25:291
2055380270,1sleep20-21swapper/223:09:262
2039024271,3sleep22039277-21unixbench_singl22:50:272
2023286271,2sleep00-21swapper/022:35:250
2011563271,3sleep12012376-21wc22:25:161
1998060270,2sleep10-21swapper/122:15:111
1996670271,3sleep11996783-21ntpq22:10:171
1992282270,1sleep00-21swapper/022:06:070
1975383270,3sleep21971509-21/usr/sbin/munin21:50:182
1954316271,3sleep21954377-21/usr/sbin/munin21:30:222
1949633272,2sleep21800823-21cyclictest21:25:342
1917662271,3sleep11917777-21/usr/sbin/munin20:55:221
1908348270,2sleep10-21swapper/120:50:141
1908348270,2sleep10-21swapper/120:50:131
1894215271,2sleep01895939-21kthreadcore20:35:180
1881910270,1sleep00-21swapper/020:24:070
1876382270,1sleep10-21swapper/120:15:431
1836866271,3sleep00-21swapper/019:40:190
1834728270,2sleep20-21swapper/219:40:132
1815922271,2sleep00-21swapper/019:20:190
2102403260,1sleep20-21swapper/223:51:082
2092050260,4sleep00-21swapper/023:41:590
2087019260,1sleep00-21swapper/023:39:310
2082652261,2sleep20-21swapper/223:35:162
2078376261,2sleep10-21swapper/123:30:191
2071924261,2sleep00-21swapper/023:25:170
2055685260,1sleep20-21swapper/223:10:112
2055364260,1sleep10-21swapper/123:09:131
2050013260,1sleep00-21swapper/023:02:440
2040229261,2sleep10-21swapper/122:55:141
2036280261,2sleep10-21swapper/122:50:181
2035275261,1sleep00-21swapper/022:50:170
2017699260,2sleep10-21swapper/122:30:221
2001872261,2sleep02001993-21ntpq22:15:170
1997722260,1sleep00-21swapper/022:14:020
1992497260,4sleep10-21swapper/122:08:501
1977035260,1sleep00-21swapper/021:55:020
1972404261,2sleep00-21swapper/021:50:160
1960358261,3sleep10-21swapper/121:39:241
1936282261,2sleep00-21swapper/021:15:180
1907830260,1sleep00-21swapper/020:46:480
1896956261,2sleep10-21swapper/120:35:271
1892201261,2sleep20-21swapper/220:32:262
1882242260,1sleep10-21swapper/120:25:131
1871724260,1sleep20-21swapper/220:15:112
1864808260,2sleep11865180-21strings20:05:201
1823278261,2sleep10-21swapper/119:25:241
1810658260,2sleep00-21swapper/019:15:170
1805900260,4sleep20-21swapper/219:13:422
1805314261,2sleep10-21swapper/119:10:261
2133174250,1sleep20-21swapper/200:20:172
2122894250,1sleep10-21swapper/100:10:221
2061165250,1sleep10-21swapper/123:15:141
2043867250,1sleep20-21swapper/222:55:192
1949033250,1sleep10-21swapper/121:25:211
1904273250,2sleep20-21swapper/220:45:182
1902217251,1sleep20-21swapper/220:40:242
1871903250,1sleep00-21swapper/020:15:140
1860130250,2sleep20-21swapper/220:00:252
1840069250,1sleep20-21swapper/219:45:132
1828930250,1sleep00-21swapper/019:31:490
1813827250,2sleep20-21swapper/219:20:122
2134882240,1sleep20-21swapper/200:25:152
2134334240,1sleep10-21swapper/100:25:021
1965690240,1sleep10-21swapper/121:45:011
1934759240,1sleep10-21swapper/121:15:151
1908644240,1sleep00-21swapper/020:50:160
1908644240,1sleep00-21swapper/020:50:160
1881982240,1sleep20-21swapper/220:25:012
1819344240,1sleep20-21swapper/219:25:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional