You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 23:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Aug 26, 2025 12:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20542242151102,36sleep20-21swapper/207:06:082
20543132147102,36sleep10-21swapper/107:07:151
20542482147101,16sleep00-21swapper/007:06:260
20544412145100,36sleep30-21swapper/307:08:533
215320821190,2sleep10-21swapper/108:45:131
2054654993926,9cyclictest2189983-21grep09:15:263
2054654993228,2cyclictest2473824-21ls12:24:593
2054654993225,4cyclictest2264622-21ps10:05:233
2054654993128,2cyclictest2062265-21kthreadcore07:15:183
2054654993127,2cyclictest2308662-21needreboot10:35:223
2054654992819,6cyclictest2250024-21grep09:55:293
2054654992723,2cyclictest0-21swapper/309:05:263
2054654992722,3cyclictest0-21swapper/312:05:123
2054654992717,7cyclictest2297555-21grep10:30:193
24117112262,14sleep02411793-21/usr/sbin/munin11:45:190
2054654992621,3cyclictest0-21swapper/312:40:003
2054654992421,2cyclictest2384260-21ssh11:25:263
24971242230,1sleep20-21swapper/212:39:012
2054654992319,3cyclictest2222940-21sh09:40:013
2054654992319,3cyclictest2222940-21sh09:40:003
2054654992319,2cyclictest0-21swapper/308:50:233
2054654992318,3cyclictest726-21in:imuxsock10:52:013
2054654992318,3cyclictest1-21systemd12:30:283
205465499231,19cyclictest2196902-21ntpq09:20:223
2054654992310,8cyclictest262-21systemd-journal12:29:363
205465499230,8cyclictest2153077-21grep08:45:133
205465499221,7cyclictest2228021-21ssh09:40:453
2054654992217,3cyclictest0-21swapper/311:10:013
2054654992214,6cyclictest2209933-21cut09:30:233
2054654992212,6cyclictest2400953-21ssh11:36:543
205465499221,18cyclictest2342624-21munin-node11:00:143
2054654992211,7cyclictest2321128-21cut10:45:173
2054654992210,7cyclictest2204004-21tr09:25:173
205465499220,18cyclictest2126944-21runrttasks08:17:013
205465499218,9cyclictest2369227-21grep11:15:253
205465499211,7cyclictest2101139-21needreboot07:50:183
2054654992117,2cyclictest0-21swapper/310:40:273
2054654992117,2cyclictest0-21swapper/308:35:143
2054654992117,2cyclictest0-21swapper/307:35:193
205465499211,6cyclictest2259598-21ssh10:04:003
2054654992115,4cyclictest2238313-21cpuspeed_turbos09:50:123
205465499211,13cyclictest2427424-21/usr/sbin/munin11:55:133
2054654992110,9cyclictest0-21swapper/311:45:153
23852952201,2sleep10-21swapper/111:27:121
23194942201,15sleep22319562-21grep10:45:002
21417412200,2sleep10-21swapper/108:30:221
205465499201,8cyclictest2171721-21ntpq09:00:213
2054654992016,3cyclictest2277394-21kthreadcore10:15:203
2054654992016,2cyclictest2076598-21dump-pmu-power07:30:003
2054654992013,5cyclictest511ktimers/311:51:493
205465499201,17cyclictest2096527-21/usr/sbin/munin07:45:273
205465499201,17cyclictest2081853-21sshd07:34:343
205465499201,16cyclictest2183747-21ssh09:12:343
205465499201,11cyclictest2391100-21tail11:30:193
205465499200,18cyclictest2339001-21/usr/sbin/munin10:55:243
205465499200,18cyclictest2273420-21ssh10:12:023
2054654991914,4cyclictest511ktimers/311:14:543
2054654991910,6cyclictest2463526-21taskset12:16:103
205465499190,17cyclictest2286448-21ntpq10:20:223
205465499190,17cyclictest2237254-21sh09:49:073
205465499190,16cyclictest2454405-21sort12:10:233
205465499190,16cyclictest2295745-21sh10:27:543
24501662188,6sleep22450180-21ssh12:09:582
23693552180,13sleep22369376-21/usr/sbin/munin11:15:272
23494992181,2sleep10-21swapper/111:04:471
205465499189,7cyclictest2102614-21df_inode07:55:143
205465499189,6cyclictest2403559-21/usr/sbin/munin11:40:143
205465499189,6cyclictest2137552-21cron08:30:003
205465499188,6cyclictest945-21snmpd08:30:173
2054654991815,2cyclictest2372774-21cut11:20:133
2054654991813,3cyclictest2112803-21gltestperf08:05:163
205465499180,16cyclictest2439691-21ssh12:00:453
238695021712,2sleep00-21swapper/011:30:000
205465499172,13cyclictest0-21swapper/307:20:233
205465499171,15cyclictest0-21swapper/308:14:193
23496282161,2sleep20-21swapper/211:05:002
23248692161,2sleep10-21swapper/110:46:161
205465499167,7cyclictest2107829-21head08:00:153
205465499167,5cyclictest703-21dbus-daemon07:10:233
205465499166,7cyclictest2151629-21ntpq08:40:193
2054654991613,2cyclictest2129368-21tr08:20:173
205465499161,2cyclictest2166597-21/usr/sbin/munin08:55:203
23798612151,9sleep22380013-21df11:25:132
20706082151,9sleep02070653-21ntpq07:20:250
205465499150,2cyclictest2087346-21/usr/sbin/munin07:40:123
24190492141,9sleep00-21swapper/011:50:120
23717822140,3sleep12327045-21kworker/u16:4+rpciod11:19:181
23636342141,8sleep20-21swapper/211:12:582
23035442141,8sleep22177687-21diskmemload10:33:302
22938232141,9sleep02293945-21/usr/sbin/munin10:25:230
22680232143,5sleep22268043-21apt-get10:10:142
21825982141,9sleep12182613-21ssh09:10:321
21765312141,9sleep22176656-21/usr/sbin/munin09:05:232
21667432141,9sleep02166834-21ps08:55:230
21475652141,2sleep00-21swapper/008:40:010
23548852130,6sleep134-21ksoftirqd/111:06:001
23062792131,8sleep12307933-21grep10:35:221
22673102131,5sleep20-21swapper/210:09:252
22657712131,8sleep02177687-21diskmemload10:06:500
22522852131,6sleep10-21swapper/109:59:291
24809592120,8sleep12480970-21sh12:28:261
24682482121,3sleep00-21swapper/012:20:190
23416402121,7sleep02341652-21ssh10:59:010
23188772121,7sleep02177687-21diskmemload10:43:580
22708892121,6sleep02271736-21sed10:10:190
22138692121,5sleep22213885-21fwupd09:32:302
24746442110,4sleep22474866-21hddtemp_smartct12:25:142
24575242110,3sleep10-21swapper/112:14:061
24274322112,4sleep10-21swapper/111:55:141
24128052111,6sleep22414519-21kthreadcore11:45:212
23947042111,3sleep00-21swapper/011:34:500
23399872111,3sleep10-21swapper/110:56:141
23029452111,3sleep10-21swapper/110:32:241
22884412111,4sleep22177687-21diskmemload10:22:552
22761292111,7sleep00-21swapper/010:15:200
22524792111,3sleep20-21swapper/209:59:502
22310682111,6sleep22231374-21kthreadcore09:45:152
22223902111,5sleep02177687-21diskmemload09:39:040
22223902111,5sleep02177687-21diskmemload09:39:040
22174212111,6sleep22219138-21kthreadcore09:35:172
22174212111,6sleep22219138-21kthreadcore09:35:172
22156482111,3sleep10-21swapper/109:35:041
22156482111,3sleep10-21swapper/109:35:041
22143772111,3sleep00-21swapper/009:33:230
22053032111,4sleep00-21swapper/009:25:530
22047672111,3sleep10-21swapper/109:25:261
21978822111,4sleep1331ktimers/109:21:061
21844312111,5sleep22177687-21diskmemload09:13:492
21569822111,5sleep02157226-21users08:45:250
21418882111,6sleep02142140-21sed08:30:260
21072222111,5sleep02107249-21munin-run08:00:000
20867152110,5sleep22086657-31find07:36:392
20858752111,5sleep12085924-21ntpq07:35:201
24951282101,4sleep02177687-21diskmemload12:35:540
24947272101,3sleep10-21swapper/112:35:281
24873332101,3sleep00-21swapper/012:30:570
24823432101,4sleep22177687-21diskmemload12:30:072
24712712101,3sleep10-21swapper/112:21:041
24656092101,2sleep20-21swapper/212:19:322
24650182101,5sleep12465028-21ssh12:18:391
24631472101,4sleep02177687-21diskmemload12:15:350
24563842101,3sleep20-21swapper/212:12:192
24493092101,6sleep12449320-21ssh12:08:411
24468822101,4sleep02177687-21diskmemload12:05:230
24410542101,3sleep10-21swapper/112:03:081
24387402101,4sleep02438869-21ntpq12:00:180
24326492101,3sleep20-21swapper/211:56:482
24314302100,3sleep00-21swapper/011:55:260
24189642101,3sleep20-21swapper/211:50:062
24092032101,5sleep22177687-21diskmemload11:42:292
24029642101,4sleep22403044-21munin-run11:40:002
24002202101,4sleep02177687-21diskmemload11:35:410
23873422101,4sleep12387366-21apt-get11:30:141
23782142101,3sleep20-21swapper/211:22:282
23622332101,2sleep10-21swapper/111:10:371
23550122101,3sleep20-21swapper/211:06:142
23337092101,2sleep00-21swapper/010:53:280
23326282101,3sleep10-21swapper/110:51:371
23324902101,4sleep22177687-21diskmemload10:51:212
23259102101,2sleep00-21swapper/010:48:070
23253742101,3sleep20-21swapper/210:47:142
23182952101,2sleep10-21swapper/110:42:551
23108772101,3sleep02177687-21diskmemload10:38:100
23015892101,2sleep00-21swapper/010:30:310
22884172101,2sleep00-21swapper/010:22:530
22872712101,2sleep10-21swapper/110:20:541
22442792101,3sleep20-21swapper/209:53:172
22428732101,4sleep02177687-21diskmemload09:50:430
22380512101,2sleep10-21swapper/109:50:041
22300352101,5sleep22177687-21diskmemload09:44:312
22280072101,3sleep10-21swapper/109:40:451
22052552101,4sleep22177687-21diskmemload09:25:472
22000302101,3sleep00-21swapper/009:24:490
21915172100,1sleep20-21swapper/209:18:142
21901702101,6sleep12190175-21ssh09:15:441
21844532100,3sleep00-21swapper/009:13:520
20971322100,1sleep10-21swapper/107:49:551
2482330291,2sleep10-21swapper/112:30:061
2474954291,3sleep02177687-21diskmemload12:25:170
2470071291,4sleep22470133-21ntpq12:20:212
2440563291,3sleep20-21swapper/212:02:252
2425087290,3sleep10-21swapper/111:52:421
2417141290,2sleep10-21swapper/111:47:331
2409953291,4sleep02409967-21ssh11:43:390
2403590291,2sleep10-21swapper/111:40:161
2400504291,3sleep12177687-21diskmemload11:36:101
2392211291,5sleep22177687-21diskmemload11:30:582
2379521291,2sleep10-21swapper/111:24:561
2377079295,1sleep00-21swapper/011:20:310
2371575290,2sleep00-21swapper/011:18:530
2364065290,6sleep02177687-21diskmemload11:13:440
2354945291,2sleep00-21swapper/011:06:070
2340318291,3sleep22177687-21diskmemload10:56:482
2295920290,2sleep10-21swapper/110:28:121
2294517291,3sleep22177687-21diskmemload10:25:442
2281451290,2sleep10-21swapper/110:18:451
2272650291,2sleep10-21swapper/110:10:401
2266586290,6sleep12266589-21ssh10:08:111
2260060290,2sleep10-21swapper/110:04:521
2259120291,3sleep22177687-21diskmemload10:03:122
2253430294,2sleep02253503-21kthreadcore10:00:170
2252501291,3sleep02177687-21diskmemload09:59:520
2235741290,2sleep10-21swapper/109:46:271
2230793291,3sleep00-21swapper/009:45:130
2227994290,4sleep02177687-21diskmemload09:40:450
2214753291,2sleep10-21swapper/109:34:001
2196838291,4sleep22196892-21ntpq09:20:232
2185680291,3sleep00-21swapper/009:15:150
2147850291,3sleep02147869-21apt-get08:40:120
2136424291,3sleep12136533-21ntpq08:25:181
2132777291,3sleep22132796-21apt-get08:25:112
2126984290,2sleep10-21swapper/108:17:331
2102937291,4sleep22104520-21cut07:55:172
2456790280,2sleep00-21swapper/012:12:570
2348638280,1sleep00-21swapper/011:03:180
2309566280,3sleep22177687-21diskmemload10:35:502
2177342280,1sleep10-21swapper/109:05:501
2162879281,2sleep20-21swapper/208:55:132
2137814280,3sleep22137833-21apt-get08:30:122
2136867281,4sleep02137096-21awk08:25:260
2128385283,2sleep016-21ksoftirqd/008:20:160
2127053280,6sleep20-21swapper/208:18:262
2116999280,3sleep265-21kcompactd008:07:202
2101875280,1sleep00-21swapper/007:51:080
2097401280,2sleep10-21swapper/107:50:111
2076263280,2sleep10-21swapper/107:26:041
2055227281,3sleep12055385-21cat07:10:161
2280907272,2sleep22273629-21kworker/u16:0+xprtiod10:17:502
2173582271,3sleep02175257-21kthreadcore09:05:210
2168520271,2sleep10-21swapper/109:00:171
2167551270,1sleep10-21swapper/108:58:571
2157886271,3sleep02158041-21diskstats08:50:140
2152865270,2sleep20-21swapper/208:45:122
2152460270,5sleep20-21swapper/208:43:112
2149439271,3sleep12149737-21kthreadcore08:40:171
2131951270,1sleep20-21swapper/208:21:182
2122007270,1sleep10-21swapper/108:12:481
2121139270,2sleep20-21swapper/208:10:212
2116237271,2sleep10-21swapper/108:05:221
2108001271,3sleep12109541-21cut08:00:181
2102009270,1sleep20-21swapper/207:52:492
2097038270,1sleep00-21swapper/007:48:420
2086694270,2sleep00-21swapper/007:36:320
2081772270,1sleep00-21swapper/007:33:350
2076558270,2sleep20-21swapper/207:29:492
2059587271,3sleep02059605-21latency_hist07:15:000
2172554260,1sleep20-21swapper/209:03:472
2168318261,2sleep00-21swapper/009:00:170
2147256260,1sleep20-21swapper/208:36:052
2142831260,1sleep10-21swapper/108:35:131
2131319261,2sleep10-21swapper/108:20:211
2123591261,2sleep00-21swapper/008:15:190
2116189260,1sleep00-21swapper/008:05:210
2111222260,1sleep00-21swapper/008:00:200
2091788260,1sleep20-21swapper/207:40:512
2091147260,3sleep00-21swapper/007:40:200
2081816260,1sleep10-21swapper/107:34:081
2075635260,1sleep00-21swapper/007:25:220
2067109261,2sleep10-21swapper/107:20:181
2066808260,2sleep20-21swapper/207:20:152
2059992261,2sleep20-21swapper/207:15:022
2059607260,2sleep10-21swapper/107:15:011
2055386261,2sleep20-21swapper/207:10:172
2161734250,1sleep10-21swapper/108:50:231
2111305250,1sleep20-21swapper/208:00:222
2102442250,1sleep10-21swapper/107:55:121
2096097250,1sleep20-21swapper/207:45:192
2158252240,1sleep20-21swapper/208:50:162
2117495240,1sleep00-21swapper/008:10:140
2087176240,1sleep10-21swapper/107:40:011
2077184240,1sleep20-21swapper/207:30:122
2061848240,1sleep00-21swapper/007:15:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional