You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-29 - 12:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 100 highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 29, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2700522460,3sleep011-21rcuc/019:05:150
266812240163,24sleep30-21swapper/319:05:023
284472235166,55sleep20-21swapper/219:07:232
285382231163,54sleep10-21swapper/119:08:341
28755999390,2cyclictest0-21swapper/222:08:292
54732530,3sleep32876099cyclictest21:55:163
3712530,4sleep232-21ksoftirqd/220:30:172
7612510,2sleep00-21swapper/020:30:180
265112510,1sleep20-21swapper/222:40:262
73432440,2sleep00-21swapper/019:30:250
28748994227,9cyclictest0-21swapper/119:33:551
2874899406,3cyclictest20090-21cut20:00:201
28748994026,8cyclictest0-21swapper/120:29:051
28748994013,24cyclictest11137-21hddtemp_smartct19:40:171
2874899401,11cyclictest23405-21hddtemp_smartct22:35:181
28748993811,5cyclictest2625-21smart_sda23:00:251
2874899372,13cyclictest27532-21munin-run00:00:011
2874899371,10cyclictest15011-21open_files22:15:251
2875599367,2cyclictest86750irq/128-enp1s0-20:26:182
28748993612,3cyclictest24288-21/usr/sbin/munin21:25:211
28748993612,3cyclictest24288-21/usr/sbin/munin21:25:211
2874899361,13cyclictest13290-21/usr/sbin/munin21:00:291
28748993610,4cyclictest7096-21hddtemp_smartct00:25:181
2874899360,20cyclictest0-21swapper/121:31:451
2874899353,22cyclictest0-21swapper/120:45:131
2874899351,27cyclictest0-21swapper/122:21:271
28748993512,3cyclictest32192-21/usr/sbin/munin00:10:161
28748993512,2cyclictest12367-21/usr/sbin/munin23:25:151
2874899351,11cyclictest8611-21diskstats19:35:151
28748993510,4cyclictest31893-21/usr/sbin/munin22:55:171
28748993510,3cyclictest29289-21df_inode19:10:181
28748993510,3cyclictest29289-21df_inode19:10:171
28748993510,3cyclictest14621-21latency_hist19:50:011
2874899350,6cyclictest0-21swapper/122:40:241
2874899350,10cyclictest0-21swapper/122:04:411
2874899346,7cyclictest0-21swapper/121:11:441
2874899345,4cyclictest5925-21latency_hist19:30:011
28748993420,8cyclictest0-21swapper/123:51:391
2874899341,31cyclictest0-21swapper/123:12:451
2874899340,5cyclictest27949-21kernelversion22:45:171
299772330,4sleep2311ktimersoftd/222:50:182
2874899339,2cyclictest17580-21irqstats23:35:181
2874899331,15cyclictest0-21swapper/120:43:261
2874899331,14cyclictest0-21swapper/121:22:231
28748993311,3cyclictest31833-21/usr/sbin/munin19:15:171
28748993311,3cyclictest31833-21/usr/sbin/munin19:15:171
28748993310,3cyclictest21164-21/usr/sbin/munin23:45:161
2874899330,31cyclictest0-21swapper/120:05:221
2874399335,14cyclictest0-21swapper/021:35:010
2874899324,22cyclictest0-21swapper/123:15:311
2874899322,2cyclictest0-21swapper/123:23:451
2874899322,21cyclictest0-21swapper/119:53:391
28748993211,3cyclictest31928-21ntp_kernel_err21:40:221
28748993211,3cyclictest20469-21cat22:30:011
28748993211,2cyclictest23662-21/usr/sbin/munin20:10:211
2874899321,10cyclictest5551-21awk23:10:011
28748993210,3cyclictest30029-21latency_hist20:25:011
28748993210,3cyclictest30029-21latency_hist20:25:011
227082320,1sleep00-21swapper/020:05:250
2874899319,3cyclictest836-21runrttasks00:15:431
2874899318,3cyclictest15499-21/usr/sbin/munin21:05:211
2874899311,8cyclictest22141-21runrttasks22:30:331
2874899311,8cyclictest22141-21runrttasks22:30:331
28748993117,8cyclictest0-21swapper/120:52:251
2874899311,6cyclictest13168-21timerwakeupswit22:10:241
2874899311,3cyclictest30334-21dnf00:05:171
28748993112,3cyclictest9312-21hwlatdetect00:30:171
2874899311,21cyclictest0-21swapper/120:38:191
28748993111,3cyclictest4379-21grep00:20:141
2874899311,13cyclictest11674-21/usr/sbin/munin00:35:221
2874899311,10cyclictest0-21swapper/120:00:021
28748993110,3cyclictest29524-21cat22:50:141
28748993110,3cyclictest12373-21cat20:55:241
28755993018,3cyclictest8886-21kworker/2:122:26:572
2875599301,2cyclictest0-21swapper/200:05:262
2874899309,3cyclictest836-21runrttasks20:19:591
28748993026,3cyclictest0-21swapper/121:18:111
2874899302,5cyclictest29499-1kworker/u9:121:45:191
28748993016,7cyclictest0-21swapper/120:30:231
28748993015,8cyclictest0-21swapper/121:54:591
28748993011,3cyclictest836-21runrttasks19:24:391
28748993010,3cyclictest14985-21fschecks_count23:30:151
2875599291,15cyclictest6980-21memory19:30:192
2874899299,3cyclictest9720-21cstates22:05:161
28748992915,7cyclictest0-21swapper/100:04:131
28748992910,3cyclictest19292-21expr23:40:131
2874399292,12cyclictest4283-21perf20:40:010
2875599281,14cyclictest0-21swapper/221:59:172
2875599281,13cyclictest0-21swapper/220:11:012
2874899288,3cyclictest5310-21cstates21:55:141
2876099274,7cyclictest14190-40dnf23:28:243
2875599271,12cyclictest0-21swapper/221:11:342
2875599270,2cyclictest0-21swapper/220:18:132
2874899271,3cyclictest30787-21cat21:40:011
2874399271,12cyclictest8675-21awk20:50:010
2875599267,2cyclictest86750irq/128-enp1s0-21:31:482
28755992614,3cyclictest8886-21kworker/2:123:15:012
2875599261,2cyclictest0-21swapper/219:13:022
2875599261,2cyclictest0-21swapper/219:13:012
2875599261,13cyclictest0-21swapper/222:02:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional