You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 04:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot4s (updated Thu Nov 20, 2025 00:59:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
190199912681,1261cyclictest27980-21users20:00:421
190199912231214,6cyclictest0-21swapper/119:25:181
190189911741140,24cyclictest0-21swapper/019:25:160
185002504458,31sleep00-21swapper/019:20:040
186652491444,32sleep10-21swapper/119:21:511
1901899197176,17cyclictest0-21swapper/020:00:410
219342790,3sleep1211ktimersoftd/122:17:451
64292600,2sleep06427-21fgrep00:15:280
312672451,4sleep131278-21sh00:08:081
1901899448,5cyclictest81rcu_preempt22:59:360
1901899447,6cyclictest83662open_files20:35:260
1901999434,9cyclictest81rcu_preempt00:22:331
19019994223,4cyclictest22-21ksoftirqd/123:50:021
263582411,35sleep126515-21sed23:30:001
1901999413,4cyclictest81rcu_preempt00:39:171
1901899401,12cyclictest22969-21ssh22:19:150
152722400,3sleep0111rcuc/000:25:140
19019993915,5cyclictest40950irq/93-i91521:29:151
12692398,27sleep10-21swapper/123:03:461
1901999388,5cyclictest81rcu_preempt22:10:041
1901999388,5cyclictest81rcu_preempt22:10:041
1901999386,10cyclictest81rcu_preempt22:59:501
1901999386,10cyclictest22-21ksoftirqd/100:15:281
1901999384,4cyclictest81rcu_preempt21:59:051
1901899380,13cyclictest11129-21ssh00:20:250
111352381,4sleep111275-21latency_hist20:45:011
1901999374,4cyclictest81rcu_preempt21:13:091
19019993731,4cyclictest137350irq/90-eth021:20:481
1901999373,11cyclictest81rcu_preempt22:30:501
19019993727,5cyclictest22-21ksoftirqd/122:05:101
19019993713,7cyclictest22-21ksoftirqd/123:23:001
1901899372,10cyclictest0-21swapper/023:22:190
68772360,8sleep11901999cyclictest22:35:261
1901999367,15cyclictest81rcu_preempt23:53:071
1901999365,4cyclictest81rcu_preempt00:34:511
1901999364,4cyclictest81rcu_preempt22:03:141
1901999362,4cyclictest81rcu_preempt00:26:251
1901999362,4cyclictest22-21ksoftirqd/122:20:271
1901999361,5cyclictest22-21ksoftirqd/123:40:241
1901899361,18cyclictest20563-21kernelversion00:30:250
1901999358,4cyclictest81rcu_preempt23:58:471
1901999358,4cyclictest81rcu_preempt22:52:441
1901999353,4cyclictest81rcu_preempt21:46:521
19019993530,3cyclictest137350irq/90-eth023:19:361
19019993529,4cyclictest137350irq/90-eth021:53:231
1901999352,4cyclictest81rcu_preempt23:30:431
19019993514,10cyclictest22-21ksoftirqd/122:43:371
19019993513,19cyclictest22-21ksoftirqd/123:40:001
1901899356,5cyclictest81rcu_preempt22:20:120
1901899353,16cyclictest14741-21iostat_ios22:10:180
1901899353,16cyclictest14741-21iostat_ios22:10:170
1901899352,4cyclictest81rcu_preempt22:04:500
19018993524,7cyclictest23493-21ssh23:25:500
1901899352,16cyclictest0-21swapper/023:02:370
19018993511,4cyclictest81rcu_preempt21:33:430
1901999347,4cyclictest81rcu_preempt23:10:401
1901999344,4cyclictest81rcu_preempt00:47:051
19019993428,4cyclictest137350irq/90-eth022:29:001
19019993420,4cyclictest22-21ksoftirqd/121:17:151
19019993420,4cyclictest22-21ksoftirqd/119:39:141
1901999341,3cyclictest22-21ksoftirqd/121:35:131
1901999341,16cyclictest15889-21df_inode22:45:151
1901899347,4cyclictest81rcu_preempt00:01:590
1901899345,4cyclictest81rcu_preempt23:42:380
1901899345,4cyclictest81rcu_preempt20:10:280
1901899344,4cyclictest81rcu_preempt21:50:200
1901899344,4cyclictest81rcu_preempt00:09:500
1901899341,21cyclictest0-21swapper/021:29:270
169612341,5sleep017022-21users22:45:310
261682331,4sleep10-21swapper/100:02:431
1901999336,4cyclictest81rcu_preempt23:05:531
1901999334,4cyclictest81rcu_preempt21:30:541
19019993327,4cyclictest137350irq/90-eth021:44:061
1901999331,4cyclictest81rcu_preempt20:50:181
1901899335,4cyclictest81rcu_preempt23:16:450
1901899334,5cyclictest81rcu_preempt22:43:260
1901899334,4cyclictest81rcu_preempt21:10:000
19018993323,6cyclictest5482-21cron20:30:000
1901899331,6cyclictest0-21swapper/000:48:100
139302331,29sleep013967-21grep23:50:190
121662331,3sleep00-21swapper/020:45:260
59432321,4sleep05948-21diskstats20:30:190
1901999322,8cyclictest32617-21df20:15:141
19019993212,4cyclictest22-21ksoftirqd/120:08:021
1901999321,16cyclictest3020-21runrttasks20:58:261
1901899324,5cyclictest81rcu_preempt23:38:270
1901899324,4cyclictest81rcu_preempt22:25:220
1901899324,4cyclictest81rcu_preempt21:15:210
1901899323,16cyclictest0-21swapper/000:36:100
1901899322,16cyclictest21173-21df_inode21:10:160
1901899322,10cyclictest23252-21ntp_states21:45:230
1901899321,4cyclictest81rcu_preempt22:50:160
300082311,5sleep00-21swapper/020:07:060
1901999317,4cyclictest81rcu_preempt00:50:151
1901999312,4cyclictest81rcu_preempt19:59:501
1901999311,6cyclictest22-21ksoftirqd/120:25:271
1901999311,10cyclictest4151-21kworker/1:100:11:051
1901899317,4cyclictest81rcu_preempt22:05:220
1901899317,11cyclictest0-21swapper/020:55:230
1901899315,4cyclictest81rcu_preempt23:58:520
1901899314,4cyclictest81rcu_preempt00:11:170
1901899313,4cyclictest81rcu_preempt21:55:140
1901899311,4cyclictest81rcu_preempt22:35:230
1901899311,4cyclictest81rcu_preempt22:30:280
1901899311,4cyclictest81rcu_preempt21:23:430
19018993111,6cyclictest3020-21runrttasks19:45:410
1901899311,13cyclictest3950-21grep23:05:290
1901999306,4cyclictest81rcu_preempt21:00:271
1901999301,4cyclictest81rcu_preempt21:09:181
1901999301,15cyclictest30582-21mailstats00:40:291
1901899306,4cyclictest81rcu_preempt23:33:240
1901899301,4cyclictest81rcu_preempt00:42:140
19018993012,6cyclictest25560-21/usr/sbin/munin19:55:220
19018993011,6cyclictest8018-21ssh23:10:200
1901899301,12cyclictest12010-21ssh23:48:080
34872291,5sleep00-21swapper/020:23:490
34872291,5sleep00-21swapper/020:23:480
217042291,5sleep00-21swapper/019:40:270
1901999292,4cyclictest81rcu_preempt19:30:051
19019992913,4cyclictest22-21ksoftirqd/119:51:281
1901899292,14cyclictest18351-21/usr/sbin/munin21:40:220
148802291,5sleep00-21swapper/020:54:530
56372281,5sleep10-21swapper/120:30:051
227032281,5sleep10-21swapper/119:49:291
19019992814,11cyclictest22-21ksoftirqd/119:41:251
1901899282,3cyclictest81rcu_preempt00:54:030
1901899281,4cyclictest81rcu_preempt19:33:270
108142281,5sleep00-21swapper/020:41:380
89902271,5sleep10-21swapper/120:37:251
1901999278,4cyclictest40950irq/93-i91520:10:141
1901999277,4cyclictest81rcu_preempt20:45:111
1901999276,4cyclictest81rcu_preempt20:20:251
1901999276,4cyclictest81rcu_preempt20:20:241
1901899279,5cyclictest17239-21df21:00:140
1901899271,14cyclictest13338-21rm21:35:210
19018992710,5cyclictest21439-21taskset19:39:240
19018992611,13cyclictest0-21swapper/020:16:510
19018992516,6cyclictest3-21ksoftirqd/019:51:320
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional