You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 11:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 100 highest latencies:
System rack9slot4s (updated Sat Jul 12, 2025 00:59:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161622622575,32sleep00-21swapper/019:22:030
162462507419,73sleep10-21swapper/119:22:591
1650599208178,6cyclictest18377-21munin-run19:30:000
894021160,2sleep18939-21aten_r4power_cu00:15:121
161622960,4sleep00-21swapper/022:08:050
143052940,9sleep11650699cyclictest00:20:171
102862940,4sleep016503-21cyclictest22:01:330
60012790,5sleep1201rcuc/123:05:011
173722640,5sleep117335-21ssh23:50:141
296402620,9sleep01650599cyclictest19:55:320
265312560,3sleep041ktimersoftd/019:50:010
187932450,8sleep01650599cyclictest21:36:580
1650699456,6cyclictest22-21ksoftirqd/100:30:321
312742411,36sleep131287-21sh00:37:531
1650699402,4cyclictest81rcu_preempt21:30:291
1650699397,4cyclictest81rcu_preempt21:16:381
1650699397,4cyclictest81rcu_preempt21:16:381
102582391,34sleep110272-21ssh21:28:351
308623813,6sleep03-21ksoftirqd/021:20:330
1650699389,5cyclictest81rcu_preempt19:50:251
16506993832,4cyclictest137350irq/90-eth023:56:341
16506993832,4cyclictest137350irq/90-eth023:39:461
1650599383,21cyclictest5816-21uniq00:45:180
1650599382,18cyclictest357-21ssh23:32:080
274142371,3sleep027424-21ssh23:26:170
262202371,4sleep026234-21ssh22:18:440
1650699375,6cyclictest81rcu_preempt00:29:111
1650699375,25cyclictest40950irq/93-i91522:25:541
16506993715,9cyclictest22-21ksoftirqd/120:25:011
1650599371,24cyclictest3730-21ssh23:01:560
158662371,6sleep115875-21awk20:45:271
1650699369,5cyclictest0-21swapper/123:21:171
1650699369,23cyclictest0-21swapper/122:37:051
16506993631,3cyclictest137350irq/90-eth023:18:001
16506993625,9cyclictest137350irq/90-eth022:32:151
16506993620,4cyclictest22-21ksoftirqd/100:50:311
1650599361,13cyclictest30530-21sendmail_mailst21:15:300
1650599361,13cyclictest30530-21sendmail_mailst21:15:290
115482361,4sleep111557-21df_inode20:35:151
248112351,4sleep124002-21/usr/sbin/munin22:50:261
221482350,7sleep00-21swapper/023:20:370
1650699356,4cyclictest81rcu_preempt20:50:281
16506993529,4cyclictest137350irq/90-eth021:23:451
16506993520,5cyclictest22-21ksoftirqd/123:00:301
1650599358,5cyclictest81rcu_preempt22:21:050
88272341,4sleep18841-21irqstats22:00:241
246062340,5sleep00-21swapper/023:57:110
230222341,29sleep123058-21hddtemp_smartct21:05:181
1650699343,4cyclictest7856-21kworker/1:300:05:221
16506993415,5cyclictest22-21ksoftirqd/100:01:581
1650699341,4cyclictest22-21ksoftirqd/122:05:121
16506993411,6cyclictest22-21ksoftirqd/123:13:111
1650599349,12cyclictest31635-21ssh21:50:260
16505993419,8cyclictest9989-21iostat_ios20:30:200
266052331,27sleep126665-21iostat21:45:201
200432339,20sleep120068-21vmstat20:55:301
1650699335,16cyclictest0-21swapper/122:48:301
16506993328,3cyclictest137350irq/90-eth022:21:221
16506993327,4cyclictest137350irq/90-eth022:15:411
1650699331,8cyclictest32021-21sensors_temp21:50:321
16506993317,12cyclictest8653-21fgrep23:40:271
16506993313,10cyclictest22-21ksoftirqd/100:47:071
1650599339,20cyclictest42032chrt22:29:410
1650599336,4cyclictest81rcu_preempt23:06:120
1650599334,4cyclictest81rcu_preempt22:50:010
1650599334,4cyclictest81rcu_preempt00:36:040
16505993319,7cyclictest18961-21python22:10:300
1650599331,8cyclictest26583-21ssh21:11:180
1650599331,20cyclictest2788-21aten_r4power_cu23:35:160
16505993311,5cyclictest0-21swapper/000:55:010
1650599331,13cyclictest18344-21cut23:50:300
98942321,4sleep19978-21awk20:30:181
326252321,26sleep132629-21diskstats20:05:171
1650699329,4cyclictest40950irq/93-i91522:58:321
16506993226,4cyclictest22-21ksoftirqd/119:30:011
16506993226,4cyclictest137350irq/90-eth000:41:201
16506993215,4cyclictest81rcu_preempt23:30:161
1650699321,4cyclictest22-21ksoftirqd/121:35:161
16506993214,5cyclictest22-21ksoftirqd/100:10:551
16506993214,15cyclictest22-21ksoftirqd/121:45:011
16506993213,6cyclictest130452sh23:45:191
16506993211,9cyclictest22-21ksoftirqd/122:10:191
1650599326,5cyclictest81rcu_preempt00:20:250
1650599325,4cyclictest81rcu_preempt23:10:220
1650599325,4cyclictest81rcu_preempt00:14:110
1650599324,4cyclictest81rcu_preempt22:45:270
1650599323,6cyclictest81rcu_preempt21:55:150
1650599323,6cyclictest81rcu_preempt21:55:150
1650599321,20cyclictest2285-21ssh00:40:500
1650599321,15cyclictest20308-21timerwakeupswit19:30:280
102212321,3sleep00-21swapper/022:35:220
39222310,4sleep13899-21ssh21:55:221
39222310,4sleep13899-21ssh21:55:221
262982310,7sleep00-21swapper/021:45:130
16506993123,5cyclictest13987-21ntp_offset20:40:261
16506993112,6cyclictest22-21ksoftirqd/122:41:081
16506993110,5cyclictest81rcu_preempt20:25:211
1650599319,10cyclictest32443-21missed_timers00:05:220
1650599317,5cyclictest81rcu_preempt23:15:270
1650599317,5cyclictest12230-21irqstats21:30:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional