You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 11:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot4s (updated Sat Jul 12, 2025 00:59:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
161622622575,32sleep00-21swapper/019:22:030
162462507419,73sleep10-21swapper/119:22:591
1650599208178,6cyclictest18377-21munin-run19:30:000
894021160,2sleep18939-21aten_r4power_cu00:15:121
161622960,4sleep00-21swapper/022:08:050
143052940,9sleep11650699cyclictest00:20:171
102862940,4sleep016503-21cyclictest22:01:330
60012790,5sleep1201rcuc/123:05:011
173722640,5sleep117335-21ssh23:50:141
296402620,9sleep01650599cyclictest19:55:320
265312560,3sleep041ktimersoftd/019:50:010
187932450,8sleep01650599cyclictest21:36:580
1650699456,6cyclictest22-21ksoftirqd/100:30:321
312742411,36sleep131287-21sh00:37:531
1650699402,4cyclictest81rcu_preempt21:30:291
1650699397,4cyclictest81rcu_preempt21:16:381
1650699397,4cyclictest81rcu_preempt21:16:381
102582391,34sleep110272-21ssh21:28:351
308623813,6sleep03-21ksoftirqd/021:20:330
1650699389,5cyclictest81rcu_preempt19:50:251
16506993832,4cyclictest137350irq/90-eth023:56:341
16506993832,4cyclictest137350irq/90-eth023:39:461
1650599383,21cyclictest5816-21uniq00:45:180
1650599382,18cyclictest357-21ssh23:32:080
274142371,3sleep027424-21ssh23:26:170
262202371,4sleep026234-21ssh22:18:440
1650699375,6cyclictest81rcu_preempt00:29:111
1650699375,25cyclictest40950irq/93-i91522:25:541
16506993715,9cyclictest22-21ksoftirqd/120:25:011
1650599371,24cyclictest3730-21ssh23:01:560
158662371,6sleep115875-21awk20:45:271
1650699369,5cyclictest0-21swapper/123:21:171
1650699369,23cyclictest0-21swapper/122:37:051
16506993631,3cyclictest137350irq/90-eth023:18:001
16506993625,9cyclictest137350irq/90-eth022:32:151
16506993620,4cyclictest22-21ksoftirqd/100:50:311
1650599361,13cyclictest30530-21sendmail_mailst21:15:300
1650599361,13cyclictest30530-21sendmail_mailst21:15:290
115482361,4sleep111557-21df_inode20:35:151
248112351,4sleep124002-21/usr/sbin/munin22:50:261
221482350,7sleep00-21swapper/023:20:370
1650699356,4cyclictest81rcu_preempt20:50:281
16506993529,4cyclictest137350irq/90-eth021:23:451
16506993520,5cyclictest22-21ksoftirqd/123:00:301
1650599358,5cyclictest81rcu_preempt22:21:050
88272341,4sleep18841-21irqstats22:00:241
246062340,5sleep00-21swapper/023:57:110
230222341,29sleep123058-21hddtemp_smartct21:05:181
1650699343,4cyclictest7856-21kworker/1:300:05:221
16506993415,5cyclictest22-21ksoftirqd/100:01:581
1650699341,4cyclictest22-21ksoftirqd/122:05:121
16506993411,6cyclictest22-21ksoftirqd/123:13:111
1650599349,12cyclictest31635-21ssh21:50:260
16505993419,8cyclictest9989-21iostat_ios20:30:200
266052331,27sleep126665-21iostat21:45:201
200432339,20sleep120068-21vmstat20:55:301
1650699335,16cyclictest0-21swapper/122:48:301
16506993328,3cyclictest137350irq/90-eth022:21:221
16506993327,4cyclictest137350irq/90-eth022:15:411
1650699331,8cyclictest32021-21sensors_temp21:50:321
16506993317,12cyclictest8653-21fgrep23:40:271
16506993313,10cyclictest22-21ksoftirqd/100:47:071
1650599339,20cyclictest42032chrt22:29:410
1650599336,4cyclictest81rcu_preempt23:06:120
1650599334,4cyclictest81rcu_preempt22:50:010
1650599334,4cyclictest81rcu_preempt00:36:040
16505993319,7cyclictest18961-21python22:10:300
1650599331,8cyclictest26583-21ssh21:11:180
1650599331,20cyclictest2788-21aten_r4power_cu23:35:160
16505993311,5cyclictest0-21swapper/000:55:010
1650599331,13cyclictest18344-21cut23:50:300
98942321,4sleep19978-21awk20:30:181
326252321,26sleep132629-21diskstats20:05:171
1650699329,4cyclictest40950irq/93-i91522:58:321
16506993226,4cyclictest22-21ksoftirqd/119:30:011
16506993226,4cyclictest137350irq/90-eth000:41:201
16506993215,4cyclictest81rcu_preempt23:30:161
1650699321,4cyclictest22-21ksoftirqd/121:35:161
16506993214,5cyclictest22-21ksoftirqd/100:10:551
16506993214,15cyclictest22-21ksoftirqd/121:45:011
16506993213,6cyclictest130452sh23:45:191
16506993211,9cyclictest22-21ksoftirqd/122:10:191
1650599326,5cyclictest81rcu_preempt00:20:250
1650599325,4cyclictest81rcu_preempt23:10:220
1650599325,4cyclictest81rcu_preempt00:14:110
1650599324,4cyclictest81rcu_preempt22:45:270
1650599323,6cyclictest81rcu_preempt21:55:150
1650599323,6cyclictest81rcu_preempt21:55:150
1650599321,20cyclictest2285-21ssh00:40:500
1650599321,15cyclictest20308-21timerwakeupswit19:30:280
102212321,3sleep00-21swapper/022:35:220
39222310,4sleep13899-21ssh21:55:221
39222310,4sleep13899-21ssh21:55:221
262982310,7sleep00-21swapper/021:45:130
16506993123,5cyclictest13987-21ntp_offset20:40:261
16506993112,6cyclictest22-21ksoftirqd/122:41:081
16506993110,5cyclictest81rcu_preempt20:25:211
1650599319,10cyclictest32443-21missed_timers00:05:220
1650599317,5cyclictest81rcu_preempt23:15:270
1650599317,5cyclictest12230-21irqstats21:30:210
1650599314,4cyclictest81rcu_preempt00:15:260
1650599313,15cyclictest0-21swapper/020:00:460
16505993112,6cyclictest4755-21python20:15:280
16505993112,6cyclictest25506-21ssh21:44:420
16505993111,8cyclictest22248-21/usr/sbin/munin19:35:300
16505993111,4cyclictest5491-21sshd22:30:230
295822301,4sleep00-21swapper/000:02:250
220612301,25sleep022072-21ssh00:28:300
1650699305,4cyclictest81rcu_preempt21:00:141
1650699304,18cyclictest24898-21diskstats21:10:151
1650699303,4cyclictest81rcu_preempt23:25:041
16506993022,5cyclictest24019-21timerandwakeup19:40:331
1650699301,4cyclictest81rcu_preempt19:49:361
16506993014,4cyclictest22-21ksoftirqd/119:55:251
1650699301,16cyclictest4230-21hddtemp_smartct20:15:161
1650699301,12cyclictest22-21ksoftirqd/119:35:301
1650599308,11cyclictest3-21ksoftirqd/021:29:280
16505993022,5cyclictest30271-21ssh22:55:320
1650599302,14cyclictest8725-21python23:40:270
1650599301,4cyclictest81rcu_preempt20:57:470
1650599301,4cyclictest81rcu_preempt00:31:400
16505993012,5cyclictest13766-21/usr/sbin/munin20:40:220
16505993012,5cyclictest0-21swapper/020:05:150
16505993011,6cyclictest2666-21ps20:10:260
1650599300,4cyclictest81rcu_preempt19:41:550
31512291,5sleep10-21swapper/120:10:361
266762291,5sleep00-21swapper/019:50:060
242522291,5sleep00-21swapper/021:09:470
168022291,5sleep00-21swapper/023:49:520
16506992914,12cyclictest22-21ksoftirqd/119:30:171
1650599291,17cyclictest6204-21gdbus22:41:520
1650599291,13cyclictest17155-21cstates20:50:130
70512281,5sleep00-21swapper/020:21:540
219912281,5sleep00-21swapper/021:00:480
16505992810,6cyclictest15726-21memory20:45:230
16505992810,6cyclictest12908-21taskset20:39:170
16506992712,11cyclictest22-21ksoftirqd/120:00:191
16505992711,13cyclictest9230-21cron20:30:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional