You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 12:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot4s (updated Thu Nov 06, 2025 00:59:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
146559912351226,6cyclictest0-21swapper/119:25:081
146549911871154,23cyclictest0-21swapper/019:25:060
146559911771158,16cyclictest0-21swapper/119:35:141
142692561517,29sleep00-21swapper/019:21:380
145222492446,31sleep10-21swapper/119:24:171
1465499166118,44cyclictest19629-21fschecks_time19:35:150
305692990,7sleep00-21swapper/020:05:120
192642970,7sleep01465499cyclictest23:54:080
125322960,6sleep012536-21unixbench_multi20:40:320
3872880,5sleep027743-21kworker/0:022:26:540
91332840,3sleep1211ktimersoftd/122:35:301
235602800,7sleep00-21swapper/022:17:250
48382790,3sleep00-21swapper/022:31:110
206282640,5sleep120620-21ssh21:40:301
276082511,32sleep027641-21hwlatdetect22:55:220
86862470,7sleep01465499cyclictest22:01:490
210492470,6sleep03-21ksoftirqd/021:40:470
35532450,8sleep01465499cyclictest23:04:170
1465499431,12cyclictest10642-21egrep23:45:130
14655994221,5cyclictest22-21ksoftirqd/123:31:461
1465499411,13cyclictest7317-21ssh00:49:010
97382380,8sleep01465499cyclictest23:44:160
309322381,4sleep130948-21ssh21:51:111
225272381,33sleep022539-21diskstats22:50:150
14655993817,8cyclictest22-21ksoftirqd/122:57:021
14655993812,11cyclictest3020-21runrttasks23:15:081
1465499385,15cyclictest0-21swapper/000:22:410
14654993812,14cyclictest24259-21runrttasks19:45:370
11952381,5sleep11205-21ssh22:28:051
1465599376,4cyclictest81rcu_preempt22:54:441
14655993720,8cyclictest22-21ksoftirqd/100:47:451
14654993710,14cyclictest10383-21awk20:35:260
37702361,5sleep122487-21diskmemload23:04:381
14655993618,14cyclictest22-21ksoftirqd/123:05:211
14655993616,8cyclictest22-21ksoftirqd/123:45:241
1465499362,11cyclictest81rcu_preempt19:31:190
1465499362,11cyclictest81rcu_preempt19:31:190
14654993611,11cyclictest0-21swapper/000:03:510
55962350,2sleep05594-21iostat21:25:200
247572350,8sleep00-21swapper/021:11:160
14655993525,6cyclictest4502-21ssh23:38:411
1465499355,4cyclictest81rcu_preempt23:07:100
1465499355,4cyclictest81rcu_preempt21:18:570
1465499353,10cyclictest81rcu_preempt22:06:510
1465499352,6cyclictest17429-21ssh00:25:210
14654993513,8cyclictest22501-21ntpdc00:30:260
44162341,4sleep04501-21/usr/sbin/munin20:20:210
1465599344,4cyclictest81rcu_preempt21:13:161
1465499349,11cyclictest24872-21if_eth021:45:200
1465499348,10cyclictest0-21swapper/000:36:060
1465499348,10cyclictest0-21swapper/000:36:060
1465499344,4cyclictest81rcu_preempt22:44:590
14654993418,5cyclictest3-21ksoftirqd/020:46:270
83632330,7sleep01465499cyclictest22:35:190
281672331,5sleep10-21swapper/122:22:061
1465599339,10cyclictest3020-21runrttasks22:19:131
1465599337,22cyclictest0-21swapper/123:42:151
1465599332,8cyclictest21299-21latency_hist00:30:021
1465599332,4cyclictest81rcu_preempt00:33:271
1465599332,27cyclictest0-21swapper/120:05:161
14655993317,12cyclictest22-21ksoftirqd/100:00:261
1465599331,30cyclictest40950irq/93-i91521:37:491
1465499339,11cyclictest3020-21runrttasks19:56:340
1465499335,4cyclictest81rcu_preempt00:40:250
1465499333,4cyclictest81rcu_preempt23:17:570
1465499332,14cyclictest9178-21ntp_states00:50:270
1465499331,4cyclictest81rcu_preempt21:51:170
236042321,26sleep023618-21ssh23:58:270
180652321,5sleep00-21swapper/022:11:270
1465599329,5cyclictest81rcu_preempt23:14:061
1465599326,5cyclictest81rcu_preempt23:25:281
1465599326,4cyclictest81rcu_preempt23:21:361
14655993224,5cyclictest22-21ksoftirqd/122:14:591
1465599322,26cyclictest0-21swapper/121:20:541
14655993217,11cyclictest22-21ksoftirqd/119:56:581
14655993214,6cyclictest22-21ksoftirqd/121:27:571
1465599321,16cyclictest7634-21perl00:15:191
1465499326,4cyclictest81rcu_preempt00:13:460
1465499322,4cyclictest81rcu_preempt20:17:120
1465499321,4cyclictest81rcu_preempt23:35:450
1465499321,4cyclictest81rcu_preempt23:29:320
1465499321,4cyclictest81rcu_preempt21:55:070
1465499321,4cyclictest81rcu_preempt21:32:230
1465599315,23cyclictest0-21swapper/100:38:551
1465599315,23cyclictest0-21swapper/100:38:541
1465599312,8cyclictest8547-21df_inode00:50:161
14655993122,5cyclictest6220-21seq22:33:121
14655993117,6cyclictest22-21ksoftirqd/100:42:231
1465499313,4cyclictest81rcu_preempt00:07:270
1465499312,14cyclictest29088-21irqstats23:30:200
1465499311,4cyclictest81rcu_preempt23:22:280
14654993112,5cyclictest0-21swapper/021:36:330
14654993111,7cyclictest944-21ntp_states21:20:250
1465499311,15cyclictest0-21swapper/000:18:220
212262301,5sleep10-21swapper/122:48:591
203672301,5sleep00-21swapper/021:02:370
203672301,5sleep00-21swapper/021:02:360
1465599308,18cyclictest0-21swapper/100:10:011
1465599303,23cyclictest0-21swapper/100:20:141
14655993017,4cyclictest22-21ksoftirqd/122:06:511
1465599301,6cyclictest81rcu_preempt21:47:151
14655993012,14cyclictest3020-21runrttasks00:05:241
1465599300,5cyclictest81rcu_preempt20:11:461
1465499302,15cyclictest10104-21ntp_states23:10:280
1465499301,4cyclictest81rcu_preempt22:20:420
1465499301,27cyclictest0-21swapper/019:50:440
1465499301,15cyclictest5693-21/usr/sbin/munin20:25:140
1465499301,12cyclictest22172-21timerandwakeup19:40:260
14654993010,4cyclictest0-21swapper/022:47:310
47592291,5sleep10-21swapper/121:58:081
256792292,22sleep125683-21ntp_states19:50:281
224132291,5sleep10-21swapper/119:40:591
203912291,5sleep10-21swapper/121:02:511
203912291,5sleep10-21swapper/121:02:511
1465599295,3cyclictest81rcu_preempt20:41:191
1465599294,18cyclictest28310-21iostat_ios21:15:221
14655992922,4cyclictest40950irq/93-i91521:30:281
1465599291,4cyclictest81rcu_preempt23:54:271
1465599291,23cyclictest20487-21diskstats23:55:181
1465599291,15cyclictest9784-21fschecks_count20:35:161
1465599291,11cyclictest82372wc20:30:221
1465499291,15cyclictest29683-21switchtime20:00:300
243742281,23sleep10-21swapper/119:46:511
185952281,5sleep10-21swapper/119:30:321
185952281,5sleep10-21swapper/119:30:321
1465599285,4cyclictest40950irq/93-i91522:40:531
1465599283,4cyclictest81rcu_preempt20:55:311
14655992816,8cyclictest22-21ksoftirqd/122:00:071
1465499281,4cyclictest81rcu_preempt20:13:070
33082271,5sleep10-21swapper/120:16:441
299972271,5sleep10-21swapper/120:01:331
165712271,5sleep10-21swapper/120:52:251
1465599279,4cyclictest14376-21ls20:45:291
1465499272,13cyclictest6204-21gdbus20:57:080
14654992716,6cyclictest22022-21grep21:05:320
1465599264,14cyclictest0-21swapper/120:21:071
1465599261,4cyclictest81rcu_preempt20:28:081
1465499267,6cyclictest16367-21/usr/sbin/munin20:50:300
1465599251,13cyclictest20749-21seq21:05:011
1465499251,8cyclictest6204-21gdbus20:30:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional