You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 11:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot4s (updated Wed Nov 26, 2025 00:59:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
56189912511222,25cyclictest5788-21aten_r4power_cu19:25:131
56189912261188,33cyclictest16279-21perl19:50:171
56179911631132,21cyclictest0-21swapper/019:25:110
51462694649,29sleep10-21swapper/119:20:421
54432548503,29sleep00-21swapper/019:23:480
561799181160,6cyclictest0-21swapper/019:50:170
2132921060,9sleep0561799cyclictest00:05:170
1226621050,2sleep00-21swapper/023:22:010
232072900,4sleep0561799cyclictest20:10:000
233652840,5sleep123367-21fschecks_time23:00:171
142492800,3sleep0111rcuc/023:57:510
561799481,13cyclictest19082-21ssh00:35:490
263522450,5sleep0111rcuc/022:30:110
5617994226,5cyclictest3-21ksoftirqd/022:08:240
5617993922,6cyclictest3-21ksoftirqd/023:37:510
5617993922,6cyclictest3-21ksoftirqd/022:40:330
5617993922,6cyclictest3-21ksoftirqd/020:17:410
5617993915,7cyclictest3-21ksoftirqd/022:47:060
5617993823,5cyclictest3-21ksoftirqd/021:25:020
5617993823,5cyclictest3-21ksoftirqd/000:48:140
5617993822,6cyclictest3-21ksoftirqd/019:42:210
5617993821,6cyclictest3-21ksoftirqd/000:51:250
5617993821,6cyclictest3-21ksoftirqd/000:15:370
241702381,4sleep124182-21sh22:27:251
162542384,30sleep113467-21diskmemload22:52:521
67992371,4sleep16813-21ssh22:43:071
5617993721,6cyclictest3-21ksoftirqd/019:38:400
5617993720,6cyclictest3-21ksoftirqd/022:50:490
5617993720,6cyclictest3-21ksoftirqd/021:19:260
5617993720,6cyclictest3-21ksoftirqd/021:01:500
5617993720,5cyclictest3-21ksoftirqd/022:27:400
5617993719,6cyclictest3-21ksoftirqd/020:03:440
81862361,4sleep08243-21df20:55:140
561899364,5cyclictest81rcu_preempt22:05:551
561899361,15cyclictest286632ssh21:58:581
5617993620,6cyclictest3-21ksoftirqd/020:34:360
5617993620,5cyclictest3-21ksoftirqd/020:13:210
5617993619,6cyclictest3-21ksoftirqd/021:20:470
5617993520,5cyclictest3-21ksoftirqd/021:38:130
5617993519,5cyclictest3-21ksoftirqd/021:13:250
3642351,31sleep1378-21ssh22:35:591
303832350,5sleep130376-21ssh23:40:301
261702351,4sleep026276-21users21:55:310
561899346,5cyclictest81rcu_preempt21:35:411
5617993419,5cyclictest3-21ksoftirqd/022:22:090
5617993418,6cyclictest3-21ksoftirqd/020:48:270
5617993418,6cyclictest3-21ksoftirqd/000:44:550
5617993418,5cyclictest3-21ksoftirqd/022:00:100
5617993418,5cyclictest3-21ksoftirqd/019:55:280
5617993418,5cyclictest3-21ksoftirqd/000:24:020
5617993414,10cyclictest3-21ksoftirqd/020:54:040
561799341,21cyclictest8458-21iostat00:25:200
318992341,28sleep130883-21/usr/sbin/munin20:30:291
291632341,29sleep029284-21df20:25:130
58332331,4sleep113467-21diskmemload23:15:261
561899338,4cyclictest81rcu_preempt00:50:581
5618993321,7cyclictest15304-21irqstats23:25:221
561899331,4cyclictest81rcu_preempt23:48:211
561799331,8cyclictest3410-21ssh21:32:210
5617993318,5cyclictest3-21ksoftirqd/020:22:260
5617993318,5cyclictest3-21ksoftirqd/019:46:350
5617993313,5cyclictest2582-21fschecks_count20:40:160
234612330,4sleep041ktimersoftd/023:34:180
201952331,28sleep120278-21memory20:00:231
146962331,28sleep114719-21open_files21:10:271
561899322,25cyclictest0-21swapper/122:01:461
5618993221,6cyclictest28348-21ssh00:12:201
561899321,7cyclictest23354-21/usr/sbin/munin20:10:301
561899321,6cyclictest81rcu_preempt20:38:081
561899321,6cyclictest81rcu_preempt20:38:081
561899321,4cyclictest81rcu_preempt21:50:451
5618993213,15cyclictest22-21ksoftirqd/123:20:061
561899321,11cyclictest3020-21runrttasks00:41:351
561899321,10cyclictest11490-21df_inode23:55:141
5617993218,4cyclictest3-21ksoftirqd/023:15:060
5617993217,5cyclictest3-21ksoftirqd/021:49:410
561799321,17cyclictest18772-21ssh00:02:240
302672321,5sleep130277-21ssh21:26:251
261282321,4sleep126142-21ssh23:36:111
22992321,27sleep12307-21sh23:11:431
22882321,27sleep12302-21ssh00:19:371
159632321,4sleep113467-21diskmemload00:00:081
561899313,14cyclictest81rcu_preempt23:00:021
5618993112,5cyclictest26954-21latency_hist20:20:001
561899311,16cyclictest9232-21sh00:25:311
5618993110,8cyclictest0-21swapper/121:20:241
561799314,5cyclictest3020-21runrttasks23:14:540
5617993114,6cyclictest3-21ksoftirqd/000:32:480
561799311,25cyclictest13450-21cron21:10:000
561799311,18cyclictest352-21udevd23:00:020
561799311,11cyclictest0-21swapper/021:50:200
35192310,9sleep1561899cyclictest00:20:171
294902311,5sleep00-21swapper/023:06:050
101922311,4sleep110203-21ssh23:53:581
101922311,4sleep110203-21ssh23:53:581
96442301,5sleep00-21swapper/019:31:250
76152301,5sleep10-21swapper/120:53:101
561899308,18cyclictest0-21swapper/122:47:061
561899306,19cyclictest0-21swapper/120:27:161
561899302,25cyclictest0-21swapper/122:20:311
561899301,7cyclictest18125-21ssh00:35:221
561899301,6cyclictest23925-21runrttasks23:34:581
5618993014,5cyclictest40950irq/93-i91500:05:161
561799303,12cyclictest6658-21df23:50:120
561799303,12cyclictest6658-21df23:50:120
561799302,15cyclictest4374-21ssh23:47:310
561799301,8cyclictest18950-21ssh22:55:240
5617993016,4cyclictest3-21ksoftirqd/022:11:420
561799301,17cyclictest29529-21ssh00:14:030
5617993010,5cyclictest0-21swapper/021:42:160
17822301,5sleep00-21swapper/020:37:320
17822301,5sleep00-21swapper/020:37:320
561899298,17cyclictest0-21swapper/121:15:111
561899294,5cyclictest40950irq/93-i91520:20:221
561899293,5cyclictest0-21swapper/123:08:171
5618992919,6cyclictest26235-21/usr/sbin/munin22:30:271
5618992916,10cyclictest22-21ksoftirqd/121:40:561
561799294,13cyclictest0-21swapper/022:35:500
561799291,12cyclictest15148-21hddtemp_smartct23:25:180
35782290,4sleep1201rcuc/120:40:291
168602291,5sleep10-21swapper/121:45:511
561899289,15cyclictest0-21swapper/100:47:011
561899289,15cyclictest0-21swapper/100:31:211
561899288,16cyclictest0-21swapper/122:16:081
561899283,19cyclictest4794-21iostat_ios20:45:211
561899281,22cyclictest0-21swapper/121:00:201
5618992811,13cyclictest0-21swapper/122:10:311
561799282,14cyclictest0-21swapper/022:15:240
313862281,5sleep00-21swapper/023:41:460
93922271,5sleep10-21swapper/120:57:081
561899273,9cyclictest12887-21ntp_states19:40:251
561899273,3cyclictest81rcu_preempt19:31:271
561899273,10cyclictest0-21swapper/119:35:131
223012271,5sleep10-21swapper/120:05:261
193432271,5sleep10-21swapper/119:59:231
561899268,6cyclictest40950irq/93-i91519:45:251
561899261,11cyclictest895-21ssh21:30:151
130812261,6sleep10-21swapper/121:06:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional