You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 14:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack9slot4s (updated Sun Nov 16, 2025 12:59:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
317239912231193,25cyclictest31682-21latency_hist07:25:011
317239911861166,16cyclictest3821-21apt-get07:35:001
317229911751151,21cyclictest0-21swapper/007:25:050
316092752707,30sleep00-21swapper/007:24:290
3172399226218,6cyclictest0-21swapper/107:25:051
3248621140,10sleep13172399cyclictest12:23:171
317229910699,4cyclictest41ktimersoftd/007:35:010
71802860,9sleep13172399cyclictest10:15:341
308512840,8sleep03172299cyclictest08:45:140
308512840,8sleep03172299cyclictest08:45:140
286902810,3sleep128689-21ssh11:45:251
3172299594,7cyclictest29446-21diskstats12:20:150
3172299543,24cyclictest5755-21aten_r4power_cu10:15:140
3172299541,15cyclictest12400-21mailstats12:35:310
59232530,7sleep03172299cyclictest11:55:240
3172299521,4cyclictest4589-21ssh11:20:240
15832520,8sleep03172299cyclictest11:50:340
3172299507,12cyclictest25677-21sh10:01:230
23552470,4sleep1201rcuc/111:18:411
283102410,7sleep03172299cyclictest12:52:470
124042400,2sleep00-21swapper/009:15:000
31723993819,6cyclictest17946-21tune2fs09:20:191
3172299381,13cyclictest0-21swapper/010:22:090
133972371,4sleep013411-21ssh09:49:350
3172399362,19cyclictest2350-21memory12:25:271
3172299361,22cyclictest31886-21uname11:15:230
113782361,31sleep111392-21ssh11:27:391
3172399357,24cyclictest0-21swapper/111:40:551
31723993525,6cyclictest12630-21sh09:48:261
3172399351,15cyclictest279952sh12:18:471
3172299351,7cyclictest8364-21ssh12:31:130
63612340,4sleep0111rcuc/010:49:190
3172399349,5cyclictest81rcu_preempt12:30:271
3172399348,22cyclictest0-21swapper/111:30:441
31723993417,13cyclictest22-21ksoftirqd/110:04:051
3172299344,4cyclictest81rcu_preempt09:34:540
3172299341,7cyclictest18739-21ssh11:01:170
3172299341,10cyclictest81rcu_preempt09:41:150
3172299341,10cyclictest81rcu_preempt08:02:490
260422340,3sleep0111rcuc/011:09:580
3172399335,5cyclictest81rcu_preempt10:38:201
3172399332,4cyclictest81rcu_preempt08:35:251
3172299337,4cyclictest81rcu_preempt10:35:230
3172299336,4cyclictest81rcu_preempt12:03:410
3172299331,6cyclictest0-21swapper/012:15:250
31722993312,17cyclictest30889-21ssh11:48:000
3172299331,16cyclictest0-21swapper/011:35:280
31722993310,8cyclictest31720-21cyclictest10:13:140
21702331,27sleep12178-21diskstats08:55:161
123772330,2sleep112375-21fschecks_count10:55:191
116432331,4sleep17482-21diskmemload12:01:011
3172399327,13cyclictest40950irq/93-i91511:00:411
3172399326,5cyclictest81rcu_preempt10:13:491
3172399326,4cyclictest81rcu_preempt09:42:451
3172399325,5cyclictest81rcu_preempt11:37:521
3172399325,5cyclictest81rcu_preempt09:19:041
3172399324,4cyclictest81rcu_preempt09:27:061
3172399323,16cyclictest5591-21udevd07:37:101
3172399322,7cyclictest7482-21diskmemload10:26:061
3172399322,16cyclictest0-21swapper/109:55:381
31723993217,12cyclictest22-21ksoftirqd/112:50:271
3172399321,4cyclictest81rcu_preempt10:43:401
3172299327,13cyclictest17008-21chrt08:07:560
3172299326,5cyclictest81rcu_preempt12:14:350
31722993215,5cyclictest29915-21aten_r4power_en10:40:160
3172299321,19cyclictest24466-21ssh09:26:360
31722993211,6cyclictest14323-21diskstats09:50:160
3172299321,10cyclictest0-21swapper/012:47:190
31722993210,5cyclictest5600-21udevd07:37:090
185312321,5sleep10-21swapper/112:09:051
31723993119,9cyclictest22-21ksoftirqd/110:45:301
3172399311,8cyclictest32220-21ssh10:08:541
31723993117,8cyclictest7114-21/usr/sbin/munin10:50:341
3172399311,4cyclictest81rcu_preempt09:30:081
3172299315,4cyclictest81rcu_preempt11:42:220
3172299311,4cyclictest81rcu_preempt11:13:050
3172299311,19cyclictest19009-21taskset08:14:130
3172299311,14cyclictest8156-21ps10:50:290
241942311,5sleep10-21swapper/110:34:161
203042311,3sleep020475-21expr10:30:160
19412311,5sleep10-21swapper/111:51:071
191042311,4sleep119117-21sh12:42:491
3172399306,4cyclictest81rcu_preempt10:22:321
3172399304,10cyclictest23008-21users11:05:321
3172399302,6cyclictest0-21swapper/112:46:031
3172399302,4cyclictest81rcu_preempt11:12:461
31723993021,6cyclictest12050-21cat12:35:251
3172399301,5cyclictest81rcu_preempt08:12:421
3172299303,15cyclictest0-21swapper/010:05:190
3172299302,13cyclictest16088-21irqstats10:25:240
3172299302,13cyclictest0-21swapper/012:09:300
31722993020,6cyclictest11889-21ssh10:55:090
3172299301,4cyclictest81rcu_preempt09:56:270
3172299301,4cyclictest81rcu_preempt09:01:160
31722993012,6cyclictest6582-21/usr/sbin/munin09:05:280
3172299301,21cyclictest0-21swapper/008:44:080
3172299301,21cyclictest0-21swapper/008:20:500
3172299301,17cyclictest0-21swapper/012:41:020
31722993011,5cyclictest2172-21ssh09:37:210
247262301,5sleep00-21swapper/008:29:520
103402300,4sleep0111rcuc/007:50:190
95902291,5sleep00-21swapper/007:49:070
3172399295,20cyclictest0-21swapper/109:37:001
3172399292,6cyclictest0-21swapper/109:52:461
3172399292,22cyclictest8313-21hddtemp_smartct09:10:191
3172399292,14cyclictest0-21swapper/112:10:151
3172399291,4cyclictest81rcu_preempt11:56:311
3172399291,4cyclictest81rcu_preempt07:55:261
3172299296,4cyclictest81rcu_preempt12:26:580
3172299293,4cyclictest81rcu_preempt07:59:160
3172299293,15cyclictest0-21swapper/011:26:510
3172299292,13cyclictest7167-21awk07:40:310
3172299291,17cyclictest0-21swapper/008:50:250
3172299291,15cyclictest20272-21/usr/sbin/munin08:15:290
217322291,5sleep00-21swapper/009:24:400
71102281,5sleep10-21swapper/111:23:261
54702281,4sleep10-21swapper/109:03:491
3172399284,4cyclictest81rcu_preempt08:23:171
3172399284,4cyclictest81rcu_preempt08:16:511
31723992817,8cyclictest22-21ksoftirqd/107:45:171
3172399281,4cyclictest81rcu_preempt08:00:041
3172299289,6cyclictest25970-21wc08:30:310
3172299286,18cyclictest7426-21kworker/0:111:30:150
3172299282,13cyclictest2797-21ntp_states08:55:260
31722992810,5cyclictest13796-21python09:15:300
302012281,5sleep10-21swapper/108:43:021
74402271,5sleep10-21swapper/109:09:441
74232271,5sleep10-21swapper/107:41:061
3172399276,10cyclictest31006-21fschecks_time08:45:171
3172399276,10cyclictest31006-21fschecks_time08:45:171
16562271,5sleep10-21swapper/108:54:341
3172399265,4cyclictest81rcu_preempt08:29:331
31723992616,6cyclictest26019-21ls08:30:321
3172299261,12cyclictest27383-21iostat_ios08:35:220
169772261,5sleep10-21swapper/108:07:391
114652261,4sleep10-21swapper/107:54:071
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional