You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot4s (updated Tue May 13, 2025 00:59:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
115239912701240,26cyclictest13388-21latency_hist19:30:001
1152299634616,14cyclictest3-21ksoftirqd/019:30:000
113262606517,73sleep00-21swapper/019:23:290
110602548467,66sleep10-21swapper/119:20:421
145302430,7sleep00-21swapper/019:30:170
11522993820,6cyclictest3-21ksoftirqd/023:31:020
222272371,31sleep122317-21sed00:10:001
1152299374,4cyclictest81rcu_preempt20:40:320
1152399361,15cyclictest0-21swapper/123:31:021
66062350,5sleep16752-21fw_packets20:35:201
25672341,10sleep00-21swapper/000:40:530
1152299344,13cyclictest26868-21missed_timers00:20:260
1152299344,13cyclictest26868-21missed_timers00:20:250
326252331,5sleep132627-21ntp_states00:35:301
120142330,5sleep1201rcuc/120:50:151
168902327,5sleep00-21swapper/022:28:570
1152399324,4cyclictest81rcu_preempt20:15:551
1152399322,4cyclictest81rcu_preempt20:46:091
1152299329,7cyclictest3-21ksoftirqd/000:45:150
11522993211,7cyclictest6848-21munin-run00:55:000
1152399316,4cyclictest81rcu_preempt22:50:511
1152399316,4cyclictest81rcu_preempt19:41:041
1152399315,4cyclictest81rcu_preempt23:42:121
1152399315,4cyclictest81rcu_preempt22:17:031
1152399314,4cyclictest81rcu_preempt21:30:581
1152399311,17cyclictest25575-21aten_r4power_vo20:00:131
1152299315,12cyclictest24243-21ls21:20:310
1152299312,24cyclictest14024-21python23:45:290
11522993119,6cyclictest10653-21python22:10:300
1152299311,4cyclictest81rcu_preempt22:46:450
1152299311,4cyclictest81rcu_preempt22:34:320
1152299311,4cyclictest81rcu_preempt21:53:040
11522993112,6cyclictest21217-21df_inode22:40:160
108062311,5sleep00-21swapper/023:38:360
200732301,5sleep10-21swapper/100:02:031
1152399306,4cyclictest81rcu_preempt23:21:111
1152399306,4cyclictest81rcu_preempt22:26:221
1152399305,4cyclictest81rcu_preempt21:29:561
1152399305,4cyclictest81rcu_preempt21:08:481
1152399305,4cyclictest81rcu_preempt19:58:521
1152399305,4cyclictest81rcu_preempt19:35:021
1152399304,4cyclictest81rcu_preempt23:39:331
1152399304,4cyclictest81rcu_preempt23:04:031
1152399304,4cyclictest81rcu_preempt22:13:261
1152399304,19cyclictest17052-21diskstats23:55:161
11523993018,7cyclictest13305-21diskstats23:45:161
1152399301,11cyclictest145272fschecks_count19:30:171
1152299307,11cyclictest0-21swapper/023:15:010
1152299302,14cyclictest3020-21runrttasks00:29:130
11522993015,4cyclictest3-21ksoftirqd/023:56:230
1152299301,4cyclictest81rcu_preempt19:54:170
11522993011,7cyclictest29430-21grep20:10:150
1152299301,15cyclictest7969-21cron20:40:000
1152299301,15cyclictest32595-21/usr/sbin/munin00:35:280
92522291,5sleep00-21swapper/022:07:470
68092291,5sleep00-21swapper/023:25:590
60602291,5sleep00-21swapper/020:34:420
53622291,5sleep00-21swapper/021:56:300
263382291,5sleep00-21swapper/021:25:310
257852291,5sleep00-21swapper/000:17:590
219502291,5sleep00-21swapper/000:07:040
211762291,5sleep00-21swapper/019:46:150
208002291,5sleep00-21swapper/021:11:290
206272291,4sleep120654-21ntp_states19:45:241
201052291,5sleep00-21swapper/000:02:240
172612291,5sleep00-21swapper/021:03:430
151102291,5sleep00-21swapper/022:24:550
1152399295,4cyclictest81rcu_preempt20:09:421
1152399295,3cyclictest81rcu_preempt19:54:391
1152399294,4cyclictest81rcu_preempt21:21:171
1152399294,4cyclictest81rcu_preempt20:12:341
1152399294,4cyclictest81rcu_preempt00:52:571
1152399293,4cyclictest81rcu_preempt22:40:091
1152399293,4cyclictest81rcu_preempt21:43:191
1152399293,4cyclictest81rcu_preempt00:45:471
1152399292,4cyclictest81rcu_preempt23:09:381
1152399291,10cyclictest29671-21runrttasks00:29:121
1152299292,15cyclictest0-21swapper/020:07:550
1152299292,14cyclictest3470-21/usr/sbin/munin20:25:300
1152299292,14cyclictest25628-21/usr/sbin/munin22:50:280
1152299292,14cyclictest2132-21/usr/sbin/munin23:15:230
11522992918,7cyclictest0-21swapper/023:50:190
11522992914,5cyclictest3-21ksoftirqd/020:00:100
11522992912,5cyclictest18388-21ntp_states21:05:270
11522992911,5cyclictest30743-21ntp_states00:30:270
1152299291,15cyclictest140882sleep020:55:150
1152299291,12cyclictest20577-21taskset22:38:160
11522992910,5cyclictest30811-21chrt23:05:190
86592281,4sleep10-21swapper/120:40:191
5602281,3sleep1592-21proc_pri23:10:281
35362281,5sleep10-21swapper/121:52:011
324672281,5sleep00-21swapper/020:16:380
279802281,5sleep10-21swapper/122:57:081
279632281,5sleep00-21swapper/022:56:590
251212281,5sleep00-21swapper/019:58:200
240172281,5sleep10-21swapper/100:14:071
229612281,5sleep00-21swapper/021:19:280
120552280,4sleep0111rcuc/022:15:200
1152399285,4cyclictest81rcu_preempt22:05:581
1152399285,4cyclictest81rcu_preempt21:03:141
1152399284,6cyclictest81rcu_preempt22:30:351
1152399284,3cyclictest81rcu_preempt22:48:321
1152399283,4cyclictest81rcu_preempt22:02:191
1152399281,19cyclictest0-21swapper/100:16:491
1152399281,11cyclictest22191-21ps21:15:291
1152399281,10cyclictest0-21swapper/121:45:141
1152299283,19cyclictest13631-21munin-run20:54:590
1152299282,13cyclictest32695-21wget21:45:130
1152299282,13cyclictest29420-21iostat_ios21:35:230
1152299282,13cyclictest29277-21ntp_states23:00:300
1152299282,13cyclictest11435-21/usr/sbin/munin23:40:170
11522992814,8cyclictest31010-21diskstats21:40:160
1152299281,14cyclictest954-21fschecks_time20:20:170
1152299281,13cyclictest16633-21/usr/sbin/munin19:35:190
11522992810,5cyclictest4282-21ls23:20:230
1152399278,5cyclictest6207-21cat23:25:251
1152399275,4cyclictest81rcu_preempt21:14:021
1152399274,14cyclictest0-21swapper/120:21:221
1152399273,4cyclictest81rcu_preempt00:43:321
1152399273,14cyclictest0-21swapper/120:31:241
11523992718,6cyclictest22-21ksoftirqd/120:26:321
1152399271,4cyclictest81rcu_preempt22:35:351
1152399271,21cyclictest1613-21/usr/sbin/munin23:15:321
1152299274,17cyclictest6574-21memory22:00:220
1152299272,12cyclictest19217-21users19:40:300
113482271,4sleep00-21swapper/020:45:570
302622261,5sleep10-21swapper/121:37:151
151662261,4sleep10-21swapper/120:56:371
147872261,5sleep10-21swapper/122:21:331
1152399263,17cyclictest26764-21iostat_ios00:20:231
1152399263,17cyclictest26764-21iostat_ios00:20:231
1152399262,4cyclictest81rcu_preempt23:53:371
1152399261,5cyclictest81rcu_preempt00:35:011
1152399261,4cyclictest81rcu_preempt21:56:321
1152299263,17cyclictest22995-21iostat_ios00:10:230
1152299251,16cyclictest0-21swapper/021:31:580
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional