You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 22:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack9slot4s (updated Tue Nov 25, 2025 12:59:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
164299912571235,19cyclictest0-21swapper/107:25:121
164289912171186,14cyclictest13781-21kworker/0:207:25:110
164299911071089,15cyclictest24487-21copy07:43:181
161112669623,31sleep00-21swapper/007:22:210
160252576530,31sleep10-21swapper/107:21:271
1642899165155,6cyclictest24496-21udevd07:43:170
2808021010,5sleep10-21swapper/109:46:011
255622860,4sleep00-21swapper/009:10:210
302602820,7sleep00-21swapper/010:22:060
16428997716,39cyclictest23710-21missed_timers10:15:250
214292720,4sleep00-21swapper/009:40:090
1642899573,17cyclictest22594-21df11:55:150
16428995710,5cyclictest0-21swapper/011:54:520
1642899564,7cyclictest27482-21diskstats12:00:170
16428995625,27cyclictest20069-21/usr/sbin/munin07:30:270
1642899562,14cyclictest20886-21/usr/sbin/munin11:20:170
1642899561,12cyclictest16692-21ssh10:08:260
1642899551,22cyclictest30434-21sed09:15:230
16428995511,9cyclictest10156-21diskstats10:35:160
1642899551,13cyclictest14443-21ssh09:32:140
16428995510,12cyclictest0-21swapper/012:50:410
1642899542,15cyclictest19509-21chrt12:25:200
16428995410,6cyclictest3681-21irqstats11:35:210
16428995311,5cyclictest4351-21memory09:55:220
1642899531,14cyclictest23722-21aten_r4power_cu12:30:110
1642899531,13cyclictest27415-21/usr/sbin/munin07:50:170
1642899531,13cyclictest27415-21/usr/sbin/munin07:50:160
1642899531,12cyclictest7570-21ssh12:12:590
1642899521,15cyclictest6686-21ls11:05:140
1642899515,18cyclictest24551-21pidof09:10:000
1642899512,6cyclictest18346-21df10:10:130
16428995113,7cyclictest2220-21dbus-daemon10:30:170
1642899511,13cyclictest8426-21basename10:00:110
1642899511,13cyclictest16315-21/usr/sbin/munin08:45:280
1642899511,12cyclictest2539-21ssh12:07:460
1642899511,11cyclictest3290-21sh09:54:590
1642899509,6cyclictest29326-21memory12:35:230
1642899503,8cyclictest2432-21diskstats09:20:170
16428995017,5cyclictest473-21seq11:31:550
1642899501,12cyclictest22175-21grep09:00:310
1642899492,24cyclictest17059-21/usr/sbin/munin11:15:270
16428994915,6cyclictest3826-21cron08:15:000
16428994817,6cyclictest3-21ksoftirqd/011:12:330
1642899481,14cyclictest9917-21df08:30:140
1642999479,16cyclictest0-21swapper/112:41:401
1642899471,14cyclictest26878-21ls09:45:190
234592460,4sleep1211ktimersoftd/112:30:001
1642899461,14cyclictest15128-21pidof11:47:000
1642899452,14cyclictest0-21swapper/012:46:510
1642899452,13cyclictest0-21swapper/011:44:190
1642899451,13cyclictest3020-21runrttasks08:52:050
1642899451,13cyclictest28990-21ssh10:55:010
1642899441,13cyclictest17843-21ssh10:43:010
1642899431,13cyclictest3948-21ssh10:28:520
253332420,7sleep10-21swapper/111:57:481
7222409,27sleep1735-21iostat_ios10:25:201
82652390,7sleep11642999cyclictest11:40:161
56312380,9sleep11642999cyclictest11:04:171
56312380,9sleep11642999cyclictest11:04:161
1642999389,4cyclictest81rcu_preempt11:20:011
1642999371,5cyclictest81rcu_preempt12:12:011
1642999371,10cyclictest81rcu_preempt10:52:441
47752361,31sleep14787-21ssh09:22:121
194532361,32sleep119119-21/usr/sbin/munin07:30:171
1642899363,5cyclictest81rcu_preempt11:00:190
1642899363,5cyclictest81rcu_preempt11:00:180
1642999356,5cyclictest81rcu_preempt12:36:391
1642999355,21cyclictest0-21swapper/112:02:141
16429993419,4cyclictest22-21ksoftirqd/111:54:211
1642999341,8cyclictest9157-21cat10:00:221
71692331,28sleep124557-21diskmemload11:39:291
228532331,5sleep122987-21wc09:05:131
1642999338,5cyclictest81rcu_preempt10:13:061
1642999336,4cyclictest81rcu_preempt12:54:261
1642999336,23cyclictest0-21swapper/109:33:191
16429993325,4cyclictest22494-21proc_pri09:40:261
16429993317,4cyclictest22-21ksoftirqd/112:17:341
16429993317,4cyclictest22-21ksoftirqd/112:17:341
1642999331,28cyclictest21835-21proc_pri09:00:291
16429993310,19cyclictest0-21swapper/111:23:071
302272321,27sleep130276-21ssh09:15:201
1661123212,4sleep122-21ksoftirqd/112:22:061
1642999322,26cyclictest0-21swapper/110:39:261
1642999322,26cyclictest0-21swapper/109:11:041
16429993215,14cyclictest22-21ksoftirqd/111:11:001
1642999321,4cyclictest81rcu_preempt10:43:501
16429993213,14cyclictest30083-21ntp_states10:55:281
16429993210,7cyclictest22301-21taskset10:47:301
215632311,3sleep10-21swapper/107:35:201
1642999313,4cyclictest81rcu_preempt08:00:531
1642999312,24cyclictest13347-21irqstats11:45:221
16429993112,7cyclictest22-21ksoftirqd/108:15:211
1642999311,17cyclictest0-21swapper/110:22:141
1642999311,16cyclictest7404-21taskset08:23:041
1642999311,16cyclictest2788-21ssh09:54:161
16429993111,15cyclictest23860-21ntpq10:15:281
1642899312,5cyclictest81rcu_preempt12:40:180
1642899312,13cyclictest7294-21df_inode09:25:150
1642899311,4cyclictest81rcu_preempt09:35:020
1642899311,4cyclictest81rcu_preempt08:26:320
66902301,5sleep10-21swapper/110:30:511
1642999303,4cyclictest81rcu_preempt10:10:001
1642999302,24cyclictest0-21swapper/109:59:431
1642999302,23cyclictest26381-21iostat11:25:211
16429993017,10cyclictest22-21ksoftirqd/112:07:031
1642999301,4cyclictest81rcu_preempt09:38:241
16429993014,4cyclictest22-21ksoftirqd/111:30:561
1642999301,24cyclictest24016-21df12:30:131
1642999301,16cyclictest7759-21/usr/sbin/munin08:25:131
1642899309,6cyclictest15211-21awk12:20:300
1642899307,5cyclictest81rcu_preempt10:45:260
1642899302,14cyclictest2220-21dbus-daemon11:29:370
16428993015,5cyclictest3-21ksoftirqd/010:51:480
1642899301,25cyclictest29113-21fschecks_count07:55:170
265432291,5sleep10-21swapper/107:48:011
1642999294,9cyclictest6128-21diskstats12:45:161
1642999293,4cyclictest19048-21/usr/sbin/munin08:55:301
1642999292,4cyclictest81rcu_preempt07:55:291
1642999291,4cyclictest81rcu_preempt08:46:321
16429992913,13cyclictest22-21ksoftirqd/108:50:181
16429992913,12cyclictest22-21ksoftirqd/111:05:151
1642999291,14cyclictest9919-21df08:30:131
1642899293,14cyclictest0-21swapper/008:08:010
1642899292,22cyclictest19926-21/usr/sbin/munin08:55:270
1642899291,4cyclictest81rcu_preempt12:17:450
1642899291,4cyclictest81rcu_preempt12:17:440
16428992911,14cyclictest32029-21chrt08:01:130
89522281,5sleep10-21swapper/109:26:161
283452281,5sleep10-21swapper/107:52:111
283452281,5sleep10-21swapper/107:52:111
1642999283,11cyclictest0-21swapper/108:39:011
1642899281,4cyclictest81rcu_preempt08:36:370
1642899281,11cyclictest21134-21aten_r4power_vo07:35:140
35982271,5sleep10-21swapper/108:12:531
1642999272,13cyclictest1462-21chrt08:06:241
1642899272,14cyclictest0-21swapper/007:47:020
147572271,5sleep10-21swapper/108:41:131
1642899261,3cyclictest81rcu_preempt08:41:450
1642899251,13cyclictest4648-21latency08:15:220
1642899241,15cyclictest0-21swapper/008:20:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional