You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 00:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack9slot4s (updated Sat Nov 22, 2025 12:59:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
241339912681,1246cyclictest352-21udevd07:35:171
241329912501201,35cyclictest21150-21kworker/0:107:35:170
241329912391201,28cyclictest0-21swapper/007:25:130
241339911281119,6cyclictest0-21swapper/107:25:121
237192565519,31sleep10-21swapper/107:21:201
239292564519,30sleep00-21swapper/007:23:330
59932880,9sleep12413399cyclictest10:22:451
38342850,7sleep12413399cyclictest10:54:341
297072850,3sleep0111rcuc/010:47:100
221422820,10sleep02413299cyclictest11:13:090
254112730,31sleep025406-21cron10:10:010
4902690,3sleep10-21swapper/111:57:471
223522580,8sleep12413399cyclictest10:40:121
19962530,8sleep02413299cyclictest10:52:020
203352520,8sleep12413399cyclictest12:18:271
255582430,4sleep1211ktimersoftd/111:50:171
24982420,8sleep02413299cyclictest10:19:490
24132994225,6cyclictest3-21ksoftirqd/009:56:090
2413299421,19cyclictest2864-21fschecks_time12:00:160
2413299411,9cyclictest0-21swapper/010:29:580
2413299392,22cyclictest26254-21munin-run12:25:000
2413299391,18cyclictest0-21swapper/010:22:200
28902381,4sleep12901-21ssh09:12:221
222782371,32sleep122289-21ssh09:32:291
40922361,31sleep14106-21ssh12:00:501
258342361,31sleep125845-21ssh09:35:391
2413299361,21cyclictest18667-21/usr/sbin/munin10:35:310
24132993610,10cyclictest10440-21ssh11:35:000
211792361,4sleep121192-21df_inode10:05:151
44952351,30sleep04506-21irqstats12:35:230
2413399354,5cyclictest81rcu_preempt11:31:021
2413399352,25cyclictest40950irq/93-i91511:02:131
2413299358,15cyclictest0-21swapper/011:05:050
2413299354,4cyclictest81rcu_preempt10:43:200
2413299352,15cyclictest4153-21ssh09:47:080
24132993515,4cyclictest0-21swapper/009:32:150
219722350,23sleep03-21ksoftirqd/011:45:500
7282341,29sleep1760-21hddtemp_smartct07:45:231
24133993421,9cyclictest22802-21cron08:45:001
24133993419,8cyclictest2228-21iostat_ios09:45:251
24133993417,6cyclictest22-21ksoftirqd/111:45:231
2413299341,20cyclictest25292-21ntp_states09:35:290
24132993410,4cyclictest81rcu_preempt12:29:560
39032331,28sleep00-21swapper/009:13:500
253152331,5sleep125331-21ssh12:23:411
2413399335,18cyclictest0-21swapper/111:08:291
2413399332,26cyclictest0-21swapper/109:56:441
2413399331,4cyclictest81rcu_preempt10:32:001
24133993312,4cyclictest22-21ksoftirqd/110:25:391
24133993312,16cyclictest14272-21needreboot12:45:261
24133993312,16cyclictest14272-21needreboot12:45:251
2413299337,6cyclictest0-21swapper/011:04:430
2413299337,5cyclictest81rcu_preempt11:40:170
2413299336,4cyclictest81rcu_preempt12:13:560
2413299335,5cyclictest81rcu_preempt09:28:030
2413299332,4cyclictest81rcu_preempt10:33:020
2413299331,8cyclictest7041-21ssh09:16:200
2413299331,20cyclictest12380-21missed_timers08:15:220
142152331,4sleep114229-21ssh12:11:341
89592321,27sleep18973-21ssh12:05:561
64802321,27sleep16508-21ssh09:15:341
326212321,27sleep032633-21rm11:57:190
2413399326,13cyclictest0-21swapper/111:23:221
2413399324,4cyclictest81rcu_preempt10:56:251
2413399324,4cyclictest81rcu_preempt10:48:041
2413399322,17cyclictest27335-21irqstats08:55:201
2413399321,9cyclictest24021-21switchtime08:45:291
2413399321,7cyclictest10594-21kernelversion09:20:251
2413299322,15cyclictest29488-21ssh11:20:300
2413299322,13cyclictest0-21swapper/011:17:380
2413299322,12cyclictest28069-21ssh10:11:520
2413299321,4cyclictest81rcu_preempt09:42:380
24132993211,6cyclictest28456-21munin-run09:00:000
2413299321,13cyclictest0-21swapper/009:54:330
70842311,4sleep00-21swapper/010:57:180
316802311,26sleep131773-21ps07:40:281
300992311,4sleep10-21swapper/109:40:281
258462311,4sleep125857-21ssh11:16:331
2413399315,4cyclictest81rcu_preempt12:50:291
24133993123,5cyclictest20549-21ssh11:10:501
24133993118,7cyclictest4431-21diskstats07:55:171
24133993116,12cyclictest22-21ksoftirqd/111:25:071
24133993116,12cyclictest22-21ksoftirqd/111:25:061
2413399311,5cyclictest40950irq/93-i91508:06:461
2413299316,4cyclictest81rcu_preempt12:05:140
2413299312,16cyclictest0-21swapper/012:41:510
2413299312,16cyclictest0-21swapper/012:41:500
24132993117,11cyclictest0-21swapper/007:31:070
24132993111,7cyclictest2177-21/usr/sbin/munin11:25:310
24132993111,7cyclictest2177-21/usr/sbin/munin11:25:300
2413299311,17cyclictest0-21swapper/012:34:400
2413299311,13cyclictest0-21swapper/007:50:390
203862311,4sleep120397-21ssh10:04:441
149202310,3sleep00-21swapper/011:39:390
13592311,4sleep11371-21ssh12:32:011
2413399304,22cyclictest0-21swapper/110:36:201
2413399302,15cyclictest0-21swapper/110:14:111
2413399301,6cyclictest8127-21ssh12:39:421
2413399301,16cyclictest0-21swapper/110:17:201
2413299308,6cyclictest31726-21grep09:05:430
2413299302,4cyclictest81rcu_preempt12:18:320
2413299302,14cyclictest31126-21df07:40:150
2413299302,14cyclictest12230-21ssh09:21:520
24132993012,5cyclictest21450-21fschecks_count08:40:170
17532301,5sleep00-21swapper/007:47:410
128872301,5sleep10-21swapper/112:44:411
128872301,5sleep10-21swapper/112:44:401
74932291,5sleep00-21swapper/008:02:240
299502291,5sleep00-21swapper/009:00:550
299502291,5sleep00-21swapper/009:00:550
2413399296,4cyclictest81rcu_preempt09:50:041
2413399293,6cyclictest0-21swapper/109:05:111
2413399292,23cyclictest10086-21diskstats08:10:171
2413399291,5cyclictest81rcu_preempt12:25:311
24133992913,12cyclictest13381-21sh08:19:591
2413299296,12cyclictest0-21swapper/010:00:110
2413299294,18cyclictest25684-21ntp_states08:50:250
2413299293,4cyclictest81rcu_preempt12:51:320
2413299293,4cyclictest81rcu_preempt12:45:590
2413299293,4cyclictest81rcu_preempt12:45:580
2413299292,4cyclictest81rcu_preempt11:51:070
200232291,5sleep00-21swapper/008:35:370
281902281,4sleep10-21swapper/107:31:291
2413399285,4cyclictest81rcu_preempt08:33:041
2413399282,10cyclictest29230-21iostat09:00:211
2413399282,10cyclictest29230-21iostat09:00:201
24133992820,5cyclictest7285-21unixbench_singl08:00:301
2413399281,7cyclictest13232-21ssh11:37:091
24133992812,12cyclictest0-21swapper/109:26:501
2413399281,14cyclictest16183-21irqstats11:40:231
2413299281,4cyclictest81rcu_preempt08:06:120
2413299281,14cyclictest24003-21/usr/sbin/munin08:45:290
170862281,4sleep00-21swapper/008:29:190
168252281,5sleep10-21swapper/108:26:371
2413399271,12cyclictest14280-21missed_timers08:20:241
2413299272,13cyclictest4629-21sh07:55:190
2413299271,13cyclictest6204-21gdbus08:31:090
2413299271,13cyclictest13650-21grep08:20:130
24132992710,6cyclictest11445-21taskset08:14:280
24133992610,11cyclictest21029-21latency_hist08:40:011
2413399259,12cyclictest0-21swapper/108:50:191
2413399251,4cyclictest81rcu_preempt07:53:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional