You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-28 - 07:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3825 @1333 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of shadow in rack #9, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack9slot4s.osadl.org (updated Tue Mar 28, 2023 00:59:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
327149912101187,20cyclictest0-21swapper/019:29:520
325992622576,32sleep10-21swapper/119:24:251
325552565520,31sleep00-21swapper/019:23:580
3271599127119,6cyclictest0-21swapper/119:29:521
61662830,8sleep13271599cyclictest23:55:181
261692690,3sleep026170-21processes23:20:290
3271499512,21cyclictest11430-21users22:40:320
3271499495,5cyclictest81rcu_preempt19:35:190
206282470,8sleep13271599cyclictest21:40:311
3271599462,9cyclictest0-21swapper/120:45:171
3271599450,12cyclictest19564-21tr00:30:241
3271499451,14cyclictest4541-21run-parts20:59:590
3271599444,5cyclictest30322-21ntpq20:40:391
239772440,8sleep03271499cyclictest23:15:220
3271599432,9cyclictest0-21swapper/120:30:321
3271599431,9cyclictest0-21swapper/123:04:391
3271599411,8cyclictest0-21swapper/122:55:551
3271599411,8cyclictest0-21swapper/121:20:241
3271599411,23cyclictest14388-21/usr/sbin/munin20:00:301
3271599401,9cyclictest0-21swapper/121:10:201
32715994017,9cyclictest32657-21sendmail23:40:001
3271599397,5cyclictest22-21ksoftirqd/122:25:351
3271599397,5cyclictest10376-21dpkg21:15:131
3271599395,6cyclictest22-21ksoftirqd/123:54:291
3271599391,8cyclictest0-21swapper/123:29:561
3271599387,6cyclictest22-21ksoftirqd/100:36:201
3271599387,5cyclictest8262-21dpkg22:35:111
3271599381,8cyclictest0-21swapper/100:17:521
32715993813,4cyclictest22-21ksoftirqd/121:32:501
3271599376,5cyclictest23594-21dpkg21:50:121
3271599375,5cyclictest22-21ksoftirqd/120:28:061
3271599371,8cyclictest0-21swapper/100:12:511
32715993713,4cyclictest22-21ksoftirqd/123:06:041
32715993713,3cyclictest22-21ksoftirqd/122:47:031
32715993712,3cyclictest22-21ksoftirqd/119:52:231
3271599368,6cyclictest2779-21snmpd20:06:241
3271599368,6cyclictest22-21ksoftirqd/122:24:001
3271599368,5cyclictest29588-21diskstats22:05:301
3271599367,6cyclictest22-21ksoftirqd/123:47:001
3271599367,19cyclictest24490-21sshd00:40:551
3271599366,6cyclictest22-21ksoftirqd/122:13:001
3271599366,6cyclictest22-21ksoftirqd/100:27:191
3271599364,5cyclictest22-21ksoftirqd/119:45:441
3271599363,11cyclictest0-21swapper/119:59:551
3271599362,8cyclictest0-21swapper/122:30:591
3271599361,8cyclictest0-21swapper/123:33:461
3271599361,8cyclictest0-21swapper/119:44:011
3271599361,7cyclictest22-21ksoftirqd/121:57:251
32715993614,4cyclictest22-21ksoftirqd/123:17:251
32715993613,4cyclictest22-21ksoftirqd/120:24:381
32715993612,4cyclictest22-21ksoftirqd/120:38:491
32715993612,4cyclictest22-21ksoftirqd/120:12:191
32715993611,4cyclictest22-21ksoftirqd/121:26:261
3271499365,13cyclictest2517-21seq19:30:030
3271499362,20cyclictest30187-21timerwakeupswit23:30:310
3271599357,6cyclictest22-21ksoftirqd/100:01:001
3271599355,6cyclictest22-21ksoftirqd/121:49:251
3271599355,6cyclictest22-21ksoftirqd/100:21:261
3271599352,16cyclictest22-21ksoftirqd/123:40:171
3271499351,21cyclictest9156-21ls00:00:380
3271599348,5cyclictest28222-21ntpq22:00:291
3271599348,5cyclictest28222-21ntpq22:00:291
3271599347,5cyclictest2779-21snmpd21:01:251
3271599346,6cyclictest22-21ksoftirqd/100:06:291
3271599346,3cyclictest81rcu_preempt20:50:361
3271599345,5cyclictest22-21ksoftirqd/120:55:181
3271599345,4cyclictest81rcu_preempt22:52:291
32715993413,4cyclictest22-21ksoftirqd/121:06:331
3271499349,5cyclictest14924-21sendmail_mailtr22:50:340
3271499343,15cyclictest27163-21apt-get23:25:210
3271499341,21cyclictest26744-21users21:55:340
3271499341,13cyclictest0-21swapper/021:32:590
3271599337,9cyclictest2779-21snmpd22:41:471
3271599337,6cyclictest27485-21smartctl00:50:211
3271599337,6cyclictest27485-21smartctl00:50:211
3271599337,5cyclictest20632-21iostat20:15:221
3271599337,5cyclictest18163-21cstates21:35:151
3271599336,5cyclictest25780-21smartctl23:20:201
3271599334,4cyclictest81rcu_preempt22:15:241
3271599334,3cyclictest81rcu_preempt00:48:401
3271599331,8cyclictest0-21swapper/119:30:331
32714993315,5cyclictest371-21apt-get23:40:120
3271499331,21cyclictest4558-21wget22:25:190
221052331,5sleep122188-21memory23:10:321
21292331,28sleep03637-21packagekitd20:50:490
3271499323,3cyclictest81rcu_preempt23:05:170
3271499321,4cyclictest81rcu_preempt23:00:210
3271499321,4cyclictest81rcu_preempt22:17:130
3271499321,4cyclictest81rcu_preempt21:16:140
3271499321,4cyclictest81rcu_preempt20:20:060
3271499321,12cyclictest27265-21df00:50:170
3271499321,12cyclictest27265-21df00:50:160
3271599311,8cyclictest0-21swapper/119:35:501
3271499315,13cyclictest2226-21dbus-daemon21:37:160
3271499313,14cyclictest15618-21apt-get00:20:210
3271499311,4cyclictest81rcu_preempt23:49:110
3271499311,4cyclictest81rcu_preempt21:50:040
3271499311,4cyclictest81rcu_preempt21:49:260
3271499311,4cyclictest81rcu_preempt21:07:570
3271499311,4cyclictest81rcu_preempt00:17:000
32714993112,6cyclictest6840-21apt-get19:40:160
3271499311,25cyclictest14982-21/usr/sbin/munin21:25:280
3271499311,18cyclictest21271-21apt-get00:35:130
3271499311,18cyclictest20181-21netstat00:30:340
32714993111,6cyclictest26115-21python00:45:370
3271499311,10cyclictest14987-21interrupts20:00:300
3271499304,4cyclictest81rcu_preempt19:49:260
3271499301,4cyclictest81rcu_preempt20:35:510
3271499301,25cyclictest7046-21/usr/sbin/munin23:55:370
32714993011,6cyclictest12648-21/usr/sbin/munin22:45:220
3271499301,14cyclictest18022-21date20:10:010
3271499301,12cyclictest0-21swapper/023:14:570
3271499300,4cyclictest81rcu_preempt21:22:470
3271499298,5cyclictest23597-21fschecks_time00:40:260
3271499294,9cyclictest5015-21diskstats21:00:170
3271499292,14cyclictest3638-21gdbus20:46:450
3271499291,4cyclictest81rcu_preempt22:37:360
3271499291,4cyclictest81rcu_preempt22:14:420
3271499291,4cyclictest81rcu_preempt20:13:550
3271499291,4cyclictest81rcu_preempt00:05:400
32714992912,5cyclictest8487-21apt-get21:10:130
32714992912,5cyclictest27986-21irqstats22:00:240
32714992912,5cyclictest27986-21irqstats22:00:230
32714992912,5cyclictest25993-21df20:30:160
32714992912,5cyclictest18093-21iostat00:25:210
3271499291,24cyclictest3445-21/usr/sbin/munin22:20:270
3271499291,17cyclictest0-21swapper/021:43:540
3271499291,15cyclictest30965-21chrt22:09:540
3271499291,15cyclictest12446-21/usr/sbin/munin00:10:300
3271499291,12cyclictest0-21swapper/020:27:120
3271499290,4cyclictest81rcu_preempt23:37:380
63762281,5sleep00-21swapper/022:30:130
3271499288,4cyclictest30325-21ntpdc20:40:390
3271499281,4cyclictest81rcu_preempt22:57:520
3271499281,14cyclictest13438-21/usr/sbin/munin19:55:300
3271499273,13cyclictest4996-21missed_timers23:50:300
3271499272,13cyclictest20638-21iostat_ios20:15:230
3271499271,12cyclictest10815-21tr19:50:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional