You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-01 - 19:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) CPU E3825 @ 1.33GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of shadow in rack #9, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack9slot4s.osadl.org (updated Sun Aug 01, 2021 12:59:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25809912081184,20cyclictest4425-21pidof07:29:001
25799911521127,22cyclictest0-21swapper/007:29:490
21242618573,30sleep00-21swapper/007:21:010
20942512418,30sleep10-21swapper/107:20:451
4262590,2sleep0427-21awk10:05:290
79232390,3sleep17920-21ntp_states10:25:311
2579993923,5cyclictest3-21ksoftirqd/010:23:220
2579993922,6cyclictest3-21ksoftirqd/012:38:530
2579993922,6cyclictest3-21ksoftirqd/012:38:530
2579993922,6cyclictest3-21ksoftirqd/009:10:080
75702381,4sleep17576-21df_inode07:35:181
257999388,7cyclictest3-21ksoftirqd/011:24:340
2579993823,5cyclictest3-21ksoftirqd/011:33:170
2579993821,6cyclictest3-21ksoftirqd/010:58:430
2579993721,6cyclictest3-21ksoftirqd/011:01:130
2579993721,6cyclictest3-21ksoftirqd/009:55:470
2579993715,4cyclictest22037-21apt-get11:05:130
2579993714,8cyclictest3-21ksoftirqd/009:09:190
2579993621,5cyclictest3-21ksoftirqd/010:11:360
2579993621,5cyclictest3-21ksoftirqd/009:52:270
2579993611,13cyclictest18242-21watchsnmpd10:55:010
59742351,4sleep15978-21irqstats07:30:321
57112351,30sleep05767-21/usr/sbin/munin08:55:280
2579993517,5cyclictest3298-21runrttasks09:26:500
2579993513,12cyclictest25059-21idleruntime08:20:360
2579993511,8cyclictest3-21ksoftirqd/009:00:360
281022340,5sleep128101-21cron08:30:001
268022341,5sleep10-21swapper/112:40:341
258099342,28cyclictest19277-21iostat_ios09:30:381
258099341,16cyclictest7598-21diskstats09:00:191
2579993418,6cyclictest29687-21wget12:50:140
2579993413,7cyclictest3-21ksoftirqd/012:17:310
257999341,20cyclictest7171-21cat07:35:010
257999341,20cyclictest11152-21diskstats10:35:180
2579993411,8cyclictest3-21ksoftirqd/007:35:320
49622331,28sleep04963-21sshd10:19:310
257999334,18cyclictest0-21swapper/011:39:190
2579993319,5cyclictest3-21ksoftirqd/012:07:530
2579993313,6cyclictest4161-21gdbus11:57:100
2579993312,13cyclictest0-21swapper/008:11:130
2579993311,8cyclictest3-21ksoftirqd/010:44:310
2580993223,5cyclictest0-21swapper/111:10:391
257999329,13cyclictest15769-21taskset12:10:590
257999323,25cyclictest7395-21hddtemp_smartct11:50:230
2579993219,4cyclictest3-21ksoftirqd/011:27:180
2579993218,5cyclictest4161-21gdbus07:47:550
2579993216,5cyclictest0-21swapper/008:15:200
2579993214,6cyclictest14090-21seq07:50:360
2579993213,8cyclictest0-21swapper/008:06:010
2579993213,14cyclictest28266-21apt-get08:30:120
2579993212,9cyclictest19210-21idleruntime09:30:360
257999321,19cyclictest6748-21latency_hist11:50:000
2579993211,6cyclictest3-21ksoftirqd/011:19:050
2579993211,6cyclictest3-21ksoftirqd/011:19:050
2579993211,6cyclictest3-21ksoftirqd/008:27:220
2579993210,7cyclictest3-21ksoftirqd/012:25:100
241722320,5sleep1201rcuc/109:45:001
258099312,24cyclictest30411-21ntp_states12:50:261
258099311,4cyclictest81rcu_preempt10:35:301
257999319,7cyclictest3-21ksoftirqd/008:40:140
257999315,19cyclictest28294-21iostat_ios12:45:200
257999314,21cyclictest17583-21iostat_ios08:00:390
2579993116,5cyclictest15600-21/usr/sbin/munin10:45:300
2579993115,9cyclictest31695-21munin-run10:05:000
2579993110,6cyclictest3-21ksoftirqd/010:34:590
131662311,5sleep113171-21irqstats12:05:281
109682311,4sleep110679-21/usr/sbin/munin12:00:201
48132301,5sleep00-21swapper/008:51:000
258099302,24cyclictest0-21swapper/109:25:141
257999309,10cyclictest0-21swapper/007:56:410
257999308,8cyclictest2577-21cyclictest12:24:570
257999308,19cyclictest23249-21sed12:30:360
257999308,12cyclictest3-21ksoftirqd/009:23:080
257999308,12cyclictest3-21ksoftirqd/008:47:200
257999301,8cyclictest0-21swapper/012:00:470
2579993015,5cyclictest0-21swapper/009:40:280
2579993013,6cyclictest23938-21apt-get11:10:140
2579993011,6cyclictest7088-21apt-get10:25:150
188332301,25sleep118837-21apt-get08:05:121
110822301,5sleep10-21swapper/109:10:191
320462290,5sleep1201rcuc/111:30:251
2580992922,4cyclictest133-21jbd2/sda2-807:54:011
258099291,4cyclictest81rcu_preempt09:09:191
257999299,9cyclictest0-21swapper/009:45:110
257999291,9cyclictest26807-21python12:40:330
2579992916,5cyclictest0-21swapper/008:35:360
2579992916,5cyclictest0-21swapper/008:35:350
2579992913,5cyclictest3418-21df11:40:210
257999291,10cyclictest0-21swapper/007:40:120
223762291,5sleep10-21swapper/108:14:261
162832291,5sleep10-21swapper/110:49:181
157042291,5sleep10-21swapper/112:10:331
100842291,3sleep10-21swapper/111:55:501
95802281,5sleep10-21swapper/107:40:211
297502281,5sleep10-21swapper/108:32:371
293362281,5sleep10-21swapper/112:49:191
258099284,18cyclictest30397-21iostat_ios11:25:261
258099281,4cyclictest81rcu_preempt12:17:311
2580992813,4cyclictest22-21ksoftirqd/110:24:211
2579992813,5cyclictest21139-21iostat_ios09:35:260
236502281,5sleep10-21swapper/111:07:081
181752281,5sleep10-21swapper/110:54:251
162102281,5sleep10-21swapper/109:21:151
15652281,4sleep11605-21hddtemp_smartct11:35:221
127232281,4sleep10-21swapper/107:49:261
12612281,5sleep10-21swapper/108:44:351
51362271,5sleep10-21swapper/108:54:201
47372271,4sleep10-21swapper/111:44:001
32142271,5sleep10-21swapper/108:49:051
314432271,5sleep10-21swapper/108:35:411
314432271,5sleep10-21swapper/108:35:401
276782271,5sleep10-21swapper/109:52:401
26352271,5sleep10-21swapper/110:10:471
258099279,14cyclictest69842chrt08:58:441
258099279,14cyclictest0-21swapper/111:50:151
258099275,4cyclictest81rcu_preempt12:26:251
258099273,4cyclictest81rcu_preempt11:19:051
258099273,4cyclictest81rcu_preempt11:19:051
258099273,3cyclictest81rcu_preempt10:43:461
258099273,18cyclictest15333-21diskstats07:55:201
258099273,12cyclictest0-21swapper/108:00:281
258099272,9cyclictest22843-21ntp_states12:30:311
258099272,3cyclictest81rcu_preempt11:45:341
258099272,20cyclictest24396-21hddtemp_smartct12:35:301
258099272,20cyclictest24396-21hddtemp_smartct12:35:301
257999277,7cyclictest3-21ksoftirqd/009:18:330
256482271,5sleep10-21swapper/109:46:071
239442271,5sleep10-21swapper/108:16:121
218882271,5sleep10-21swapper/109:36:211
216582271,5sleep10-21swapper/111:01:141
197892271,5sleep10-21swapper/110:56:361
197362271,5sleep10-21swapper/112:22:501
103822271,5sleep10-21swapper/110:31:551
258099262,3cyclictest81rcu_preempt09:58:261
2580992617,6cyclictest22-21ksoftirqd/111:20:481
245432261,5sleep10-21swapper/108:20:191
143952261,5sleep10-21swapper/109:17:111
258099254,4cyclictest81rcu_preempt10:09:141
258099253,17cyclictest30578-21iostat_ios10:00:221
2580992513,9cyclictest0-21swapper/110:15:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional