You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-10-21 - 21:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) Processor E3940 @ 1.60GHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackaslot8s.osadl.org (updated Thu Oct 21, 2021 12:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
52182177127,27sleep00-21swapper/007:07:070
51402176102,28sleep20-21swapper/207:06:082
53312172101,27sleep10-21swapper/107:08:361
35412160103,29sleep30-21swapper/307:05:083
280752590,3sleep328053-21ssh10:35:323
214862500,3chrt5887-21cyclictest10:58:251
197232490,1sleep00-21swapper/010:56:150
319132480,3sleep231914-21turbostat.cron10:40:002
143402420,1sleep20-21swapper/210:25:002
322722400,2sleep132270-21ssh09:15:191
188972380,2sleep00-21swapper/012:19:200
116332360,1sleep30-21swapper/312:12:373
116332360,1sleep30-21swapper/312:12:363
588999358,13cyclictest7830-21munin-run07:15:010
5892993216,9cyclictest55350irq/142-enp3s0-09:50:232
588999329,11cyclictest16422-21perf07:34:590
588999326,12cyclictest28722-21sh11:05:000
228702320,2sleep20-21swapper/212:22:022
118162320,11sleep012-21ksoftirqd/011:17:290
142662300,9sleep334-21ksoftirqd/309:55:323
5891992925,3cyclictest211rcuc/111:03:471
588999294,13cyclictest1951-21munin-run12:04:590
247352290,8sleep012-21ksoftirqd/009:36:400
5891992825,2cyclictest211rcuc/111:11:091
5891992824,3cyclictest211rcuc/110:19:301
588999281,15cyclictest13630-21munin-run12:15:000
588999281,15cyclictest13630-21munin-run12:15:000
589299278,6cyclictest28-21ksoftirqd/212:00:232
589299278,10cyclictest28-21ksoftirqd/210:19:312
589299271,7cyclictest25772-21apt-get12:25:172
589299271,13cyclictest2600-21ssh11:10:152
589199272,5cyclictest22-21ksoftirqd/110:46:441
5891992723,3cyclictest211rcuc/112:07:361
5891992723,3cyclictest211rcuc/109:30:381
5891992722,3cyclictest211rcuc/112:19:101
588999271,12cyclictest0-21swapper/011:35:000
589299262,10cyclictest28-21ksoftirqd/210:50:162
5892992612,2cyclictest141rcu_preempt11:22:592
5892992610,4cyclictest28-21ksoftirqd/209:18:592
589199267,9cyclictest22-21ksoftirqd/107:25:121
589199266,10cyclictest22-21ksoftirqd/112:37:371
589199264,14cyclictest22-21ksoftirqd/109:14:311
5891992622,3cyclictest211rcuc/111:33:261
588999261,13cyclictest23118-21munin-run07:50:000
95062250,2sleep10-21swapper/111:15:261
82552250,2sleep00-21swapper/007:15:240
70182250,11sleep228-21ksoftirqd/211:40:342
5893992517,4cyclictest25561-21apt-key09:10:013
589299257,7cyclictest141rcu_preempt11:55:192
589299255,12cyclictest28-21ksoftirqd/209:25:342
589299252,16cyclictest22170-21ssh09:35:182
5892992510,8cyclictest141rcu_preempt12:17:362
589199256,8cyclictest22-21ksoftirqd/111:20:441
589199255,8cyclictest22-21ksoftirqd/110:25:281
589199255,5cyclictest22-21ksoftirqd/111:59:041
589199255,13cyclictest22-21ksoftirqd/109:59:441
589199254,13cyclictest22-21ksoftirqd/111:54:431
589199251,7cyclictest22-21ksoftirqd/110:53:251
589199251,7cyclictest22-21ksoftirqd/109:40:391
5891992516,7cyclictest22-21ksoftirqd/110:40:011
589199250,6cyclictest30583-21ls11:05:271
588999258,15cyclictest0-21swapper/008:25:000
588999258,15cyclictest0-21swapper/008:24:590
38472250,11sleep122-21ksoftirqd/112:34:011
235612250,2sleep3331rcuc/310:31:583
589299249,8cyclictest28-21ksoftirqd/211:38:042
589299249,3cyclictest141rcu_preempt10:25:012
589299249,12cyclictest26129-21ssh09:10:142
589299248,4cyclictest28-21ksoftirqd/212:09:232
589299243,15cyclictest27878-21munin-node10:55:182
589299242,16cyclictest26038-21ssh11:30:132
5892992412,2cyclictest141rcu_preempt09:59:082
5892992411,8cyclictest55250irq/141-enp3s0-11:15:162
589199246,7cyclictest22-21ksoftirqd/110:32:481
589199246,6cyclictest22-21ksoftirqd/108:50:331
589199244,7cyclictest22-21ksoftirqd/111:26:361
5891992421,2cyclictest211rcuc/109:47:551
5891992420,3cyclictest211rcuc/110:22:201
5891992418,4cyclictest12132-21/usr/sbin/munin07:25:011
589199241,7cyclictest22-21ksoftirqd/112:11:091
589199241,7cyclictest22-21ksoftirqd/112:11:091
589199241,6cyclictest22-21ksoftirqd/110:12:171
588999241,12cyclictest25531-21munin-run09:10:000
88812230,9sleep334-21ksoftirqd/312:38:003
589299239,8cyclictest141rcu_preempt12:12:322
589299239,8cyclictest141rcu_preempt12:12:322
589299237,8cyclictest141rcu_preempt10:04:132
589299235,3cyclictest141rcu_preempt10:49:512
589299235,10cyclictest28-21ksoftirqd/210:13:402
589299232,10cyclictest28-21ksoftirqd/210:08:222
5892992312,2cyclictest141rcu_preempt10:31:172
5892992311,6cyclictest172942chrt11:50:172
5892992310,9cyclictest55350irq/142-enp3s0-12:30:132
589199237,11cyclictest211rcuc/111:45:281
589199236,6cyclictest22-21ksoftirqd/110:42:391
589199235,9cyclictest22-21ksoftirqd/108:30:131
589199233,6cyclictest22-21ksoftirqd/112:25:261
5891992319,3cyclictest211rcuc/109:50:301
5891992319,3cyclictest211rcuc/109:29:501
5891992318,3cyclictest211rcuc/110:00:281
589199230,6cyclictest22-21ksoftirqd/112:00:151
5889992311,8cyclictest29793-21apt-get08:05:140
588999230,11cyclictest775-21http09:45:000
237052230,2sleep30-21swapper/309:35:353
10412230,1sleep10-21swapper/111:35:311
589399229,9cyclictest14838-21apt-get10:25:223
589299227,4cyclictest141rcu_preempt09:20:172
589299222,14cyclictest656-21snmpd12:35:182
5892992211,7cyclictest28-21ksoftirqd/211:45:142
5892992210,8cyclictest32111-21apt-get08:10:152
589199225,10cyclictest22-21ksoftirqd/111:40:471
589199222,8cyclictest22-21ksoftirqd/110:07:211
5891992218,3cyclictest211rcuc/109:35:091
589199221,7cyclictest22-21ksoftirqd/112:21:101
589199221,6cyclictest22-21ksoftirqd/109:24:501
5891992214,6cyclictest23692-21df07:50:171
5891992213,7cyclictest15889-21ls07:30:301
5891992212,7cyclictest16425-21ls08:45:281
5889992215,5cyclictest4524-21ssh10:44:030
5889992210,3cyclictest141rcu_preempt12:05:190
173772220,1sleep00-21swapper/009:30:220
110192220,2chrt0-21swapper/310:49:523
99792210,2sleep00-21swapper/008:30:330
5893992112,7cyclictest737-21cron09:45:003
5893992112,6cyclictest6671-21df_inode08:25:223
5893992111,7cyclictest26519-21df_inode09:10:183
5893992111,7cyclictest17826-21df10:55:233
589399210,19cyclictest15219-21memory11:20:233
589299219,2cyclictest141rcu_preempt11:03:472
589299216,6cyclictest281ksoftirqd/208:35:302
5892992112,6cyclictest55250irq/141-enp3s0-10:40:182
5892992110,5cyclictest28-21ksoftirqd/209:45:012
589199213,5cyclictest22-21ksoftirqd/107:35:291
589199211,7cyclictest22-21ksoftirqd/108:35:161
589199211,7cyclictest22-21ksoftirqd/107:55:201
5891992111,7cyclictest23472-21cut09:05:001
5891992111,7cyclictest23472-21cut09:04:591
588999211,4cyclictest0-21swapper/008:03:470
589399206,12cyclictest218-21systemd-journal10:53:233
5893992013,4cyclictest28475-21sh10:08:553
5893992012,5cyclictest34-21ksoftirqd/311:40:193
5893992011,7cyclictest498-21dbus-daemon12:20:003
5893992011,7cyclictest23357-21cat11:00:173
589399201,17cyclictest0-21swapper/309:34:593
5893992011,6cyclictest28779-21mailstats08:00:303
5893992010,7cyclictest20312-21cat11:25:173
589299209,2cyclictest141rcu_preempt09:32:432
589299208,4cyclictest28-21ksoftirqd/209:45:002
5892992013,4cyclictest21458-21rt-features11:25:282
5892992012,6cyclictest8661-21expr08:30:192
5892992012,6cyclictest24436-21sed09:05:282
589299201,11cyclictest28-21ksoftirqd/211:07:242
589199206,8cyclictest22-21ksoftirqd/107:14:201
589199204,7cyclictest12733-21chrt08:40:081
589199201,7cyclictest22-21ksoftirqd/108:17:211
5891992016,3cyclictest211rcuc/108:55:461
5891992015,3cyclictest211rcuc/108:08:571
5891992015,3cyclictest211rcuc/107:45:311
5891992011,6cyclictest24762-21basename09:05:311
5891992010,7cyclictest32436-21df08:10:171
589199200,7cyclictest22-21ksoftirqd/108:03:091
588999208,8cyclictest10610-21apt-get08:35:160
588999207,3cyclictest141rcu_preempt10:05:000
588999202,3cyclictest55802ssh11:12:020
5889992012,6cyclictest29079-21find11:05:110
5889992012,6cyclictest12404-21cat09:25:410
5889992011,6cyclictest18628-21expr11:50:290
5889992010,7cyclictest26510-21apt-get10:35:180
5889992010,7cyclictest17161-21df07:35:180
589399199,7cyclictest656-21snmpd11:08:253
589399192,15cyclictest7364-21cpuspeed_turbos09:50:283
589399191,9cyclictest13104-21ssh09:26:183
5893991912,5cyclictest3376-21expr10:15:183
5893991912,5cyclictest27143-21pluginstate11:30:273
5893991912,5cyclictest1839-21cat08:15:143
5893991912,5cyclictest10828-21idleruntime07:20:233
5893991912,4cyclictest32267-21sh10:40:103
5893991911,6cyclictest6157-21cat08:25:003
5893991911,6cyclictest6157-21cat08:24:593
5893991911,6cyclictest23567-21cut11:55:243
5893991910,7cyclictest29497-21cut12:00:213
589299198,2cyclictest28-21ksoftirqd/208:45:172
589299196,9cyclictest27659-21apt-get08:00:172
589299194,11cyclictest16836-21apt-get07:35:172
5892991914,3cyclictest14281-21vmstat08:40:332
5892991913,4cyclictest28-21ksoftirqd/208:20:002
5892991911,6cyclictest7734-21seq07:14:032
5892991911,6cyclictest21318-21wc07:45:172
5892991910,6cyclictest20285-21awk08:55:262
589299190,6cyclictest184-21jbd2/sda2-809:00:232
589299190,6cyclictest184-21jbd2/sda2-809:00:232
589199196,7cyclictest22-21ksoftirqd/107:17:401
589199195,8cyclictest22-21ksoftirqd/107:41:271
589199191,7cyclictest22-21ksoftirqd/108:20:291
589199191,7cyclictest22-21ksoftirqd/108:20:291
5891991915,3cyclictest211rcuc/108:25:581
5889991912,5cyclictest20107-21expr12:20:160
5889991912,5cyclictest14204-21seq10:51:380
5889991912,4cyclictest27071-21mailstats12:25:290
5889991911,6cyclictest3431-21expr10:15:180
5889991911,6cyclictest29527-21sed10:10:000
5889991911,6cyclictest19330-21cat07:40:180
5889991911,6cyclictest16213-21sensors_temp10:25:370
589399189,6cyclictest11201-21cat11:45:123
589399188,3cyclictest138432grep07:25:353
589399187,9cyclictest0-21swapper/312:22:143
589399187,9cyclictest0-21swapper/311:53:123
589399187,7cyclictest2059-21idleruntime09:45:233
589399186,9cyclictest0-21swapper/310:00:043
589399181,5cyclictest0-21swapper/309:17:253
5893991813,3cyclictest9308-21sh09:23:393
5893991813,3cyclictest6641-21latency07:10:323
5893991813,3cyclictest1937-21sh12:31:423
5893991812,4cyclictest5367-21sh12:07:033
5893991812,4cyclictest5152-21sh11:11:323
5893991812,4cyclictest28755-21sh12:27:043
5893991812,4cyclictest14450irq/133-ahci[0011:20:003
5893991812,4cyclictest12727-21sh10:23:033
5893991811,5cyclictest8327-21expr07:15:243
5893991811,5cyclictest4932-21sendmail11:40:003
5893991811,5cyclictest15285-21expr08:45:163
5893991811,5cyclictest13085-21expr08:40:193
5893991811,5cyclictest11444-21needreboot08:35:283
5893991810,6cyclictest19637-21expr08:55:183
5893991810,6cyclictest19536-21tune2fs07:40:213
589299187,7cyclictest12381-21apt-get07:25:172
589299186,2cyclictest141rcu_preempt07:20:192
589299181,5cyclictest0-21swapper/208:25:252
5892991813,3cyclictest8068-21/usr/sbin/munin07:15:372
5892991813,2cyclictest14450irq/133-ahci[0007:55:002
5892991812,4cyclictest31669-21latency_hist08:10:002
5892991812,4cyclictest20119-21proc_pri07:40:292
5892991810,6cyclictest26916-21grep07:55:302
588999189,6cyclictest20254-21grep11:25:170
588999187,8cyclictest20631-21apt-get10:30:190
588999182,14cyclictest5607-21cut12:35:230
588999182,12cyclictest23367-21apt-get07:50:120
588999182,11cyclictest32072-21kworker/u8:009:55:140
5889991814,3cyclictest14646-21sh08:45:010
5889991814,2cyclictest19529-21cstates08:55:180
5889991813,3cyclictest30508-21hwlatdetect10:10:250
5889991813,3cyclictest15086-21interrupts11:20:210
5889991812,4cyclictest9498-21sh10:48:020
5889991812,4cyclictest2372-21cut09:45:270
5889991812,3cyclictest641-21runrttasks09:51:100
5889991811,5cyclictest15388-21expr08:45:170
5889991811,5cyclictest13641-21seq11:46:220
5889991811,5cyclictest10532-21expr07:20:190
5889991811,4cyclictest25530-21diskmemload09:20:400
5889991810,6cyclictest4683-21basename12:35:000
5889991810,6cyclictest32635-21if_err_enp3s011:35:240
589399172,10cyclictest23367-21apt-get07:50:163
5893991712,4cyclictest34-21ksoftirqd/308:15:003
5893991711,4cyclictest25530-21diskmemload10:11:163
5893991711,4cyclictest21471-21ls07:45:183
5893991711,4cyclictest16703-21apt-config07:35:003
5893991710,5cyclictest9572-21which08:30:303
5893991710,5cyclictest17862-21cat08:50:273
5892991712,3cyclictest4845-21kernelversion08:20:222
5892991712,3cyclictest4845-21kernelversion08:20:222
5892991710,6cyclictest656-21snmpd08:53:022
5892991710,5cyclictest15030-21sed07:30:202
588999179,6cyclictest22345-21uname09:00:260
588999179,6cyclictest22345-21uname09:00:250
588999178,7cyclictest913-21grep08:10:290
588999177,7cyclictest26436-21cat09:10:170
588999176,7cyclictest23050-21apt-get11:55:150
588999171,3cyclictest31840-21apt-get09:15:120
5889991712,3cyclictest9675-21sh10:20:220
5889991711,4cyclictest25530-21diskmemload11:40:140
5889991711,4cyclictest19096-21latency_hist08:55:000
5889991710,5cyclictest2453-21basename08:15:210
5889991710,5cyclictest12339-21seq07:25:030
589399168,6cyclictest30952-21smartctl08:05:283
589399161,6cyclictest0-21swapper/307:30:233
5893991611,3cyclictest26037-21cut07:55:223
5893991611,3cyclictest23534-21latency_hist09:05:013
5893991611,3cyclictest23534-21latency_hist09:05:013
588999169,5cyclictest7172-21uname08:25:270
588999151,10cyclictest25505-21apt-get07:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional