You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-06-16 - 21:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Atom(TM) Processor E3940 @ 1.60GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack5slot3s.osadl.org (updated Wed Jun 16, 2021 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
173352238162,25sleep20-21swapper/207:07:192
173082234156,25sleep00-21swapper/007:06:570
173142202166,24sleep10-21swapper/107:07:021
172372202164,25sleep30-21swapper/307:06:033
872521340,4sleep11779899cyclictest10:25:151
17803996158,2cyclictest0-21swapper/211:31:102
121732600,2sleep20-21swapper/211:45:182
3082490,1sleep00-21swapper/007:40:240
140442420,1sleep20-21swapper/210:35:272
1779299324,13cyclictest12375-21munin-run10:35:010
1780399301,6cyclictest8149-21ps11:35:242
1779299305,12cyclictest12618-21munin-run08:10:010
1780399291,15cyclictest0-21swapper/210:15:292
1779899291,25cyclictest581-21systemd-journal08:30:001
1779899282,12cyclictest13360-21munin-run11:50:011
1779899281,13cyclictest3250-21munin-run12:40:021
1779299281,12cyclictest29424-21awk07:35:010
1779899272,6cyclictest0-21swapper/109:34:341
1779899271,9cyclictest0-21swapper/111:21:141
1779899271,24cyclictest0-21swapper/111:12:081
1779899271,17cyclictest0-21swapper/108:33:541
1779899271,17cyclictest0-21swapper/108:08:271
1779899271,10cyclictest0-21swapper/112:10:271
247592261,3sleep224791-21ntp_offset11:00:282
1780399261,11cyclictest21354-21tail07:15:202
17798992614,3cyclictest0-21swapper/107:57:341
1779899261,17cyclictest0-21swapper/108:44:051
1779899261,16cyclictest0-21swapper/110:48:061
1779899260,17cyclictest0-21swapper/109:41:171
1779899260,17cyclictest0-21swapper/109:28:021
197022251,20sleep219847-21/usr/sbin/munin09:35:282
1780399254,10cyclictest29810-40dnf11:10:342
17803992516,6cyclictest19617-21dnf10:50:142
17803992512,6cyclictest0-21swapper/211:44:192
1779899254,6cyclictest0-21swapper/112:07:211
1779899252,16cyclictest0-21swapper/108:49:331
1779899251,9cyclictest0-21swapper/112:34:441
1779899251,8cyclictest0-21swapper/112:19:491
1779899251,6cyclictest0-21swapper/109:21:121
17798992514,2cyclictest0-21swapper/107:50:191
17798992513,3cyclictest0-21swapper/111:25:361
1779899251,16cyclictest0-21swapper/110:20:531
1779899251,16cyclictest0-21swapper/109:48:291
1779899251,16cyclictest0-21swapper/109:48:291
1779899251,16cyclictest0-21swapper/108:14:271
1779899251,16cyclictest0-21swapper/107:14:051
1779899250,15cyclictest0-21swapper/109:05:141
1779299257,15cyclictest24963-21idleruntime-cro07:25:010
1779299251,13cyclictest1914-21hddtemp_smartct12:35:160
1780899247,7cyclictest20334-21ntp_offset10:50:213
1780399244,9cyclictest5354-21fschecks_count09:05:162
1780399242,11cyclictest17129-21ntp_states09:30:222
1780399241,7cyclictest31550-21latency_hist10:05:022
1779899248,3cyclictest25107-21tr09:50:191
1779899242,15cyclictest0-21swapper/109:57:581
1779899242,10cyclictest0-21swapper/107:23:121
1779899241,3cyclictest21043-21seq10:52:191
1779899241,20cyclictest32389-21latency_hist11:20:021
1779899241,15cyclictest0-21swapper/112:23:211
1779899241,15cyclictest0-21swapper/111:58:351
1779899241,15cyclictest0-21swapper/110:11:461
1779899241,15cyclictest0-21swapper/110:08:561
1779899241,15cyclictest0-21swapper/109:39:241
1779899241,15cyclictest0-21swapper/109:01:461
1779899241,15cyclictest0-21swapper/108:57:131
1779899241,15cyclictest0-21swapper/108:24:461
1779899241,15cyclictest0-21swapper/108:16:411
1779899241,15cyclictest0-21swapper/107:45:431
1779899241,15cyclictest0-21swapper/107:41:271
1779899241,15cyclictest0-21swapper/107:39:211
17798992411,2cyclictest31014-21sshd10:00:321
1779899241,11cyclictest8396-21rt-features09:10:231
1779899241,10cyclictest26984-21awk11:05:201
1780899238,4cyclictest19283-21dump-pmu-power08:29:563
1779899239,3cyclictest25721-21latency_hist11:05:011
1779899237,3cyclictest23503-21sleep11:00:011
17798992314,1cyclictest0-21swapper/108:52:171
17798992313,3cyclictest0-21swapper/107:18:221
17798992313,2cyclictest0-21swapper/107:25:301
17798992313,2cyclictest0-21swapper/107:25:301
17798992312,3cyclictest0-21swapper/109:17:491
1779899231,15cyclictest0-21swapper/112:04:171
1779899231,15cyclictest0-21swapper/112:04:161
1779899231,15cyclictest0-21swapper/111:35:311
1779899231,14cyclictest0-21swapper/108:39:301
1779899231,14cyclictest0-21swapper/108:39:301
1779899231,14cyclictest0-21swapper/107:33:361
1779899231,10cyclictest12935-21df_inode10:35:151
1779899231,10cyclictest10667-21unixbench_multi11:40:281
1779899231,10cyclictest0-21swapper/111:50:171
1779899231,10cyclictest0-21swapper/110:40:181
164122230,4sleep339-21rcuc/309:30:153
1780899225,8cyclictest30169-21needreboot12:25:213
1780899221,6cyclictest0-21swapper/311:10:333
1780899221,10cyclictest3483-21latency_hist10:15:023
1780399223,9cyclictest0-21swapper/212:28:502
1780399221,6cyclictest20019-21kworker/u8:009:55:142
1780399221,6cyclictest0-21swapper/210:05:252
1780399221,6cyclictest0-21swapper/209:43:052
1780399221,5cyclictest0-21swapper/209:46:302
1780399221,5cyclictest0-21swapper/209:46:302
1780399220,15cyclictest0-21swapper/210:31:262
1779899228,3cyclictest9959-21awk08:00:271
17798992212,3cyclictest0-21swapper/111:30:511
1779899220,15cyclictest0-21swapper/110:18:041
1779299223,5cyclictest0-21swapper/009:26:260
1779299222,12cyclictest0-21swapper/010:20:020
1779299221,10cyclictest27748-21kworker/u8:212:00:130
1779299221,10cyclictest27748-21kworker/u8:212:00:130
285452210,5sleep339-21rcuc/309:55:273
1780899216,7cyclictest18837-21strings07:10:223
1780899214,3cyclictest2728-21cut07:45:233
1780899213,10cyclictest29639-21/usr/sbin/munin07:35:333
1780899212,9cyclictest17420-21users11:55:273
17808992111,4cyclictest1009-1kworker/u9:111:34:413
1780399215,3cyclictest86850irq/129-enp1s0-08:35:272
1780399215,3cyclictest86850irq/129-enp1s0-08:35:272
1780399213,4cyclictest9283-21latency08:00:202
1780399212,5cyclictest14908-21ntp_states09:25:232
1780399211,7cyclictest836-21runrttasks10:25:262
1780399211,7cyclictest0-21swapper/212:20:322
1780399211,6cyclictest0-21swapper/212:15:132
1780399211,6cyclictest0-21swapper/211:09:102
1780399211,5cyclictest26207-21cut07:25:282
1780399211,5cyclictest26207-21cut07:25:282
1780399211,13cyclictest0-21swapper/210:42:512
1780399211,13cyclictest0-21swapper/210:10:462
1780399211,11cyclictest0-21swapper/212:00:142
1780399211,11cyclictest0-21swapper/212:00:132
1780399211,11cyclictest0-21swapper/211:16:012
1780399211,10cyclictest3966-21ls11:25:262
1779299217,7cyclictest1004-21munin-run12:35:020
1779299216,3cyclictest16724-21hddtemp_smartct09:30:160
1779299213,9cyclictest12069-21sort11:45:160
1779299212,11cyclictest842-21lldpd07:28:460
1779299212,11cyclictest842-21lldpd07:28:460
1779299211,9cyclictest0-21swapper/007:38:310
1779299211,3cyclictest0-21swapper/011:41:030
1779299211,2cyclictest0-21swapper/012:21:570
1779299211,11cyclictest18276-21/usr/sbin/munin09:35:230
17792992110,2cyclictest496-21jbd2/dm-0-808:57:120
181992201,3sleep318369-21df12:00:133
181992201,3sleep318369-21df12:00:133
181502201,3sleep018274-21ps10:45:250
1780899209,3cyclictest32619-21/usr/sbin/munin11:20:283
1780899207,3cyclictest21935-21ps07:15:283
1780899206,5cyclictest9388-21runrttasks09:15:143
1780899204,8cyclictest0-21swapper/312:08:443
1780899202,15cyclictest24825-21cut12:15:133
1780899201,10cyclictest18288-21acpi09:35:133
1780399205,4cyclictest27748-21kworker/u8:211:20:242
1780399203,9cyclictest1081-21latency_hist12:35:022
1780399203,3cyclictest86550irq/126-enp1s0-11:50:202
1780399201,9cyclictest0-21swapper/208:52:112
1780399201,6cyclictest0-21swapper/210:47:282
1780399201,6cyclictest0-21swapper/209:50:292
1780399201,6cyclictest0-21swapper/209:24:322
1780399201,5cyclictest0-21swapper/212:12:462
1780399201,4cyclictest6931-21strings10:20:192
1780399201,4cyclictest0-21swapper/209:16:542
1780399201,13cyclictest0-21swapper/211:56:452
1780399201,12cyclictest0-21swapper/209:12:162
1780399201,11cyclictest0-21swapper/212:35:202
1780399201,10cyclictest22095-21tail10:55:162
17803992010,6cyclictest836-21runrttasks08:09:312
1779899206,4cyclictest0-21swapper/112:26:341
1779899201,2cyclictest0-21swapper/110:35:001
1779299209,3cyclictest21735-21/usr/sbin/munin08:30:280
1779299207,8cyclictest0-21swapper/009:44:070
1779299207,4cyclictest0-21swapper/010:53:500
1779299207,4cyclictest0-21swapper/010:22:190
1779299205,6cyclictest0-21swapper/010:29:330
1779299203,8cyclictest6682-21kworker/0:211:36:450
1779299203,2cyclictest86550irq/126-enp1s0-12:26:170
1779299202,16cyclictest24287-21sshd12:12:320
1779299202,10cyclictest9392-21/usr/sbin/munin09:15:400
1779299202,10cyclictest20751-21/usr/sbin/munin07:15:160
1779299201,11cyclictest4937-21/usr/sbin/munin09:05:160
1779299201,11cyclictest28168-21/usr/sbin/munin11:10:270
1779299201,11cyclictest26182-21/usr/sbin/munin08:40:200
1779299200,4cyclictest0-21swapper/009:21:280
1779299200,11cyclictest0-21swapper/011:50:330
1780899196,7cyclictest86750irq/128-enp1s0-10:35:193
1780899196,3cyclictest7351-21ps07:55:293
1780899196,3cyclictest23806-21sh09:45:253
1780899196,3cyclictest23806-21sh09:45:253
1780899195,7cyclictest11928-21cut11:45:163
1780899195,4cyclictest4633-21seq09:04:073
1780899195,4cyclictest338-21ps10:05:243
1780899195,2cyclictest0-21swapper/310:30:063
1780899192,5cyclictest30469-21ntp_states10:00:253
1780899191,9cyclictest17698-21expr10:45:183
1780899191,9cyclictest0-21swapper/311:52:493
1780899191,6cyclictest0-21swapper/311:36:003
1780899191,12cyclictest0-21swapper/311:41:533
1780899191,10cyclictest3709-21/usr/sbin/munin10:15:253
1780899191,10cyclictest15367-21tail09:25:273
1780899191,10cyclictest12842-21/usr/sbin/munin08:10:223
17808991910,3cyclictest86850irq/129-enp1s0-09:40:123
1780899190,3cyclictest0-21swapper/311:25:273
1780399197,3cyclictest21490-21cat08:29:562
1780399191,9cyclictest86850irq/129-enp1s0-07:10:232
1780399191,9cyclictest21878-21/usr/sbin/munin12:05:272
1780399191,6cyclictest0-21swapper/208:45:182
1779299198,3cyclictest23959-21/usr/sbin/munin08:35:270
1779299198,3cyclictest23959-21/usr/sbin/munin08:35:270
1779299197,4cyclictest0-21swapper/009:49:540
1779299197,4cyclictest0-21swapper/009:49:530
1779299192,9cyclictest784-21NetworkManager11:07:550
1779299192,2cyclictest86550irq/126-enp1s0-10:00:300
1779299191,6cyclictest22096-21head10:55:160
1779299191,4cyclictest0-21swapper/011:30:140
17792991914,3cyclictest19828-21latency_hist07:15:010
1779299191,3cyclictest7662-21kworker/u8:010:40:120
1779299191,10cyclictest30394-21/usr/sbin/munin11:15:210
1779299191,10cyclictest0-21swapper/012:09:520
1779299191,10cyclictest0-21swapper/011:59:090
1779299191,10cyclictest0-21swapper/010:05:400
1779299191,10cyclictest0-21swapper/009:52:530
1779299190,4cyclictest0-21swapper/010:39:420
83662181,2sleep00-21swapper/008:00:030
1780899186,3cyclictest28785-21fschecks_count08:45:153
1780899186,3cyclictest12315-21idleruntime09:20:153
1780899185,5cyclictest0-21swapper/312:23:123
1780899185,4cyclictest26182-21/usr/sbin/munin08:40:213
1780899183,3cyclictest836-21runrttasks07:50:193
1780899183,12cyclictest7166-21/usr/sbin/munin09:10:123
1780899181,9cyclictest0-21swapper/308:15:323
1780899181,8cyclictest25637-21hddtemp_smartct09:50:293
1780899181,8cyclictest24066-21diskstats11:00:193
1780899181,8cyclictest1307-21cpuspeed_turbos12:35:113
1780899181,8cyclictest0-21swapper/310:55:233
1780899181,6cyclictest9386-21cut08:00:213
1780899181,5cyclictest31372-21cat11:15:223
17808991812,3cyclictest24489-21grep07:20:333
1780899181,15cyclictest0-21swapper/312:30:083
1780399183,12cyclictest23321-21grep08:30:272
17803991812,3cyclictest6448-21expr07:55:182
1780399181,15cyclictest0-21swapper/208:17:252
1779299185,7cyclictest1075-21kworker/u8:310:12:460
1779299181,9cyclictest0-21swapper/012:16:500
1779299181,9cyclictest0-21swapper/011:01:140
1779299181,11cyclictest0-21swapper/011:26:220
1779299181,11cyclictest0-21swapper/009:56:450
1779299181,10cyclictest16847-21sshd08:17:310
1779299180,6cyclictest0-21swapper/009:12:000
1780899175,3cyclictest28929-21sed07:30:323
1780899171,8cyclictest0-21swapper/309:05:213
1780899171,7cyclictest15633-21/usr/sbin/munin10:40:193
1780899171,7cyclictest0-21swapper/310:20:143
17808991713,3cyclictest0-21swapper/310:26:293
1780899171,2cyclictest0-21swapper/308:05:233
17808991712,3cyclictest25204-21find08:35:283
17808991712,3cyclictest25204-21find08:35:283
17808991711,3cyclictest31954-21sort07:40:153
17808991711,3cyclictest26236-21expr11:05:123
1780899171,10cyclictest0-21swapper/312:11:323
17808991710,4cyclictest22101-21/usr/sbin/munin08:30:163
1780399175,3cyclictest13546-21idleruntime08:10:202
1780399174,3cyclictest86750irq/128-enp1s0-08:55:402
1780399174,2cyclictest86750irq/128-enp1s0-07:30:322
1780399172,1cyclictest0-21swapper/207:21:392
1780399171,5cyclictest5017-21ntp_states07:50:232
1780399171,3cyclictest30486-21kernelversion07:35:242
17803991713,3cyclictest0-21swapper/209:00:092
1780399171,14cyclictest0-21swapper/208:20:052
17803991711,3cyclictest26182-21/usr/sbin/munin08:40:172
1779299172,4cyclictest3311-21sed07:45:310
1779299171,5cyclictest8106-21sshd07:59:300
1779299171,5cyclictest30622-21/usr/sbin/munin08:50:220
1779299171,3cyclictest3469-21if_enp2s009:00:200
17792991712,3cyclictest13715-21sed08:10:220
17792991710,4cyclictest0-21swapper/011:20:260
1780899161,9cyclictest0-21swapper/308:50:183
1780899161,5cyclictest25198-21/usr/sbin/munin07:25:203
1780899161,5cyclictest25198-21/usr/sbin/munin07:25:203
1780899161,13cyclictest1486-21grep08:55:333
17808991610,3cyclictest17367-21cat08:20:163
17803991612,3cyclictest0-21swapper/207:47:332
1779299164,7cyclictest0-21swapper/008:25:180
1779299161,6cyclictest0-21swapper/007:50:210
17792991611,3cyclictest17588-21expr08:20:150
1779299160,6cyclictest0-21swapper/008:45:270
12992161,2sleep20-21swapper/207:42:562
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional