Class: Jets::Builders::CodeBuilder

Inherits:
Object
  • Object
show all
Extended by:
Memoist
Includes:
AwsServices, Util
Defined in:
lib/jets/builders/code_builder.rb

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from AwsServices

#apigateway, #aws_lambda, #aws_options, #cfn, #dynamodb, #logs, #s3, #s3_resource, #sns, #sqs, #sts

Methods included from AwsServices::StackStatus

#lookup, #stack_exists?, #stack_in_progress?

Methods included from AwsServices::GlobalMemoist

included

Constructor Details

#initialize ⇒ CodeBuilder

Returns a new instance of CodeBuilder.



21
22
23
24
25
26
 
# File 'lib/jets/builders/code_builder.rb', line 21

def initialize
  # Expanding to the full path and capture now.
  # Dir.chdir gets called later and we'll lose this info.
  @full_project_path = File.expand_path(Jets.root) + "/"
  @version_purger = Purger.new
end

Instance Attribute Details

#full_project_path ⇒ Object (readonly)

Returns the value of attribute full_project_path.



20
21
22
 
# File 'lib/jets/builders/code_builder.rb', line 20

def full_project_path
  @full_project_path
end

Class Method Details

.tmp_code ⇒ Object

Group all the path settings together here



344
345
346
 
# File 'lib/jets/builders/code_builder.rb', line 344

def self.tmp_code
  Jets::Cfn::Builder.tmp_code
end

Instance Method Details

#asset_host ⇒ Object

Different url for these. Examples:

asset_host  https://demo-dev-s3bucket-lw5vq7ht8ip4.s3.us-west-2.amazonaws.com/jets/public/packs/media/images/boltops-0dd1c6bd.png
s3_base_url https://s3-us-west-2.amazonaws.com/demo-dev-s3bucket-lw5vq7ht8ip4/jets/packs/media/images/boltops-0dd1c6bd.png

Interesting: asset_host works but s3_base_url does not for CORs. IE: reactjs or vuejs requests Thinking AWS configures the non-subdomain url endpoint to be more restrictive.

Allow user to set assets.asset_host

Jets.application.configure do
  config.assets.asset_host = "https://cloudfront.com/my/base/path"
end



213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/jets/builders/code_builder.rb', line 213

def asset_host
  assets = Jets.config.assets
  return assets.base_url if assets.base_url && assets.base_url != "s3_endpoint"

  # By default, will use the s3 url endpoint directly by convention
  region = Jets.aws.region
  asset_base_url = region == 'us-east-1' ?
    "https://#{s3_bucket}.s3.amazonaws.com" :
    "https://#{s3_bucket}.s3.#{region}.amazonaws.com"

  "#{asset_base_url}/jets/public" # s3_base_url
end

#assets_precompile ⇒ Object 



169
170
171
172
173
 
# File 'lib/jets/builders/code_builder.rb', line 169

def assets_precompile
  return if skip_assets?
  return unless gemfile_include?("sprockets-jets")
  sh "jets assets:precompile"
end

#build ⇒ Object 



28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/jets/builders/code_builder.rb', line 28

def build
  @version_purger.purge
  cache_check_message

  clean_start
  assets_precompile
  run_webpack # easier to do before we copy the project because node and yarn has been likely setup in the that dir
  copy_project
  copy_ruby_version_file
  Dir.chdir("#{stage_area}/code") do
    # These commands run from project root
    code_setup
    package_ruby
    code_finish
  end
end

#build_lambda_layer ⇒ Object 



325
326
327
328
329
 
# File 'lib/jets/builders/code_builder.rb', line 325

def build_lambda_layer
  return unless Jets.gem_layer?
  lambda_layer = LambdaLayer.new
  lambda_layer.build
end

#cache_check_message ⇒ Object 



331
332
333
334
335
 
# File 'lib/jets/builders/code_builder.rb', line 331

def cache_check_message
  if File.exist?("#{Jets.build_root}/cache")
    puts "The #{Jets.build_root}/cache folder exists. Incrementally re-building the jets using the cache.  To clear the cache: rm -rf #{Jets.build_root}/cache"
  end
end

#calculate_md5s ⇒ Object

Resolves the chicken-and-egg problem with md5 checksums. The handlers need to reference files with the md5 checksum. The files are the:

jets/code/rack-checksum.zip
jets/code/opt-checksum.zip

We compute the checksums before we generate the node shim handlers.



52
53
54
 
# File 'lib/jets/builders/code_builder.rb', line 52

def calculate_md5s
  Md5.compute! # populates Md5.checksums hash
end

#check_agree ⇒ Object 



320
321
322
323
 
# File 'lib/jets/builders/code_builder.rb', line 320

def check_agree
  agree = Jets::Api::Agree.new
  agree.prompt
end

#check_code_size! ⇒ Object 



115
116
117
 
# File 'lib/jets/builders/code_builder.rb', line 115

def check_code_size!
  CodeSize.check!
end

#clean_start ⇒ Object

Cleans out non-cached files like code-*.zip in Jets.build_root for a clean start. Also ensure that the /tmp/jets/project build root exists.

Most files are kept around after the build process for inspection and debugging. So we have to clean out the files. But we only want to clean out some of the files.



244
245
246
247
 
# File 'lib/jets/builders/code_builder.rb', line 244

def clean_start
  Dir.glob("#{Jets.build_root}/code/code-*.zip").each { |f| FileUtils.rm_f(f) }
  FileUtils.mkdir_p(Jets.build_root) # /tmp/jets/demo
end

#code_finish ⇒ Object 



94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/jets/builders/code_builder.rb', line 94

def code_finish
  # Reconfigure code
  store_s3_base_url
  disable_webpacker_middleware

  # Code prep and zipping
  check_code_size!
  calculate_md5s # must be called before create_zip_files and generate_shims because checksums need to be populated
  # generate_shims and create_zip_files use checksums
  #
  # Notes:
  #
  # Had moved calculate_md5s to fix a what thought was a subtle issue https://github.com/tongueroo/jets/pull/424
  # But am unsure about that the fix now. This essentially reverts that issue.
  #
  # Fix in https://github.com/tongueroo/jets/pull/459
  #
  generate_shims # the generated handlers/data.yml has rack_zip key
  create_zip_files
end

#code_setup ⇒ Object 



90
91
92
 
# File 'lib/jets/builders/code_builder.rb', line 90

def code_setup
  reconfigure_development_webpacker
end

#copy_project ⇒ Object

Copy project into temporary directory. Do this so we can keep the project directory untouched and we can also remove a bunch of unnecessary files like logs before zipping it up.



252
253
254
255
256
257
258
259
260
261
262
263
 
# File 'lib/jets/builders/code_builder.rb', line 252

def copy_project
  headline "Copying current project directory to temporary build area: #{"#{stage_area}/code"}"
  FileUtils.rm_rf("#{build_area}/stage") # clear out from previous build's stage area
  FileUtils.mkdir_p("#{build_area}/stage")
  FileUtils.rm_rf("#{stage_area}/code") # remove current code folder
  move_node_modules(Jets.root, Jets.build_root)
  begin
    Jets::Util.cp_r(@full_project_path, "#{stage_area}/code")
  ensure
    move_node_modules(Jets.build_root, Jets.root) # move node_modules directory back
  end
end

#copy_ruby_version_file ⇒ Object 



337
338
339
340
341
 
# File 'lib/jets/builders/code_builder.rb', line 337

def copy_ruby_version_file
  ruby_version_path = Jets.root.join(".ruby-version")
  return unless File.exists?(ruby_version_path)
  FileUtils.cp_r(ruby_version_path, build_area)
end

#create_zip_files ⇒ Object 



64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/jets/builders/code_builder.rb', line 64

def create_zip_files
  folders = Md5.stage_folders
  # Md5.stage_folders ["stage/bundled", "stage/code"]
  folders.each do |folder|
    zip = Md5Zip.new(folder)
    if exist_on_s3?(zip.md5_name)
      puts "Already exists: s3://#{s3_bucket}/jets/code/#{zip.md5_name}"
    else
      zip = Md5Zip.new(folder)
      zip.create
    end
  end
end

#dir_size(folder) ⇒ Object

Thanks stackoverflow.com/questions/9354595/recursively-getting-the-size-of-a-directory Seems to overestimate a little bit but close enough.



121
122
123
124
125
126
 
# File 'lib/jets/builders/code_builder.rb', line 121

def dir_size(folder)
  Dir.glob(File.join(folder, '**', '*'))
    .select { |f| File.file?(f) }
    .map{ |f| File.size(f) }
    .inject(:+)
end

#disable_webpacker_middleware ⇒ Object 



163
164
165
166
167
 
# File 'lib/jets/builders/code_builder.rb', line 163

def disable_webpacker_middleware
  full_path = "#{"#{stage_area}/code"}/config/disable-webpacker-middleware.txt"
  FileUtils.mkdir_p(File.dirname(full_path))
  FileUtils.touch(full_path)
end

#exist_on_s3?(filename) ⇒ Boolean

Returns:

  • (Boolean) 


78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/jets/builders/code_builder.rb', line 78

def exist_on_s3?(filename)
  return false if ENV['JETS_NO_INTERNET']
  s3_key = "jets/code/#{filename}"
  begin
    Jets.logger.debug "Checking s3://#{s3_bucket}/#{s3_key}"
    s3.head_object(bucket: s3_bucket, key: s3_key)
    true
  rescue Aws::S3::Errors::NotFound, Aws::S3::Errors::Forbidden
    false
  end
end

#gemfile_include?(name) ⇒ Boolean

Returns:

  • (Boolean) 


226
227
228
229
230
231
232
233
234
235
236
 
# File 'lib/jets/builders/code_builder.rb', line 226

def gemfile_include?(name)
  # Old code, leaving around for now:
  # Thanks: https://stackoverflow.com/questions/4195735/get-list-of-gems-being-used-by-a-bundler-project
  # webpacker_loaded = Gem.loaded_specs.keys.include?("webpacker")
  # return unless webpacker_loaded

  # Checking this way because when using jets standalone for Afterburner mode we don't want to run into
  # bundler gem collisions.  TODO: figure out the a better way to handle the collisions.
  lines = IO.readlines("#{Jets.root}/Gemfile")
  lines.detect { |l| l =~ /gem ['"]#{name}/ && l !~ /^\s*?#/ }
end

#generate_shims ⇒ Object 



56
57
58
59
60
61
62
 
# File 'lib/jets/builders/code_builder.rb', line 56

def generate_shims
  headline "Generating shims in the handlers folder."
  # Crucial that the Dir.pwd is in the tmp_code because for
  # Jets::Builders::app_files because Jets.boot set ups
  # autoload_paths and this is how project classes are loaded.
  Jets::Builders::HandlerGenerator.build!
end

#move_node_modules(source_folder, dest_folder) ⇒ Object

Move the node modules to the tmp build folder to speed up project copying. A little bit risky because a ctrl-c in the middle of the project copying results in a missing node_modules but user can easily rebuild that.

Tesing shows 6.623413 vs 0.027754 speed improvement.



270
271
272
273
274
275
276
 
# File 'lib/jets/builders/code_builder.rb', line 270

def move_node_modules(source_folder, dest_folder)
  source = "#{source_folder}/node_modules"
  dest = "#{dest_folder}/node_modules"
  if File.exist?(source)
    FileUtils.mv(source, dest)
  end
end

#package_ruby ⇒ Object 



304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
 
# File 'lib/jets/builders/code_builder.rb', line 304

def package_ruby
  if ENV['JETS_SKIP_PACKAGE']
    puts "Skip packaging ruby".color(:yellow) # useful for developing handlers
    return
  end
  return unless Jets.gem_layer?

  check_agree
  ruby_packager.install
  rack_packager.install
  ruby_packager.finish # by this time we have a /tmp/jets/demo/stage/code/vendor/gems
  rack_packager.finish

  build_lambda_layer
end

#rack_packager ⇒ Object 



299
300
301
 
# File 'lib/jets/builders/code_builder.rb', line 299

def rack_packager
  RackPackager.new("#{tmp_code}/rack")
end

#reconfigure_development_webpacker ⇒ Object

Bit hacky but this saves the user from accidentally forgetting to change this when they deploy a jets project in development mode



280
281
282
283
284
285
286
287
288
289
290
291
292
 
# File 'lib/jets/builders/code_builder.rb', line 280

def reconfigure_development_webpacker
  return unless Jets.env.development?
  return unless gemfile_include?("jetpacker")
  headline "Reconfiguring webpacker development settings for AWS Lambda."

  webpacker_yml = "#{"#{stage_area}/code"}/config/webpacker.yml"
  return unless File.exist?(webpacker_yml)

  config = Jets::Util::Yamler.load_file(webpacker_yml)
  config["development"]["compile"] = false # force this to be false for deployment
  new_yaml = YAML.dump(config)
  IO.write(webpacker_yml, new_yaml)
end

#ruby_packager ⇒ Object 



294
295
296
 
# File 'lib/jets/builders/code_builder.rb', line 294

def ruby_packager
  RubyPackager.new(tmp_code)
end

#run_webpack ⇒ Object

This happens in the current app directory not the tmp code for simplicity. This is because the node and yarn has likely been set up correctly there.



177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/jets/builders/code_builder.rb', line 177

def run_webpack
  return if skip_assets?
  return unless gemfile_include?("jetpacker")

  headline "Compling assets in current project directory"
  sh("yarn install")

  ENV['WEBPACKER_ASSET_HOST'] = asset_host if Jets.config.assets.base_url
  webpack_command = File.exist?("#{Jets.root}/bin/webpack") ?
      "bin/webpack" :
      `which webpack`.strip
  sh "JETS_ENV=#{Jets.env} #{webpack_command}"
end

#s3_base_url ⇒ Object 



143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
 
# File 'lib/jets/builders/code_builder.rb', line 143

def s3_base_url
  # Allow user to set assets.base_url
  #
  #   Jets.application.configure do
  #     config.assets.base_url = "https://cloudfront.com/my/base/path"
  #   end
  #
  return Jets.config.assets.base_url if Jets.config.assets.base_url

  # Note: subdomain form works with CORs but the subfolder form does not. Using subfolder form.
  region = Jets.aws.region
  asset_base_url = region == 'us-east-1' ?
    "https://#{s3_bucket}.s3.amazonaws.com/jets" :
    "https://#{s3_bucket}.s3-#{region}.amazonaws.com/jets"
end

#s3_bucket ⇒ Object 



159
160
161
 
# File 'lib/jets/builders/code_builder.rb', line 159

def s3_bucket
  Jets.aws.s3_bucket
end

#skip_assets? ⇒ Boolean

Returns:

  • (Boolean) 


191
192
193
194
195
196
197
 
# File 'lib/jets/builders/code_builder.rb', line 191

def skip_assets?
  if ENV['JETS_SKIP_ASSETS']
    puts "Skip compiling assets".color(:yellow) # useful for debugging
    return true
  end
  Jets.config.mode == "job"
end

#store_s3_base_url ⇒ Object

Store s3 base url is needed for asset serving from s3 later. Need to package this as part of the code so we have a reference to it. At this point the minimal stack exists, so we can grab it with the AWS API. We do not want to grab this as part of the live request because it is slow.



132
133
134
135
136
 
# File 'lib/jets/builders/code_builder.rb', line 132

def store_s3_base_url
  return if Jets.config.mode == "job"
  return unless gemfile_include?("sprockets-jets")
  write_s3_base_url("#{stage_area}/code/config/s3_base_url.txt")
end

#tmp_code ⇒ Object 



348
349
350
 
# File 'lib/jets/builders/code_builder.rb', line 348

def tmp_code
  self.class.tmp_code
end

#write_s3_base_url(full_path) ⇒ Object 



138
139
140
141
 
# File 'lib/jets/builders/code_builder.rb', line 138

def write_s3_base_url(full_path)
  FileUtils.mkdir_p(File.dirname(full_path))
  IO.write(full_path, s3_base_url)
end