Class: Jets::Gems::Check

Inherits:
Object
  • Object
show all
Extended by:
Memoist
Defined in:
lib/jets/gems/check.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ Check

Returns a new instance of Check.



9
10
11
12
 
# File 'lib/jets/gems/check.rb', line 9

def initialize(options={})
  @options = options
  @missing_gems = [] # keeps track of gems that are not found in any of the serverlessgems source
end

Instance Attribute Details

#missing_gems ⇒ Object (readonly)

Returns the value of attribute missing_gems.



8
9
10
 
# File 'lib/jets/gems/check.rb', line 8

def missing_gems
  @missing_gems
end

Instance Method Details

#agree ⇒ Object 



79
80
81
 
# File 'lib/jets/gems/check.rb', line 79

def agree
  Agree.new
end

#compiled_gem_paths ⇒ Object

Use pre-compiled gem because the gem could have development header shared object file dependencies. The shared dependencies are packaged up as part of the pre-compiled gem so it is available in the Lambda execution environment.

Example paths: Macosx:

opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/nokogiri-1.8.1
opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/byebug-9.1.0

Official AWS Lambda Linux AMI:

opt/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0-static/nokogiri-1.8.1

Circleci Ubuntu based Linux:

opt/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0/pg-0.21.0



167
168
169
 
# File 'lib/jets/gems/check.rb', line 167

def compiled_gem_paths
  Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/*/extensions/**/**/*.{so,bundle}")
end

#compiled_gems ⇒ Object

Context, observations, and history:

Two ways to check if gem is compiled.

1. compiled_gem_paths - look for .so and .bundle extension files in the folder itself.
2. gemspec - uses the gemspec metadata.

Observations:

* The gemspec approach generally finds more compiled gems than the compiled_gem_paths approach.
* So when using the compiled_gem_paths some compiled are missed and not properly detected like http-parser.
* However, some gemspec found compiled gems like json are weird and they don't work when they get replaced.

History:

* Started with compiled_gem_paths approach
* Tried to gemspec approach, but ran into json-2.1.0 gem issues. bundler removes? http://bit.ly/39T8uln
* Went to selective checking approach with `cli: true` option. This helped gather more data.
  * jets deploy - compiled_gem_paths
  * jets gems:check - gemspec_compiled_gems
* Going back to compiled_gem_paths with:
  * Using the `weird_gem?` check to filter out gems removed by bundler. Note: Only happens with specific versions of json.
* Keeping compiled_gem_paths for Jets Afterburner mode. Default to gemspec_compiled_gems otherwise



109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/jets/gems/check.rb', line 109

def compiled_gems
  # @use_gemspec option  finds compile gems with Gem::Specification
  # The normal build process does not use this and checks the file system.
  # So @use_gemspec is only used for this command:
  #
  #   jets gems:check
  #
  # This is because it seems like some gems like json are remove and screws things up.
  # We'll filter out for the json gem as a hacky workaround, unsure if there are more
  # gems though that exhibit this behavior.
  if @options[:use_gemspec] == false
    # Afterburner mode
    compiled_gems = compiled_gem_paths.map { |p| gem_name_from_path(p) }.uniq
    # Double check that the gems are also in the gemspec list since that
    # one is scoped to Bundler and will only included gems used in the project.
    # This handles the possiblity of stale gems leftover from previous builds
    # in the cache.
    # TODO: figure out if we need
    # compiled_gems.select { |g| gemspec_compiled_gems.include?(g) }
  else
    # default when use_gemspec not set
    #
    #     jets deploy
    #     jets gems:check
    #
    gems = gemspec_compiled_gems
    gems += other_compiled_gems
    gems.uniq
  end
end

#gem_name_from_path(path) ⇒ Object

Input: opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/byebug-9.1.0 Output: byebug-9.1.0



173
174
175
176
 
# File 'lib/jets/gems/check.rb', line 173

def gem_name_from_path(path)
  regexp = %r{opt/ruby/gems/\d+\.\d+\.\d+/extensions/.*?/.*?/(.*?)/}
  path.match(regexp)[1] # gem_name
end

#gems_source ⇒ Object 



19
20
21
 
# File 'lib/jets/gems/check.rb', line 19

def gems_source
  ENV['SG_API'] || Jets.config.gems.source
end

#gemspec_compiled_gems ⇒ Object

So can also check for compiled gems with Gem::Specification But then also includes the json gem, which then bundler removes? We’ll figure out the the json gems. gist.github.com/tongueroo/16f4aa5ac5393424103347b0e529495e

This is a faster way to check but am unsure if there are more gems than just json that exhibit this behavior. So only using this technique for this commmand:

jets gems:check

Thanks: gist.github.com/aelesbao/1414b169a79162b1d795 and

https://stackoverflow.com/questions/5165950/how-do-i-get-a-list-of-gems-that-are-installed-that-have-native-extensions



190
191
192
193
194
 
# File 'lib/jets/gems/check.rb', line 190

def gemspec_compiled_gems
  specs = Gem::Specification.each.select { |spec| spec.extensions.any?  }
  specs.reject! { |spec| weird_gem?(spec.name) }
  specs.map(&:full_name)
end

#missing? ⇒ Boolean

Returns:

  • (Boolean) 


44
45
46
 
# File 'lib/jets/gems/check.rb', line 44

def missing?
  !@missing_gems.empty?
end

#missing_message ⇒ Object 



48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/jets/gems/check.rb', line 48

def missing_message
  template = "Your project requires compiled gems that are not currently available.  Unavailable pre-compiled gems:\n<% missing_gems.each do |gem| %>\n* <%= gem -%>\n<% end %>\n\nYour current serverlessgems source: \#{gems_source}\n\nJets is unable to build a deployment package that will work on AWS Lambda without the required pre-compiled gems. To remedy this, you can:\n\n* Use another gem that does not require compilation.\n* Create your own custom layer with the gem: http://rubyonjets.com/docs/extras/custom-lambda-layers/\n<% if agree.yes? -%>\n* No need to report this to us, as we've already been notified.\n<% elsif agree.no? -%>\n* You have choosen not to report data to serverlessgems so we will not be notified about these missing gems.\n* You can edit ~/.jets/agree to change this.\n* Reporting gems generally allows Serverless Gems to build the missing gems within a few minutes.\n* You can try redeploying again after a few minutes.\n* Non-reported gems may take days or even longer to be built.\n<% end -%>\n\nCompiled gems usually take some time to figure out how to build as they each depend on different libraries and packages.\nMore info: http://rubyonjets.com/docs/serverlessgems/\n\n"
  erb = ERB.new(template, nil, '-') # trim mode https://stackoverflow.com/questions/4632879/erb-template-removing-the-trailing-line
  erb.result(binding)
end

#other_compiled_gems ⇒ Object

note 2.7.0/gems vs 2.7.0/extensions

/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-darwin/
/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/

Also the platform is appended to the gem folder now

/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-darwin
/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-linux



150
151
152
153
 
# File 'lib/jets/gems/check.rb', line 150

def other_compiled_gems
  paths = Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/#{Jets::Gems.ruby_folder}/gems/*{-darwin,-linux}")
  paths.map { |p| File.basename(p).sub(/-x\d+.*-(darwin|linux)/,'') }
end

#run(exit_early: false) ⇒ Object

Checks whether the gem is found at the serverlessgems source.



24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/jets/gems/check.rb', line 24

def run(exit_early: false)
  puts "Checking projects gems for binary serverlessgems..."
  compiled_gems.each do |gem_name|
    puts "Checking #{gem_name}..." if @options[:cli]
    exist = Jets::Gems::Exist.new
    @missing_gems << gem_name unless exist.check(gem_name)
  end

  if exit_early && !@missing_gems.empty?
    # Exits early if not all the linux gems are available.
    # Better to error now than deploy a broken package to AWS Lambda.
    # Provide users with message about using their own serverlessgems source.
    puts missing_message
    Report.new(@options).report(@missing_gems) if agree.yes?
    exit 1
  end

  compiled_gems
end

#run! ⇒ Object 



14
15
16
17
 
# File 'lib/jets/gems/check.rb', line 14

def run!
  puts "Gems source: #{gems_source}" if @options[:show_source]
  run(exit_early: true)
end

#weird_gem?(name) ⇒ Boolean

Filter out the weird special case gems that bundler deletes? Probably to fix some bug.

$ bundle show json
The gem json has been deleted. It was installed at:
/home/ec2-user/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/json-2.1.0

Returns:

  • (Boolean) 


203
204
205
206
207
 
# File 'lib/jets/gems/check.rb', line 203

def weird_gem?(name)
  command = "bundle show #{name} 2>&1"
  output = `#{command}`
  output.include?("has been deleted")
end