Class: Jets::Gems::Check

Inherits:
Object
  • Object
show all
Extended by:
Memoist
Defined in:
lib/jets/gems/check.rb

Constant Summary collapse

GEM_REGEXP =

note 2.7.0/gems vs 2.7.0/extensions

/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-darwin/
/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/

Also the platform is appended to the gem folder now

/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-darwin
/tmp/jets/demo/stage/opt/ruby/gems/2.7.0/gems/nokogiri-1.11.1-x86_64-linux

On new 2021 macbook with m1 chip: the gems are being saved in a folder like so:

nokogiri-1.12.5-arm64-darwin

The GEM_REGEXP accounts for this case.

/-(arm|x)\d+.*-(darwin|linux)/

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ Check

Returns a new instance of Check.



9
10
11
12
 
# File 'lib/jets/gems/check.rb', line 9

def initialize(options={})
  @options = options
  @missing_gems = [] # keeps track of gems that are not found in any of the serverlessgems source
end

Instance Attribute Details

#missing_gems ⇒ Object (readonly)

Returns the value of attribute missing_gems.



8
9
10
 
# File 'lib/jets/gems/check.rb', line 8

def missing_gems
  @missing_gems
end

Instance Method Details

#agree ⇒ Object 



84
85
86
 
# File 'lib/jets/gems/check.rb', line 84

def agree
  Agree.new
end

#compiled_gem_paths ⇒ Object

Use pre-compiled gem because the gem could have development header shared object file dependencies. The shared dependencies are packaged up as part of the pre-compiled gem so it is available in the Lambda execution environment.

Example paths: Macosx:

opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/nokogiri-1.8.1
opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/byebug-9.1.0

Official AWS Lambda Linux AMI:

opt/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0-static/nokogiri-1.8.1

Circleci Ubuntu based Linux:

opt/ruby/gems/2.5.0/extensions/x86_64-linux/2.5.0/pg-0.21.0



189
190
191
 
# File 'lib/jets/gems/check.rb', line 189

def compiled_gem_paths
  Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/*/extensions/**/**/*.{so,bundle}")
end

#compiled_gems ⇒ Object

Context, observations, and history:

Two ways to check if gem is compiled.

1. compiled_gem_paths - look for .so and .bundle extension files in the folder itself.
2. gemspec - uses the gemspec metadata.

Observations:

* The gemspec approach generally finds more compiled gems than the compiled_gem_paths approach.
* So when using the compiled_gem_paths some compiled are missed and not properly detected like http-parser.
* However, some gemspec found compiled gems like json are weird and they don't work when they get replaced.

History:

* Started with compiled_gem_paths approach
* Tried to gemspec approach, but ran into json-2.1.0 gem issues. bundler removes? http://bit.ly/39T8uln
* Went to selective checking approach with `cli: true` option. This helped gather more data.
  * jets deploy - compiled_gem_paths
  * jets gems:check - gemspec_compiled_gems
* Going back to compiled_gem_paths with:
  * Using the `weird_gem?` check to filter out gems removed by bundler. Note: Only happens with specific versions of json.
* Keeping compiled_gem_paths for Jets Afterburner mode. Default to gemspec_compiled_gems otherwise



114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/jets/gems/check.rb', line 114

def compiled_gems
  # @use_gemspec option  finds compile gems with Gem::Specification
  # The normal build process does not use this and checks the file system.
  # So @use_gemspec is only used for this command:
  #
  #   jets gems:check
  #
  # This is because it seems like some gems like json are remove and screws things up.
  # We'll filter out for the json gem as a hacky workaround, unsure if there are more
  # gems though that exhibit this behavior.
  if @options[:use_gemspec] == false
    # Afterburner mode
    compiled_gems = compiled_gem_paths.map { |p| gem_name_from_path(p) }.uniq
    # Double check that the gems are also in the gemspec list since that
    # one is scoped to Bundler and will only included gems used in the project.
    # This handles the possiblity of stale gems leftover from previous builds
    # in the cache.
    # TODO: figure out if we need
    # compiled_gems.select { |g| gemspec_compiled_gems.include?(g) }
  else
    # default when use_gemspec not set
    #
    #     jets deploy
    #     jets gems:check
    #
    gems = gemspec_compiled_gems
    gems += other_compiled_gems
    gems += registered_compiled_gems
    gems.uniq
  end
end

#gem_name_from_path(path) ⇒ Object

Input: opt/ruby/gems/2.5.0/extensions/x86_64-darwin-16/2.5.0-static/byebug-9.1.0 Output: byebug-9.1.0



195
196
197
198
 
# File 'lib/jets/gems/check.rb', line 195

def gem_name_from_path(path)
  regexp = %r{opt/ruby/gems/\d+\.\d+\.\d+/extensions/.*?/.*?/(.*?)/}
  path.match(regexp)[1] # gem_name
end

#gems_source ⇒ Object 



19
20
21
 
# File 'lib/jets/gems/check.rb', line 19

def gems_source
  ENV['SG_API'] || Jets.config.gems.source
end

#gemspec_compiled_gems ⇒ Object

So can also check for compiled gems with Gem::Specification But then also includes the json gem, which then bundler removes? We’ll figure out the the json gems. gist.github.com/tongueroo/16f4aa5ac5393424103347b0e529495e

This is a faster way to check but am unsure if there are more gems than just json that exhibit this behavior. So only using this technique for this commmand:

jets gems:check

Thanks: gist.github.com/aelesbao/1414b169a79162b1d795 and

https://stackoverflow.com/questions/5165950/how-do-i-get-a-list-of-gems-that-are-installed-that-have-native-extensions



212
213
214
215
216
 
# File 'lib/jets/gems/check.rb', line 212

def gemspec_compiled_gems
  specs = Gem::Specification.each.select { |spec| spec.extensions.any?  }
  specs.reject! { |spec| weird_gem?(spec.name) }
  specs.map(&:full_name)
end

#missing? ⇒ Boolean

Returns:

  • (Boolean) 


46
47
48
 
# File 'lib/jets/gems/check.rb', line 46

def missing?
  !@missing_gems.empty?
end

#missing_message ⇒ Object 



50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/jets/gems/check.rb', line 50

def missing_message
  template = <<-EOL
Your project requires compiled gems that are not currently available.  Unavailable pre-compiled gems:
<% missing_gems.each do |missing_gem|
   available = missing_gem['available'].reject { |v| missing_gem['gem_name'].include?(v) }
%>
* Unavailable: <%= missing_gem['gem_name'] -%> Available versions: <%= available.join(' ') %>
<% end %>
Your current serverlessgems source: #{gems_source}

Jets is unable to build a deployment package that will work on AWS Lambda without the required pre-compiled gems. To remedy this, you can:

* Use another gem that does not require compilation.
* Create your own custom layer with the gem: http://rubyonjets.com/docs/extras/custom-lambda-layers/
<% if agree.yes? -%>
* No need to report this to us, as we've already been notified.
* Usually, missing gems can be built within a few minutes.
* Some gems may take days or even longer.
<% elsif agree.no? -%>
* You have choosen not to report data to serverlessgems so we will not be notified about these missing gems.
* You can edit ~/.jets/agree to change this.
* Reporting gems generally allows Serverless Gems to build the missing gems within a few minutes.
* You can try redeploying again after a few minutes.
* Non-reported gems may take days or even longer to be built.
<% end -%>

Compiled gems usually take some time to figure out how to build as they each depend on different libraries and packages.
More info: http://rubyonjets.com/docs/serverlessgems/

EOL
  erb = ERB.new(template, trim_mode: '-') # trim mode https://stackoverflow.com/questions/4632879/erb-template-removing-the-trailing-line
  erb.result(binding)
end

#other_compiled_gems ⇒ Object 



160
161
162
163
 
# File 'lib/jets/gems/check.rb', line 160

def other_compiled_gems
  paths = Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/#{Jets::Gems.ruby_folder}/gems/*{-darwin,-linux}")
  paths.map { |p| File.basename(p).sub(GEM_REGEXP,'') }
end

#registered_compiled_gems ⇒ Object 



165
166
167
168
169
170
171
172
173
174
175
 
# File 'lib/jets/gems/check.rb', line 165

def registered_compiled_gems
  registered = Jets::Gems::Registered.new
  registered_gems = registered.all # no version numbers in this list

  paths = Dir.glob("#{Jets.build_root}/stage/opt/ruby/gems/#{Jets::Gems.ruby_folder}/gems/*")
  project_gems = paths.map { |p| File.basename(p).sub(GEM_REGEXP,'') }
  project_gems.select do |name|
    name_only = name.sub(/-\d+\.\d+\.\d+.*/,'')
    registered_gems.include?(name_only)
  end
end

#run(exit_early: false) ⇒ Object

Checks whether the gem is found at the serverlessgems source.



24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/jets/gems/check.rb', line 24

def run(exit_early: false)
  puts "Checking projects gems for binary serverlessgems..."
  compiled_gems.each do |gem_name|
    puts "Checking #{gem_name}..." if @options[:verbose]
    exist = Jets::Gems::Exist.new
    data = exist.check(gem_name)
    @missing_gems << data unless data["exist"]
  end

  if exit_early && !@missing_gems.empty?
    # Exits early if not all the linux gems are available.
    # Better to error now than deploy a broken package to AWS Lambda.
    # Provide users with message about using their own serverlessgems source.
    puts missing_message
    names = @missing_gems.map {|i| i['gem_name']}
    Report.new(@options).report(names) if agree.yes?
    exit 1
  end

  compiled_gems
end

#run! ⇒ Object 



14
15
16
17
 
# File 'lib/jets/gems/check.rb', line 14

def run!
  puts "Gems source: #{gems_source}" if @options[:verbose]
  run(exit_early: true)
end

#weird_gem?(name) ⇒ Boolean

Filter out the weird special case gems that bundler deletes? Probably to fix some bug.

$ bundle show json
The gem json has been deleted. It was installed at:
/home/ec2-user/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/json-2.1.0

Returns:

  • (Boolean) 


225
226
227
228
229
 
# File 'lib/jets/gems/check.rb', line 225

def weird_gem?(name)
  command = "bundle show #{name} 2>&1"
  output = `#{command}`
  output.include?("has been deleted")
end