Module: Vanagon::Utilities

Extended by:
Utilities
Included in:
Component, Component::Source::Http, Driver, Platform, Project, Utilities
Defined in:
lib/vanagon/utilities.rb,
lib/vanagon/utilities/shell_utilities.rb,
lib/vanagon/utilities/extra_files_signer.rb

Defined Under Namespace

Modules: ExtraFilesSigner, ShellUtilities

Instance Method Summary collapse

Instance Method Details

#erb_file(erbfile, outfile = nil, remove_orig = false, opts = { :binding => binding }) ⇒ Object

Helper method that takes a template and writes the evaluated contents to a file on disk

Parameters:

  • erbfile (String)
  • outfile (String) (defaults to: nil)
  • remove_orig (true, false) (defaults to: false)
  • opts (Hash) (defaults to: { :binding => binding }) 


305
306
307
308
309
310
311
312
 
# File 'lib/vanagon/utilities.rb', line 305

def erb_file(erbfile, outfile = nil, remove_orig = false, opts = { :binding => binding })
  outfile ||= File.join(Dir.mktmpdir, File.basename(erbfile).sub(File.extname(erbfile), ""))
  output = erb_string(erbfile, opts[:binding])
  File.open(outfile, 'w') { |f| f.write output }
  VanagonLogger.info "Generated: #{outfile}"
  FileUtils.rm_rf erbfile if remove_orig
  outfile
end

#erb_string(erbfile, b = binding) ⇒ String

Helper method that takes a template file and runs it through ERB

Parameters:

  • erbfile (String) —

    template to be evaluated

  • b (Binding) (defaults to: binding) —

    binding to evaluate the template under

Returns:

  • (String) —

    the evaluated template



291
292
293
294
295
296
297
 
# File 'lib/vanagon/utilities.rb', line 291

def erb_string(erbfile, b = binding)
  template = File.read(erbfile)
  message  = ERB.new(template, trim_mode: "-")
  message.result(b)
    .gsub(/[\n]+{3,}/, "\n\n")
    .squeeze("\s")
end

#ex(command) ⇒ String

Similar to rake’s sh, the passed command will be executed and an exception will be raised on command failure. However, in contrast to rake’s sh, this method returns the output of the command instead of a boolean.

Parameters:

  • command (String) —

    The command to be executed

Returns:

  • (String) —

    The standard output of the executed command

Raises:



121
122
123
124
125
126
127
 
# File 'lib/vanagon/utilities.rb', line 121

def ex(command)
  ret = %x(#{command})
  unless $CHILD_STATUS.success?
    raise Vanagon::Error, "'#{command}' did not succeed"
  end
  ret
end

#find_program_on_path(command, required = true) ⇒ String, false Also known as: which

Similar to the command-line utility which, the method will search the PATH for the passed command and return the full path to the command if it exists.

Parameters:

  • command (String) —

    Command to search for on PATH

  • required (true, false) (defaults to: true) —

    Whether or not to raise an exception if the command cannot be found

Returns:

  • (String, false) —

    Returns either the full path to the command or false if the command cannot be found

Raises:

  • (RuntimeError) —

    If the command is required and cannot be found



137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
 
# File 'lib/vanagon/utilities.rb', line 137

def find_program_on_path(command, required = true)
  extensions = ENV['PATHEXT'] ? ENV['PATHEXT'].split(';') : ['']
  ENV['PATH'].split(File::PATH_SEPARATOR).each do |path_elem|
    extensions.each do |ext|
      location = File.join(path_elem, "#{command}#{ext}")
      return location if FileTest.executable?(location)
    end
  end

  if required
    fail "Could not find '#{command}'. Please install (or ensure it is on $PATH), and try again."
  else
    return false
  end
end

#get_md5sum(file) ⇒ String

Deprecated.

Please use #get_sum instead, this will be removed in a future vanagon release.

Utility to get the md5 sum of a file

Parameters:

  • file (String) —

    file to md5sum

Returns:

  • (String) —

    md5sum of the given file



24
25
26
 
# File 'lib/vanagon/utilities.rb', line 24

def get_md5sum(file)
  get_sum(file, 'md5')
end

#get_sum(file, type = 'md5') ⇒ String

Generic file summing utility

Parameters:

  • file (String) —

    file to sum

  • type (String) (defaults to: 'md5') —

    type of sum to provide, defaults to md5

Returns:

  • (String) —

    sum of the given file

Raises:

  • (RuntimeError) —

    raises an exception if the given sum type is not supported



34
35
36
37
38
39
40
41
 
# File 'lib/vanagon/utilities.rb', line 34

def get_sum(file, type = 'md5')
  Digest.const_get(type.upcase).file(file).hexdigest.to_s

# If Digest::const_get fails, it'll raise a LoadError when it tries to
# pull in the subclass `type`. We catch that error, and fail instead.
rescue LoadError
  fail "Don't know how to produce a sum of type: '#{type}' for '#{file}'"
end

#http_request(url, type, payload = {}.to_json, header = nil) ⇒ Hash

uses http_request_generic and returns the body as parsed by JSON. body cannot be parsed as JSON

Parameters:

  • url (String) —

    The url to make the request against (needs to be parsable by URI

  • type (String) —

    One of the supported request types (currently ‘get’, ‘post’, ‘delete’)

  • payload (String) (defaults to: {}.to_json) —

    The request body data payload used for POST and PUT

  • header (Hash) (defaults to: nil) —

    Send additional information in the HTTP request header

Returns:

  • (Hash) —

    The response in JSON format

Raises:

  • (RuntimeError, Vanagon::Error) —

    an exception is raised if the response



95
96
97
98
99
100
 
# File 'lib/vanagon/utilities.rb', line 95

def http_request(url, type, payload = {}.to_json, header = nil)
  response = http_request_generic(url, type, payload, header)
  JSON.parse(response.body)
rescue JSON::ParserError => e
  raise Vanagon::Error.wrap(e, "#{url} handed us a response that doesn't look like JSON.")
end

#http_request_code(url, type, payload = {}.to_json, header = nil) ⇒ String

uses http_request_generic and returns the response code.

Parameters:

  • url (String) —

    The url to make the request against (needs to be parsable by URI

  • type (String) —

    One of the supported request types (currently ‘get’, ‘post’, ‘delete’)

  • payload (String) (defaults to: {}.to_json) —

    The request body data payload used for POST and PUT

  • header (Hash) (defaults to: nil) —

    Send additional information in the HTTP request header

Returns:

  • (String) —

    The response code eg 202, 200 etc



108
109
110
111
 
# File 'lib/vanagon/utilities.rb', line 108

def http_request_code(url, type, payload = {}.to_json, header = nil)
  response = http_request_generic(url, type, payload, header)
  response.code
end

#http_request_generic(url, type, payload = {}.to_json, header = nil) ⇒ Net::HTTPAccepted

Simple wrapper around Net::HTTP. Will make a request of the given type to the given url and return the response object

action is not supported, or if there is a problem with the http request

Parameters:

  • url (String) —

    The url to make the request against (needs to be parsable by URI

  • type (String) —

    One of the supported request types (currently ‘get’, ‘post’, ‘delete’)

  • payload (String) (defaults to: {}.to_json) —

    The request body data payload used for POST and PUT

  • header (Hash) (defaults to: nil) —

    Send additional information in the HTTP request header

Returns:

  • (Net::HTTPAccepted) —

    The response object

Raises:



53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/vanagon/utilities.rb', line 53

def http_request_generic(url, type, payload = {}.to_json, header = nil) # rubocop:disable Metrics/AbcSize
  uri = URI.parse(url)
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true if uri.scheme == 'https'

  case type.downcase
  when "get"
    request = Net::HTTP::Get.new(uri.request_uri)
  when "post"
    request = Net::HTTP::Post.new(uri.request_uri)
    request.body = payload
  when "put"
    request = Net::HTTP::Put.new(uri.request_uri)
    request.body = payload
  when "delete"
    request = Net::HTTP::Delete.new(uri.request_uri)
  else
    fail "ACTION: #{type} not supported by #http_request method. Maybe you should add it?"
  end

  # Add any headers to the request
  if header && header.is_a?(Hash)
    header.each do |key, val|
      request[key] = val
    end
  end

  http.request(request)
rescue Errno::ETIMEDOUT, Timeout::Error, Errno::EINVAL, Errno::ECONNRESET,
  EOFError, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError,
  Net::ProtocolError => e
  raise Vanagon::Error.wrap(e, "Problem reaching #{url}. Is #{uri.host} down?")
end

#local_command(command, return_command_output: false) ⇒ true, String

Runs the command on the local host

Parameters:

  • command (String) —

    command to run locally

  • return_command_output (Boolean) (defaults to: false) —

    whether or not command output should be returned

Returns:

  • (true, String) —

    Returns true if the command was successful or the output of the command if return_command_output is true

Raises:

  • (RuntimeError) —

    If the command fails an exception is raised



263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/vanagon/utilities.rb', line 263

def local_command(command, return_command_output: false)
  clean_environment do
    VanagonLogger.info "Executing '#{command}' locally"
    if return_command_output
      ret = %x(#{command}).chomp
      if $CHILD_STATUS.success?
        return ret
      else
        raise "Local command (#{command}) failed."
      end
    else
      Kernel.system(command)
      $CHILD_STATUS.success? or raise "Local command (#{command}) failed."
    end
  end
end

#remote_ssh_command(target, command, port = 22, return_command_output: false) ⇒ true, String

Runs the command on the given host via ssh call

Parameters:

  • target (String) —

    ssh host to run command on (user@machine)

  • command (String) —

    command to run on the target

  • port (Integer) (defaults to: 22) —

    port number for ssh (default 22)

  • return_command_output (Boolean) (defaults to: false) —

    whether or not command output should be returned

Returns:

  • (true, String) —

    Returns true if the command was successful or the output of the command if return_command_output is true

Raises:

  • (RuntimeError) —

    If there is no target given or the command fails an exception is raised



241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/vanagon/utilities.rb', line 241

def remote_ssh_command(target, command, port = 22, return_command_output: false)
  VanagonLogger.info "Executing '#{command}' on '#{target}'"
  if return_command_output
    ret = %x(#{ssh_command(port)} -T #{target} '#{command.gsub("'", "'\\\\''")}').chomp
    if $CHILD_STATUS.success?
      return ret
    else
      raise "Remote ssh command (#{command}) failed on '#{target}'."
    end
  else
    Kernel.system("#{ssh_command(port)} -T #{target} '#{command.gsub("'", "'\\\\''")}'")
    $CHILD_STATUS.success? or raise "Remote ssh command (#{command}) failed on '#{target}'."
  end
end

#retry_with_timeout(tries = 5, timeout = 1, &blk) ⇒ true

Method to retry a ruby block and fail if the command does not succeed within the number of tries and timeout.

Parameters:

  • tries (Integer) (defaults to: 5) —

    number of times to try calling the block

  • timeout (Integer) (defaults to: 1) —

    number of seconds to run the block before timing out

Returns:

  • (true) —

    If the block succeeds, true is returned

Raises:

  • (Vanagon::Error) —

    if the block fails after the retries are exhausted, an error is raised



161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/vanagon/utilities.rb', line 161

def retry_with_timeout(tries = 5, timeout = 1, &blk)
  error = nil
  tries.to_i.times do
    Timeout::timeout(timeout.to_i) do
      begin
        yield
        return true
      rescue StandardError => e
        VanagonLogger.error 'An error was encountered evaluating block. Retrying..'
        error = e
      end
    end
  end

  message = "Block failed maximum number of #{tries} tries"
  unless error.nil?
    message += "\n with error #{error.message}" + "\n#{error.backtrace.join("\n")}"
  end
  message += "\nExiting..."
  raise error, message unless error.nil?
  raise Vanagon::Error, "Block failed maximum number of #{tries} tries"
end

#rsync_from(source, target, dest, port = 22, extra_flags = []) ⇒ String

Retrieves the desired file/directory from the destination using rsync

Parameters:

  • source (String) —

    path on target to retrieve from

  • target (String) —

    ssh host to retrieve from (user@machine)

  • dest (String) —

    path on local host to place the source

  • port (Integer) (defaults to: 22) —

    port number for ssh (default 22)

  • extra_flags (Array) (defaults to: []) —

    any additional flags to pass to rsync

Returns:

  • (String) —

    output of rsync command



223
224
225
226
227
228
229
230
 
# File 'lib/vanagon/utilities.rb', line 223

def rsync_from(source, target, dest, port = 22, extra_flags = [])
  rsync = find_program_on_path('rsync')
  flags = "-rHlv -O --no-perms --no-owner --no-group"
  unless extra_flags.empty?
    flags << " " << extra_flags.join(" ")
  end
  ex("#{rsync} -e '#{ssh_command(port)}' #{flags} #{target}:#{source} #{dest}")
end

#rsync_to(source, target, dest, port = 22, extra_flags = ["--ignore-existing"]) ⇒ String

Sends the desired file/directory to the destination using rsync

Parameters:

  • source (String) —

    file or directory to send

  • target (String) —

    ssh host to send to (user@machine)

  • dest (String) —

    path on target to place the source

  • extra_flags (Array) (defaults to: ["--ignore-existing"]) —

    any additional flags to pass to rsync

  • port (Integer) (defaults to: 22) —

    Port number for ssh (default 22)

Returns:

  • (String) —

    output of rsync command



192
193
194
195
196
197
198
199
 
# File 'lib/vanagon/utilities.rb', line 192

def rsync_to(source, target, dest, port = 22, extra_flags = ["--ignore-existing"])
  rsync = find_program_on_path('rsync')
  flags = "-rHlv --no-perms --no-owner --no-group"
  unless extra_flags.empty?
    flags << " " << extra_flags.join(" ")
  end
  ex("#{rsync} -e '#{ssh_command(port)}' #{flags} #{source} #{target}:#{dest}")
end

#ssh_command(port = 22) ⇒ String

Hacky wrapper to add on the correct flags for ssh to be used in ssh and rsync methods

Parameters:

  • port (Integer) (defaults to: 22) —

    Port number for ssh (default 22)

Returns:

  • (String) —

    start of ssh command, including flags for ssh keys



205
206
207
208
209
210
211
212
213
 
# File 'lib/vanagon/utilities.rb', line 205

def ssh_command(port = 22)
  ssh = find_program_on_path('ssh')
  args = ENV['VANAGON_SSH_KEY'] ? " -i #{ENV['VANAGON_SSH_KEY']}" : ""
  args << " -p #{port} "
  args << " -o UserKnownHostsFile=/dev/null"
  args << " -o StrictHostKeyChecking=no"
  args << " -o ForwardAgent=yes" if ENV['VANAGON_SSH_AGENT']
  return ssh + args
end