Module: Pkg::Util::Net

Defined in:
lib/packaging/util/net.rb

Overview

Utility methods for handling network calls and interactions

Class Method Summary collapse

Class Method Details

.add_param_to_uri(uri, param) ⇒ Object

Add a parameter to a given uri. If we were sane we’d use encode_www_form(params) of URI, but because we’re not, because that will http encode it, which isn’t what we want since we’re require the encoding provided by escapeHTML of CGI, since this is being transfered in the xml of a jenkins job via curl and DEAR JEEBUS WHAT HAVE WE DONE.



369
370
371
372
373
374
 
# File 'lib/packaging/util/net.rb', line 369

def add_param_to_uri(uri, param)
  require 'uri'
  uri = URI.parse(uri)
  uri.query = [uri.query, param].compact.join('&')
  uri.to_s
end

.check_host(host, options = { required: true }) ⇒ Object

Check that the current host matches the one we think it should



21
22
23
24
25
26
 
# File 'lib/packaging/util/net.rb', line 21

def check_host(host, options = { required: true })
  return true if hostname == host

  fail "Error: #{hostname} does not match #{host}" if options[:required]
  return nil
end

.check_host_gpg(hosts, gpg) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to check for gpg keys

    If the host needs a special username it should be passed in as user@host

  • gpg —

    • The gpg secret key to look for

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/packaging/util/net.rb', line 51

def check_host_gpg(hosts, gpg)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_execute(host, "gpg --list-secret-keys #{gpg} > /dev/null 2&>1",
                     { extra_options: '-oBatchMode=yes' })
    rescue StandardError
      errs << host
    end
  end
  return errs
end

.check_host_ssh(hosts) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to try ssh-ing into

    If the host needs a special username it should be passed in as user@host

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/packaging/util/net.rb', line 33

def check_host_ssh(hosts)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_execute(host, 'exit', { extra_options: '-oBatchMode=yes' })
    rescue StandardError
      errs << host
    end
  end
  return errs
end

.curl_form_data(uri, form_data = [], options = {}) ⇒ Object

This is fairly absurd. We’re implementing curl by shelling out. What do I wish we were doing? Using a sweet ruby wrapper around curl, such as Curb or Curb-fu. However, because we’re using clean build systems and trying to make this portable with minimal system requirements, we can’t very well depend on libraries that aren’t in the ruby standard libaries. We could also do this using Net::HTTP but that set of libraries is a rabbit hole to go down when what we’re trying to accomplish is posting multi-part form data that includes file uploads to jenkins. It gets hairy fairly quickly, but, as they say, pull requests accepted.

This method takes three arguments 1) String - the URL to post to 2) Array - Ordered array of name=VALUE curl form parameters 3) Hash - Options to be set



248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/packaging/util/net.rb', line 248

def curl_form_data(uri, form_data = [], options = {})
  curl = Pkg::Util::Tool.check_tool("curl")
  #
  # Begin constructing the post string.
  # First, assemble the form_data arguments
  #
  post_string = "-i "
  form_data.each do |param|
    post_string << "#{param} "
  end

  # Add the uri
  post_string << "'#{uri}'"

  # If this is quiet, we're going to silence all output
  begin
    stdout, _, retval = Pkg::Util::Execution.capture3("#{curl} #{post_string}")
    if options[:quiet]
      stdout = ''
    end
    return stdout, retval
  rescue RuntimeError => e
    puts e
    return false
  end
end

.escape_html(uri) ⇒ Object 



359
360
361
362
 
# File 'lib/packaging/util/net.rb', line 359

def escape_html(uri)
  require 'cgi'
  CGI.escapeHTML(uri)
end

.fetch_uri(uri, target) ⇒ Object

This simple method does an HTTP get of a URI and writes it to a file in a slightly more platform agnostic way than curl/wget



7
8
9
10
11
12
 
# File 'lib/packaging/util/net.rb', line 7

def fetch_uri(uri, target)
  require 'open-uri'
  if Pkg::Util::File.file_writable?(File.dirname(target))
    File.open(target, 'w') { |f| f.puts(URI.open(uri).read) }
  end
end

.hostname ⇒ Object

Get the hostname of the current host



15
16
17
18
 
# File 'lib/packaging/util/net.rb', line 15

def hostname
  require 'socket'
  Socket.gethostname
end

.print_url_info(url_string) ⇒ Object

Use the provided URL string to print important information with ASCII emphasis



289
290
291
292
293
 
# File 'lib/packaging/util/net.rb', line 289

def print_url_info(url_string)
  puts "\n////////////////////////////////////////////////////////////////////////////////\n\n
  Build submitted. To view your build progress, go to\n#{url_string}\n\n
////////////////////////////////////////////////////////////////////////////////\n\n"
end

.remote_buildparams(host, build) ⇒ Object

Given a BuildInstance object and a host, send its params to the host. Return the remote path to the params.



405
406
407
408
409
410
411
 
# File 'lib/packaging/util/net.rb', line 405

def remote_buildparams(host, build)
  params_file = build.config_to_yaml
  params_file_name = File.basename(params_file)
  params_dir = Pkg::Util.rand_string
  Pkg::Util::Net.rsync_to(params_file, host, "/tmp/#{params_dir}/")
  "/tmp/#{params_dir}/#{params_file_name}"
end

.remote_bundle_install_command ⇒ Object 



396
397
398
399
400
401
 
# File 'lib/packaging/util/net.rb', line 396

def remote_bundle_install_command
  rvm_ruby_version = ENV['RVM_RUBY_VERSION'] || '2.7.5'
  export_packaging_location = "export PACKAGING_LOCATION='#{ENV['PACKAGING_LOCATION']}';" if ENV['PACKAGING_LOCATION'] && !ENV['PACKAGING_LOCATION'].empty?
  export_vanagon_location = "export VANAGON_LOCATION='#{ENV['VANAGON_LOCATION']}';" if ENV['VANAGON_LOCATION'] && !ENV['VANAGON_LOCATION'].empty?
  "source /usr/local/rvm/scripts/rvm; rvm use ruby-#{rvm_ruby_version}; #{export_packaging_location} #{export_vanagon_location} bundle install --path .bundle/gems ;"
end

.remote_create_latest_symlink(package_name, dir, platform_ext, options = {}) ⇒ Object

Create a symlink indicating the latest version of a package

Parameters:

  • package_name (String) —

    The name of the package you want to symlink to, e.g. ‘puppet-agent’, ‘facter’, etc.

  • dir (String) —

    The directory you want to find the latest package and create the symlink in.

  • platform_ext (String) —

    The type of files you want to consider, e.g. ‘dmg’, ‘msi’, etc.

  • options (Hash) (defaults to: {}) —

    Additional optional params: @option :arch [String] Architecture you want to narrow your search by. @option :excludes [Array] Strings you want to exclude from your search,

    e.g. 'agent' if only searching for 'puppet'.



322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
 
# File 'lib/packaging/util/net.rb', line 322

def remote_create_latest_symlink(package_name, dir, platform_ext, options = {})
  ls_cmd = "ls -1 *.#{platform_ext} | grep -v latest | grep -v rc | grep -P '#{package_name}-\\d' "

  # store this in a separate var to avoid side affects
  full_package_name = String.new(package_name)

  if options[:arch]
    ls_cmd << "| grep #{options[:arch]}"
    full_package_name << "-#{options[:arch]}"
  end
  if options[:excludes]
    options[:excludes].each do |excl|
      ls_cmd << "| grep -v #{excl} "
    end
  end
  ls_cmd << '| sort --version-sort | tail -1'
  cmd = <<-CMD
    if [ ! -d '#{dir}' ] ; then
      echo "directory '#{dir}' does not exist, not creating latest package link"
      exit 0
    fi
    pushd '#{dir}'
    link_target=$(#{ls_cmd})
    if [ -z "$link_target" ] ; then
      echo "Unable to find a link target for '#{full_package_name}' in '#{dir}'; skipping link creation"
      exit 0
    fi
    echo "creating link to '$link_target'"
    ln -sf "$link_target" #{full_package_name}-latest.#{platform_ext}
  CMD

  _, err = Pkg::Util::Net.remote_execute(
    Pkg::Config.staging_server, cmd, { capture_output: true }
  )
  warn err
end

.remote_execute(target_host, command, user_options = {}) ⇒ Object 



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/packaging/util/net.rb', line 64

def remote_execute(target_host, command, user_options = {})
  option_defaults = {
    capture_output: false,
    extra_options: '',
    fail_fast: true,
    trace: false
  }
  options = option_defaults.merge(user_options)

  ssh = Pkg::Util::Tool.check_tool('ssh')

  # we pass some pretty complicated commands in via ssh. We need this to fail
  # if any part of the remote ssh command fails.
  shell_flags = ''
  shell_flags += 'set -e;' if options[:fail_fast]
  shell_flags += 'set -x;' if options[:trace]
  shell_commands = "#{shell_flags}#{command}"

  remote_command = "#{ssh} #{options[:extra_options]} -t #{target_host} " +
                   "'#{shell_commands.gsub("'", "'\\\\''")}'"

  # This is NOT a good way to support this functionality.
  if ENV['DRYRUN']
    puts "[DRY-RUN] Executing '#{command}' on #{target}"
    puts "[DRY-RUN] #{cmd}"
    return ''
  end

  # We're forced to make different calls depending on the capture_output option
  # because something about our #capture3 method screws up gpg. This should
  # be untangled.
  if options[:capture_output]
    stdout, stderr, exitstatus = Pkg::Util::Execution.capture3(remote_command)
    Pkg::Util::Execution.success?(exitstatus) or
      raise "Remote ssh command (\"#{remote_command}\") failed."
    return stdout, stderr
  end

  # Pkg::Util::Execution.capture3 reports its command but Kernel.system does not
  # Let's print it out for some amount of consistency.
  puts "Remote Execute: '#{remote_command}'"
  Kernel.system(remote_command)
  Pkg::Util::Execution.success? or
    raise "Remote ssh command (\"#{remote_command}\") failed."
end

.remote_set_immutable(host, files) ⇒ Object

Remotely set the immutable bit on a list of files



306
307
308
 
# File 'lib/packaging/util/net.rb', line 306

def remote_set_immutable(host, files)
  Pkg::Util::Net.remote_execute(host, "sudo chattr +i #{files.join(' ')}")
end

.remote_set_ownership(host, owner, group, files) ⇒ Object 



295
296
297
298
 
# File 'lib/packaging/util/net.rb', line 295

def remote_set_ownership(host, owner, group, files)
  remote_cmd = "for file in #{files.join(' ')}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chown #{owner}:#{group} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_execute(host, remote_cmd)
end

.remote_set_permissions(host, permissions, files) ⇒ Object 



300
301
302
303
 
# File 'lib/packaging/util/net.rb', line 300

def remote_set_permissions(host, permissions, files)
  remote_cmd = "for file in #{files.join(' ')}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chmod #{permissions} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_execute(host, remote_cmd)
end

.remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false) ⇒ Object

Deprecated method implemented as a shim to the new ‘remote_execute` method



113
114
115
116
117
118
119
120
121
 
# File 'lib/packaging/util/net.rb', line 113

def remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false) # rubocop:disable Metrics/ParameterLists
  puts "Warn: \"remote_ssh_cmd\" call in packaging is deprecated. Use \"remote_execute\" instead."
  remote_execute(target, command, {
                   capture_output: capture_output,
                   extra_options: extra_options,
                   fail_fast: fail_fast,
                   trace: trace
})
end

.remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil) ⇒ Object

We take a tar argument for cases where ‘tar` isn’t best, e.g. Solaris. We also take an optional argument of the tarball containing the git bundle to use.



379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
 
# File 'lib/packaging/util/net.rb', line 379

def remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil)
  unless tar = tar_cmd
    tar = 'tar'
  end
  tarball ||= Pkg::Util::Git.bundle(treeish)
  tarball_name = File.basename(tarball).gsub('.tar.gz', '')
  Pkg::Util::Net.rsync_to(tarball, host, '/tmp')
  appendix = Pkg::Util.rand_string
  git_bundle_directory = File.join('/tmp', "#{Pkg::Config.project}-#{appendix}")
  command = <<~DOC
    #{tar} -zxvf /tmp/#{tarball_name}.tar.gz -C /tmp/ ;
    git clone --recursive /tmp/#{tarball_name} #{git_bundle_directory} ;
  DOC
  Pkg::Util::Net.remote_execute(host, command)
  return git_bundle_directory
end

.rsync_cmd(origin_path, opts = {}) ⇒ String

Construct a valid rsync command

Parameters:

  • origin_path (String, Pathname) —

    the path to sync from; if opts is not passed, then the parent directory of ‘origin_path` will be used to construct a target path to sync to.

  • opts (Hash) (defaults to: {}) —

    additional options that can be used to construct the rsync command.

Options Hash (opts):

  • :bin (String) — default: 'rsync' —

    the path to rsync (can be relative or fully qualified).

  • :origin_host (String) —

    the remote host to sync data from; cannot be specified alongside :target_host

  • :target_host (String) —

    the remote host to sync data to; cannot be specified alongside :origin_host.

  • :extra_flags (String) — default: ["--ignore-existing"] —

    extra flags to use when constructing an rsync command

  • :dryrun (String) — default: false —

    tell rsync to perform a trial run with no changes made.

Returns:

  • (String) —

    a rsync command that can be used in shell or ssh methods

Raises:

  • (ArgumentError) —

    if opts and opts names are both defined.

  • (ArgumentError) —

    if :origin_path exists without opts, opts, remote target is defined.



144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/packaging/util/net.rb', line 144

def rsync_cmd(origin_path, opts = {})
  options = {
    bin: 'rsync',
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: false
}.merge(opts)
  origin = Pathname.new(origin_path)
  target = options[:target_path] || origin.parent

  raise(ArgumentError, "Cannot sync between two remote hosts") if
    options[:origin_host] && options[:target_host]

  raise(ArgumentError, "Cannot sync path '#{origin}' because both origin_host and target_host are nil. Perhaps you need to set TEAM=release ?") unless
    options[:origin_host] || options[:target_host]

  cmd = %W[
    #{options[:bin]}
    --recursive
    --hard-links
    --links
    --verbose
    --omit-dir-times
    --no-perms
    --no-owner
    --no-group
  ] + [*options[:extra_flags]]

  cmd << '--dry-run' if options[:dryrun]
  cmd << Pkg::Util.pseudo_uri(path: origin, host: options[:origin_host])
  cmd << Pkg::Util.pseudo_uri(path: target, host: options[:target_host])

  cmd.uniq.compact.join("\s")
end

.rsync_exec(source, opts = {}) ⇒ Object

A generic rsync execution method that wraps rsync_cmd in a call to Pkg::Util::Execution#capture3()



183
184
185
186
187
188
189
190
191
192
193
194
195
 
# File 'lib/packaging/util/net.rb', line 183

def rsync_exec(source, opts = {})
  options = {
    bin: Pkg::Util::Tool.check_tool('rsync'),
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: ENV['DRYRUN']
}.merge(opts.delete_if { |_, value| value.nil? })

  stdout, = Pkg::Util::Execution.capture3(rsync_cmd(source, options), true)
  stdout
end

.rsync_from(source, origin_host, dest, opts = {}) ⇒ Object

A wrapper method to maintain the existing interface for executing incoming rsync commands with minimal changes to existing code.



212
213
214
215
216
217
218
219
220
221
 
# File 'lib/packaging/util/net.rb', line 212

def rsync_from(source, origin_host, dest, opts = {})
  rsync_exec(
    source,
    origin_host: origin_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] }) ⇒ Object

A wrapper method to maintain the existing interface for executing outbound rsync commands with minimal changes to existing code.



199
200
201
202
203
204
205
206
207
208
 
# File 'lib/packaging/util/net.rb', line 199

def rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] })
  rsync_exec(
    source,
    target_host: target_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.s3sync_to(source, target_bucket, target_directory = "", flags = []) ⇒ Object 



223
224
225
226
227
228
229
230
231
232
 
# File 'lib/packaging/util/net.rb', line 223

def s3sync_to(source, target_bucket, target_directory = "", flags = [])
  s3cmd = Pkg::Util::Tool.check_tool('s3cmd')

  if Pkg::Util::File.file_exists?(File.join(ENV['HOME'], '.s3cfg'))
    stdout, = Pkg::Util::Execution.capture3("#{s3cmd} sync #{flags.join(' ')} '#{source}' s3://#{target_bucket}/#{target_directory}/")
    stdout
  else
    fail "#{File.join(ENV['HOME'], '.s3cfg')} does not exist. It is required to ship files using s3cmd."
  end
end

.uri_status_code(uri) ⇒ Object 



275
276
277
278
279
280
281
282
283
284
285
 
# File 'lib/packaging/util/net.rb', line 275

def uri_status_code(uri)
  data = [
    '--request GET',
    '--silent',
    '--location',
    '--write-out "%{http_code}"',
    '--output /dev/null'
  ]
  stdout, = Pkg::Util::Net.curl_form_data(uri, data)
  stdout
end