Module: Pkg::Util::Net

Defined in:
lib/packaging/util/net.rb

Overview

Utility methods for handling network calls and interactions

Class Method Summary collapse

Class Method Details

.add_param_to_uri(uri, param) ⇒ Object

Add a parameter to a given uri. If we were sane we’d use encode_www_form(params) of URI, but because we’re not, because that will http encode it, which isn’t what we want since we’re require the encoding provided by escapeHTML of CGI, since this is being transfered in the xml of a jenkins job via curl and DEAR JEEBUS WHAT HAVE WE DONE.



367
368
369
370
371
372
 
# File 'lib/packaging/util/net.rb', line 367

def add_param_to_uri(uri, param)
  require 'uri'
  uri = URI.parse(uri)
  uri.query = [uri.query, param].compact.join('&')
  uri.to_s
end

.check_host(host, options = { required: true }) ⇒ Object

Check that the current host matches the one we think it should



23
24
25
26
27
28
 
# File 'lib/packaging/util/net.rb', line 23

def check_host(host, options = { required: true })
  return true if hostname == host

  fail "Error: #{hostname} does not match #{host}" if options[:required]
  return nil
end

.check_host_gpg(hosts, gpg) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to check for gpg keys

    If the host needs a special username it should be passed in as user@host

  • gpg —

    • The gpg secret key to look for

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/packaging/util/net.rb', line 53

def check_host_gpg(hosts, gpg)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_execute(host, "gpg --list-secret-keys #{gpg} > /dev/null 2&>1",
                     { extra_options: '-oBatchMode=yes' })
    rescue
      errs << host
    end
  end
  return errs
end

.check_host_ssh(hosts) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to try ssh-ing into

    If the host needs a special username it should be passed in as user@host

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/packaging/util/net.rb', line 35

def check_host_ssh(hosts)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_execute(host, 'exit', { extra_options: '-oBatchMode=yes' })
    rescue
      errs << host
    end
  end
  return errs
end

.curl_form_data(uri, form_data = [], options = {}) ⇒ Object

This is fairly absurd. We’re implementing curl by shelling out. What do I wish we were doing? Using a sweet ruby wrapper around curl, such as Curb or Curb-fu. However, because we’re using clean build systems and trying to make this portable with minimal system requirements, we can’t very well depend on libraries that aren’t in the ruby standard libaries. We could also do this using Net::HTTP but that set of libraries is a rabbit hole to go down when what we’re trying to accomplish is posting multi-part form data that includes file uploads to jenkins. It gets hairy fairly quickly, but, as they say, pull requests accepted.

This method takes three arguments 1) String - the URL to post to 2) Array - Ordered array of name=VALUE curl form parameters 3) Hash - Options to be set



247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
 
# File 'lib/packaging/util/net.rb', line 247

def curl_form_data(uri, form_data = [], options = {})
  curl = Pkg::Util::Tool.check_tool("curl")
  #
  # Begin constructing the post string.
  # First, assemble the form_data arguments
  #
  post_string = "-i "
  form_data.each do |param|
    post_string << "#{param} "
  end

  # Add the uri
  post_string << "'#{uri}'"

  # If this is quiet, we're going to silence all output
  begin
    stdout, _, retval = Pkg::Util::Execution.capture3("#{curl} #{post_string}")
    if options[:quiet]
      stdout = ''
    end
    return stdout, retval
  rescue RuntimeError => e
    puts e
    return false
  end
end

.escape_html(uri) ⇒ Object 



357
358
359
360
 
# File 'lib/packaging/util/net.rb', line 357

def escape_html(uri)
  require 'cgi'
  CGI.escapeHTML(uri)
end

.fetch_uri(uri, target) ⇒ Object

This simple method does an HTTP get of a URI and writes it to a file in a slightly more platform agnostic way than curl/wget



9
10
11
12
13
14
 
# File 'lib/packaging/util/net.rb', line 9

def fetch_uri(uri, target)
  require 'open-uri'
  if Pkg::Util::File.file_writable?(File.dirname(target))
    File.open(target, 'w') { |f| f.puts(open(uri).read) }
  end
end

.hostname ⇒ Object

Get the hostname of the current host



17
18
19
20
 
# File 'lib/packaging/util/net.rb', line 17

def hostname
  require 'socket'
  Socket.gethostname
end

.print_url_info(url_string) ⇒ Object

Use the provided URL string to print important information with ASCII emphasis



288
289
290
291
292
 
# File 'lib/packaging/util/net.rb', line 288

def print_url_info(url_string)
  puts "\n////////////////////////////////////////////////////////////////////////////////\n\n
  Build submitted. To view your build progress, go to\n#{url_string}\n\n
////////////////////////////////////////////////////////////////////////////////\n\n"
end

.remote_buildparams(host, build) ⇒ Object

Given a BuildInstance object and a host, send its params to the host. Return the remote path to the params.



402
403
404
405
406
407
408
 
# File 'lib/packaging/util/net.rb', line 402

def remote_buildparams(host, build)
  params_file = build.config_to_yaml
  params_file_name = File.basename(params_file)
  params_dir = Pkg::Util.rand_string
  Pkg::Util::Net.rsync_to(params_file, host, "/tmp/#{params_dir}/")
  "/tmp/#{params_dir}/#{params_file_name}"
end

.remote_bundle_install_command ⇒ Object 



394
395
396
397
398
 
# File 'lib/packaging/util/net.rb', line 394

def remote_bundle_install_command
  export_packaging_location = ''
  export_packaging_location = "export PACKAGING_LOCATION='#{ENV['PACKAGING_LOCATION']}';" if ENV['PACKAGING_LOCATION'] && !ENV['PACKAGING_LOCATION'].empty?
  command = "source /usr/local/rvm/scripts/rvm; rvm use ruby-2.5.1; #{export_packaging_location} bundle install --path .bundle/gems ;"
end

.remote_create_latest_symlink(package_name, dir, platform_ext, options = {}) ⇒ Object

Create a symlink indicating the latest version of a package

Parameters:

  • package_name (String) —

    The name of the package you want to symlink to, e.g. ‘puppet-agent’, ‘facter’, etc.

  • dir (String) —

    The directory you want to find the latest package and create the symlink in.

  • platform_ext (String) —

    The type of files you want to consider, e.g. ‘dmg’, ‘msi’, etc.

  • options (Hash) (defaults to: {}) —

    Additional optional params: @option :arch [String] Architecture you want to narrow your search by. @option :excludes [Array] Strings you want to exclude from your search,

    e.g. 'agent' if only searching for 'puppet'.



321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
 
# File 'lib/packaging/util/net.rb', line 321

def remote_create_latest_symlink(package_name, dir, platform_ext, options = {})
  ls_cmd = "ls -1 *.#{platform_ext} | grep -v latest | grep -v rc | grep -P '#{package_name}-\\d' "

  # store this in a separate var to avoid side affects
  full_package_name = String.new(package_name)

  if options[:arch]
    ls_cmd << "| grep #{options[:arch]}"
    full_package_name << "-#{options[:arch]}"
  end
  if options[:excludes]
    options[:excludes].each do |excl|
      ls_cmd << "| grep -v #{excl} "
    end
  end
  ls_cmd << '| sort --version-sort | tail -1'
  cmd = "if [ ! -d '\#{dir}' ] ; then\necho \"directory '\#{dir}' does not exist, not creating latest package link\"\nexit 0\nfi\npushd '\#{dir}'\nlink_target=$(\#{ls_cmd})\nif [ -z \"$link_target\" ] ; then\necho \"Unable to find a link target for '\#{full_package_name}' in '\#{dir}'; skipping link creation\"\nexit 0\nfi\necho \"creating link to '$link_target'\"\nln -sf \"$link_target\" \#{full_package_name}-latest.\#{platform_ext}\n"

  _, err = Pkg::Util::Net.remote_execute(
       Pkg::Config.staging_server, cmd, { capture_output: true })
  $stderr.puts err
end

.remote_execute(target_host, command, user_options = {}) ⇒ Object 



66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
 
# File 'lib/packaging/util/net.rb', line 66

def remote_execute(target_host, command, user_options = {})
  option_defaults = {
    capture_output: false,
    extra_options: '',
    fail_fast: true,
    trace: false
  }
  options = option_defaults.merge(user_options)

  ssh = Pkg::Util::Tool.check_tool('ssh')

  # we pass some pretty complicated commands in via ssh. We need this to fail
  # if any part of the remote ssh command fails.
  shell_flags = ''
  shell_flags += 'set -e;' if options[:fail_fast]
  shell_flags += 'set -x;' if options[:trace]
  shell_commands = "#{shell_flags}#{command}"

  remote_command = "#{ssh} #{options[:extra_options]} -t #{target_host} " +
                   "'#{shell_commands.gsub("'", "'\\\\''")}'"

  # This is NOT a good way to support this functionality.
  if ENV['DRYRUN']
    puts "[DRY-RUN] Executing '#{command}' on #{target}"
    puts "[DRY-RUN] #{cmd}"
    return ''
  end

  # We're forced to make different calls depending on the capture_output option
  # because something about our #capture3 method screws up gpg. This should
  # be untangled.
  if options[:capture_output]
    stdout, stderr, exitstatus = Pkg::Util::Execution.capture3(remote_command)
    Pkg::Util::Execution.success?(exitstatus) or
      raise "Remote ssh command (\"#{remote_command}\") failed."
    return stdout, stderr
  end

  # Pkg::Util::Execution.capture3 reports its command but Kernel.system does not
  # Let's print it out for some amount of consistency.
  puts "Remote Execute: '#{remote_command}'"
  Kernel.system(remote_command)
  Pkg::Util::Execution.success? or
    raise "Remote ssh command (\"#{remote_command}\") failed."
end

.remote_set_immutable(host, files) ⇒ Object

Remotely set the immutable bit on a list of files



305
306
307
 
# File 'lib/packaging/util/net.rb', line 305

def remote_set_immutable(host, files)
  Pkg::Util::Net.remote_execute(host, "sudo chattr +i #{files.join(" ")}")
end

.remote_set_ownership(host, owner, group, files) ⇒ Object 



294
295
296
297
 
# File 'lib/packaging/util/net.rb', line 294

def remote_set_ownership(host, owner, group, files)
  remote_cmd = "for file in #{files.join(" ")}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chown #{owner}:#{group} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_execute(host, remote_cmd)
end

.remote_set_permissions(host, permissions, files) ⇒ Object 



299
300
301
302
 
# File 'lib/packaging/util/net.rb', line 299

def remote_set_permissions(host, permissions, files)
  remote_cmd = "for file in #{files.join(" ")}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chmod #{permissions} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_execute(host, remote_cmd)
end

.remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false) ⇒ Object

Deprecated method implemented as a shim to the new ‘remote_execute` method



115
116
117
118
119
120
121
122
 
# File 'lib/packaging/util/net.rb', line 115

def remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false)  # rubocop:disable Style/ParameterLists
  puts "Warn: \"remote_ssh_cmd\" call in packaging is deprecated. Use \"remote_execute\" instead."
  remote_execute(target, command, {
                   capture_output: capture_output,
                   extra_options: extra_options,
                   fail_fast: fail_fast,
                   trace: trace })
end

.remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil) ⇒ Object

We take a tar argument for cases where ‘tar` isn’t best, e.g. Solaris. We also take an optional argument of the tarball containing the git bundle to use.



377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
 
# File 'lib/packaging/util/net.rb', line 377

def remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil)
  unless tar = tar_cmd
    tar = 'tar'
  end
  tarball ||= Pkg::Util::Git.bundle(treeish)
  tarball_name = File.basename(tarball).gsub('.tar.gz', '')
  Pkg::Util::Net.rsync_to(tarball, host, '/tmp')
  appendix = Pkg::Util.rand_string
  git_bundle_directory = File.join('/tmp', "#{Pkg::Config.project}-#{appendix}")
  command = "\#{tar} -zxvf /tmp/\#{tarball_name}.tar.gz -C /tmp/ ;\ngit clone --recursive /tmp/\#{tarball_name} \#{git_bundle_directory} ;\n"
  Pkg::Util::Net.remote_execute(host, command)
  return git_bundle_directory
end

.rsync_cmd(origin_path, opts = {}) ⇒ String

Construct a valid rsync command

Parameters:

  • origin_path (String, Pathname) —

    the path to sync from; if opts is not passed, then the parent directory of ‘origin_path` will be used to construct a target path to sync to.

  • opts (Hash) (defaults to: {}) —

    additional options that can be used to construct the rsync command.

Options Hash (opts):

  • :bin (String) — default: 'rsync' —

    the path to rsync (can be relative or fully qualified).

  • :origin_host (String) —

    the remote host to sync data from; cannot be specified alongside :target_host

  • :target_host (String) —

    the remote host to sync data to; cannot be specified alongside :origin_host.

  • :extra_flags (String) — default: ["--ignore-existing"] —

    extra flags to use when constructing an rsync command

  • :dryrun (String) — default: false —

    tell rsync to perform a trial run with no changes made.

Returns:

  • (String) —

    a rsync command that can be used in shell or ssh methods

Raises:

  • (ArgumentError) —

    if opts and opts names are both defined.

  • (ArgumentError) —

    if :origin_path exists without opts, opts, remote target is defined.



145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/packaging/util/net.rb', line 145

def rsync_cmd(origin_path, opts = {})
  options = {
    bin: 'rsync',
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: false }.merge(opts)
  origin = Pathname.new(origin_path)
  target = options[:target_path] || origin.parent

  raise(ArgumentError, "Cannot sync between two remote hosts") if
    options[:origin_host] && options[:target_host]

  raise(ArgumentError, "Cannot sync path '#{origin}' because both origin_host and target_host are nil. Perhaps you need to set TEAM=release ?") unless
    options[:origin_host] || options[:target_host]

  cmd = %W(
    #{options[:bin]}
    --recursive
    --hard-links
    --links
    --verbose
    --omit-dir-times
    --no-perms
    --no-owner
    --no-group
  ) + [*options[:extra_flags]]

  cmd << '--dry-run' if options[:dryrun]
  cmd << Pkg::Util.pseudo_uri(path: origin, host: options[:origin_host])
  cmd << Pkg::Util.pseudo_uri(path: target, host: options[:target_host])

  cmd.uniq.compact.join("\s")
end

.rsync_exec(source, opts = {}) ⇒ Object

A generic rsync execution method that wraps rsync_cmd in a call to Pkg::Util::Execution#capture3()



183
184
185
186
187
188
189
190
191
192
193
194
 
# File 'lib/packaging/util/net.rb', line 183

def rsync_exec(source, opts = {})
  options = {
    bin: Pkg::Util::Tool.check_tool('rsync'),
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: ENV['DRYRUN'] }.merge(opts.delete_if { |_, value| value.nil? })

  stdout, _, _ = Pkg::Util::Execution.capture3(rsync_cmd(source, options), true)
  stdout
end

.rsync_from(source, origin_host, dest, opts = {}) ⇒ Object

A wrapper method to maintain the existing interface for executing incoming rsync commands with minimal changes to existing code.



211
212
213
214
215
216
217
218
219
220
 
# File 'lib/packaging/util/net.rb', line 211

def rsync_from(source, origin_host, dest, opts = {})
  rsync_exec(
    source,
    origin_host: origin_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] }) ⇒ Object

A wrapper method to maintain the existing interface for executing outbound rsync commands with minimal changes to existing code.



198
199
200
201
202
203
204
205
206
207
 
# File 'lib/packaging/util/net.rb', line 198

def rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] })
  rsync_exec(
    source,
    target_host: target_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.s3sync_to(source, target_bucket, target_directory = "", flags = []) ⇒ Object 



222
223
224
225
226
227
228
229
230
231
 
# File 'lib/packaging/util/net.rb', line 222

def s3sync_to(source, target_bucket, target_directory = "", flags = [])
  s3cmd = Pkg::Util::Tool.check_tool('s3cmd')

  if Pkg::Util::File.file_exists?(File.join(ENV['HOME'], '.s3cfg'))
    stdout, _, _ = Pkg::Util::Execution.capture3("#{s3cmd} sync #{flags.join(' ')} '#{source}' s3://#{target_bucket}/#{target_directory}/")
    stdout
  else
    fail "#{File.join(ENV['HOME'], '.s3cfg')} does not exist. It is required to ship files using s3cmd."
  end
end

.uri_status_code(uri) ⇒ Object 



274
275
276
277
278
279
280
281
282
283
284
 
# File 'lib/packaging/util/net.rb', line 274

def uri_status_code(uri)
  data = [
    '--request GET',
    '--silent',
    '--location',
    '--write-out "%{http_code}"',
    '--output /dev/null'
  ]
  stdout, _ = Pkg::Util::Net.curl_form_data(uri, data)
  stdout
end