Module: Pkg::Util::Net

Defined in:
lib/packaging/util/net.rb

Overview

Utility methods for handling network calls and interactions

Class Method Summary collapse

Class Method Details

.add_param_to_uri(uri, param) ⇒ Object

Add a parameter to a given uri. If we were sane we’d use encode_www_form(params) of URI, but because we’re not, because that will http encode it, which isn’t what we want since we’re require the encoding provided by escapeHTML of CGI, since this is being transfered in the xml of a jenkins job via curl and DEAR JEEBUS WHAT HAVE WE DONE.



345
346
347
348
349
350
 
# File 'lib/packaging/util/net.rb', line 345

def add_param_to_uri(uri, param)
  require 'uri'
  uri = URI.parse(uri)
  uri.query = [uri.query, param].compact.join('&')
  uri.to_s
end

.check_host(host, args = { :required => true }) ⇒ Object

Check that the current host matches the one we think it should



23
24
25
26
27
28
29
30
 
# File 'lib/packaging/util/net.rb', line 23

def check_host(host, args = { :required => true })
  if hostname == host
    return true
  else
    fail "#{hostname} does not match #{host}" if args[:required]
    return nil
  end
end

.check_host_gpg(hosts, gpg) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to check for gpg keys

    If the host needs a special username it should be passed in as user@host

  • gpg —

    • The gpg secret key to look for

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



55
56
57
58
59
60
61
62
63
64
65
 
# File 'lib/packaging/util/net.rb', line 55

def check_host_gpg(hosts, gpg)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_ssh_cmd(host, "gpg --list-secret-keys #{gpg} > /dev/null 2&>1", false, '-oBatchMode=yes')
    rescue
      errs << host
    end
  end
  return errs
end

.check_host_ssh(hosts) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to try ssh-ing into

    If the host needs a special username it should be passed in as user@host

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/packaging/util/net.rb', line 37

def check_host_ssh(hosts)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_ssh_cmd(host, 'exit', false, '-oBatchMode=yes')
    rescue
      errs << host
    end
  end
  return errs
end

.curl_form_data(uri, form_data = [], options = {}) ⇒ Object

This is fairly absurd. We’re implementing curl by shelling out. What do I wish we were doing? Using a sweet ruby wrapper around curl, such as Curb or Curb-fu. However, because we’re using clean build systems and trying to make this portable with minimal system requirements, we can’t very well depend on libraries that aren’t in the ruby standard libaries. We could also do this using Net::HTTP but that set of libraries is a rabbit hole to go down when what we’re trying to accomplish is posting multi-part form data that includes file uploads to jenkins. It gets hairy fairly quickly, but, as they say, pull requests accepted.

This method takes three arguments 1) String - the URL to post to 2) Array - Ordered array of name=VALUE curl form parameters 3) Hash - Options to be set



226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
 
# File 'lib/packaging/util/net.rb', line 226

def curl_form_data(uri, form_data = [], options = {})
  curl = Pkg::Util::Tool.check_tool("curl")
  #
  # Begin constructing the post string.
  # First, assemble the form_data arguments
  #
  post_string = "-i "
  form_data.each do |param|
    post_string << "#{param} "
  end

  # Add the uri
  post_string << "'#{uri}'"

  # If this is quiet, we're going to silence all output
  begin
    stdout, _, retval = Pkg::Util::Execution.capture3("#{curl} #{post_string}")
    if options[:quiet]
      stdout = ''
    end
    return stdout, retval
  rescue RuntimeError => e
    puts e
    return false
  end
end

.escape_html(uri) ⇒ Object 



335
336
337
338
 
# File 'lib/packaging/util/net.rb', line 335

def escape_html(uri)
  require 'cgi'
  CGI.escapeHTML(uri)
end

.fetch_uri(uri, target) ⇒ Object

This simple method does an HTTP get of a URI and writes it to a file in a slightly more platform agnostic way than curl/wget



9
10
11
12
13
14
 
# File 'lib/packaging/util/net.rb', line 9

def fetch_uri(uri, target)
  require 'open-uri'
  if Pkg::Util::File.file_writable?(File.dirname(target))
    File.open(target, 'w') { |f| f.puts(open(uri).read) }
  end
end

.hostname ⇒ Object

Get the hostname of the current host



17
18
19
20
 
# File 'lib/packaging/util/net.rb', line 17

def hostname
  require 'socket'
  Socket.gethostname
end

.print_url_info(url_string) ⇒ Object

Use the provided URL string to print important information with ASCII emphasis



267
268
269
270
271
 
# File 'lib/packaging/util/net.rb', line 267

def print_url_info(url_string)
  puts "\n////////////////////////////////////////////////////////////////////////////////\n\n
  Build submitted. To view your build progress, go to\n#{url_string}\n\n
////////////////////////////////////////////////////////////////////////////////\n\n"
end

.remote_buildparams(host, build) ⇒ Object

Given a BuildInstance object and a host, send its params to the host. Return the remote path to the params.



380
381
382
383
384
385
386
 
# File 'lib/packaging/util/net.rb', line 380

def remote_buildparams(host, build)
  params_file = build.config_to_yaml
  params_file_name = File.basename(params_file)
  params_dir = Pkg::Util.rand_string
  Pkg::Util::Net.rsync_to(params_file, host, "/tmp/#{params_dir}/")
  "/tmp/#{params_dir}/#{params_file_name}"
end

.remote_bundle_install_command ⇒ Object 



372
373
374
375
376
 
# File 'lib/packaging/util/net.rb', line 372

def remote_bundle_install_command
  export_packaging_location = ''
  export_packaging_location = "export PACKAGING_LOCATION='#{ENV['PACKAGING_LOCATION']}';" if ENV['PACKAGING_LOCATION'] && !ENV['PACKAGING_LOCATION'].empty?
  command = "source /usr/local/rvm/scripts/rvm; rvm use ruby-2.4.1; #{export_packaging_location} bundle install --path .bundle/gems ;"
end

.remote_create_latest_symlink(package_name, dir, platform_ext, options = {}) ⇒ Object

Create a symlink indicating the latest version of a package

Parameters:

  • package_name (String) —

    The name of the package you want to symlink to, e.g. ‘puppet-agent’, ‘facter’, etc.

  • dir (String) —

    The directory you want to find the latest package and create the symlink in.

  • platform_ext (String) —

    The type of files you want to consider, e.g. ‘dmg’, ‘msi’, etc.

  • options (Hash) (defaults to: {}) —

    Additional optional params: @option :arch [String] Architecture you want to narrow your search by. @option :excludes [Array] Strings you want to exclude from your search,

    e.g. 'agent' if only searching for 'puppet'.



300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
 
# File 'lib/packaging/util/net.rb', line 300

def remote_create_latest_symlink(package_name, dir, platform_ext, options = {})
  ls_cmd = "ls -1 *.#{platform_ext} | grep -v latest | grep -v rc | grep -P '#{package_name}-\\d' "

  # store this in a separate var to avoid side affects
  full_package_name = String.new(package_name)

  if options[:arch]
    ls_cmd << "| grep #{options[:arch]}"
    full_package_name << "-#{options[:arch]}"
  end
  if options[:excludes]
    options[:excludes].each do |excl|
      ls_cmd << "| grep -v #{excl} "
    end
  end
  ls_cmd << '| sort --version-sort | tail -1'
  cmd = <<-CMD
    if [ ! -d '#{dir}' ] ; then
      echo "directory '#{dir}' does not exist, not creating latest package link"
      exit 0
    fi
    pushd '#{dir}'
    link_target=$(#{ls_cmd})
    if [ -z "$link_target" ] ; then
      echo "Unable to find a link target for '#{full_package_name}' in '#{dir}'; skipping link creation"
      exit 0
    fi
    echo "creating link to '$link_target'"
    ln -sf "$link_target" #{full_package_name}-latest.#{platform_ext}
  CMD

  _, err = Pkg::Util::Net.remote_ssh_cmd(Pkg::Config.staging_server, cmd, true)
  $stderr.puts err
end

.remote_set_immutable(host, files) ⇒ Object

Remotely set the immutable bit on a list of files



284
285
286
 
# File 'lib/packaging/util/net.rb', line 284

def remote_set_immutable(host, files)
  Pkg::Util::Net.remote_ssh_cmd(host, "sudo chattr +i #{files.join(" ")}")
end

.remote_set_ownership(host, owner, group, files) ⇒ Object 



273
274
275
276
 
# File 'lib/packaging/util/net.rb', line 273

def remote_set_ownership(host, owner, group, files)
  remote_cmd = "for file in #{files.join(" ")}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chown #{owner}:#{group} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_ssh_cmd(host, remote_cmd)
end

.remote_set_permissions(host, permissions, files) ⇒ Object 



278
279
280
281
 
# File 'lib/packaging/util/net.rb', line 278

def remote_set_permissions(host, permissions, files)
  remote_cmd = "for file in #{files.join(" ")}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chmod #{permissions} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_ssh_cmd(host, remote_cmd)
end

.remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false) ⇒ Object

rubocop:disable Style/ParameterLists



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/packaging/util/net.rb', line 67

def remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false)  # rubocop:disable Style/ParameterLists

  ssh = Pkg::Util::Tool.check_tool('ssh')

  # we pass some pretty complicated commands in via ssh. We need this to fail
  # if any part of the remote ssh command fails.
  shell_flags = ''
  shell_flags += 'set -e;' if fail_fast
  shell_flags += 'set -x;' if trace
  shell_commands = "#{shell_flags}#{command}"

  cmd = "#{ssh} #{extra_options} -t #{target} '#{shell_commands.gsub("'", "'\\\\''")}'"

  # This is NOT a good way to support this functionality.
  # It needs to be refactored into a set of methods that
  # other methods can use to safely and deterministically
  # support dry-run operations.
  # But I need to ship packages RIGHT NOW.
  # - Ryan McKern, 13/01/2016
  if ENV['DRYRUN']
    puts "[DRY-RUN] Executing '#{command}' on #{target}"
    puts "[DRY-RUN] #{cmd}"
    return
  end

  puts "Executing '#{command}' on #{target}"
  if capture_output
    stdout, stderr, exitstatus = Pkg::Util::Execution.capture3(cmd)
    Pkg::Util::Execution.success?(exitstatus) or raise "Remote ssh command failed."
    return stdout, stderr
  else
    Kernel.system(cmd)
    Pkg::Util::Execution.success? or raise "Remote ssh command failed."
  end
end

.remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil) ⇒ Object

We take a tar argument for cases where tar isn’t best, e.g. Solaris. We also take an optional argument of the tarball containing the git bundle to use.



355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
 
# File 'lib/packaging/util/net.rb', line 355

def remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil)
  unless tar = tar_cmd
    tar = 'tar'
  end
  tarball ||= Pkg::Util::Git.bundle(treeish)
  tarball_name = File.basename(tarball).gsub('.tar.gz', '')
  Pkg::Util::Net.rsync_to(tarball, host, '/tmp')
  appendix = Pkg::Util.rand_string
  git_bundle_directory = File.join('/tmp', "#{Pkg::Config.project}-#{appendix}")
  command = <<-DOC
#{tar} -zxvf /tmp/#{tarball_name}.tar.gz -C /tmp/ ;
git clone --recursive /tmp/#{tarball_name} #{git_bundle_directory} ;
DOC
  Pkg::Util::Net.remote_ssh_cmd(host, command)
  return git_bundle_directory
end

.rsync_cmd(origin_path, opts = {}) ⇒ String

Construct a valid rsync command

Parameters:

  • origin_path (String, Pathname) —

    the path to sync from; if opts is not passed, then the parent directory of origin_path will be used to construct a target path to sync to.

  • opts (Hash) (defaults to: {}) —

    additional options that can be used to construct the rsync command.

Options Hash (opts):

  • :bin (String) — default: 'rsync' —

    the path to rsync (can be relative or fully qualified).

  • :origin_host (String) —

    the remote host to sync data from; cannot be specified alongside :target_host

  • :target_host (String) —

    the remote host to sync data to; cannot be specified alongside :origin_host.

  • :extra_flags (String) — default: ["--ignore-existing"] —

    extra flags to use when constructing an rsync command

  • :dryrun (String) — default: false —

    tell rsync to perform a trial run with no changes made.

Returns:

  • (String) —

    a rsync command that can be used in shell or ssh methods

Raises:

  • (ArgumentError) —

    if opts and opts names are both defined.

  • (ArgumentError) —

    if :origin_path exists without opts, opts, remote target is defined.



124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
 
# File 'lib/packaging/util/net.rb', line 124

def rsync_cmd(origin_path, opts = {})
  options = {
    bin: 'rsync',
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: false }.merge(opts)
  origin = Pathname.new(origin_path)
  target = options[:target_path] || origin.parent

  raise(ArgumentError, "Cannot sync between two remote hosts") if
    options[:origin_host] && options[:target_host]

  raise(ArgumentError, "Cannot sync path '#{origin}' because both origin_host and target_host are nil. Perhaps you need to set TEAM=release ?") unless
    options[:origin_host] || options[:target_host]

  cmd = %W(
    #{options[:bin]}
    --recursive
    --hard-links
    --links
    --verbose
    --omit-dir-times
    --no-perms
    --no-owner
    --no-group
  ) + [*options[:extra_flags]]

  cmd << '--dry-run' if options[:dryrun]
  cmd << Pkg::Util.pseudo_uri(path: origin, host: options[:origin_host])
  cmd << Pkg::Util.pseudo_uri(path: target, host: options[:target_host])

  cmd.uniq.compact.join("\s")
end

.rsync_exec(source, opts = {}) ⇒ Object

A generic rsync execution method that wraps rsync_cmd in a call to Pkg::Util::Execution#capture3()



162
163
164
165
166
167
168
169
170
171
172
173
 
# File 'lib/packaging/util/net.rb', line 162

def rsync_exec(source, opts = {})
  options = {
    bin: Pkg::Util::Tool.check_tool('rsync'),
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: ENV['DRYRUN'] }.merge(opts.delete_if { |_, value| value.nil? })

  stdout, _, _ = Pkg::Util::Execution.capture3(rsync_cmd(source, options), true)
  stdout
end

.rsync_from(source, origin_host, dest, opts = {}) ⇒ Object

A wrapper method to maintain the existing interface for executing incoming rsync commands with minimal changes to existing code.



190
191
192
193
194
195
196
197
198
199
 
# File 'lib/packaging/util/net.rb', line 190

def rsync_from(source, origin_host, dest, opts = {})
  rsync_exec(
    source,
    origin_host: origin_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] }) ⇒ Object

A wrapper method to maintain the existing interface for executing outbound rsync commands with minimal changes to existing code.



177
178
179
180
181
182
183
184
185
186
 
# File 'lib/packaging/util/net.rb', line 177

def rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] })
  rsync_exec(
    source,
    target_host: target_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.s3sync_to(source, target_bucket, target_directory = "", flags = []) ⇒ Object 



201
202
203
204
205
206
207
208
209
210
 
# File 'lib/packaging/util/net.rb', line 201

def s3sync_to(source, target_bucket, target_directory = "", flags = [])
  s3cmd = Pkg::Util::Tool.check_tool('s3cmd')

  if Pkg::Util::File.file_exists?(File.join(ENV['HOME'], '.s3cfg'))
    stdout, _, _ = Pkg::Util::Execution.capture3("#{s3cmd} sync #{flags.join(' ')} '#{source}' s3://#{target_bucket}/#{target_directory}/")
    stdout
  else
    fail "#{File.join(ENV['HOME'], '.s3cfg')} does not exist. It is required to ship files using s3cmd."
  end
end

.uri_status_code(uri) ⇒ Object 



253
254
255
256
257
258
259
260
261
262
263
 
# File 'lib/packaging/util/net.rb', line 253

def uri_status_code(uri)
  data = [
    '--request GET',
    '--silent',
    '--location',
    '--write-out "%{http_code}"',
    '--output /dev/null'
  ]
  stdout, _ = Pkg::Util::Net.curl_form_data(uri, data)
  stdout
end