Module: Pkg::Util::Net

Defined in:
lib/packaging/util/net.rb

Overview

Utility methods for handling network calls and interactions

Class Method Summary collapse

Class Method Details

.add_param_to_uri(uri, param) ⇒ Object

Add a parameter to a given uri. If we were sane we’d use encode_www_form(params) of URI, but because we’re not, because that will http encode it, which isn’t what we want since we’re require the encoding provided by escapeHTML of CGI, since this is being transfered in the xml of a jenkins job via curl and DEAR JEEBUS WHAT HAVE WE DONE.



340
341
342
343
344
345
 
# File 'lib/packaging/util/net.rb', line 340

def add_param_to_uri(uri, param)
  require 'uri'
  uri = URI.parse(uri)
  uri.query = [uri.query, param].compact.join('&')
  uri.to_s
end

.check_host(host, args = { :required => true }) ⇒ Object

Check that the current host matches the one we think it should



23
24
25
26
27
28
29
30
 
# File 'lib/packaging/util/net.rb', line 23

def check_host(host, args = { :required => true })
  if hostname == host
    return true
  else
    fail "#{hostname} does not match #{host}" if args[:required]
    return nil
  end
end

.check_host_gpg(hosts, gpg) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to check for gpg keys

    If the host needs a special username it should be passed in as user@host

  • gpg —

    • The gpg secret key to look for

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



55
56
57
58
59
60
61
62
63
64
65
 
# File 'lib/packaging/util/net.rb', line 55

def check_host_gpg(hosts, gpg)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_ssh_cmd(host, "gpg --list-secret-keys #{gpg} > /dev/null 2&>1", false, '-oBatchMode=yes')
    rescue
      errs << host
    end
  end
  return errs
end

.check_host_ssh(hosts) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to try ssh-ing into

    If the host needs a special username it should be passed in as user@host

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/packaging/util/net.rb', line 37

def check_host_ssh(hosts)
  errs = []
  Array(hosts).flatten.each do |host|
    begin
      remote_ssh_cmd(host, 'exit', false, '-oBatchMode=yes')
    rescue
      errs << host
    end
  end
  return errs
end

.curl_form_data(uri, form_data = [], options = {}) ⇒ Object

This is fairly absurd. We’re implementing curl by shelling out. What do I wish we were doing? Using a sweet ruby wrapper around curl, such as Curb or Curb-fu. However, because we’re using clean build systems and trying to make this portable with minimal system requirements, we can’t very well depend on libraries that aren’t in the ruby standard libaries. We could also do this using Net::HTTP but that set of libraries is a rabbit hole to go down when what we’re trying to accomplish is posting multi-part form data that includes file uploads to jenkins. It gets hairy fairly quickly, but, as they say, pull requests accepted.

This method takes three arguments 1) String - the URL to post to 2) Array - Ordered array of name=VALUE curl form parameters 3) Hash - Options to be set



221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
 
# File 'lib/packaging/util/net.rb', line 221

def curl_form_data(uri, form_data = [], options = {})
  curl = Pkg::Util::Tool.check_tool("curl")
  #
  # Begin constructing the post string.
  # First, assemble the form_data arguments
  #
  post_string = "-i "
  form_data.each do |param|
    post_string << "#{param} "
  end

  # Add the uri
  post_string << "'#{uri}'"

  # If this is quiet, we're going to silence all output
  begin
    stdout, _, retval = Pkg::Util::Execution.capture3("#{curl} #{post_string}")
    if options[:quiet]
      stdout = ''
    end
    return stdout, retval
  rescue RuntimeError => e
    puts e
    return false
  end
end

.escape_html(uri) ⇒ Object 



330
331
332
333
 
# File 'lib/packaging/util/net.rb', line 330

def escape_html(uri)
  require 'cgi'
  CGI.escapeHTML(uri)
end

.fetch_uri(uri, target) ⇒ Object

This simple method does an HTTP get of a URI and writes it to a file in a slightly more platform agnostic way than curl/wget



9
10
11
12
13
14
 
# File 'lib/packaging/util/net.rb', line 9

def fetch_uri(uri, target)
  require 'open-uri'
  if Pkg::Util::File.file_writable?(File.dirname(target))
    File.open(target, 'w') { |f| f.puts(open(uri).read) }
  end
end

.hostname ⇒ Object

Get the hostname of the current host



17
18
19
20
 
# File 'lib/packaging/util/net.rb', line 17

def hostname
  require 'socket'
  Socket.gethostname
end

.print_url_info(url_string) ⇒ Object

Use the provided URL string to print important information with ASCII emphasis



262
263
264
265
266
 
# File 'lib/packaging/util/net.rb', line 262

def print_url_info(url_string)
  puts "\n////////////////////////////////////////////////////////////////////////////////\n\n
  Build submitted. To view your build progress, go to\n#{url_string}\n\n
////////////////////////////////////////////////////////////////////////////////\n\n"
end

.remote_bootstrap(host, treeish, tar_cmd = nil, tarball = nil) ⇒ Object

We take a tar argument for cases where ‘tar` isn’t best, e.g. Solaris. We also take an optional argument of the tarball containing the git bundle to use.



350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
 
# File 'lib/packaging/util/net.rb', line 350

def remote_bootstrap(host, treeish, tar_cmd = nil, tarball = nil)
  unless tar = tar_cmd
    tar = 'tar'
  end
  tarball ||= Pkg::Util::Git.bundle(treeish)
  tarball_name = File.basename(tarball).gsub('.tar.gz', '')
  Pkg::Util::Net.rsync_to(tarball, host, '/tmp')
  appendix = Pkg::Util.rand_string
  command = <<-DOC
#{tar} -zxvf /tmp/#{tarball_name}.tar.gz -C /tmp/ ;
git clone --recursive /tmp/#{tarball_name} /tmp/#{Pkg::Config.project}-#{appendix} ;
cd /tmp/#{Pkg::Config.project}-#{appendix} ;
bundle_prefix= ;
if [[ -r Gemfile ]]; then
  source /usr/local/rvm/scripts/rvm; rvm use ruby-2.4.1; bundle install --path .bundle/gems ;
  bundle_prefix='bundle exec' ;
fi ;
$bundle_prefix rake package:bootstrap
DOC
  Pkg::Util::Net.remote_ssh_cmd(host, command)
  "/tmp/#{Pkg::Config.project}-#{appendix}"
end

.remote_buildparams(host, build) ⇒ Object

Given a BuildInstance object and a host, send its params to the host. Return the remote path to the params.



375
376
377
378
379
380
381
 
# File 'lib/packaging/util/net.rb', line 375

def remote_buildparams(host, build)
  params_file = build.config_to_yaml
  params_file_name = File.basename(params_file)
  params_dir = Pkg::Util.rand_string
  Pkg::Util::Net.rsync_to(params_file, host, "/tmp/#{params_dir}/")
  "/tmp/#{params_dir}/#{params_file_name}"
end

.remote_create_latest_symlink(package_name, dir, platform_ext, options = {}) ⇒ Object

Create a symlink indicating the latest version of a package

Parameters:

  • package_name (String) —

    The name of the package you want to symlink to, e.g. ‘puppet-agent’, ‘facter’, etc.

  • dir (String) —

    The directory you want to find the latest package and create the symlink in.

  • platform_ext (String) —

    The type of files you want to consider, e.g. ‘dmg’, ‘msi’, etc.

  • options (Hash) (defaults to: {}) —

    Additional optional params: @option :arch [String] Architecture you want to narrow your search by. @option :excludes [Array] Strings you want to exclude from your search,

    e.g. 'agent' if only searching for 'puppet'.



295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
 
# File 'lib/packaging/util/net.rb', line 295

def remote_create_latest_symlink(package_name, dir, platform_ext, options = {})
  ls_cmd = "ls -1 *.#{platform_ext} | grep -v latest | grep -v rc | grep #{package_name} "

  # store this in a separate var to avoid side affects
  full_package_name = String.new(package_name)

  if options[:arch]
    ls_cmd << "| grep #{options[:arch]}"
    full_package_name << "-#{options[:arch]}"
  end
  if options[:excludes]
    options[:excludes].each do |excl|
      ls_cmd << "| grep -v #{excl} "
    end
  end
  ls_cmd << '| sort --version-sort | tail -1'
  cmd = <<-CMD
    if [ ! -d '#{dir}' ] ; then
      echo "directory '#{dir}' does not exist, not creating latest package link"
      exit 0
    fi
    pushd '#{dir}'
    link_target=$(#{ls_cmd})
    if [ -z "$link_target" ] ; then
      echo "Unable to find a link target for '#{full_package_name}' in '#{dir}'; skipping link creation"
      exit 0
    fi
    echo "creating link to '$link_target'"
    ln -sf "$link_target" #{full_package_name}-latest.#{platform_ext}
  CMD

  _, err = Pkg::Util::Net.remote_ssh_cmd(Pkg::Config.staging_server, cmd, true)
  $stderr.puts err
end

.remote_set_immutable(host, files) ⇒ Object

Remotely set the immutable bit on a list of files



279
280
281
 
# File 'lib/packaging/util/net.rb', line 279

def remote_set_immutable(host, files)
  Pkg::Util::Net.remote_ssh_cmd(host, "sudo chattr +i #{files.join(" ")}")
end

.remote_set_ownership(host, owner, group, files) ⇒ Object 



268
269
270
271
 
# File 'lib/packaging/util/net.rb', line 268

def remote_set_ownership(host, owner, group, files)
  remote_cmd = "for file in #{files.join(" ")}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chown #{owner}:#{group} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_ssh_cmd(host, remote_cmd)
end

.remote_set_permissions(host, permissions, files) ⇒ Object 



273
274
275
276
 
# File 'lib/packaging/util/net.rb', line 273

def remote_set_permissions(host, permissions, files)
  remote_cmd = "for file in #{files.join(" ")}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chmod #{permissions} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_ssh_cmd(host, remote_cmd)
end

.remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true) ⇒ Object 



67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
 
# File 'lib/packaging/util/net.rb', line 67

def remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true)
  ssh = Pkg::Util::Tool.check_tool('ssh')

  # we pass some pretty complicated commands in via ssh. We need this to fail
  # if any part of the remote ssh command fails.
  command = "set -e; #{command}" if fail_fast
  cmd = "#{ssh} #{extra_options} -t #{target} '#{command.gsub("'", "'\\\\''")}'"

  # This is NOT a good way to support this functionality.
  # It needs to be refactored into a set of methods that
  # other methods can use to safely and deterministically
  # support dry-run operations.
  # But I need to ship packages RIGHT NOW.
  # - Ryan McKern, 13/01/2016
  if ENV['DRYRUN']
    puts "[DRY-RUN] Executing '#{command}' on #{target}"
    puts "[DRY-RUN] #{cmd}"
    return
  end

  puts "Executing '#{command}' on #{target}"
  if capture_output
    stdout, stderr, exitstatus = Pkg::Util::Execution.capture3(cmd)
    Pkg::Util::Execution.success?(exitstatus) or raise "Remote ssh command failed."
    return stdout, stderr
  else
    Kernel.system(cmd)
    Pkg::Util::Execution.success? or raise "Remote ssh command failed."
  end
end

.rsync_cmd(origin_path, opts = {}) ⇒ String

Construct a valid rsync command

Parameters:

  • origin_path (String, Pathname) —

    the path to sync from; if opts is not passed, then the parent directory of ‘origin_path` will be used to construct a target path to sync to.

  • opts (Hash) (defaults to: {}) —

    additional options that can be used to construct the rsync command.

Options Hash (opts):

  • :bin (String) — default: 'rsync' —

    the path to rsync (can be relative or fully qualified).

  • :origin_host (String) —

    the remote host to sync data from; cannot be specified alongside :target_host

  • :target_host (String) —

    the remote host to sync data to; cannot be specified alongside :origin_host.

  • :extra_flags (String) — default: ["--ignore-existing"] —

    extra flags to use when constructing an rsync command

  • :dryrun (String) — default: false —

    tell rsync to perform a trial run with no changes made.

Returns:

  • (String) —

    a rsync command that can be used in shell or ssh methods

Raises:

  • (ArgumentError) —

    if opts and opts names are both defined.

  • (ArgumentError) —

    if :origin_path exists without opts, opts, remote target is defined.



119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/packaging/util/net.rb', line 119

def rsync_cmd(origin_path, opts = {})
  options = {
    bin: 'rsync',
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: false }.merge(opts)
  origin = Pathname.new(origin_path)
  target = options[:target_path] || origin.parent

  raise(ArgumentError, "Cannot sync between two remote hosts") if
    options[:origin_host] && options[:target_host]

  raise(ArgumentError, "Cannot sync path '#{origin}' to itself") unless
    options[:origin_host] || options[:target_host]

  cmd = %W(
    #{options[:bin]}
    --recursive
    --hard-links
    --links
    --verbose
    --omit-dir-times
    --no-perms
    --no-owner
    --no-group
  ) + [*options[:extra_flags]]

  cmd << '--dry-run' if options[:dryrun]
  cmd << Pkg::Util.pseudo_uri(path: origin, host: options[:origin_host])
  cmd << Pkg::Util.pseudo_uri(path: target, host: options[:target_host])

  cmd.uniq.compact.join("\s")
end

.rsync_exec(source, opts = {}) ⇒ Object

A generic rsync execution method that wraps rsync_cmd in a call to Pkg::Util::Execution#capture3()



157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/packaging/util/net.rb', line 157

def rsync_exec(source, opts = {})
  options = {
    bin: Pkg::Util::Tool.check_tool('rsync'),
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: ENV['DRYRUN'] }.merge(opts.delete_if { |_, value| value.nil? })

  stdout, _, _ = Pkg::Util::Execution.capture3(rsync_cmd(source, options), true)
  stdout
end

.rsync_from(source, origin_host, dest, opts = {}) ⇒ Object

A wrapper method to maintain the existing interface for executing incoming rsync commands with minimal changes to existing code.



185
186
187
188
189
190
191
192
193
194
 
# File 'lib/packaging/util/net.rb', line 185

def rsync_from(source, origin_host, dest, opts = {})
  rsync_exec(
    source,
    origin_host: origin_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] }) ⇒ Object

A wrapper method to maintain the existing interface for executing outbound rsync commands with minimal changes to existing code.



172
173
174
175
176
177
178
179
180
181
 
# File 'lib/packaging/util/net.rb', line 172

def rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] })
  rsync_exec(
    source,
    target_host: target_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.s3sync_to(source, target_bucket, target_directory = "", flags = []) ⇒ Object 



196
197
198
199
200
201
202
203
204
205
 
# File 'lib/packaging/util/net.rb', line 196

def s3sync_to(source, target_bucket, target_directory = "", flags = [])
  s3cmd = Pkg::Util::Tool.check_tool('s3cmd')

  if Pkg::Util::File.file_exists?(File.join(ENV['HOME'], '.s3cfg'))
    stdout, _, _ = Pkg::Util::Execution.capture3("#{s3cmd} sync #{flags.join(' ')} '#{source}' s3://#{target_bucket}/#{target_directory}/")
    stdout
  else
    fail "#{File.join(ENV['HOME'], '.s3cfg')} does not exist. It is required to ship files using s3cmd."
  end
end

.uri_status_code(uri) ⇒ Object 



248
249
250
251
252
253
254
255
256
257
258
 
# File 'lib/packaging/util/net.rb', line 248

def uri_status_code(uri)
  data = [
    '--request GET',
    '--silent',
    '--location',
    '--write-out "%{http_code}"',
    '--output /dev/null'
  ]
  stdout, _ = Pkg::Util::Net.curl_form_data(uri, data)
  stdout
end