Class: Pkg::ManageArtifactory

Inherits:
Object
  • Object
show all
Defined in:
lib/packaging/artifactory.rb

Overview

The Artifactory class This class provides automation to access the artifactory repos maintained by the Release Engineering team at Puppet. It has the ability to both push artifacts to the repos, and to retrieve them back from the repos.

Constant Summary collapse

ARTIFACTORY_CLEANUP_SKIP_PROPERTY =

The Artifactory property that the artifactCleanup user plugin https://github.com/jfrog/artifactory-user-plugins/tree/master/cleanup/artifactCleanup uses to tell it to not clean a particular artifact

'cleanup.skip'
DEFAULT_REPO_TYPE = 
'generic'
DEFAULT_REPO_BASE = 
'development'

Instance Method Summary collapse

Constructor Details

#initialize(project, project_version, opts = {}) ⇒ ManageArtifactory

Returns a new instance of ManageArtifactory.

Parameters:

  • project (String) —

    The name of the project this package is for

  • project_version (String) —

    The version of the project we want the package for. This can be one of three things:

    1) the final tag of the project the packages  were built from
2) the long git sha the project the packages were built from
3) the EZBake generated development sha where the packages live
  • :artifactory_uri (Hash) —

    a customizable set of options

  • :repo_base (Hash) —

    a customizable set of options



31
32
33
34
35
36
37
38
39
 
# File 'lib/packaging/artifactory.rb', line 31

def initialize(project, project_version, opts = {})
  @artifactory_uri = opts[:artifactory_uri] || 'https://artifactory.delivery.puppetlabs.net/artifactory'
  @repo_base = opts[:repo_base] || DEFAULT_REPO_BASE

  @project = project
  @project_version = project_version

  Artifactory.endpoint = @artifactory_uri
end

Instance Method Details

#all_package_names(platform_data, platform_tag) ⇒ Array

Returns An array containing all packages for the given project, project_version, and platform_tag.

Parameters:

  • platform_data (Hash) —

    The hash of the platform data that needs to be parsed

  • platform_tag (String) —

    The tag that the data we want belongs to

Returns:

  • (Array) —

    An array containing all packages for the given project, project_version, and platform_tag



267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
 
# File 'lib/packaging/artifactory.rb', line 267

def all_package_names(platform_data, platform_tag)
  packages = [platform_data[platform_tag][:artifact]]
  packages << platform_data[platform_tag][:additional_artifacts]
  packages.flatten!
  packages.reject! { |package| package.nil? || package.empty? }
  packages.map { |package| File.basename(package) }
rescue StandardError
  fail_message = <<-DOC
  Package name could not be found from loaded yaml data. Either this package
  does not exist, or '#{platform_tag}' is not present in this dataset.

  The following are available platform tags for '#{@project}' '#{@project_version}':
#{platform_data.keys.sort}
  DOC
  raise fail_message
end

#artifact_paths(package) ⇒ Object

Return a list of artifact paths for package. Returns empty array if none.



196
197
198
199
200
201
202
 
# File 'lib/packaging/artifactory.rb', line 196

def artifact_paths(package)
  check_authorization
  Artifactory::Resource::Artifact.search(
    name: File.basename(package),
    artifactory_uri: @artifactory_uri
  )
end

#copy_artifact(artifact, target_repo, target_path, target_debian_component = nil) ⇒ Object

Copy an artifact to a target repo/path

Parameters:

  • artifact (Artifactory::Resource::Artifact) —

    The artifact to be copied

  • target_repo (String) —

    The repository to copy the artifact to

  • target_path (String) —

    The path in the target repository to copy the artifact to

  • target_debian_component (String) (defaults to: nil) —

    ‘deb.component` property to set on the copied artifact defaults to `Pkg::Paths.debian_component_from_path(target_path)`



589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
 
# File 'lib/packaging/artifactory.rb', line 589

def copy_artifact(artifact, target_repo, target_path, target_debian_component = nil)
  filename = File.basename(artifact.download_uri)
  artifactory_target_path = "#{target_repo}/#{target_path}"
  puts "Copying #{artifact.download_uri} to #{artifactory_target_path}"
  begin
    artifact.copy(artifactory_target_path)
  rescue Artifactory::Error::HTTPError
    warn "Could not copy #{artifactory_target_path}. Source and destination are the same. Skipping..."
  end

  if File.extname(filename) == '.deb'
    target_debian_component ||= Pkg::Paths.debian_component_from_path(target_path)
    copied_artifact_search = search_with_path(filename, target_repo, target_path)
    fail "Error: what the hell, could not find just-copied package #{filename} under #{target_repo}/#{target_path}" if copied_artifact_search.empty?
    copied_artifact = copied_artifact_search.first
    properties = { 'deb.component' => target_debian_component }
    copied_artifact.properties(properties)
  end
end

#copy_final_pe_tarballs(pe_version, repo, remote_path, target_path) ⇒ Object

When we ship a new PE release we copy final tarballs to archives/releases

Parameters:

  • pe_version (String) —

    pe final tag

  • repo (String) —

    repo the tarballs live

  • remote_path (String) —

    path to tarballs in the repo

  • target_path (String) —

    path copy tarballs to, assumes same repo



568
569
570
571
572
573
574
575
576
577
578
579
580
 
# File 'lib/packaging/artifactory.rb', line 568

def copy_final_pe_tarballs(pe_version, repo, remote_path, target_path)
  check_authorization
  final_tarballs = Artifactory::Resource::Artifact.search(name: pe_version, repos: repo, exact_match: false)
  final_tarballs.each do |artifact|
    next unless artifact.download_uri.include? remote_path
    next if artifact.download_uri.include? "-rc"
    filename = File.basename(artifact.download_uri)
    # Artifactory does NOT like when you use `File.join`, so let's concatenate!
    full_target_path = "#{repo}/#{target_path}/#{filename}"
    puts "INFO: Copying #{filename} to #{full_target_path} . . ."
    artifact.copy(full_target_path)
  end
end

#deb_list_contents(platform_tag) ⇒ String

Returns The contents of the debian list file to enable the debian artifactory repos for the specified project and version.

Parameters:

  • platform_tag (String) —

    The platform to generate the list contents for

Returns:

  • (String) —

    The contents of the debian list file to enable the debian artifactory repos for the specified project and version



105
106
107
108
109
110
111
 
# File 'lib/packaging/artifactory.rb', line 105

def deb_list_contents(platform_tag)
  data = platform_specific_data(platform_tag)
  if data[:package_format] == 'deb'
    return "deb #{@artifactory_uri}/#{data[:repo_name]} #{data[:codename]} #{data[:repo_subdirectories]}"
  end
  raise "The platform '#{platform_tag}' is not an apt-based system."
end

#deploy_package(package) ⇒ Object

Parameters:

  • package (String) —

    The full relative path to the package to be shipped, relative from the current working directory



216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
 
# File 'lib/packaging/artifactory.rb', line 216

def deploy_package(package)
  platform_tag = Pkg::Paths.tag_from_artifact_path(package) || DEFAULT_REPO_TYPE
  data = platform_specific_data(platform_tag)

  check_authorization
  artifact = Artifactory::Resource::Artifact.new(local_path: package)
  artifact_md5 = Digest::MD5.file(package).hexdigest
  headers = { "X-Checksum-Md5" => artifact_md5 }
  artifact.upload(
    data[:repo_name],
    File.join(data[:repo_subdirectories], File.basename(package)),
    deploy_properties(platform_tag, File.basename(package)),
    headers
  )
end

#deploy_properties(platform_tag, file_name) ⇒ String

Returns Any required extra bits that we need for the curl command used to deploy packages to artifactory

These are a few examples from chef/artifactory-client. These could potentially be very powerful, but we should decide how to use them.

status: 'DEV',
rating: 5,
branch: 'master'

Currently we are including everything that would be included in the yaml file that is generated at package build time.

Parameters:

  • platform_tag (String) —

    The platform tag to generate deploy properties for

Returns:

  • (String) —

    Any required extra bits that we need for the curl command used to deploy packages to artifactory

    These are a few examples from chef/artifactory-client. These could potentially be very powerful, but we should decide how to use them.

    status: 'DEV',
rating: 5,
branch: 'master'

    Currently we are including everything that would be included in the yaml file that is generated at package build time.



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
 
# File 'lib/packaging/artifactory.rb', line 170

def deploy_properties(platform_tag, file_name)
  data = platform_specific_data(platform_tag)

  # TODO This method should be returning the entire contents of the yaml
  # file in hash form to include as metadata for these artifacts. In this
  # current iteration, the hash isn't formatted properly and the attempt to
  # deploy to Artifactory bails out. I'm leaving this in so that we at least
  # have multiple places to remind us that it needs to happen.
  #properties_hash = Pkg::Config.config_to_hash
  properties_hash = {}
  if data[:package_format] == 'deb'
    architecture = data[:architecture]
    # set arch correctly for noarch packages
    if file_name =~ /_all\.deb$/
      architecture = 'all'
    end
    properties_hash.merge!({
      'deb.distribution' => data[:codename],
      'deb.component' => data[:repo_subdirectories],
      'deb.architecture' => architecture,
    })
  end
  properties_hash
end

#download_artifact(artifact_name, repo, path, target: '.', filename: nil) ⇒ Object

Download an artifact based on name, repo, and path to artifact

Parameters:

  • artifact_name (String) —

    name of artifact to download

  • repo (String) —

    repo the artifact lives

  • path (String) —

    path to artifact in the repo

  • target (String) (defaults to: '.') —

    directory to download artifact to. Defaults to ‘.’

  • filename (String) (defaults to: nil) —

    Filename to save artifact as. Defaults to artifact_name



525
526
527
528
529
530
531
532
 
# File 'lib/packaging/artifactory.rb', line 525

def download_artifact(artifact_name, repo, path, target: '.', filename: nil)
  filename ||= artifact_name
  artifacts = search_with_path(artifact_name, repo, path)
  return nil if artifacts.empty?
  # Only download the first of the artifacts since we're saving them to
  # the same location anyways
  artifacts.first.download(target, filename: filename)
end

#download_beta_pe_tarballs(beta_tag, repo, remote_path, local_path) ⇒ Object

Download beta pe tarballs to local path based on tag, repo, and path on artifactory

Parameters:

  • beta_tag (String) —

    rc tag of beta release ex. 2019.2.0-rc10

  • repo (String) —

    repo the tarballs live

  • remote_path (String) —

    path to tarballs in the repo

  • local_path (String) —

    local path to download tarballs to



554
555
556
557
558
559
560
561
 
# File 'lib/packaging/artifactory.rb', line 554

def download_beta_pe_tarballs(beta_tag, repo, remote_path, local_path)
  check_authorization
  pattern = "#{remote_path}/*-#{beta_tag}-*"
  artifacts = Artifactory::Resource::Artifact.pattern_search(repo: repo, pattern: pattern)
  artifacts.each do |artifact|
    artifact.download(local_path)
  end
end

#download_final_pe_tarballs(pe_version, repo, remote_path, local_path) ⇒ Object

Download final pe tarballs to local path based on name, repo, and path on artifactory

Parameters:

  • pe_version (String) —

    pe final tag

  • repo (String) —

    repo the tarballs live

  • remote_path (String) —

    path to tarballs in the repo

  • local_path (String) —

    local path to download tarballs to



539
540
541
542
543
544
545
546
547
 
# File 'lib/packaging/artifactory.rb', line 539

def download_final_pe_tarballs(pe_version, repo, remote_path, local_path)
  check_authorization
  artifacts = Artifactory::Resource::Artifact.search(name: pe_version, repos: repo, exact_match: false)
  artifacts.each do |artifact|
    next unless artifact.download_uri.include? remote_path
    next if artifact.download_uri.include? "-rc"
    artifact.download(local_path)
  end
end

#download_packages(staging_directory, manifest, remote_path = '') ⇒ Object

Parameters:

  • staging_directory (String) —

    location to download packages to

  • manifest (File) —

    JSON file listing which packages to download and the corresponding md5sums

  • remote_path (String) (defaults to: '') —

    Optional partial path on the remote host containing packages Used to specify which subdirectories packages will be downloaded from.



357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
 
# File 'lib/packaging/artifactory.rb', line 357

def download_packages(staging_directory, manifest, remote_path = '')
  check_authorization
  download_repositories = %w[rpm_enterprise__local debian_enterprise__local]
  manifest.each do |dist, packages|
    if packages.nil?
      warn "Package list for #{dist} is empty, skipping"
      next
    end
    packages.each_value do |info|
      package_file_name = info['filename']
      puts format(
        "Searching Artifactory [%s]%s for %s (md5: %s)",
             download_repositories.join(', '),
             remote_path.empty? ? '' : "/#{remote_path}",
             package_file_name,
             info['md5']
      )
      artifacts = Artifactory::Resource::Artifact.checksum_search(
        md5: info['md5'],
        repos: download_repositories,
        name: package_file_name
      )

      artifact_to_download = artifacts.select do |artifact|
        artifact.download_uri.include? remote_path
      end.first

      # If we found matching artifacts, but not in the correct
      # path, copy the artifact to the correct path This should
      # help us keep repos up to date with the packages we are
      # expecting to be there while helping us avoid 'could not
      # find package' errors
      if artifact_to_download.nil? && !artifacts.empty?
        artifact_to_copy = artifacts.first
        copy_artifact(artifact_to_copy, artifact_to_copy.repo,
                      "#{remote_path}/#{dist}/#{package_file_name}")

        # Now, search again to make sure we find them in the correct path.
        artifacts = Artifactory::Resource::Artifact.checksum_search(
          md5: info['md5'],
          repos: download_repositories,
          name: package_file_name
        )
        artifact_to_download = artifacts.select do |artifact|
          artifact.download_uri.include? remote_path
        end.first
      end

      if artifact_to_download.nil?
        message = "Error: could not find package #{package_file_name} " \
                  "with md5sum #{info['md5']}"
        message += " in #{remote_path}" unless remote_path.empty?
        raise message
      end

      full_staging_path = "#{staging_directory}/#{dist}"
      puts "Downloading #{artifact_to_download.download_uri} to " \
           "#{File.join(full_staging_path, package_file_name)}"
      artifact_to_download.download(full_staging_path, filename: package_file_name)
    end
  end
end

#location_for(platform_tag) ⇒ Array

Returns An array containing three items, first being the main repo name for the platform_tag, the second being the subdirectories of the repo leading to the artifact we want to install, and the third being the alternate subdirectories for a given repo. This last option is only currently used for debian platforms, where the path to the repo specified in the list file is different than the full path to the repo.

Parameters:

  • platform_tag (String) —

    The platform tag string for the repo we need information on. If generic information is needed, pass in ‘generic`

Returns:

  • (Array) —

    An array containing three items, first being the main repo name for the platform_tag, the second being the subdirectories of the repo leading to the artifact we want to install, and the third being the alternate subdirectories for a given repo. This last option is only currently used for debian platforms, where the path to the repo specified in the list file is different than the full path to the repo.



49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/packaging/artifactory.rb', line 49

def location_for(platform_tag)
  toplevel_repo = DEFAULT_REPO_TYPE
  repo_subdirectories = File.join(@repo_base, @project, @project_version)

  unless platform_tag == DEFAULT_REPO_TYPE
    format = Pkg::Platforms.package_format_for_tag(platform_tag)
    platform, version, architecture = Pkg::Platforms.parse_platform_tag(platform_tag)
  end

  case format
  when 'rpm'
    toplevel_repo = 'rpm'
    repo_subdirectories = File.join(repo_subdirectories, "#{platform}-#{version}-#{architecture}")
  when 'deb'
    toplevel_repo = 'debian__local'
    repo_subdirectories = File.join(repo_subdirectories, "#{platform}-#{version}")
  when 'swix', 'dmg', 'svr4', 'ips'
    repo_subdirectories = File.join(repo_subdirectories, "#{platform}-#{version}-#{architecture}")
  when 'msi'
    repo_subdirectories = File.join(repo_subdirectories, "#{platform}-#{architecture}")
  end

  [toplevel_repo, repo_subdirectories]
end

#package_exists_on_artifactory?(package) ⇒ Boolean

Check if a package exists on artifactory Return true if package already exists on artifactory. #artifact_paths, above, is more useful since it will inform where they are. This is kept for backward compatibility.

Parameters:

  • package (String) —

    The full relative path to the package to be checked, relative from the current working directory

Returns:

  • (Boolean) 


210
211
212
 
# File 'lib/packaging/artifactory.rb', line 210

def package_exists_on_artifactory?(package)
  artifact_paths(package).any?
end

#package_name(platform_data, platform_tag) ⇒ String

Returns The name of the package for the given project, project_version, and platform_tag.

Parameters:

  • platform_data (Hash) —

    The hash of the platform data that needs to be parsed

  • platform_tag (String) —

    The tag that the data we want belongs to

Returns:

  • (String) —

    The name of the package for the given project, project_version, and platform_tag



249
250
251
252
253
254
255
256
257
258
259
260
 
# File 'lib/packaging/artifactory.rb', line 249

def package_name(platform_data, platform_tag)
  return File.basename(platform_data[platform_tag][:artifact])
rescue StandardError
  fail_message = <<-DOC
  Package name could not be found from loaded yaml data. Either this package
  does not exist, or '#{platform_tag}' is not present in this dataset.

  The following are available platform tags for '#{@project}' '#{@project_version}':
#{platform_data.keys.sort}
  DOC
  raise fail_message
end

#platform_specific_data(platform_tag) ⇒ Hash

Returns a hash of data specific to this platform tag

Parameters:

  • platform_tag (String) —

    The platform tag specific to the information we need. If only the generic information is needed, pass in ‘generic`

Returns:

  • (Hash) —

    Returns a hash of data specific to this platform tag



77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/packaging/artifactory.rb', line 77

def platform_specific_data(platform_tag)
  unless platform_tag == DEFAULT_REPO_TYPE
    platform, version, architecture = Pkg::Platforms.parse_platform_tag(platform_tag)
    package_format = Pkg::Platforms.package_format_for_tag(platform_tag)
    if package_format == 'deb'
      codename = Pkg::Platforms.codename_for_platform_version(platform, version)
    end
  end

  repo_name, repo_subdirectories = location_for(platform_tag)
  full_artifactory_path = File.join(repo_name, repo_subdirectories)

  {
    platform: platform,
    platform_version: version,
    architecture: architecture,
    codename: codename,
    package_format: package_format,
    repo_name: repo_name,
    repo_subdirectories: repo_subdirectories,
    full_artifactory_path: full_artifactory_path
  }
end

#populate_pe_repos(manifest, target_path) ⇒ Object

When we cut a new PE branch, we need to copy the pe components into <pe_version>/repos,feature,release/<platform>

Parameters:

  • manifest (File) —

    JSON file containing information about which packages to download and their corresponding md5sums

  • target_path (String) —

    path on artifactory to copy components to Example: <pe_version>/release



615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
 
# File 'lib/packaging/artifactory.rb', line 615

def populate_pe_repos(manifest, target_path)
  check_authorization
  manifest.each do |dist, packages|
    puts "Copying #{dist} packages..."
    packages.each_value do |info|
      filename = info['filename']
      artifact = Artifactory::Resource::Artifact.checksum_search(
        md5: (info['md5']).to_s,
        repos: ['rpm_enterprise__local', 'debian_enterprise__local'],
        name: filename
      ).first
      if artifact.nil?
        raise "Error: could not find package #{filename} with md5sum #{info['md5']}"
      end
      copy_artifact(artifact, artifact.repo, "#{target_path}/#{dist}/#{filename}")
    end
  end
end

#prevent_artifact_cleanup(repo, directory, pe_build_version) ⇒ Object

Start by clearing the ARTIFACTORY_CLEANUP_SKIP_PROPERTY on all artifacts in a single repo/directory location. This allows all artifacts in the directory to be cleaned. Once cleared, set ARTIFACTORY_CLEANUP_SKIP_PROPERTY on those matching pe_build_version, presumably the latest. This prevents those artifacts from being deleted.

Parameters:

  • repo (String) —

    Artifactory repository that contains the specified directory

  • directory (String) —

    Artifactory directory in repo containing the artifacts from which to set the ‘cleanup.skip’ property setting to false

  • pe_build_version (String) —

    Set ‘cleanup.skip’ property on artifacts that contain this string in their file inside the directory.



476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
 
# File 'lib/packaging/artifactory.rb', line 476

def prevent_artifact_cleanup(repo, directory, pe_build_version)
  # Clean up any trailing slashes on directory, just in case
  directory.sub!(/(\/)+$/, '')

  all_artifacts_pattern = "#{directory}/*"
  latest_artifacts_pattern = "#{directory}/*#{pe_build_version}*"

  all_artifacts = Artifactory::Resource::Artifact.pattern_search(
    repo: repo,
    pattern: all_artifacts_pattern
  )
  latest_artifacts = Artifactory::Resource::Artifact.pattern_search(
    repo: repo,
    pattern: latest_artifacts_pattern
  )

  # Clear cleanup.skip on all artifacts in directory
  puts "Clearing #{ARTIFACTORY_CLEANUP_SKIP_PROPERTY} in #{repo}/#{all_artifacts_pattern}"
  all_artifacts.each do |artifact|
    artifact.properties(ARTIFACTORY_CLEANUP_SKIP_PROPERTY => false)
  end

  # Set cleanup.skip on all artifacts in directory matching *pe_build_version*
  puts "Setting #{ARTIFACTORY_CLEANUP_SKIP_PROPERTY} in #{repo}/#{latest_artifacts_pattern}"
  latest_artifacts.each do |artifact|
    artifact.properties(ARTIFACTORY_CLEANUP_SKIP_PROPERTY => true)
  end
end

#promote_package(pkg, ref, platform_tag, repository, debian_component = nil) ⇒ Object

Promotes a build based on build SHA or tag (or SNAPSHOT version, for ezbake) Depending on if it’s an RPM or Deb package promote accordingly ‘promote’ by copying the package(s) to the enterprise directory on artifactory

Parameters:

  • pkg (String) —

    the package name ex. puppet-agent

  • ref (String) —

    tag or SHA of package(s) to be promoted

  • platform_tag (String) —

    the platform tag of the artifact ex. el-7-x86_64, ubuntu-18.04-amd64

  • repository (String) —

    the repository to promote the artifact to. Will prepend ‘rpm_’ or ‘debian_’ to the repositories depending on package type

  • debian_component (String) (defaults to: nil) —

    the debian component to promote packages into. Optional.



297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
 
# File 'lib/packaging/artifactory.rb', line 297

def promote_package(pkg, ref, platform_tag, repository, debian_component = nil)
  # load package metadata
  yaml_content = retrieve_yaml_data(pkg, ref)
  yaml_data = YAML::safe_load(yaml_content, [Symbol])

  # get the artifact name
  artifact_names = all_package_names(yaml_data[:platform_data], platform_tag)
  artifact_names.each do |artifact_name|
    artifact_search_results = Artifactory::Resource::Artifact.search(
      name: artifact_name, :artifactory_uri => @artifactory_uri
    )

    if artifact_search_results.empty?
      raise "Error: could not find PKG=#{pkg} at REF=#{ref} for #{platform_tag}"
    end
    artifact_to_promote = artifact_search_results[0]

    # This makes an assumption that we're using some consistent repo names
    # but need to either prepend 'rpm_' or 'debian_' based on package type
    case File.extname(artifact_name)
    when '.rpm'
      promotion_path = "rpm_#{repository}/#{platform_tag}/#{artifact_name}"
    when '.deb'
      promotion_path = "debian_#{repository}/#{platform_tag}/#{artifact_name}"
      properties = { 'deb.component' => debian_component } unless debian_component.nil?
    else
      raise "Error: Unknown promotion repository for #{artifact_name}! Only .rpm and .deb files are supported!"
    end

    begin
      source_path = artifact_to_promote.download_uri.sub(@artifactory_uri, '')
      puts "promoting #{artifact_name} from #{source_path} to #{promotion_path}"
      artifact_to_promote.copy(promotion_path)
      unless properties.nil?
        artifacts = Artifactory::Resource::Artifact.search(name: artifact_name, :artifactory_uri => @artifactory_uri)
        promoted_artifact = artifacts.select { |artifact| artifact.download_uri =~ %r{#{promotion_path}} }.first
        promoted_artifact.properties(properties)
      end
    rescue Artifactory::Error::HTTPError => e
      if e.message =~ /(destination and source are the same|user doesn't have permissions to override)/i
        puts "Skipping promotion of #{artifact_name}; it has already been promoted"
      else
        puts e.message.to_s
        raise e
      end
    rescue StandardError => e
      puts "Something went wrong promoting #{artifact_name}!"
      raise e
    end
  end
end

#purge_copied_pe_tarballs(tarball_path, pe_repo) ⇒ Object

Remove shipped PE tarballs from artifactory Used when compose fails, we only want the tarball shipped to artifactory if all platforms succeed Identify which packages were created and shipped based on md5sum and remove them

Parameters:

  • tarball_path (String) —

    the local path to the tarballs that were shipped

  • pe_repo (String) —

    the artifactory repo the tarballs were shipped to



679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
 
# File 'lib/packaging/artifactory.rb', line 679

def purge_copied_pe_tarballs(tarball_path, pe_repo)
  check_authorization
  Dir.foreach("#{tarball_path}/") do |pe_tarball|
    next if ['.', '..'].include?(pe_tarball)
    md5 = Digest::MD5.file("#{tarball_path}/#{pe_tarball}").hexdigest
    artifacts_to_delete = Artifactory::Resource::Artifact.checksum_search(md5: md5, repos: pe_repo, name: pe_tarball)
    next if artifacts_to_delete.nil?
    begin
      artifacts_to_delete.each do |artifact|
        puts "Removing #{pe_tarball} from #{pe_repo}... "
        artifact.delete
      end
    rescue Artifactory::Error::HTTPError
      warn "Error: cannot remove #{pe_tarball}, do you have the right permissions?"
    end
  end
end

#remove_promoted_packages(manifest, remote_path, package, repos) ⇒ Object

Remove promoted artifacts if promotion is reverted, use information provided in manifest

Parameters:

  • manifest (File) —

    JSON file containing information about what packages to download and the corresponding md5sums

  • remote_path (String) —

    path on artifactory to promoted packages ex. 2019.1/repos/

  • package (String) —

    package name ex. puppet-agent

  • repos (Array) —

    the repos the promoted artifacts live



654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
 
# File 'lib/packaging/artifactory.rb', line 654

def remove_promoted_packages(manifest, remote_path, package, repos)
  check_authorization
  manifest.each_value do |packages|
    packages.each do |package_name, info|
      next unless package_name == package
      filename = info['filename']
      artifacts = Artifactory::Resource::Artifact.checksum_search(
        md5: (info['md5']).to_s,
        repos: repos,
        name: filename
      )
      artifacts.each do |artifact|
        next unless artifact.download_uri.include? remote_path
        puts "Removing reverted package #{artifact.download_uri}"
        artifact.delete
      end
    end
  end
end

#retrieve_yaml_data(pkg, ref) ⇒ String

Returns The contents of the YAML file.

Parameters:

  • pkg (String) —

    The package to download YAML for i.e. ‘puppet-agent’ or ‘puppetdb’

  • ref (String) —

    The git ref (sha or tag) we want the YAML for

Returns:

  • (String) —

    The contents of the YAML file



237
238
239
240
241
242
 
# File 'lib/packaging/artifactory.rb', line 237

def retrieve_yaml_data(pkg, ref)
  yaml_url = "#{@artifactory_uri}/#{DEFAULT_REPO_TYPE}/#{DEFAULT_REPO_BASE}/#{pkg}/#{ref}/#{ref}.yaml"
  URI.parse(yaml_url).read
rescue StandardError
  raise "Failed to load YAML data for #{pkg} at #{ref} from #{yaml_url}!"
end

#rpm_repo_contents(platform_tag) ⇒ String

Returns The contents of the rpm repo file to enable the rpm artifactory repo for the specified project and version.

Parameters:

  • platform_tag (String) —

    The platform to generate the repo file contents for

Returns:

  • (String) —

    The contents of the rpm repo file to enable the rpm artifactory repo for the specified project and version



117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/packaging/artifactory.rb', line 117

def rpm_repo_contents(platform_tag)
  data = platform_specific_data(platform_tag)
  if data[:package_format] == 'rpm'
    return <<-DOC
  [Artifactory #{@project} #{@project_version} for #{platform_tag}]
  name=Artifactory Repository for #{@project} #{@project_version} for #{platform_tag}
  baseurl=#{@artifactory_uri}/#{data[:repo_name]}/#{data[:repo_subdirectories]}
  enabled=1
  gpgcheck=0
  #Optional - if you have GPG signing keys installed, use the below flags to verify the repository metadata signature:
  #gpgkey=#{@artifactory_uri}/#{data[:repo_name]}/#{data[:repo_subdirectories]}/repomd.xml.key
  #repo_gpgcheck=1
    DOC
  end
  raise "The platform '#{platform_tag}' is not a yum-based system"
end

#search_with_path(artifact_id, repo, path) ⇒ Array<Artifactory::Resource::Artifact>

Search for artifacts matching ‘artifact_name` in `repo` with path matching `path`

Parameters:

  • artifact_name (String) —

    name of artifact to download

  • repo (String) —

    repo the artifact lives

  • path (String) —

    path to artifact in the repo

Returns:

  • (Array<Artifactory::Resource::Artifact>) —

    A list of artifacts that match the query



513
514
515
516
517
 
# File 'lib/packaging/artifactory.rb', line 513

def search_with_path(artifact_id, repo, path)
  check_authorization
  artifacts = Artifactory::Resource::Artifact.search(name: artifact_id, repos: repo)
  artifacts.select { |artifact| artifact.download_uri.include? path }
end

#ship_pe_tarballs(local_tarball_directory, target_repo, ship_paths) ⇒ Object

Ship PE tarballs to specified artifactory repo and paths

Parameters:

  • local_tarball_directory (String) —

    the local directory containing the tarballs

  • target_repo (String) —

    the artifactory repo to ship the tarballs to

  • ship_paths (Array) —

    the artifactory path(s) to ship the tarballs to within the target_repo



425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
 
# File 'lib/packaging/artifactory.rb', line 425

def ship_pe_tarballs(local_tarball_directory, target_repo, ship_paths)
  check_authorization
  ship_paths.each do |path|
    Dir.foreach(local_tarball_directory) do |pe_tarball|
      next if ['.', '..'].include?(pe_tarball)
      begin
        puts "Uploading #{pe_tarball} to #{target_repo}/#{path}#{pe_tarball}"
        artifact = Artifactory::Resource::Artifact.new(
          local_path: "#{local_tarball_directory}/#{pe_tarball}"
        )
        artifact.upload(target_repo, "#{path}#{pe_tarball}")
      rescue Errno::EPIPE
        warn "Warning: Could not upload #{pe_tarball} to #{target_repo}/#{path}. Skipping."
        next
      end
    end
  end
end

#teardown_repo(repos, pattern) ⇒ Object

Remove all artifacts in repo based on pattern, used when we purge all artifacts in release/ after PE release

Parameters:

  • repos (Array) —

    repos that we want to search for artifacts in

  • pattern (String) —

    pattern for artifacts that should be deleted ex. ‘2019.1/release//`



637
638
639
640
641
642
643
644
645
646
 
# File 'lib/packaging/artifactory.rb', line 637

def teardown_repo(repos, pattern)
  check_authorization
  repos.each do |repo|
    artifacts = Artifactory::Resource::Artifact.pattern_search(repo: repo, pattern: pattern)
    artifacts.each do |artifact|
      puts "Deleting #{artifact.download_uri}"
      artifact.delete
    end
  end
end

#upload_file(local_path, target_repo, target_path, properties = {}, headers = {}) ⇒ Object

Upload file to Artifactory

Parameters:

  • local_path (String) —

    local path to file to upload

  • target_repo (String) —

    repo on artifactory to upload to

  • target_path (String) —

    path within target_repo to upload to

  • properties (Hash) (defaults to: {}) —

    Optional property names and values to assign the uploaded file For example, this would set both the ‘cleanup.skip’ and ‘deb.component’ properties: { “cleanup.skip” => true, “deb.component” => ‘bionic’ }

  • headers (Hash) (defaults to: {}) —

    Optional upload headers, most likely checksums, for the upload request “X-Checksum-Md5” and “X-Checksum-Sha1” are typical



453
454
455
456
457
458
459
460
461
462
463
464
 
# File 'lib/packaging/artifactory.rb', line 453

def upload_file(local_path, target_repo, target_path, properties = {}, headers = {})
  fail "Error: Couldn't find file at #{local_path}." unless File.exist? local_path
  check_authorization
  artifact = Artifactory::Resource::Artifact.new(local_path: local_path)
  full_upload_path = File.join(target_path, File.basename(local_path))
  begin
    puts "Uploading #{local_path} to #{target_repo}/#{full_upload_path} . . ."
    artifact.upload(target_repo, full_upload_path, properties, headers)
  rescue Artifactory::Error::HTTPError
    fail "Error: Upload failed. Ensure path #{target_path} exists in the #{target_repo} repository."
  end
end