Vara

A gem for building products to be consumed by Ops Manager.

Complementary tools

While Vara focuses on building a .pivotal from a complete product repository, you may find some of the following tools complement your workflow with Vara:

  • Vöxtur to automatically update metadata_parts/binaries.yml from a built release tarball.

Usage

Build .pivotal file

In most circumstances, you will only need to run vara build-pivotal PRODUCT_DIR, which calls other lower-level vara commands.

vara build-pivotal ~/workspace/p-runtime

will build a .pivotal file using the stemcell, release tarball, and compiled packages as specified by the product metadata. If the stemcell, release tarball, or compiled packages do not exist on disk, they will be downloaded.

The product file will be named #{product_name}-#{product_version}.pivotal, where both variables come from the product metadata YAML.

The product file's contents will look like:

cf-1.1.0.0.pivotal
├── compiled_packages
│   └── cf-158.1-dev-bosh-vsphere-esxi-ubuntu-1471.2.tgz
├── content_migrations
│   └── migrations.yml
├── metadata
│   └── cf.yml
├── releases
│   └── cf-158.1-dev.tgz
└── stemcells
    └── bosh-stemcell-1471_2-vsphere-esxi-ubuntu.tgz

The content_migrations/migrations.yml is copied from the product folder's content_migrations/migrations.yml. The metadata/cf.yml file is copied from the product folder's metadata/cf.yml if it exists. Otherwise the metadata YAML is assembled from metadata_parts/{handcraft,binaries}.yml. The release file is copied from the product folder's releases folder. The stemcell file is copied from the product folder's stemcells folder.

The PRODUCT_DIR

Vara expects a product directory that resembles the layout of the .pivotal zip contents.

p-runtime/
├── .gitignore
├── compiled_packages
│   └── .gitkeep
├── content_migrations
│   └── .gitkeep
├── content_migrations_parts
│   ├── base.yml
│   └── migrations
│       ├── from_1.1.0.0.yml
│       └── from_1.2.0.0.yml
├── metadata
│   └── .gitkeep
├── metadata_parts
│   ├── binaries.yml
│   └── handcraft.yml
├── releases
│   └── .gitkeep
└── stemcells
    └── .gitkeep

The contents of compiled_packages, releases, and stemcells are large tarballs. Large binary files are discouraged from being checked into git and other source control systems. To work around this limitation, vara examines the metadata YAML and downloads these files from a URL if necessary.

The compiled_packages, releases, and stemcells directories are populated during the download-artifacts step of building a .pivotal. The metadata directory is populated during the build-metadata step. The content_migrations directory is populated during the build-migrations step.

The .gitignore file should include these entries to prevent large binary files from being committed by mistake. These entries also prevent vara from failing various operations due to a dirty git working tree.

releases/
stemcells/
compiled_packages/
*.pivotal
*.pivotal.yml
*.pivotal.md5

Re-generate Metadata .yml

Using metadata_parts

To only build the product metadata file from the two separate metadata_parts files:

vara build-metadata ~/workspace/p-runtime

This will create a metadata/${PRODUCT_NAME}.yml file composed of the two files located in ${product_dir}/metadata_parts:

  • binaries.yml (includes stemcells, release, and optional compiled_packages information)
  • handcraft.yml (includes everything else, including product name (e.g. cf), product_version (e.g. 1.2.0.0)

(where PRODUCT_NAME is derived from the product name entry in handcraft.yml)

*[These files were split in order to maintain the comments and the formatting of contained in the handcraft.yml file while allowing the information contained in binaries.yml to be updated by utilities such as vara-update-metadata]

Include the following entry in .gitignore if vara dynamically generates the metadata .yml.

metadata/*.yml

Automatically increasing a product's prerelease version

You can set the product_version to something like 1.2.3.4$PRERELEASE_VERSION$ and during metadata generation, vara will expand that into something like 1.2.3.4.alpha.88.dedbeef where:

  • alpha is the "cycle" as specified by the --cycle flag to vara build-metadata, which defaults to alpha
  • 88 is the number of commits in the current branch
  • dedbeef is the short commit hash of the current revision on the current branch

Specifying a product version

You can set the product_version to something specific like 1.2.3-build.1 or 1.2.3 by specifying the --version flag, e.g.:

vara build-metadata --version=1.2.3-build.1 .
vara build-migrations --version=1.2.3 .
vara build-pivotal --version=1.2.3-rc.3 .

Specifying a final version

You can remove the $PRERELEASE_VERSION$ from the product version by using the --final flag. So for a product version that is in handcraft.yml as 1.2.3.4$PRERELEASE_VERSION$ running

vara build-pivotal . --final
vara build-metadata . --final
vara build-migrations . --final

would result in a product version of 1.2.3.4.

Specifying an external release

You can update multiple releases when building the pivotal by passing one or more external release files using the --external-releases flag.

vara build-pivotal . --external-releases=some-release.yml
vara build-pivotal . --external-release=some-release.yml,another-release.yml

An external release file contains an array of manifest snippets that replace entries in metadata_parts/binaries.yml. For example a sample external release file could be the following. This would update the cf entry in binaries.yml when running the pivotal.

- file: cf-214.tgz
  name: cf
  version: '214'
  md5: 20cd54b93c23a7fb232d5c25b2082666
  url: http://bosh.io/d/github.com/cloudfoundry/cf-release?v=214

The flag will use the specified files to replace the entries that matches the name property of an already existing release inside of the binaries.yml file.

Lint product metadata

Syntax

vara lint <Product-Metadata>

Description

Running vara lint ~/workspace/p-runtime/cf.yml will inspect the product template for malformed data. vara build-pivotal executes vara lint before running vara zip-pivotal, therefore if malformed data exists, the resulting .pivotal file will not be created.

Re-generate content migrations

To only compose the content migrations file from the files in the content_migrations_parts directory:

vara build-migrations ~/workspace/p-runtime

This will create a content_migrations/migrations.yml file composed of the files in ${product_dir}/content_migrations_parts:

  • base.yml
    • Typically includes the product, installation_version, and to_version keys
    • Optionally includes the migrations key, which is allowed to have any number of entries underneath it
  • migrations/from_1.2.3.4.yml
    • Can be named anything you want with a .yml extension, but by convention is from_${version}.yml
    • Content is the hash of a single entry to be inserted under the migrations key (e.g. has from_version and rules as top-level keys)

A migration file in the migrations folder may have a base.yml file like:

product: example-product
installation_version: "1.1"
to_version: "1.2.0.0$PRERELEASE_VERSION$"

and a rule entry like:

from_version: 1.1.0.0
rules:
- type: update
  selector: "product_version"
  to: "1.2.0.0$PRERELEASE_VERSION$"

and then $PRERELEASE_VERSION$ will be replaced in both places using the same rules as above in the metadata file.

Include the following entry in .gitignore if vara dynamically generates the metadata .yml.

content_migrations/*.yml

A Warning on Auto-Generated Prerelease versions

Ops Manager does not allow wildcards in the from_version field of migrations. Therefore, if you have deployed a prerelease version, you will not be able to upgrade to another prerelease version unless you explicitly write a migration between the two. We expect that the normal workflow will not include upgrading between prereleases.

Any prerelease products published externally will require an additional stanza in future migrations.yml for upgradability.

Download missing artifacts

Running vara download-artifacts ~/workspace/p-runtime/cf.yml will download any missing compiled_packages, releases, and stemcells, inserting them into the respective directories. For stemcells, vara assumes that the stemcell is a public bosh stemcell and infers the URL based on the file attribute of the stemcell hash. For releases and compiled packages, vara uses the url attribute of the respective hashes in the product metadata.

For example:

releases:
- file: cf-172.tgz
  name: cf
  version: '172'
  md5: 777fe352515612841a3d96af12054947
  url: https://example.s3.amazonaws.com/cf-172.tgz

Checksum Validation

Artifacts can use either MD5 or SHA1 checksums for release and stemcell validation. For example:

releases:
- file: cf-172.tgz
  name: cf
  version: '172'
  md5: 777fe352515612841a3d96af12054947
  sha1: 7d7ea80ccb59ce57c047e0c91c865172d98f1ba0
  url: https://example.s3.amazonaws.com/cf-172.tgz

You must specify at least one checksum.

AWS S3 support

Release artifacts can be downloaded from an S3 bucket. For example:

releases:
- file: cf-172.tgz
  name: cf
  version: '172'
  md5: 777fe352515612841a3d96af12054947
  sha1: 7d7ea80ccb59ce57c047e0c91c865172d98f1ba0
  aws:
    access_key_id: <some-access-key>
    secret_access_key: <some-secret-key>
    region: us-west-1
    bucket_name: <some-bucket>
    filename: path/to/s3/object.tgz

You should only specify url or aws, not both.

Zipping the .pivotal

If your metadata file has been generated and your artifacts have been downloaded, running vara zip-pivotal ~/workspace/p-runtime will put all of those contents into a new .pivotal in your working directory.

Copyright (c) 2014-2015 Pivotal Software, Inc. All rights reserved. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.