avst-cloud gem

Automated creation, bootstrapping and provisioning of servers. Currently supports AWS, Azure and Rackspace

Prerequisites

Make sure ruby 2.0 is installed. The application depends on several gems listed in avst-cloud.gemspec file. Bundle install command will install all dependencies. The most important ones are Fog.io and Derelict for provider integration.

Installation from source

  • Download source
  • Navigate to folder
  • Run bundle install
  • Check the example, modify, try

Server creation example

AWS


    # Provide your AWS credentials, region, you can specify custom logger on conn object (defaults to: Logger.new(STDOUT))
    conn = AvstCloud::AwsConnection.new(provider_user, provider_pass, region)

    # Creates server on aws, if server exists and it is stopped it will start it
    #
    # server_name - name of newly created server
    # flavour - aws flavour name, defaults to t2.micro
    # os - operating system of ami provided, defaults to ubuntu-14
    # ami_image_id - aws ami id, defaults to ami-f0b11187 running ubuntu-14
    # key_name - aws ssh access key name, admin.pem
    # ssh_key  - lcoal aws ssh access key path - just to make sure you have it :) 
    # subnet_id - aws subnet id
    # security_group_ids - list of aws security groups
    # ebs_size - disk size
    # hdd_device_path - hdd device path, may differ per ami/os
    # availability_zone - aws availability zone
    # additional_hdds - hash of additional disks to attach to server, must contain device_name (e.g. /dev/sda1) and ebs_size per each hdd
    additional_hdds:
        'AppDisk':
            device_name: '/dev/sdf'
            ebs_size: '100'
        'DatabaseDisk':
            device_name: '/dev/sdg'
            ebs_size: '50'
            volume_type: 'gp2' #default value
    # vpc - virtual private cloud, defaults to nil, make sure you adjust subned_id and security_group_ids accordingly when setting this option

    server = conn.create_server(server_name, flavour, os, key_name, ssh_key, subnet_id, security_group_ids, ebs_size, hdd_device_path, ami_image_id, availability_zone)

    # Stop server
    server.stop

    # Destroy server 
    server.destroy

    # Returns setver status
    server.status

Rackspace

additional_hdds - hash of additional disks to attach to server, must contain device_name (e.g. /dev/sda1) and ebs_size per each hdd

additional_hdds:
    'AppDisk':
        device_name: '/dev/sdf'
        ebs_size: '100'
        volume_type: 'SSD' #default value
    'DatabaseDisk':
        device_name: '/dev/sdg'
        ebs_size: '50'


    conn = AvstCloud::RackspaceConnection.new(provider_user, provider_pass, region)
    # image_id - int representing rackspace image id e.g. '4' - "2GB-standard"
    server = conn.create_server(server_name, image_id)

    # Returns setver status
    server.status

    # Stop server
    server.stop

    # Destroy server 
    server.destroy

Azure

See bin/avst-cloud-azure for examples

Azure RM

Using new Azure Resource Manager way to manage resources. See bin/avst-cloud-azure-rm for examples

Connecting to existing server


conn = AvstCloud::AwsConnection.new(provider_user, provider_pass, region)
# server_name - hostname
# access_user - user to access server as
# ssh_key - full path to access key for your server, or password
# os - if access_user is nil and os provided, it will try to use standard user based on os
server = conn.server(server_name, access_user, ssh_key, os)

Bootstrap

Bootstrap executes pre_upload_commands, custom_file_uploads and post_upload_commands on the server. 


    server = FogAws::Server.new(provider_user, provider_pass, region)
    # Creates server
    server = conn.create_server(server_name, flavour, os, key_name, ssh_key, subnet_id, security_group_ids, ebs_size, hdd_device_path, ami_image_id, availability_zone)

    # Commands will run as "sudo su -c \"#{cmd}\""
    # List of commands running on server before files upload
    pre_upload_commands = [
      "echo 'I was here' >> /tmp/was_here",
    ]

    avst_cloud_base ="#{File.expand_path("../../", __FILE__)}"

    # To allow root user to connect to github and run capistrano provisioning in the next stage
    # Define hash of from_local_path and destination on the server
    custom_file_uploads = {
        "#{avst_cloud_base}/files/id_rsa" => "/tmp/.",
        "#{avst_cloud_base}/files/known_hosts" => "/tmp/."
    }

    post_upload_commands = [
        "mkdir /home/ubuntu/.ssh",
        "mv /tmp/id_rsa /home/ubuntu/.ssh/.",
        "mv /tmp/known_hosts /home/ubuntu/.ssh/.",
        "chmod 0600 /home/ubuntu/.ssh/known_hosts",
        "chmod 0600 /home/ubuntu/.ssh/id_rsa",
        "echo 'done here'"
    ]

    remote_server_debug = true
    debug_structured_log = false

    server.bootstrap(pre_upload_commands, custom_file_uploads, post_upload_commands, remote_server_debug, debug_structured_log)

Provisioning

Uses Capistrano to download source from git repo (branch or tag) to destination_folder (defaults to /var/opt/puppet/current/ ) on the server. Make sure git is installed on the server and user has access to git repo provided. Make sure destination_folder exists and user (aws - ubuntu - ubuntu, centos - ec2-user, debian - admin) has ownership of it.

If puppet_runner param is defined:

  • create avst_cloud_tmp_folder on the server
  • uploads the content of local folder config/custom_system_config to it and then moves it to destination_folder, in subfolder hiera-configs the only files uploaded may be server_name.yaml, server_name_facts.yaml and puppetfile_dictionary.yaml
  • if puppet_runner_prepare is defined, it will run it
  • if download_dependencies_command is defined, it will run it, in the common case should be 'r10k puppetfile install'
  • clear /etc/puppet folder
  • link /var/opt/puppet/current to /etc/puppet
  • links /var/opt/puppet/current/hiera.yaml to /etc/hiera.yaml
  • if config/keys folder exists it will upload it to /etc/puppet/config
  • executes command passed as puppet_runner param in /etc/puppet folder on the server

If custom_provisioning_command is defined, it will run the command on the server.


    avst_cloud_tmp_folder = "/tmp/avst_cloud_tmp_#{Time.now.to_i}"
    reference = nil
    git = "ssh://[email protected]:7999/pup/base_puppet_templates.git"
    branch = "master"
    puppet_runner = nil 
    puppet_runner_prepare = nil
    avst_cloud_config_dir = nil
    download_dependencies_command = nil
    custom_provisioning_command = "echo 'done' >> /tmp/done",
    destination_folder = nil
    server.provision(git, branch, server_tmp_folder, reference, custom_provisioning_commands, puppet_runner, puppet_runner_prepare, destination_folder, avst_cloud_config_dir, download_dependencies_command)

Cleanup

Cleans after provisioning. Runs yum clean all or apt-get clean based on os. Also removes avst_cloud_tmp_folder. You can specify custom commands to run afterwards.


  # avst_cloud_tmp_folder = "/tmp/avst_cloud_tmp_*"
  # server_name - name of newly created server
  # custom_cleanup_commands - list of commands to run
  # cloud_operating_system - os
  # remote_server_debug - enable debug

  server.post_provisioning_cleanup(custom_commands, cloud_operating_system, remote_server_debug, server_tmp_folder)

Example no puppet run

See example executable bin/avst-cloud

Example puppet run

Make sure that provided repo contains templates and base setup. An example is available in Adaptavist github account. Make sure ruby 2.0.0, git, puppet, puppet-runner,... is installed. Check full example in bin/avst-cloud-puppet


# url to git repo you want to upload to the server
git = "[email protected]:Adaptavist/base_puppet_templates.git" # url to git repo
branch = "master" # branch
reference = nil # tag
# In this example we are using puppet-runner to apply our puppet configs, check doco
puppet_runner = "puppet-runner start"
avst_cloud_config_dir = 'config'
download_dependencies_command = 'gem install r10k -v 2.6.4; r10k puppetfile install'
puppet_runner_prepare = "puppet-runner prepare -c ./hiera-configs -d ./hiera -f ./environments/production/modules/hosts/facts.d -t ./hiera-configs -r ./hiera-configs/puppetfile_dictionary.yaml -o ./Puppetfile -e /var/opt/puppet/secure/keys"

custom_provisioning_commands = ["echo 'done' >> /tmp/done"]
# defaults to /var/opt/puppet
destination_folder = nil

server.provision(git, branch, server_tmp_folder, reference, custom_provisioning_commands, puppet_runner, puppet_runner_prepare, destination_folder. avst_cloud_config_dir, download_dependencies_command)