Class: Beaker::AwsSdk

Inherits:
Hypervisor show all
Defined in:
lib/beaker/hypervisor/aws_sdk.rb

Overview

This is an alternate EC2 driver that implements direct API access using Amazon’s AWS-SDK library: aws.amazon.com/documentation/sdkforruby/

It is built for full control, to reduce any other layers beyond the pure vendor API.

Constant Summary

Constants inherited from Hypervisor

Hypervisor::CHARMAP

Constants included from HostPrebuiltSteps

HostPrebuiltSteps::APT_CFG, HostPrebuiltSteps::ETC_HOSTS_PATH, HostPrebuiltSteps::ETC_HOSTS_PATH_SOLARIS, HostPrebuiltSteps::IPS_PKG_REPO, HostPrebuiltSteps::NTPSERVER, HostPrebuiltSteps::ROOT_KEYS_SCRIPT, HostPrebuiltSteps::ROOT_KEYS_SYNC_CMD, HostPrebuiltSteps::SLEEPWAIT, HostPrebuiltSteps::SLES_PACKAGES, HostPrebuiltSteps::TRIES, HostPrebuiltSteps::UNIX_PACKAGES, HostPrebuiltSteps::WINDOWS_PACKAGES

Instance Method Summary collapse

Methods inherited from Hypervisor

#configure, create, #generate_host_name, #validate

Methods included from HostPrebuiltSteps

#add_el_extras, #apt_get_update, #copy_file_to_remote, #copy_ssh_to_root, #disable_iptables, #disable_se_linux, #enable_root_login, #epel_info_for!, #get_domain_name, #get_ip, #hack_etc_hosts, #package_proxy, #proxy_config, #set_etc_hosts, #sync_root_keys, #timesync, #validate_host

Constructor Details

#initialize(hosts, options) ⇒ AwsSdk

Initialize AwsSdk hypervisor driver

Parameters:

  • hosts (Array<Beaker::Host>) —

    Array of Beaker::Host objects

  • options (Hash<String, String>) —

    Options hash



17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 17

def initialize(hosts, options)
  @hosts = hosts
  @options = options
  @logger = options[:logger]

  # Get fog credentials from the local .fog file
  creds = load_fog_credentials(@options[:dot_fog])

  config = {
    :access_key_id => creds[:access_key],
    :secret_access_key => creds[:secret_key],
    :logger => Logger.new($stdout),
    :log_level => :debug,
    :log_formatter => AWS::Core::LogFormatter.colored,
    :max_retries => 12,
  }
  AWS.config(config)

  @ec2 = AWS::EC2.new()
end

Instance Method Details

#backoff_sleep(tries) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Calculates and waits a back-off period based on the number of tries

Logs each backupoff time and retry value to the console.

Parameters:

  • tries (Number) —

    number of tries to calculate back-off period



254
255
256
257
258
259
260
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 254

def backoff_sleep(tries)
  # Exponential with some randomization
  sleep_time = 2 ** tries
  @logger.notify("aws-sdk: Sleeping #{sleep_time} seconds for attempt #{tries}.")
  sleep sleep_time
  nil
end

#cleanup ⇒ void

This method returns an undefined value.

Cleanup all earlier provisioned hosts on EC2 using the AWS::EC2 library

It goes without saying, but a #cleanup does nothing without a #provision method call first.



70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 70

def cleanup
  @logger.notify("aws-sdk: Cleanup, iterating across all hosts and terminating them")
  @hosts.each do |host|
    # This was set previously during provisioning
    instance = host['instance']

    # Only attempt a terminate if the instance actually is set by provision
    # and the instance actually 'exists'.
    if !instance.nil? and instance.exists?
      instance.terminate
    end
  end

  nil #void
end

#configure_hosts ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Configure /etc/hosts for each node



227
228
229
230
231
232
233
234
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 227

def configure_hosts
  base = "127.0.0.1\tlocalhost localhost.localdomain\n"
  @hosts.each do |host|
    hn = host.hostname
    etc_hosts = base + "#{host['private_ip']}\t#{hn} #{hn.split(".")[0]}\n"
    set_etc_hosts(host, etc_hosts)
  end
end

#create_group(region, ports) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Create a new security group

Parameters:

  • region (AWS::EC2::Region) —

    the AWS region control object

  • ports (Array<Number>) —

    an array of port numbers

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 356

def create_group(region, ports)
  name = group_id(ports)
  @logger.notify("aws-sdk: Creating group #{name} for ports #{ports.to_s}")
  group = region.security_groups.create(name,
                                        :description => "Custom Beaker security group for #{ports.to_a}")

  unless ports.is_a? Set
    ports = Set.new(ports)
  end

  ports.each do |port|
    group.authorize_ingress(:tcp, port)
  end

  group
end

#ensure_group(region, ports) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return an existing group, or create new one

Parameters:

  • region (AWS::EC2::Region) —

    the AWS region control object

  • ports (Array<Number>) —

    an array of port numbers

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



337
338
339
340
341
342
343
344
345
346
347
348
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 337

def ensure_group(region, ports)
  @logger.notify("aws-sdk: Ensure security group exists for ports #{ports.to_s}, create if not")
  name = group_id(ports)

  group = region.security_groups.filter('group-name', name).first

  if group.nil?
    group = create_group(region, ports)
  end

  group
end

#ensure_key_pair(region) ⇒ AWS::EC2::KeyPair

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the KeyPair for this host, creating it if needed

Parameters:

  • region (AWS::EC2::Region) —

    region to create the key pair in

Returns:

  • (AWS::EC2::KeyPair) —

    created key_pair



300
301
302
303
304
305
306
307
308
309
310
311
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 300

def ensure_key_pair(region)
  @logger.notify("aws-sdk: Ensure key pair exists, create if not")
  key_pairs = region.key_pairs
  pair_name = key_name()
  kp = key_pairs[pair_name]
  unless kp.exists?
    ssh_string = public_key()
    kp = key_pairs.import(pair_name, ssh_string)
  end

  kp
end

#group_id(ports) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return a reproducable security group identifier based on input ports

Parameters:

  • ports (Array<Number>) —

    array of port numbers

Returns:

  • (String) —

    group identifier



318
319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 318

def group_id(ports)
  if ports.nil? or ports.empty?
    raise ArgumentError, "Ports list cannot be nil or empty"
  end

  unless ports.is_a? Set
    ports = Set.new(ports)
  end

  # Lolwut, #hash is inconsistent between ruby processes
  "Beaker-#{Zlib.crc32(ports.inspect)}"
end

#key_name ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Generate a reusable key name from the local hosts hostname

Returns:

  • (String) —

    safe key name for current host



282
283
284
285
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 282

def key_name
  safe_hostname = Socket.gethostname.gsub('.', '-')
  "Beaker-#{local_user}-#{safe_hostname}"
end

#launch_all_nodes ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Launch all nodes

This is where the main launching work occurs for each node. Here we take care of feeding the information from the image required into the config for the new host, we perform the launch operation and ensure that the instance is properly tagged for identification.



95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 95

def launch_all_nodes
  # Load the ec2_yaml spec file
  ami_spec = YAML.load_file(@options[:ec2_yaml])["AMI"]

  # Iterate across all hosts and launch them, adding tags along the way
  @logger.notify("aws-sdk: Iterate across all hosts in configuration and launch them")
  @hosts.each do |host|
    amitype = host['vmname'] || host['platform']
    amisize = host['amisize'] || 'm1.small'

    # Use snapshot provided for this host
    image_type = host['snapshot']
    if not image_type
      raise RuntimeError, "No snapshot/image_type provided for EC2 provisioning"
    end
    ami = ami_spec[amitype]
    ami_region = ami[:region]

    # Main region object for ec2 operations
    region = @ec2.regions[ami_region]

    # Grab image object
    image_id = ami[:image][image_type.to_sym]
    @logger.notify("aws-sdk: Checking image #{image_id} exists and getting its root device")
    image = region.images[image_id]
    if image.nil? and not image.exists?
      raise RuntimeError, "Image not found: #{image_id}"
    end

    # Transform the images block_device_mappings output into a format
    # ready for a create.
    orig_bdm = image.block_device_mappings()
    @logger.notify("aws-sdk: Image block_device_mappings: #{orig_bdm.to_hash}")
    block_device_mappings = []
    orig_bdm.each do |device_name, rest|
      block_device_mappings << {
        :device_name => device_name,
        :ebs => {
          # This is required to override the images default for
          # delete_on_termination, forcing all volumes to be deleted once the
          # instance is terminated.
          :delete_on_termination => true,
        }
      }
    end

    # Launch the node, filling in the blanks from previous work.
    @logger.notify("aws-sdk: Launch instance")
    config = {
      :count => 1,
      :image_id => image_id,
      :monitoring_enabled => true,
      :key_pair => ensure_key_pair(region),
      :security_groups => [ensure_group(region, Beaker::EC2Helper.amiports(host['roles']))],
      :instance_type => amisize,
      :disable_api_termination => false,
      :instance_initiated_shutdown_behavior => "terminate",
      :block_device_mappings => block_device_mappings,
    }
    instance = region.instances.create(config)

    # Persist the instance object for this host, so later it can be
    # manipulated by 'cleanup' for example.
    host['instance'] = instance

    # Define tags for the instance
    @logger.notify("aws-sdk: Add tags")
    instance.add_tag("jenkins_build_url", :value => @options[:jenkins_build_url])
    instance.add_tag("Name", :value => host.name)
    instance.add_tag("department", :value => @options[:department])
    instance.add_tag("project", :value => @options[:project])

    @logger.notify("aws-sdk: Launched #{host.name} (#{amitype}:#{amisize}) using snapshot/image_type #{image_type}")
  end

  nil
end

#load_fog_credentials(dot_fog = '.fog') ⇒ Hash<Symbol, String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return a hash containing the fog credentials for EC2

Parameters:

  • dot_fog (String) (defaults to: '.fog') —

    dot fog path

Returns:

  • (Hash<Symbol, String>) —

    ec2 credentials



378
379
380
381
382
383
384
385
386
387
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 378

def load_fog_credentials(dot_fog = '.fog')
  fog = YAML.load_file( dot_fog )

  default = fog[:default]

  creds = {}
  creds[:access_key] = default[:aws_access_key_id]
  creds[:secret_key] = default[:aws_secret_access_key]
  creds
end

#local_user ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the local user running this tool

Returns:

  • (String) —

    username of local user



291
292
293
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 291

def local_user
  ENV['USER']
end

#populate_dns ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Populate the hosts IP address and vmhostname entry from the EC2 dns_name



210
211
212
213
214
215
216
217
218
219
220
221
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 210

def populate_dns
  # Obtain the IP addresses and dns_name for each host
  @hosts.each do |host|
    instance = host['instance']
    host['vmhostname'] = instance.dns_name
    host['ip'] = instance.ip_address
    host['private_ip'] = instance.private_ip_address
    @logger.notify("aws-sdk: name: #{host.name} vmhostname: #{host['vmhostname']} ip: #{host['ip']} private_ip: #{host['private_ip']}")
  end

  nil
end

#provision ⇒ void

This method returns an undefined value.

Provision all hosts on EC2 using the AWS::EC2 API



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 41

def provision
  start_time = Time.now

  # Perform the main launch work
  launch_all_nodes()

  # Wait for each node to reach status :running
  wait_for_status(:running)

  # Grab the ip addresses and dns from EC2 for each instance to use for ssh
  populate_dns()

  # Set the hostname for each box
  set_hostnames()

  # Configure /etc/hosts on each host
  configure_hosts()

  @logger.notify("aws-sdk: Provisioning complete in #{Time.now - start_time} seconds")

  nil #void
end

#public_key ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Retrieve the public key locally from the executing users ~/.ssh directory

Returns:

  • (String) —

    contents of public key



266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 266

def public_key
  filename = File.expand_path('~/.ssh/id_rsa.pub')
  unless File.exists? filename
    filename = File.expand_path('~/.ssh/id_dsa.pub')
    unless File.exists? filename
      raise RuntimeError, 'Expected either ~/.ssh/id_rsa.pub or ~/.ssh/id_dsa.pub but found neither'
    end
  end

  File.read(filename)
end

#set_hostnames ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Set the hostname of all instances to be the hostname defined in the beaker configuration.



241
242
243
244
245
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 241

def set_hostnames
  @hosts.each do |host|
    host.exec(Command.new("hostname #{host['vmhostname']}"))
  end
end

#wait_for_status(status) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Waits until all boxes reach the desired status

Parameters:

  • status (Symbol) —

    EC2 state to wait for, :running :stopped etc.



178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 178

def wait_for_status(status)
  # Wait for each node to reach status :running
  @logger.notify("aws-sdk: Now wait for all hosts to reach state #{status}")
  @hosts.each do |host|
    instance = host['instance']
    name = host.name

    @logger.notify("aws-sdk: Wait for status #{status} for node #{name}")

    # Here we keep waiting for the machine state to reach ':running' with an
    # exponential backoff for each poll.
    # TODO: should probably be a in a shared method somewhere
    for tries in 1..10
      if instance.status == status
        # Always sleep, so the next command won't cause a throttle
        backoff_sleep(tries)
        break
      elsif tries == 10
        raise "Instance never reached state #{status}"
      end
      backoff_sleep(tries)
    end

    # Set the IP to be the dns_name of the host, yes I know its not an IP.
    host['ip'] = instance.dns_name
  end
end