Class: Beaker::AwsSdk

Inherits:
Hypervisor show all
Defined in:
lib/beaker/hypervisor/aws_sdk.rb

Overview

This is an alternate EC2 driver that implements direct API access using Amazon's AWS-SDK library: SDK For Ruby

It is built for full control, to reduce any other layers beyond the pure vendor API.

Constant Summary collapse

ZOMBIE =

anything older than 3 hours is considered a zombie

3
PING_SECURITY_GROUP_NAME = 
'beaker-ping'

Constants inherited from Hypervisor

Hypervisor::CHARMAP

Constants included from HostPrebuiltSteps

HostPrebuiltSteps::APT_CFG, HostPrebuiltSteps::CUMULUS_PACKAGES, HostPrebuiltSteps::DEBIAN_PACKAGES, HostPrebuiltSteps::ETC_HOSTS_PATH, HostPrebuiltSteps::ETC_HOSTS_PATH_SOLARIS, HostPrebuiltSteps::FREEBSD_PACKAGES, HostPrebuiltSteps::IPS_PKG_REPO, HostPrebuiltSteps::NTPSERVER, HostPrebuiltSteps::OPENBSD_PACKAGES, HostPrebuiltSteps::PSWINDOWS_PACKAGES, HostPrebuiltSteps::ROOT_KEYS_SCRIPT, HostPrebuiltSteps::ROOT_KEYS_SYNC_CMD, HostPrebuiltSteps::SLEEPWAIT, HostPrebuiltSteps::SLES10_PACKAGES, HostPrebuiltSteps::SLES_PACKAGES, HostPrebuiltSteps::TRIES, HostPrebuiltSteps::UNIX_PACKAGES, HostPrebuiltSteps::WINDOWS_PACKAGES

Instance Method Summary collapse

Methods inherited from Hypervisor

#configure, create, #generate_host_name, #proxy_package_manager, #validate

Methods included from HostPrebuiltSteps

#add_el_extras, #additive_hash_merge, #apt_get_update, #check_and_install_packages_if_needed, #construct_env, #copy_file_to_remote, #copy_ssh_to_root, #disable_iptables, #disable_se_linux, #enable_root_login, #epel_info_for, #get_domain_name, #get_ip, #hack_etc_hosts, #package_proxy, #proxy_config, #set_env, #set_etc_hosts, #sync_root_keys, #timesync, #validate_host

Methods included from DSL::Patterns

#block_on

Constructor Details

#initialize(hosts, options) ⇒ AwsSdk

Initialize AwsSdk hypervisor driver

Parameters:

  • hosts (Array<Beaker::Host>) —

    Array of Beaker::Host objects

  • options (Hash<String, String>) —

    Options hash



20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 20

def initialize(hosts, options)
  @hosts = hosts
  @options = options
  @logger = options[:logger]

  # Get fog credentials from the local .fog file
  creds = load_fog_credentials(@options[:dot_fog])

  config = {
    :access_key_id => creds[:access_key],
    :secret_access_key => creds[:secret_key],
    :logger => Logger.new($stdout),
    :log_level => :debug,
    :log_formatter => AWS::Core::LogFormatter.colored,
    :max_retries => 12,
  }
  AWS.config(config)

  @ec2 = AWS::EC2.new()
end

Instance Method Details

#add_tags ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Add metadata tags to all instances



458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 458

def add_tags
  @hosts.each do |host|
    instance = host['instance']

    # Define tags for the instance
    @logger.notify("aws-sdk: Add tags for #{host.name}")
    instance.add_tag("jenkins_build_url", :value => @options[:jenkins_build_url])
    instance.add_tag("Name", :value => host.name)
    instance.add_tag("department", :value => @options[:department])
    instance.add_tag("project", :value => @options[:project])
    instance.add_tag("created_by", :value => @options[:created_by])
  end

  nil
end

#backoff_sleep(tries) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Calculates and waits a back-off period based on the number of tries

Logs each backupoff time and retry value to the console.

Parameters:

  • tries (Number) —

    number of tries to calculate back-off period



597
598
599
600
601
602
603
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 597

def backoff_sleep(tries)
  # Exponential with some randomization
  sleep_time = 2 ** tries
  @logger.notify("aws-sdk: Sleeping #{sleep_time} seconds for attempt #{tries}.")
  sleep sleep_time
  nil
end

#cleanup ⇒ void

This method returns an undefined value.

Cleanup all earlier provisioned hosts on EC2 using the AWS::EC2 library

It goes without saying, but a #cleanup does nothing without a #provision method call first.



90
91
92
93
94
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 90

def cleanup
  # Provisioning should have set the host 'instance' values.
  kill_instances(@hosts.map{|h| h['instance']}.select{|x| !x.nil?})
  nil
end

#configure_hosts ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Configure /etc/hosts for each node



496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 496

def configure_hosts
  @hosts.each do |host|
    etc_hosts = "127.0.0.1\tlocalhost localhost.localdomain\n"
    name = host.name
    domain = get_domain_name(host)
    ip = host['private_ip']
    etc_hosts += "#{ip}\t#{name} #{name}.#{domain} #{host['dns_name']}\n"
    @hosts.each do |neighbor|
      if neighbor == host
        next
      end
      name = neighbor.name
      domain = get_domain_name(neighbor)
      ip = neighbor['ip']
      etc_hosts += "#{ip}\t#{name} #{name}.#{domain} #{neighbor['dns_name']}\n"
    end
    set_etc_hosts(host, etc_hosts)
  end
end

#create_group(rv, ports) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Create a new security group

Accepts a region or VPC for group creation.

Parameters:

  • rv (AWS::EC2::Region, AWS::EC2::VPC) —

    the AWS region or vpc control object

  • ports (Array<Number>) —

    an array of port numbers

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 739

def create_group(rv, ports)
  name = group_id(ports)
  @logger.notify("aws-sdk: Creating group #{name} for ports #{ports.to_s}")
  group = rv.security_groups.create(name,
                                    :description => "Custom Beaker security group for #{ports.to_a}")

  unless ports.is_a? Set
    ports = Set.new(ports)
  end

  ports.each do |port|
    group.authorize_ingress(:tcp, port)
  end

  group
end

#create_instance(host, ami_spec, subnet_id) ⇒ AWS::EC2::Instance

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Create an EC2 instance for host, tag it, and return it.

Returns:

  • (AWS::EC2::Instance) —

    ]



229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 229

def create_instance(host, ami_spec, subnet_id)
  amitype = host['vmname'] || host['platform']
  amisize = host['amisize'] || 'm1.small'
  vpc_id = host['vpc_id'] || @options['vpc_id'] || nil

  if vpc_id and !subnet_id
    raise RuntimeError, "A subnet_id must be provided with a vpc_id"
  end

  # Use snapshot provided for this host
  image_type = host['snapshot']
  if not image_type
    raise RuntimeError, "No snapshot/image_type provided for EC2 provisioning"
  end
  ami = ami_spec[amitype]
  ami_region = ami[:region]

  # Main region object for ec2 operations
  region = @ec2.regions[ami_region]

  # If we haven't defined a vpc_id then we use the default vpc for the provided region
  if !vpc_id
    @logger.notify("aws-sdk: filtering available vpcs in region by 'isDefault")
    filtered_vpcs = region.client.describe_vpcs(:filters => [{:name => 'isDefault', :values => ['true']}])
    if !filtered_vpcs[:vpc_set].empty?
      vpc_id = filtered_vpcs[:vpc_set].first[:vpc_id]
    else #there's no default vpc, use nil
      vpc_id = nil
    end
  end

  # Grab the vpc object based upon provided id
  vpc = vpc_id ? region.vpcs[vpc_id] : nil

  # Grab image object
  image_id = ami[:image][image_type.to_sym]
  @logger.notify("aws-sdk: Checking image #{image_id} exists and getting its root device")
  image = region.images[image_id]
  if image.nil? and not image.exists?
    raise RuntimeError, "Image not found: #{image_id}"
  end

  @logger.notify("Image Storage Type: #{image.root_device_type}")

  # Transform the images block_device_mappings output into a format
  # ready for a create.
  block_device_mappings = []
  if image.root_device_type == :ebs
    orig_bdm = image.block_device_mappings()
    @logger.notify("aws-sdk: Image block_device_mappings: #{orig_bdm.to_hash}")
    orig_bdm.each do |device_name, rest|
      block_device_mappings << {
        :device_name => device_name,
        :ebs => {
          # Change the default size of the root volume.
          :volume_size => host['volume_size'] || rest[:volume_size],
          # This is required to override the images default for
          # delete_on_termination, forcing all volumes to be deleted once the
          # instance is terminated.
          :delete_on_termination => true,
        }
      }
    end
  end

  security_group = ensure_group(vpc || region, Beaker::EC2Helper.amiports(host))
  #check if ping is enabled
  ping_security_group = ensure_ping_group(vpc || region)

  msg = "aws-sdk: launching %p on %p using %p/%p%s" %
        [host.name, amitype, amisize, image_type,
         subnet_id ? ("in %p" % subnet_id) : '']
  @logger.notify(msg)
  config = {
    :count => 1,
    :image_id => image_id,
    :monitoring_enabled => true,
    :key_pair => ensure_key_pair(region),
    :security_groups => [security_group, ping_security_group],
    :instance_type => amisize,
    :disable_api_termination => false,
    :instance_initiated_shutdown_behavior => "terminate",
    :subnet => subnet_id,
  }
  config[:block_device_mappings] = block_device_mappings if image.root_device_type == :ebs
  region.instances.create(config)
end

#create_ping_group(rv) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Create a new ping enabled security group

Accepts a region or VPC for group creation.

Parameters:

  • rv (AWS::EC2::Region, AWS::EC2::VPC) —

    the AWS region or vpc control object

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



721
722
723
724
725
726
727
728
729
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 721

def create_ping_group(rv)
  @logger.notify("aws-sdk: Creating group #{PING_SECURITY_GROUP_NAME}")
  group = rv.security_groups.create(PING_SECURITY_GROUP_NAME,
                                    :description => "Custom Beaker security group to enable ping")

  group.allow_ping

  group
end

#enable_root(host) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Enables root access for a host when username is not root



530
531
532
533
534
535
536
537
538
539
540
541
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 530

def enable_root(host)
  if host['user'] != 'root'
    if host['platform'] =~ /f5-/
      enable_root_f5(host)
    else
      copy_ssh_to_root(host, @options)
      (host, @options)
      host['user'] = 'root'
    end
    host.close
  end
end

#enable_root_f5(host) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method does not support other platforms

Enables root access for a host on an f5 platform

Returns:

  • nil



548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 548

def enable_root_f5(host)
  for tries in 1..10
    begin
      #This command is problematic as the F5 is not always done loading
      if host.exec(Command.new("modify sys db systemauth.disablerootlogin value false"), :acceptable_exit_codes => [0,1]).exit_code == 0 \
          and host.exec(Command.new("modify sys global-settings gui-setup disabled"), :acceptable_exit_codes => [0,1]).exit_code == 0 \
          and host.exec(Command.new("save sys config"), :acceptable_exit_codes => [0,1]).exit_code == 0
        backoff_sleep(tries)
        break
      elsif tries == 10
        raise "Instance was unable to be configured"
      end
    rescue Beaker::Host::CommandFailure => e
      @logger.debug("Instance not yet configured (#{e})")
    end
    backoff_sleep(tries)
  end
  host['user'] = 'root'
  host.close
  sha256 = Digest::SHA256.new
  password = sha256.hexdigest((1..50).map{(rand(86)+40).chr}.join.gsub(/\\/,'\&\&'))
  host['ssh'] = {:password => password}
  host.exec(Command.new("echo -e '#{password}\\n#{password}' | tmsh modify auth password admin"))
  @logger.notify("f5: Configured admin password to be #{password}")
end

#enable_root_on_hosts ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Enables root for instances with custom username like ubuntu-amis



520
521
522
523
524
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 520

def enable_root_on_hosts
  @hosts.each do |host|
    enable_root(host)
  end
end

#ensure_group(vpc, ports) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return an existing group, or create new one

Accepts a VPC as input for checking & creation.

Parameters:

  • vpc (AWS::EC2::VPC) —

    the AWS vpc control object

  • ports (Array<Number>) —

    an array of port numbers

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



701
702
703
704
705
706
707
708
709
710
711
712
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 701

def ensure_group(vpc, ports)
  @logger.notify("aws-sdk: Ensure security group exists for ports #{ports.to_s}, create if not")
  name = group_id(ports)

  group = vpc.security_groups.filter('group-name', name).first

  if group.nil?
    group = create_group(vpc, ports)
  end

  group
end

#ensure_key_pair(region) ⇒ AWS::EC2::KeyPair

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the KeyPair for this host, creating it if needed

Parameters:

  • region (AWS::EC2::Region) —

    region to create the key pair in

Returns:

  • (AWS::EC2::KeyPair) —

    created key_pair



643
644
645
646
647
648
649
650
651
652
653
654
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 643

def ensure_key_pair(region)
  @logger.notify("aws-sdk: Ensure key pair exists, create if not")
  key_pairs = region.key_pairs
  pair_name = key_name()
  kp = key_pairs[pair_name]
  unless kp.exists?
    ssh_string = public_key()
    kp = key_pairs.import(pair_name, ssh_string)
  end

  kp
end

#ensure_ping_group(vpc) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return an existing group, or create new one

Accepts a VPC as input for checking & creation.

Parameters:

  • vpc (AWS::EC2::VPC) —

    the AWS vpc control object

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



681
682
683
684
685
686
687
688
689
690
691
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 681

def ensure_ping_group(vpc)
  @logger.notify("aws-sdk: Ensure security group exists that enables ping, create if not")

  group = vpc.security_groups.filter('group-name', PING_SECURITY_GROUP_NAME).first

  if group.nil?
    group = create_ping_group(vpc)
  end

  group
end

#group_id(ports) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return a reproducable security group identifier based on input ports

Parameters:

  • ports (Array<Number>) —

    array of port numbers

Returns:

  • (String) —

    group identifier



661
662
663
664
665
666
667
668
669
670
671
672
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 661

def group_id(ports)
  if ports.nil? or ports.empty?
    raise ArgumentError, "Ports list cannot be nil or empty"
  end

  unless ports.is_a? Set
    ports = Set.new(ports)
  end

  # Lolwut, #hash is inconsistent between ruby processes
  "Beaker-#{Zlib.crc32(ports.inspect)}"
end

#instance_by_id(id) ⇒ AWS::EC2::Instance

Provided an id return an instance object. Instance object will respond to methods described here: AWS Instance Object.

Parameters:

  • id (String) —

    The id of the instance to return

Returns:

  • (AWS::EC2::Instance) —

    An AWS::EC2 instance object



124
125
126
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 124

def instance_by_id(id)
  @ec2.instances[id]
end

#instances ⇒ Array<AWS::EC2::Instance>

Return all instances currently on ec2.

Returns:

  • (Array<AWS::EC2::Instance>) —

    An array of AWS::EC2 instance objects

See Also:



131
132
133
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 131

def instances
  @ec2.instances
end

#key_name ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Generate a reusable key name from the local hosts hostname

Returns:

  • (String) —

    safe key name for current host



625
626
627
628
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 625

def key_name
  safe_hostname = Socket.gethostname.gsub('.', '-')
  "Beaker-#{local_user}-#{safe_hostname}"
end

#kill_instances(instances) ⇒ void

This method returns an undefined value.

Kill all instances.

Parameters:

  • instances (Enumerable<EC2::Instance>) 


74
75
76
77
78
79
80
81
82
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 74

def kill_instances(instances)
  instances.each do |instance|
    if !instance.nil? and instance.exists?
      @logger.notify("aws-sdk: killing EC2 instance #{instance.id}")
      instance.terminate
    end
  end
  nil
end

#kill_zombie_volumes ⇒ Object

Destroy any volumes marked 'available', INCLUDING THOSE YOU DON'T OWN! Use with care.



199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 199

def kill_zombie_volumes
  # Occasionaly, tearing down ec2 instances leaves orphaned EBS volumes behind -- these stack up quickly.
  # This simply looks for EBS volumes that are not in use
  # Note: don't use volumes.each here as that funtion doesn't allow proper rescue from error states
  @logger.notify("aws-sdk: Kill Zombie Volumes!")
  volume_count = 0
  @ec2.regions.each do |region|
    @logger.debug "Reviewing: #{region.name}"
    volumes = @ec2.regions[region.name].volumes.map { |vol| vol.id }
    volumes.each do |vol|
      begin
        vol = @ec2.regions[region.name].volumes[vol]
        if ( vol.status.to_s =~ /available/ )
          @logger.debug "Tear down available volume: #{vol.id}"
          vol.delete()
          volume_count += 1
        end
      rescue AWS::EC2::Errors::InvalidVolume::NotFound => e
        @logger.debug "Failed to remove volume: #{vol.id}, #{e}"
      end
    end
  end
  @logger.notify "Freed #{volume_count} volume(s)"

end

#kill_zombies(max_age = ZOMBIE, key = key_name) ⇒ Object

Shutdown and destroy ec2 instances idenfitied by key that have been alive longer than ZOMBIE hours.

Parameters:

  • max_age (Integer) (defaults to: ZOMBIE) —

    The age in hours that a machine needs to be older than to be considered a zombie

  • key (String) (defaults to: key_name) —

    The key_name to match for



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 170

def kill_zombies(max_age = ZOMBIE, key = key_name)
  @logger.notify("aws-sdk: Kill Zombies! (keyname: #{key}, age: #{max_age} hrs)")
  #examine all available regions
  kill_count = 0
  time_now = Time.now.getgm #ec2 uses GM time
  @ec2.regions.each do |region|
    @logger.debug "Reviewing: #{region.name}"
    # Note: don't use instances.each here as that funtion doesn't allow proper rescue from error states
    instances = @ec2.regions[region.name].instances
    instances.each do |instance|
      begin
        if (instance.key_name =~ /#{key}/)
          @logger.debug "Examining #{instance.id} (keyname: #{instance.key_name}, launch time: #{instance.launch_time}, status: #{instance.status})"
          if ((time_now - instance.launch_time) >  max_age*60*60) and instance.status.to_s !~ /terminated/
            @logger.debug "Kill! #{instance.id}: #{instance.key_name} (Current status: #{instance.status})"
            instance.terminate()
            kill_count += 1
          end
        end
      rescue AWS::Core::Resource::NotFound, AWS::EC2::Errors => e
        @logger.debug "Failed to remove instance: #{instance.id}, #{e}"
      end
    end
  end

  @logger.notify "#{key}: Killed #{kill_count} instance(s)"
end

#launch_all_nodes ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Create EC2 instances for all hosts, tag them, and wait until they're running. When a host provides a subnet_id, create the instance in that subnet, otherwise prefer a CONFIG subnet_id. If neither are set but there is a CONFIG subnet_ids list, attempt to create the host in each specified subnet, which might fail due to capacity constraints, for example. Specifying both a CONFIG subnet_id and subnet_ids will provoke an error.



368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 368

def launch_all_nodes
  @logger.notify("aws-sdk: launch all hosts in configuration")
  ami_spec = YAML.load_file(@options[:ec2_yaml])["AMI"]
  global_subnet_id = @options['subnet_id']
  global_subnets = @options['subnet_ids']
  if global_subnet_id and global_subnets
    raise RuntimeError, 'Config specifies both subnet_id and subnet_ids'
  end
  no_subnet_hosts = []
  specific_subnet_hosts = []
  some_subnet_hosts = []
  @hosts.each do |host|
    if global_subnet_id or host['subnet_id']
      specific_subnet_hosts.push(host)
    elsif global_subnets
      some_subnet_hosts.push(host)
    else
      no_subnet_hosts.push(host)
    end
  end
  instances = [] # Each element is {:instance => i, :host => h}
  begin
    @logger.notify("aws-sdk: launch instances not particular about subnet")
    launch_nodes_on_some_subnet(some_subnet_hosts, global_subnets, ami_spec,
                                instances)
    @logger.notify("aws-sdk: launch instances requiring a specific subnet")
    specific_subnet_hosts.each do |host|
      subnet_id = host['subnet_id'] || global_subnet_id
      instance = create_instance(host, ami_spec, subnet_id)
      instances.push({:instance => instance, :host => host})
    end
    @logger.notify("aws-sdk: launch instances requiring no subnet")
    no_subnet_hosts.each do |host|
      instance = create_instance(host, ami_spec, nil)
      instances.push({:instance => instance, :host => host})
    end
    wait_for_status(:running, instances)
  rescue Exception => ex
    @logger.notify("aws-sdk: exception - #{ex}")
    kill_instances(instances.map{|x| x[:instance]})
    raise ex
  end
  # At this point, all instances should be running since wait
  # either returns on success or throws an exception.
  if instances.empty?
    raise RuntimeError, "Didn't manage to launch any EC2 instances"
  end
  # Assign the now known running instances to their hosts.
  instances.each {|x| x[:host]['instance'] = x[:instance]}
  nil
end

#launch_nodes_on_some_subnet(hosts, subnets, ami_spec, instances_created) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

For each host, create an EC2 instance in one of the specified subnets and push it onto instances_created. Each subnet will be tried at most once for each host, and more than one subnet may be tried if capacity constraints are encountered. Each Hash in instances_created will contain an :instance and :host value.

Parameters:

  • hosts (Enumerable<Host>)
  • subnets (Enumerable<String>)
  • ami_spec (Hash)
  • instances_created —

    Enumerable<HashSymbol=>EC2::Instance,Host>



329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 329

def launch_nodes_on_some_subnet(hosts, subnets, ami_spec, instances_created)
  # Shuffle the subnets so we don't always hit the same one
  # first, and cycle though the subnets independently of the
  # host, so we stick with one that's working.  Try each subnet
  # once per-host.
  if subnets.nil? or subnets.empty?
    return
  end
  subnet_i = 0
  shuffnets = subnets.shuffle
  hosts.each do |host|
    instance = nil
    shuffnets.length.times do
      begin
        subnet_id = shuffnets[subnet_i]
        instance = create_instance(host, ami_spec, subnet_id)
        instances_created.push({:instance => instance, :host => host})
        break
      rescue AWS::EC2::Errors::InsufficientInstanceCapacity => ex
        @logger.notify("aws-sdk: hit #{subnet_id} capacity limit; moving on")
        subnet_i = (subnet_i + 1) % shuffnets.length
      end
    end
    if instance.nil?
      raise RuntimeError, "unable to launch host in any requested subnet"
    end
  end
end

#load_fog_credentials(dot_fog = '.fog') ⇒ Hash<Symbol, String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return a hash containing the fog credentials for EC2

Parameters:

  • dot_fog (String) (defaults to: '.fog') —

    dot fog path

Returns:

  • (Hash<Symbol, String>) —

    ec2 credentials



761
762
763
764
765
766
767
768
769
770
771
772
773
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 761

def load_fog_credentials(dot_fog = '.fog')
  fog = YAML.load_file( dot_fog )

  default = fog[:default]

  creds = {}
  creds[:access_key] = default[:aws_access_key_id]
  creds[:secret_key] = default[:aws_secret_access_key]
  raise "You must specify an aws_access_key_id in your .fog file (#{dot_fog}) for ec2 instances!" unless creds[:access_key]
  raise "You must specify an aws_secret_access_key in your .fog file (#{dot_fog}) for ec2 instances!" unless creds[:secret_key]

  creds
end

#local_user ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the local user running this tool

Returns:

  • (String) —

    username of local user



634
635
636
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 634

def local_user
  ENV['USER']
end

#log_instances(key = key_name, status = /running/) ⇒ Object

Print instances to the logger. Instances will be from all regions associated with provided key name and limited by regex compared to instance status. Defaults to running instances.

Parameters:

  • key (String) (defaults to: key_name) —

    The key_name to match for

  • status (Regex) (defaults to: /running/) —

    The regular expression to match against the instance's status



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 102

def log_instances(key = key_name, status = /running/)
  instances = []
  @ec2.regions.each do |region|
    @logger.debug "Reviewing: #{region.name}"
    @ec2.regions[region.name].instances.each do |instance|
      if (instance.key_name =~ /#{key}/) and (instance.status.to_s =~ status)
        instances << instance
      end
    end
  end
  output = ""
  instances.each do |instance|
    output << "#{instance.id} keyname: #{instance.key_name}, dns name: #{instance.dns_name}, private ip: #{instance.private_ip_address}, ip: #{instance.ip_address}, launch time #{instance.launch_time}, status: #{instance.status}\n"
  end
  @logger.notify("aws-sdk: List instances (keyname: #{key})")
  @logger.notify("#{output}")
end

#populate_dns ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Populate the hosts IP address from the EC2 dns_name



478
479
480
481
482
483
484
485
486
487
488
489
490
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 478

def populate_dns
  # Obtain the IP addresses and dns_name for each host
  @hosts.each do |host|
    @logger.notify("aws-sdk: Populate DNS for #{host.name}")
    instance = host['instance']
    host['ip'] = instance.ip_address ? instance.ip_address : instance.private_ip_address
    host['private_ip'] = instance.private_ip_address
    host['dns_name'] = instance.dns_name
    @logger.notify("aws-sdk: name: #{host.name} ip: #{host['ip']} private_ip: #{host['private_ip']} dns_name: #{instance.dns_name}")
  end

  nil
end

#provision ⇒ void

This method returns an undefined value.

Provision all hosts on EC2 using the AWS::EC2 API



44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 44

def provision
  start_time = Time.now

  # Perform the main launch work
  launch_all_nodes()

  # Add metadata tags to each instance
  add_tags()

  # Grab the ip addresses and dns from EC2 for each instance to use for ssh
  populate_dns()

  #enable root if user is not root
  enable_root_on_hosts()

  # Set the hostname for each box
  set_hostnames()

  # Configure /etc/hosts on each host
  configure_hosts()

  @logger.notify("aws-sdk: Provisioning complete in #{Time.now - start_time} seconds")

  nil #void
end

#public_key ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Retrieve the public key locally from the executing users ~/.ssh directory

Returns:

  • (String) —

    contents of public key



609
610
611
612
613
614
615
616
617
618
619
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 609

def public_key
  filename = File.expand_path('~/.ssh/id_rsa.pub')
  unless File.exists? filename
    filename = File.expand_path('~/.ssh/id_dsa.pub')
    unless File.exists? filename
      raise RuntimeError, 'Expected either ~/.ssh/id_rsa.pub or ~/.ssh/id_dsa.pub but found neither'
    end
  end

  File.read(filename)
end

#security_group_by_id(id) ⇒ AWS::EC2::SecurityGroup

Provided an id return a security group object Security object will respond to methods described here: AWS SecurityGroup Object.

Parameters:

  • id (String) —

    The id of the security group to return

Returns:

  • (AWS::EC2::SecurityGroup) —

    An AWS::EC2 security group object



154
155
156
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 154

def security_group_by_id(id)
  @ec2.security_groups[id]
end

#security_groups ⇒ Array<AWS::EC2::SecurityGroup>

Return all security groups currently on ec2.

Returns:

  • (Array<AWS::EC2::SecurityGroup>) —

    An array of AWS::EC2 security group objects

See Also:

  • AwsSdk#security_goup_by_id


161
162
163
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 161

def security_groups
  @ec2.security_groups
end

#set_hostnames ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Set the hostname of all instances to be the hostname defined in the beaker configuration.



579
580
581
582
583
584
585
586
587
588
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 579

def set_hostnames
  @hosts.each do |host|
    if host['platform'] =~ /el-7/
      # on el-7 hosts, the hostname command doesn't "stick" randomly
      host.exec(Command.new("hostnamectl set-hostname #{host.name}"))
    else
      host.exec(Command.new("hostname #{host.name}"))
    end
  end
end

#vpc_by_id(id) ⇒ AWS::EC2::VPC

Provided an id return a VPC object. VPC object will respond to methods described here: AWS VPC Object.

Parameters:

  • id (String) —

    The id of the VPC to return

Returns:

  • (AWS::EC2::VPC) —

    An AWS::EC2 vpc object



139
140
141
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 139

def vpc_by_id(id)
  @ec2.vpcs[id]
end

#vpcs ⇒ Array<AWS::EC2::VPC>

Return all VPCs currently on ec2.

Returns:

  • (Array<AWS::EC2::VPC>) —

    An array of AWS::EC2 vpc objects

See Also:



146
147
148
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 146

def vpcs
  @ec2.vpcs
end

#wait_for_status(status, instances) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Wait until all instances reach the desired state. Each Hash in instances must contain an :instance and :host value.

Parameters:

  • status (Symbol) —

    EC2 state to wait for, :running :stopped etc.

  • instances —

    Enumerable<HashSymbol=>EC2::Instance,Host>



427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 427

def wait_for_status(status, instances)
  # Wait for each node to reach status :running
  @logger.notify("aws-sdk: Waiting for all hosts to be #{status}")
  instances.each do |x|
    name = x[:name]
    instance = x[:instance]
    @logger.notify("aws-sdk: Wait for node #{name} to be #{status}")
    # Here we keep waiting for the machine state to reach ':running' with an
    # exponential backoff for each poll.
    # TODO: should probably be a in a shared method somewhere
    for tries in 1..10
      begin
        if instance.status == status
          # Always sleep, so the next command won't cause a throttle
          backoff_sleep(tries)
          break
        elsif tries == 10
          raise "Instance never reached state #{status}"
        end
      rescue AWS::EC2::Errors::InvalidInstanceID::NotFound => e
        @logger.debug("Instance #{name} not yet available (#{e})")
      end
      backoff_sleep(tries)
    end
  end
end