Module: Beaker::DSL::Structure

Included in:
Beaker::DSL
Defined in:
lib/beaker/dsl/structure.rb

Overview

These are simple structural elements necessary for writing understandable tests and ensuring cleanup actions happen. If using a third party test runner they are unnecessary.

To include this in your own test runner a method #logger should be available to yield a logger that implements Logger's interface. As well as a method #teardown_procs that yields an array.

Examples:

Structuring a test case.

test_name 'Look at me testing things!' do
  teardown do
    ...clean up actions...
  end

  step 'Prepare the things' do
    ...setup steps...
  end

  step 'Test the things' do
    ...tests...
  end

  step 'Expect this to fail' do
    expect_failure('expected to fail due to PE-1234') do
    assert_equal(400, response.code, 'bad response code from API call')
  end
end

Instance Method Summary collapse

Instance Method Details

#confine(type, criteria, host_array = nil, &block) ⇒ Array<Host>

Note:

This will modify the TestCase#hosts member in place unless an array of hosts is passed into it and TestCase#logger yielding an object that responds like Logger#warn, as well as Outcomes#skip_test, and optionally TestCase#hosts.

Limit the hosts a test case is run against

Examples:

Basic usage to confine to debian OSes.

confine :to, :platform => 'debian'

Confining to anything but Windows and Solaris

confine :except, :platform => ['windows', 'solaris']

Using additional block to confine to Solaris global zone.

confine :to, :platform => 'solaris' do |solaris|
  on( solaris, 'zonename' ) =~ /global/
end

Confining to an already defined subset of hosts

confine :to, {}, agents

Confining from an already defined subset of hosts

confine :except, {}, agents

Confining to all ubuntu agents + all non-agents

confine :to, { :platform => 'ubuntu' }, agents

Confining to any non-windows agents + all non-agents

confine :except, { :platform => 'windows' }, agents

Raises:

  • (SkipTest)

    Raises skip test if there are no valid hosts for this test case after confinement.



258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
# File 'lib/beaker/dsl/structure.rb', line 258

def confine(type, criteria, host_array = nil, &block)
  hosts_to_modify = Array( host_array || hosts )
  hosts_not_modified = hosts - hosts_to_modify #we aren't examining these hosts
  case type
  when :except
    if criteria and ( not criteria.empty? )
      hosts_to_modify = hosts_to_modify - select_hosts(criteria, hosts_to_modify, &block) + hosts_not_modified
    else
      # confining to all hosts *except* provided array of hosts
      hosts_to_modify = hosts_not_modified
    end
    if hosts_to_modify.empty?
      logger.warn "No suitable hosts without: #{criteria.inspect}"
      skip_test "No suitable hosts found without #{criteria.inspect}"
    end
  when :to
    if criteria and ( not criteria.empty? )
      hosts_to_modify = select_hosts(criteria, hosts_to_modify, &block) + hosts_not_modified
    else
      # confining to only hosts in provided array of hosts
    end
    if hosts_to_modify.empty?
      logger.warn "No suitable hosts with: #{criteria.inspect}"
      skip_test "No suitable hosts found with #{criteria.inspect}"
    end
  else
    raise "Unknown option #{type}"
  end
  self.hosts = hosts_to_modify
  hosts_to_modify
end

#confine_block(type, criteria, host_array = nil, &block) ⇒ Object

Ensures that host restrictions as specifid by type, criteria and host_array are confined to activity within the passed block. TestCase#hosts is reset after block has executed.

See Also:



295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
# File 'lib/beaker/dsl/structure.rb', line 295

def confine_block(type, criteria, host_array = nil, &block)
  host_array = Array( host_array || hosts )
  original_hosts = self.hosts.dup
  confine(type, criteria, host_array)

  yield

rescue Beaker::DSL::Outcomes::SkipTest => e
  # I don't like this much, but adding options to confine is a breaking change
  # to the DSL that would involve a major version bump
  if e.message !~ /No suitable hosts found/
    # a skip generated from the provided block, pass it up the chain
    raise e
  end
ensure
  self.hosts = original_hosts
end

#expect_failure(explanation, &block) ⇒ Object

Wrap an assert that is supposed to fail due to a product bug, an undelivered feature, or some similar situation.

This converts failing asserts into passing asserts (so we can continue to run the test even though there are underlying product bugs), and converts passing asserts into failing asserts (so we know when the underlying product bug has been fixed).

Pass an assert as a code block, and pass an explanatory message as a parameter. The assert's logic will be inverted (so passes turn into fails and fails turn into passes).

Examples:

Typical usage

expect_failure('expected to fail due to PE-1234') do
  assert_equal(400, response.code, 'bad response code from API call')
end

Output when a product bug would normally cause the assert to fail

Warning: An assertion was expected to fail, and did.
This is probably due to a known product bug, and is probably not a problem.
Additional info: 'expected to fail due to PE-6995'
Failed assertion: 'bad response code from API call.
<400> expected but was <409>.'

Output when the product bug has been fixed

<RuntimeError: An assertion was expected to fail, but passed.
This is probably because a product bug was fixed, and "expect_failure()"
needs to be removed from this assert.
Additional info: 'expected to fail due to PE-6996'>

Raises:

  • (RuntimeError)

    if the code block passed to this method does not raise a Assertions (i.e., if the assert passes)

Author:

  • Chris Cowell-Shah (ccs@puppetlabs.com)



181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
# File 'lib/beaker/dsl/structure.rb', line 181

def expect_failure(explanation, &block)
  begin
    yield if block_given?  # code block should contain an assert that you expect to fail
  rescue Beaker::DSL::Assertions, Minitest::Assertion => failed_assertion
    # Yay! The assert in the code block failed, as expected.
    # Swallow the failure so the test passes.
    logger.notify 'An assertion was expected to fail, and did. ' +
                    'This is probably due to a known product bug, ' +
                    'and is probably not a problem. ' +
                    "Additional info: '#{explanation}' " +
                    "Failed assertion: '#{failed_assertion}'"
    return
  end
  # Uh-oh! The assert in the code block unexpectedly passed.
  fail('An assertion was expected to fail, but passed. ' +
           'This is probably because a product bug was fixed, and ' +
           '"expect_failure()" needs to be removed from this test. ' +
           "Additional info: '#{explanation}'")
end

#manual_step(step_name) ⇒ Object

Provides a method to help manual tests. So we can use beaker to set up the environment, then prompt a user to manually check the setup.



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
# File 'lib/beaker/dsl/structure.rb', line 60

def manual_step step_name
  require 'readline'
  logger.notify "\n* #{step_name}\n"
  if(!@options.has_key?(:exec_manual_tests))
    # if the option -exec-manual-tests is not set then this has executed outside of a manual tests
    # so we raise an error to avoid issues
    raise('--exec-manual-tests option not set, this means a manual_step was used outside a manual_test')
  end

  set_current_step_name(step_name)
  # Here we prompt the user to tell us if the step passed or failed
  loop do 
    input = Readline.readline('Did this step pass, Y/n? ', true).squeeze(" ").strip.downcase
    if %w(y yes).include?(input)
      break
    elsif %w(n no).include?(input)
      # if the step failed, the user can enter a fail message.
      # we loops to ensure they give use a fail message
      fail_message = ''
      loop do
        fail_message = Readline.readline('What was the reason for failure? ', true).squeeze(" ").strip
        if fail_message == ''
          # if nothing is entered we tell the user to enter something
          puts "No reason for failure given, please enter reason for failure."
        else
          break
        end
      end
      raise Beaker::DSL::FailTest, fail_message
    else
      # if something other than Y or n is returned we ask them again
      puts "Please enter Y or n."
    end
  end
end

#manual_test(manual_test_name, &block) ⇒ Object

Provides a method to mark manual tests. If the –exec-manual-tests param is not set then we skip the test this is so manual tests do not execute by mistake



102
103
104
105
106
107
108
109
110
111
112
# File 'lib/beaker/dsl/structure.rb', line 102

def manual_test manual_test_name, &block
  if(@options.has_key?(:exec_manual_tests) && @options[:exec_manual_tests] == true)
    # here the option is set so we run the test as normal
    test_name manual_test_name, &block 
  else
    # here no option was set so we log the test name and skip it
    test_name manual_test_name
    raise( Beaker::DSL::SkipTest,
           '--exec-manual-tests option not set, so skipping manual test' )
  end
end

#select_hosts(criteria, host_array = nil, &block) ⇒ Array<Host>

Return a set of hosts that meet the given criteria



331
332
333
334
335
336
337
338
339
340
341
342
343
344
# File 'lib/beaker/dsl/structure.rb', line 331

def select_hosts(criteria, host_array = nil, &block)
  hosts_to_select_from = host_array || hosts
  criteria.each_pair do |property, value|
    hosts_to_select_from = hosts_to_select_from.select do |host|
      inspect_host host, property, value
    end
  end
  if block_given?
    hosts_to_select_from = hosts_to_select_from.select do |host|
      yield host
    end
  end
  hosts_to_select_from
end

#step(step_name, &block) ⇒ Object

Provides a method to help structure tests into coherent steps.



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# File 'lib/beaker/dsl/structure.rb', line 38

def step step_name, &block
  logger.notify "\n* #{step_name}\n"
  set_current_step_name(step_name)
  if block_given?
    begin
      logger.with_indent do
        yield
      end
    rescue Exception => e
      if(@options.has_key?(:debug_errors) && @options[:debug_errors] == true)
        logger.info("Exception raised during step execution and debug-errors option is set, entering pry. Exception was: #{e.inspect}")
        logger.info("HINT: Use the pry 'backtrace' and 'up' commands to navigate to the test code")
        binding.pry
      end
      raise e
    end
  end
end

#teardown(&block) ⇒ Object

Declare a teardown process that will be called after a test case is complete.

Examples:

Always remove /etc/puppet/modules

teardown do
  on(master, puppet_resource('file', '/etc/puppet/modules',
    'ensure=absent', 'purge=true'))
end


138
139
140
# File 'lib/beaker/dsl/structure.rb', line 138

def teardown &block
  @teardown_procs << block
end

#test_name(my_name, &block) ⇒ Object

Provides a method to name tests.



119
120
121
122
123
124
125
126
127
# File 'lib/beaker/dsl/structure.rb', line 119

def test_name my_name, &block
  logger.notify "\n#{my_name}\n"
  set_current_test_name(my_name)
  if block_given?
    logger.with_indent do
      yield
    end
  end
end