Module: Telly

Defined in:
lib/telly.rb

Overview

telly.rb

This module provides functions to add test results in testrail from a finished beaker run.

It takes in a beaker junit file and a TestRail Testrun ID

It matches the beaker tests with TestRail testcases by looking for the

test case ID in the beaker script. The combination of a test run and a test case
allows the script to add a result for a particular instance of a test case.
In TestRail parlance, this is confusingly called a test.

From the TestRail API docs:

"In TestRail, tests are part of a test run and the test cases are part of the
related test suite. So, when you create a new test run, TestRail creates a test
for each test case found in the test suite of the run. You can therefore think
of a test as an “instance” of a test case which can have test results, comments 
and a test status.""

Constant Summary collapse

TESTRAIL_URL = 
'https://testrail.ops.puppetlabs.net/'
CREDENTIALS_FILE = 
'~/.testrail_credentials.yaml'
TESTCASE_ID_REGEX =

Used for extracted the test case ID from beaker scripts

/.*(?<jira_ticket>\w+-\d+).*[cC](?<testrun_id>\d+)/
PASSED =

Testrail Status IDs

1
BLOCKED = 
2
FAILED = 
5

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.beaker_test_path(junit_file_path, junit_result) ⇒ String

Calculates the path to a beaker test file by combining the junit file path with the test name from the junit results. Makes the assumption that junit folder that beaker creates will always be 2 directories up from the beaker script base directory. TODO somewhat hacky, maybe a config/command line option

Examples:

load_junit_results(“~/junit/latest/beaker_junit.xml”)

Parameters:

  • junit_file_path (String) —

    Path to a junit xml file

  • junit_result (String) —

    Path to a junit xml file

Returns:

  • (String) —

    The path to the beaker script from the junit test result



282
283
284
285
286
287
 
# File 'lib/telly.rb', line 282

def Telly.beaker_test_path(junit_file_path, junit_result)
  beaker_folder_path = junit_result[:classname]
  test_filename = junit_result[:name]

  File.join(File.dirname(junit_file_path), "../../", beaker_folder_path, test_filename)
end

.get_testrail_api(credentials) ⇒ TestRail::APIClient

Returns a testrail API object that talks to testrail

Examples:

api = get_testrail_api(load_credentials)

Parameters:

  • credentials (Hash) —

    A hash containing at least two keys, testrail_username and testrail_password

Returns:



107
108
109
110
111
112
113
 
# File 'lib/telly.rb', line 107

def Telly.get_testrail_api(credentials)
  client = TestRail::APIClient.new(TESTRAIL_URL)
  client.user = credentials["testrail_username"]
  client.password = credentials["testrail_password"]

  return client
end

.load_credentials(credentials_file) ⇒ Hash

Load testrail credentials from file

Examples:

password = load_credentials()

Returns:

  • (Hash) —

    Contains testrail_username and testrail_password



86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/telly.rb', line 86

def Telly.load_credentials(credentials_file)
  begin
    YAML.load_file(File.expand_path(credentials_file))  
  rescue
    puts "Error: Could not find #{credentials_file}"
    puts "Create #{credentials_file} with the following:"
    puts "testrail_username: your.username\ntestrail_password: yourpassword"

    exit
    
  end
end

.load_junit_results(junit_file) ⇒ Hash

Loads the results of a beaker run. Returns hash of failures, passes, and skips that each hold a list of junit xml objects

Examples:

load_junit_results(“~/junit/latest/beaker_junit.xml”)

Parameters:

  • junit_file (String) —

    Path to a junit xml file

Returns:

  • (Hash) —

    A hash containing xml objects for the failures, skips, and passes



244
245
246
247
248
249
250
251
252
 
# File 'lib/telly.rb', line 244

def Telly.load_junit_results(junit_file)
  junit_doc = Nokogiri::XML(File.read(junit_file))

  failures = junit_doc.xpath('//testcase[failure]')
  skips = junit_doc.xpath('//testcase[skip]')
  passes = junit_doc.xpath('//testcase[not(failure) and not(skip)]')

  return {failures: failures, skips: skips, passes: passes}
end

.main(options) ⇒ Void

Run the importer

Examples:

password = Telly::main(parse_opts)

Parameters:

  • An (Hash) —

    optparse object

Returns:

  • (Void) 


55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/telly.rb', line 55

def Telly.main(options)
  # Get pass/fail/skips from junit file
  results = load_junit_results(options[:junit_file])

  puts "Run results:"
  puts "#{results[:passes].length} Passing"
  puts "#{results[:failures].length} Failing or Erroring"
  puts "#{results[:skips].length} Skipped"

  # Set results in testrail
  bad_results = set_testrail_results(results, options[:junit_file], options[:testrun_id])

  # Print error messages
  if not bad_results.empty?
    puts "Error: There were problems processing these test scripts:"
    bad_results.each do |test_script, error|
      puts "#{test_script}:\n\t#{error}"
    end
  end
end

.make_testrail_time(seconds_string) ⇒ String

Returns a string that testrail accepts as an elapsed time Input from beaker is a float in seconds, so it rounds it to the nearest second, and adds an ‘s’ at the end

Testrail throws an exception if it gets “0s”, so it returns a minimum of “1s”

Examples:

puts make_testrail_time(“2.34”) # “2s”

Parameters:

  • seconds_string (String) —

    A string that contains only a number, the elapsed time of a test

Returns:

  • (String) —

    The elapsed time of the test run, rounded and with an ‘s’ appended



221
222
223
224
225
226
227
228
 
# File 'lib/telly.rb', line 221

def Telly.make_testrail_time(seconds_string)
  # If time is 0, make it 1
  rounded_time = [seconds_string.to_f.round, 1].max
  # Test duration
  time_elapsed = "#{rounded_time}s"

  return time_elapsed
end

.set_testrail_results(results, junit_file, testrun_id) ⇒ Void

Sets the results in testrail. Tests that have testrail API exceptions are kept track of in bad_results

Parameters:

  • results (Hash) —

    A hash of lists of xml objects from the junit output file.

  • junit_file (String) —

    The path to the junit xml file Needed for determining the path of the test file in add_failure, etc

  • testrun_id (String) —

    The TestRail test run ID

Returns:

  • (Void) 


125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
# File 'lib/telly.rb', line 125

def Telly.set_testrail_results(results, junit_file, testrun_id)
  credentials = load_credentials(CREDENTIALS_FILE)
  api = get_testrail_api(credentials)

  # Results that couldn't be set in testrail for some reason
  bad_results = {}

  # passes
  results[:passes].each do |junit_result|
    begin
      submit_result(api, PASSED, junit_result, junit_file, testrun_id)    
    rescue TestRail::APIError => e
      bad_results[junit_result[:name]] = e.message
    end
  end

  # Failures
  results[:failures].each do |junit_result|
    begin
      submit_result(api, FAILED, junit_result, junit_file, testrun_id)    
    rescue TestRail::APIError => e
      bad_results[junit_result[:name]] = e.message
    end
  end

  # Skips
  results[:skips].each do |junit_result|
    begin
      submit_result(api, BLOCKED, junit_result, junit_file, testrun_id)    
    rescue TestRail::APIError => e
      bad_results[junit_result[:name]] = e.message
    end
  end

  return bad_results
end

.submit_result(api, status, junit_result, junit_file, testrun_id) ⇒ Void

Submits a test result to TestRail

Examples:

submit_result(api, BLOCKED, junit_result, junit_file, testrun_id)

Parameters:

  • api (TestRail::APIClient) —

    TestRail API object

  • status (int) —

    The testrail status to set

  • junit_result (Nokogiri::XML::Element) —

    The nokogiri node that holds the junit result

  • junit_file (String) —

    Path to the junit file the test result originated from

  • testrun_id (String) —

    The testrun ID

Returns:

  • (Void)

Raises:

  • (TestRail::APIError) —

    When there is a problem with the API request, testrail raises this exception. Should be caught for error reporting



176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
 
# File 'lib/telly.rb', line 176

def Telly.submit_result(api, status, junit_result, junit_file, testrun_id)
  test_file_path = beaker_test_path(junit_file, junit_result)

  puts junit_result.class
  testcase_id = testcase_id_from_beaker_script(test_file_path)

  time_elapsed = make_testrail_time(junit_result[:time])

  # Make appropriate comment for testrail
  case status
  when FAILED
    error_message = junit_result.xpath('./failure').first[:message]
    testrail_comment = "Failed with message:\n#{error_message}"
  when BLOCKED
    skip_message = junit_result.xpath('system-out').first.text
    testrail_comment = "Skipped with message:\n#{skip_message}"
  else
    testrail_comment = "Passed"
  end

  puts "\nSetting result for test case: #{testcase_id}"
  puts "Adding comment:\n#{testrail_comment}"

  api.send_post("add_result_for_case/#{testrun_id}/#{testcase_id}", 
    {
      status_id: status,
      comment: testrail_comment,
      elapsed: time_elapsed,
    }
  )
end

.testcase_id_from_beaker_script(beaker_file) ⇒ String

Extracts the test case id from the test script

Examples:

testcase_id_from_beaker_script(“~/tests/test_the_things.rb”) # 1234

Parameters:

  • beaker_file (String) —

    Path to a beaker test script

Returns:

  • (String) —

    The test case ID



262
263
264
265
266
267
 
# File 'lib/telly.rb', line 262

def Telly.testcase_id_from_beaker_script(beaker_file)
  # Find first matching line
  match = File.readlines(beaker_file).map { |line| line.match(TESTCASE_ID_REGEX) }.compact.first

  match[:testrun_id]
end

Instance Method Details

#do_stub_test(credentials) ⇒ Object 



37
38
39
40
41
 
# File 'lib/telly.rb', line 37

def do_stub_test(credentials)
  api = get_testrail_api(credentials)

  api.send_post
end