test_center plugin 🎯

Have you ever spent too much time trying to fix fragile tests only to give up with nothing real to show? Use the fastlane actions from test_center to remove internal and external interference from your tests, so that you can focus on what makes 💰: features that customers love 😍.

Quick Start | Overview | Issues and Feedback | Contributing | License

Quick Start

This project is a fastlane plugin. To get started with fastlane-plugin-test_center, add it to your project by running:

fastlane add_plugin test_center

Add this example 'lane' to your Fastfile, change MY_XCODE_PROJECT_FILEPATH to point to your project path, and change the option scheme: AtomicBoy in the call to multi_scan to be the name of your Xcode projects Scheme:


################################################################################
# An example of how one can use the plugin's :multi_scan action to run tests
# that have not yet passed (up to 3 times). If, after the 3 runs of the tests, there
# are still failing tests, print out the number of tests that are still failing.
#
# For a walkthrough to write a lane that can run tests up to 3 times, suppress
# the failing tests in the Xcode project, and create a Github Pull Request, see:
# https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/docs/WALKTHROUGH.md
################################################################################

MY_XCODE_PROJECT_FILEPATH = File.absolute_path('../AtomicBoy/AtomicBoy.xcodeproj')
lane :sweep do
  test_run_block = lambda do |testrun_info|
    failed_test_count = testrun_info[:failed].size

    if failed_test_count > 0
      UI.important('The run of tests would finish with failures due to fragile tests here.')

      try_attempt = testrun_info[:try_count]
      if try_attempt < 3
        UI.header('Since we are using :multi_scan, we can re-run just those failing tests!')
      end
    end
  end

  result = multi_scan(
    project: MY_XCODE_PROJECT_FILEPATH,
    try_count: 3,
    fail_build: false,
    scheme: 'AtomicBoy',
    testrun_completed_block: test_run_block,
    parallel_testrun_count: 4
  )
  unless result[:failed_testcount].zero?
    UI.message("There are #{result[:failed_testcount]} legitimate failing tests")
  end
end

Overview

This plugin makes testing your iOS app easier by providing you actions that give you greater control over everthing related to testing your app.

multi_scan began when I engineered an action to only re-run the failed tests in order to determine which ones were truly failing, or just failing randomly due to a fragile infrastructure. This action morphed into an entire plugin with many actions related to tests.

This fastlane plugin includes the following actions:

multi_scan: gives you control over how your tests are exercised.
suppress_tests_from_junit: from a test report, suppresses tests in your project.
suppress_tests: from a provided list, suppresses tests in your project.
suppressed_tests: returns a list of the suppressed tests in your project.
tests_from_junit: from a test report, returns lists of passing and failed tests.
tests_from_xctestrun: from an xctestrun file, returns a list of tests for each of its test targets.
collate_junit_reports: combines multiple junit test reports into one report.
collate_html_reports: combines multiple html test reports into one report.
collate_json_reports: combines multiple json test reports into one report.
collate_test_result_bundles: combines multiple test_result bundles into one test_result bundle.

multi_scan 🎉

Use :multi_scan intead of :scan to improve the usefulness of iOS test results, reduce test run time, inspect partial results periodically during a test run, and provide better results reporting.

Improving Usefulness

Over time, your tests can change the state of your application in unexpected ways that cause other tests to fail randomly. Or, the tools and infrastructure for testing are the root causes of random test failures. The test results may not truly reflect how the product code is working.

Rather than wasting time trying to account for unstable tools, or trying to tweak your test code ad-nauseum to get a passing result reliably, just use the :try_count option to run :scan multiple times, running only the tests that failed each time. This ensures that any fragility is ironed out over a number of "tries". The end result is that only the truly failing tests appear.

Another issue that can cause tests to incorrectly fail comes from an issue with the iOS Simulator. If you provide a huge number of tests to the iOS Simulator, it can exhaust the available resources and cause it to fail large numbers of tests. You can get around this by running your tests in batches using the :batch_count option in order to lighten the load on the simulator.

Reduce Test Run Time

Make better use of your Mac resources by running batches of test runs in parallel iOS Simulators running simultaneously. Use the :parallel_testrun_count option to specify 2 to 6 simulators, each running a subset of your tests. It is not recommended to run more than 6 simulators in parallel as the service that backs the simulators can fail to connect to them.

Note: while Xcode provides the option to run testsuites in parallel, this does not help if one testsuite has 100 testcases and another has 300 testcases. That's because each of those testsuites will run on their own iOS Simulator and you have to wait for the Simulator with the most testcases: 300 testcases in this example.

`multi_scan`, on the other hand, can split those tests into 4 batches of 100 using a `:parallel_testrun_count` of 4. You only have to wait for the iOS Simulators to finish 100 testcases.

Inspect Partial Results

If you have a large number of tests, and you want to inspect the overall status of how test runs are progressing, you can use the :testrun_completed_block callback to bailout early or make adjustments on how your tests are exercised.

Better Results Reporting

Do you have multiple test targets and the normal operation of :scan is providing you a test report that implies that all the tests ran in just one test target? Don't worry, :multi_scan has fixed that. It will provide a separate test report for each test target. It can handle JUnit, HTML, JSON, and Apple's test_result bundles.

test_result bundles are particularly useful because they contain screenshots of the UI when a UI test fails so you can review what was actually there compared to what you expected.

Invocation based tests

If your tests are invocation based like Kiwi you need to set :invocation_based_tests to handle these tests, because unlike XCTests the list of tests cannot be deterimned before running and also you can't try an exact test (The reruns run the whole file where the test failed).