Polytrix - the Polyglot Testing Matrix

Polytrix is a polyglot test runner and documentation generator. It aims to let you run sample code written in any language. It's especially useful if you want to run similar code samples in multiple languages, a project that has been ported to several languages, or API clients for the same service that are provided in several languages.

Polytrix was influenced by a number of polyglot projects, including Travis-CI, Docco, Slate, and polyglot test-suites like the JSON Schema Test Suite and the JSON-LD Test Suite.

The user-experience was heavily influenced by test-kitchen. Polytrix aims to do for multi-language development of command-line applications what test-kitchen has done for multi-platform development of infrastructure.

Installing Polytrix

Polytrix is distributed as a Ruby Gem. It should be installed using Bundler by adding this line to your Gemfile:

gem 'polytrix', '~> 0.1'

And then running bundle install.

It can also be installed without Bundler by running gem install polytrix.

Getting Help

Polytrix is primarily intented to be run as a standalone utility, though it does have an API for integrating with RSpec or other Ruby-based test frameworks.

Note: You may need to prefix commands with bundle exec if you installed Polytrix using Bundler.

If you need a quick reminder of what the polytrix command gives you, then use the help subcommand:

$ bundle exec polytrix help
Commands:s
  polytrix bootstrap [(all|<REGEX>)] [opts] # Bootstraps the code samples for one or more tests
  polytrix code2doc [(all|<REGEX>)] [opts]  # Converts annotated code samples to Markdown or reStructuredText
  polytrix clean [(all|<REGEX>)] [opts]     # Removes test results for one or more tests
  polytrix clone [(all|<REGEX>)] [opts]     # Clones the code samples from git for one or more tests
  polytrix exec [(all|<REGEX>)] [opts]      # Executes code samples for one or more tests
  polytrix help [COMMAND]                   # Describe available commands or one specific command
  polytrix list [(all|<REGEX>)] [opts]      # List all tests
  polytrix test [all|<REGEX>)] [opts]       # Test one or more tests
  polytrix verify [(all|<REGEX>)] [opts]    # Verify the execution results of one or more tests
  polytrix version                          # Print Polytrix's version information

The polytrix.yml file

The Polytrix test suites and the implementors that should be tested are defined in a polytrix.yml file. A basic file looks like this:

polytrix:
  implementors:
    ruby_samples:
      language: ruby
      bootstrap_cmd: bundle install
      exec_cmd: bundle exec ruby
      git:
        repo: https://github.com/polytrix/ruby_samples
    python_samples:
      language: python
      bootstrap_script: scripts/pip_install
      exec_script: scripts/pip_install
      git:
        repo: https://github.com/polytrix/python_samples
    java_samples:
      language: java
      # The default is to look for scripts/bootstrap and scripts/exec
      # or try to use the default bootstrap/exec behavior for the language
      git:
        repo: 'https://github.com/polytrix/java_samples'

  suites:
    katas:
      code_samples:
        hello_world:
          - validate:
              stdout: Hello, World!
        quine:
          - validate:
              quine_validator: true
    utilities:
      code_samples:
        word_count:
          - name: short text
            input: |
              I am a word count utility
            validate:
              stdout: 6 words
          - name: small file
            input_file: fixtures/small_file.txt
            validate:
              stdout: 10 words
          - name: large file
            input_file: fixtures/large_file.txt
            validate:
              stdout: 1000 words

Defining Implementors

Polytrix tests scenarios across one or more implementors. The implementors may each be in a different language, but you can also have multiple implementors for the same langauge. The structure used to define implementors within polytrix.yml is described below.

Implementor Object

Field Name	Type	Description
name	`string`	The name of the implementor. (If omitted, this attribute’s value defaults to the key name associated with this object.)
directory	`string`	The location of the implementor project containing code samples.
language	`string`	The primary programming language of the code samples in this implementor.
git	Git Object	Defines how to clone the project via Git.
scripts	Scripts Object	Defines scripts to bootstrap, compile, and execute the samples.

Git Object

Field Name	Type	Description
repo	`string`	Required. The git repo to clone.
branch	`string`	The git branch to clone.
to	`string`	The local directory to clone into.

Scripts Object

Field Name	Type	Description
directory	`directory`	The directory that contains the script files. Default: `scripts/`.
bootstrap_script	`file`	The file to execute to bootstrap the samples. Default: `bootstrap`
compile_script	`file`	The file to execute to compile the samples. Default: `compile`
exec_script	`file`	The file to use as an execute wrapper script for running samples. Default: `exec`
bootstrap_cmd	`string`	An inline script to execute instead of executing bootstrap_file.
compile_cmd	`string`	An inline script to execute instead of executing bootstrap_file.
exec_cmd	`string`	An inline script to use as an execution wrapper, instead of searching for a `exec` script.

Example Implementor Definition

implementors:
  ruby_samples:
    directory: samples/ruby
    language: ruby
    git:
      repo: https://github.com/polytrix/ruby_samples
      branch: master
      to: samples/ruby
    scripts:
      bootstrap_cmd: bundle install
      exec_cmd: bundle exec ruby "$@"

Attaching Code Samples

Polytrix tests are divided into suites, code samples, and scenarios. Each tuple (suite, code sample, scenario) is a test.

The structure used to define tests is:

Suite Object

Field Name	Type	Description
name	`string`	The name of the suite. (If omitted, this attribute’s value defaults to hash key associated with this object.)
input	`string`	The content to send to the code sample as standard input.
input_file	`file`	Reads the specified file and sets input.
code_samples	Hash of Code Samples	Defines the code samples to test

Code Sample Object

The code sample object determines which scenario to runwhich determines which code sample will be run. It can also override the default values set in the Suite Object.

If the Scenario Object is a list, it will be automatically split into distinct but related scenarios. This is useful for testing the same code sample with a variety of different input values.

Field Name	Type	Description
name	`string`	The name of the suite. (If omitted, this attribute’s value defaults to hash key associated with this object.)
scenarios	List of Scenario Objects	Defines the scenarios to test for the code sample. Default: An scenario named default with no input.

Scenario Object

Field Name	Type	Description
name	`string`	Required. The scenario name.
input	`string`	The content to send to the code sample as standard input.
input_file	`file`	Reads the specified file and sets input.
language	Hash of Scenario Objects	Defines the scenarios within the suite

Example Suite Definition

  suites:
    katas:
      scenarios:
        hello_world:
          validate:
            stdout: Hello, World!
        quine:
          validate:
            quine_validator: true
    utilities:
      scenarios:
        word_count:
          - name: short text
            input: |
              I am a word count utility
            validate:
              stdout: 6 words
          - name: small file
            input_file: fixtures/small_file.txt
            validate:
              stdout: 10 words
          - name: large file
            input_file: fixtures/large_file.txt
            validate:
              stdout: 1000 words

Plugins

TBD

Custom Validators

TBD

Usage

Cloning implementors

Bootstrapping implementors

Executing scenarios

Verifying scenario results

Full test

</code>