Toys User Guide

Toys is a configurable command line tool. Write commands in Ruby using a simple DSL, and Toys will provide the command line executable and take care of all the details such as argument parsing, online help, and error reporting.

Toys is designed for software developers, IT professionals, and other power users who want to write and organize scripts to automate their workflows. It can also be used as a replacement for Rake, providing a more natural command line interface for your project's build tasks.

Unlike most command line frameworks, Toys is not primarily designed to help you build and ship a custom command line executable written in Ruby. Rather, it provides a single executable called toys whose functionality you can define by writing configuration files. (You can, however, build your own custom command line executable using the related toys-core library.)

If this is your first time using Toys, we recommend starting with the README, which includes a tutorial that introduces how to install Toys, write and execute tools, and even use Toys to replace Rake. The tutorial will likely give you enough information to start using Toys effectively.

This user's guide is also structured like an extended tutorial, but it is much longer and covers all the features of Toys in much more depth. Read it when you're ready to unlock all the capabilities of Toys to create sophisticated command line tools.

Conceptual overview

Toys is a command line framework. It provides an executable called toys with basic functions such as argument parsing and online help. You provide the actual behavior of the Toys executable by writing Toys files.

Toys is a multi-command executable. You can define any number of commands, called tools, which can be invoked by passing the tool name as an argument to the toys executable. Tools are arranged in a hierarchy; you can define namespaces that have subtools.

Tools can recognize command line arguments in the form of flags and positional arguments. Flags can optionally take values, while positional arguments can be required or optional. Flags can be organized into flag groups that support marking flags or flag combinations as required.

The configuration of a tool can include descriptions, for the tool itself, and for each command line argument. These descriptions are displayed in the tool's online help screen. Descriptions come in long and short forms, which appear in different styles of help.

Toys searches for tools in specifically-named Toys files and Toys directories. It searches for these in the current directory, in parent directories, and in the Toys search path.

Toys provides various features to help you write tools. This includes providing a logger for each tool, mixins that provide common functions a tool can call (such as to control subprocesses and style output), and templates which are prefabricated tools that you can configure for your needs. It also integrates with testing frameworks, making it easy to write and run tests.

Finally, Toys provides useful built-in behavior, including automatically providing flags to display help screens and set verbosity. It also includes a built-in set of system tools that let you inspect and configure the Toys system itself.

The Toys command line

In this section, you will learn how Toys parses its command line, identifies a tool to run, and interprets flags and other command line arguments.

The general form of the toys command line is:

toys [TOOL...] [FLAGS...] [ARGS...]

Tools

The tool name consists of all the command line arguments until the first argument that begins with a hyphen (which is interpreted as a flag), until no tool with that name exists (in which case the argument is treated as the first positional argument), or until there are no more arguments.

For example, in the following command:

       |----TOOL----|
$ toys system version

The tool name is system version. Notice that the tool name can have multiple words. Tools are arranged hierarchically. In this case, system is a namespace for tools related to the Toys system, and version is one of its subtools. It prints the current Toys version.

The words in a tool name can be delimited with spaces as shown above, or alternately periods or colons. The following commands also invoke the tool system version:

$ toys system.version
$ toys system:version

In the following command:

       |TOOL| |ARG|
$ toys system frodo

There is no subtool frodo under the system namespace, so Toys works backward until it finds an existing tool. In this case, the system namespace itself does exist, so Toys runs it as the tool, and passes it frodo as an argument.

Namespaces such as system are themselves tools and can be executed like any other tool. In the above case, it takes the argument frodo, determines it has no subtool of that name, and prints an error message. More commonly, though, you might execute a namespace without arguments:

$ toys system

This displays the online help screen for the system namespace, which includes a list of all its subtools and what they do.

It is also legitimate for the tool name to be empty. This invokes the root tool, the toplevel namespace:

$ toys

Like any namespace, invoking the root tool displays its help screen, including showing the list of all its subtools.

One last example:

$ toys frodo

If there is no tool called frodo in the toplevel namespace, then once again, frodo is interpreted as an argument to the root tool. The root tool responds by printing an error message that the frodo tool does not exist.

Flags

Flags are generally arguments that begin with a hyphen, and are used to set options for a tool.

Each tool recognizes a specific set of flags. If you pass an unknown flag to a tool, the tool will generally display an error message.

Toys follows the typical unix conventions for flags, specifically those covered by Ruby's OptionParser library. You can provide short (single-character) flags with a single hyphen, or long flags with a double hyphen. Some flags can also take values. Following are a few examples.

Here we pass a single short flag (for verbose output).

$ toys system -v

Here we pass multiple long flags (for verbose output and recursive subtool search).

$ toys system --verbose --recursive

You can combine short flags. The following passes both the -v and -r flags (i.e. it has the same effect as the previous example.)

$ toys system -vr

Long flags can be abbreviated, as long as the abbreviation is not ambiguous. For example, there is only one flag (--recursive) beginning with the string --rec, so you can use the shortened form.

$ toys --rec

However, there are two flags (--version and --verbose) beginning with --ver, so it cannot be used as an abbreviation. This will cause an error:

$ toys --ver

Some flags take values. The root tool supports the --search flag to search for tools that have the given keyword.

$ toys --search=build
$ toys --search build

The short form of the search flag -s also takes a value.

$ toys -s build
$ toys -sbuild

If a double hyphen -- appears by itself in the arguments, it disables flag parsing from that point. Any further arguments are treated as positional arguments, even if they begin with hyphens. For example:

       |--FLAG--|   |---ARG---|
$ toys --verbose -- --recursive

That will cause --recursive to be treated as a positional argument. (In this case, as we saw earlier, the root tool will respond by printing an error message that no tool named --recursive exists.)

Note that a single hyphen by itself - is not considered a flag, nor does it disable flag parsing. It is treated as a normal positional argument.

Standard flags

For the most part, each tool specifies which flags and arguments it recognizes. However, Toys adds a few standard flags globally to every tool. (It is possible for individual tools to override these flags, but most tools should support them.) These standard flags include:

--help (also -?) which displays the full help screen for the tool.
--usage which displays a shorter usage screen for the tool.
--verbose (also -v) which increases the verbosity. This affects the tool's logging display, increasing the number of log levels shown. This flag can be issued multiple times.
--quiet (also -q) which decreases the verbosity. This affects the tool's logging display, decreasing the number of log levels shown. This flag can also be issued multiple times.

Namespace tools (tools that have subtools but no explicit functionality of their own) always behave as though --help is invoked. (They do recognize the flag, but it has no additional effect.) Namespaces also support the following additional flags:

--all which displays all subtools, including hidden subtools and namespaces.
--no-recursive which displays only immediate subtools, instead of the default behavior of showing all subtools recursively.
--search=TERM which displays only subtools whose name or description contain the specified search term.
--tools which displays just the list of subtools rather than the entire help screen.

Finally, the root tool also supports:

--version which displays the current Toys version.

Positional arguments

Any arguments not recognized as flags or flag arguments, are interpreted as positional arguments. Positional arguments are recognized in order and can be required or optional.

Each tool recognizes a specific set of positional arguments. If you do not pass a value for a required argument, or you pass too many arguments, the tool will generally display an error message.

For example, the built-in do tool runs multiple tools in sequence. It recognizes any number of positional arguments. Those arguments specify which tools to run and what arguments to pass to them. If, for example, you had a build tool and a test tool, you could run them in sequence with:

          |---ARGS---|
$ toys do build , test

The three arguments build and , and test are positional arguments to the do tool. (The do tool uses , to delimit the tools that it should run.)

Most tools allow flags and positional arguments to be interspersed. A flag will be recognized even if it appears after some of the positional arguments.

However, this approach would not work for the do tool because its common case is to pass flags down to the steps it runs. (That is, do wants most arguments to be treated as positional even if they look like flags.) So do stops recognizing flags once it encounters its first positional argument. That is, you could do this:

          |------------ARGS-----------|
$ toys do build --staging , test --help

Each tool can choose which behavior it will support, whether or not to enforce flags before positional args.

You can also, of course, stop recognizing flags on the command line by passing -- as an argument.

Tab completion

If you are using the Bash shell, Toys provides custom tab completion. See this section for instructions on installing tab completion.

Toys will complete tool and subtool names, flags, values passed to flags, and positional argument values, and it will respect the current context. For example, if you type:

$ toys <TAB><TAB>

The tab completion will show you a list of reasonable things that could appear next, including the defined tool names (such as system and do) as well as all the flags supported by the root tool (such as --help and -v). And of course, if you start typing something, tab completion will limit the display to matching completions. The following displays only flags, i.e. completions that begin with a hyphen:

$ toys -<TAB><TAB>

And if you type the following:

$ toys sys<TAB>

It is likely only one tool name starts with sys, so completion will automatically type the rest of system for you.

The tab completion for Toys also supports values passed to flags and positional args. As we shall see later, when you define a flag or a positional argument, you can specify how completions are computed.

Note: Because of the highly dynamic nature of Toys in which tools, flags, and arguments can be highly customized, the completion implementation actually requires executing Toys so it can analyze your tool configurations. This unfortunately means paying some upfront latency as the Ruby interpreter starts up. So you can expect a slight pause when evaluating tab completion for Toys, at least in comparison with most other tab completions.

Defining tools

So far we've been experimenting only with the built-in tools provided by Toys. In this section, you will learn how to define tools by writing a Toys file. We will cover how to write tools, including specifying the functionality of the tool, the flags and arguments it takes, and how its description appears in the help screen.

Basic Toys syntax

A file named .toys.rb (note the leading period) in the current working directory is called a Toys file. It defines tools available in that directory and its subdirectories.

The format of a Toys file is a Ruby DSL that includes directives, methods, and nested blocks. The actual DSL is specified in the Toys::DSL::Tool class.

To create a tool, write a tool block, giving the tool a name. Within the block, use directives to set the properties of the tool, including descriptions and the flags and arguments recognized by the tool. The actual functionality of the tool is set by defining a run method.

Let's start with an example:

tool "greet" do
  desc "Print a friendly greeting."
  long_desc "Prints a friendly greeting. You can customize whom to" \
              " greet, and how friendly it should be.",
            "",
            "Example:",
            ["    toys greet --shout ruby"]

  optional_arg :whom, default: "world", desc: "Whom to greet."
  flag :shout, "-s", "--shout", desc: "Greet loudly."

  def run
    greeting = "Hello, #{whom}!"
    greeting = greeting.upcase if shout
    puts greeting
  end
end

Its results should be mostly self-evident. But let's unpack a few details.

Tool descriptions

Each tool can have a short description and/or a long description. The short description is a generally a single string that is displayed with the tool name, at the top of its help page or in a subtool list. The long description generally includes multiple strings, which are displayed in multiple lines in the "description" section of the tool's help page. Long descriptions can include blank lines to separate paragraphs visually.

By default, each description string/line is word-wrapped when displayed. In the long description example above, the first line is a bit longer than 80 characters, and may be word-wrapped if displayed on an 80-character terminal.

If you need to control the wrapping behavior, pass an array of strings for that line. Each array element will be considered a unit for wrapping purposes, and will not be split. The example command in the long description above illustrates how to prevent a line from being word-wrapped. This is also a useful technique for preserving spaces and indentation.

For more details, see the reference documentation for Toys::DSL::Tool#desc and Toys::DSL::Tool#long_desc.

Positional arguments

Tools can recognize any number of positional arguments. Each argument must have a name, which is a key that the tool can use to obtain the argument's value at execution time. Arguments can also have various properties controlling how values are validated and expressed.

The above example uses the directive Toys::DSL::Tool#optional_arg to declare an optional argument named :whom. If the argument is provided on the command line e.g.

$ toys greet ruby
Hello, ruby!

Then the option :whom is set to the string "ruby". Otherwise, if the argument is omitted, e.g.

$ toys greet
Hello, world!

Then the option :whom is set to the default value "world".

If the option name is symbol representing a valid method name (that doesn't collide with a method already present), Toys will provide a method that you can use to retrieve the value. In the above example, we retrieve the value for the option :whom by calling the method whom. If the option name cannot be made into a method, you can retrieve the value by calling Toys::Context#get.

An argument can also be required, which means it must be provided on the command line; otherwise the tool will report a usage error. You can declare a required argument using the directive Toys::DSL::Tool#required_arg.

Parsing required and optional arguments

When command line arguments are parsed, the required arguments are matched first, in order, followed by the optional arguments. For example:

tool "args-demo" do
  optional_arg :arg2
  required_arg :arg1

  def run
    puts "Options data is #{options.inspect}"
  end
end

If a user runs

$ toys args-demo foo
Options data is {arg1: "foo", arg2: nil}

Then the required argument :arg1 will be set to "foo", and the optional argument :arg2 will not be set (i.e. it will remain nil).

If the user runs:

$ toys args-demo foo bar
Options data is {arg1: "foo", arg2: "bar"}

Then :arg1 is set to "foo", and :arg2 is set to "bar".

Running the following:

$ toys args-demo

Will produce a usage error, because no value is set for the required argument :arg1. Similarly, running:

$ toys args-demo foo bar baz

Will also produce an error, since the tool does not define an argument to match "baz".

Optional arguments can declare a default value to be used if the argument is not provided on the command line. For example:

tool "args-demo" do
  required_arg :arg1
  optional_arg :arg2, default: "the-default"

  def run
    puts "Options data is #{options.inspect}"
  end
end

Now running the following:

$ toys args-demo foo
Options data is {arg1: "foo", arg2: "the-default"}

Will set the required argument to "foo" as usual, and the optional argument, because it is not provided, will default to "the-default" instead of nil.

Remaining arguments

Normally, unmatched arguments will result in an error message. However, you can provide an "argument" to match all remaining unmatched arguments at the end, using the directive Toys::DSL::Tool#remaining_args. For example:

tool "args-demo" do
  required_arg :arg1
  optional_arg :arg2
  remaining_args :arg3

  def run
    puts "Options data is #{options.inspect}"
  end
end

Now, we can see how the remaining arguments (if any) are collected by :arg3:

$ toys args-demo foo bar baz qux
Options data is {arg1: "foo", arg2: "bar", arg3: ["baz", "qux"]}

$ toys args-demo foo
Options data is {arg1: "foo", arg2: nil, arg3: []}

Tools can include any number of required_arg and optional_arg directives, declaring any number of required and optional arguments. But tools can have at most only one remaining_args directive.

Descriptions and the args DSL

Positional arguments can also have short and long descriptions, which are displayed in online help. Set descriptions via the desc: and long_desc: arguments to the argument directive. The desc: argument takes a single string description, while the long_desc: argument takes an array of strings. Here is an example:

required_arg :arg,
             desc: "This is a short description for the arg",
             long_desc: ["Long descriptions can have multiple lines.",
                         "This is the second line."]

See the above section on Descriptions for more information on how descriptions are rendered and word wrapped.

Because long descriptions may be unwieldly to write as a hash argument in this way, Toys provides an alternate syntax for defining arguments using a block.

required_arg :arg do
  desc "This is a short description for the arg"
  long_desc "Long desc can be set as multiple lines together,",
            "like this second line."
  long_desc "Or you can call long_desc again to add more lines."
end

For detailed info on configuring an argument using a block, see the Toys::DSL::PositionalArg class.

Argument acceptors

Finally, positional arguments can use acceptors to define how to validate arguments and convert them to Ruby objects for your tool to consume. By default, Toys will accept any argument string, and expose it to your tool as a raw string. However, you can provide an acceptor to change this behavior.

Acceptors are part of the OptionParser interface, and are described under the type coercion section. For example, you can provide the Integer class as an acceptor, which will validate that the argument is a well-formed integer, and convert it to an integer during parsing:

tool "acceptor-demo" do
  required_arg :age, accept: Integer
  def run
    puts "Next year I will be #{age + 1}"  # Age is an integer
  end
end

If you pass a non-integer for this argument, Toys will report a usage error.

You can use any of the ready-to-use coercion types provided by OptionParser, including the special types such as OptionParser::DecimalInteger and OptionParser::OctalInteger.

You can also create custom acceptors. See the section below on Custom Acceptors for more information.

Argument completions

Shell tab completion supports positional arguments, and arguments can be configured to present a set of completion candidates for themselves.

By default, an argument does not provide any completions for itself. To change that, set the completion option. Currently there are three ways to set the completion:

Provide a static set of possible values, as an array of strings.
Specify that values should be paths in the file system by setting the symbol :file_system.
Provide a Proc that returns possible values.

The following are two example arguments, one that supports a static set of completions and the other that supports file paths.

required_arg :language, complete: ["ruby", "elixir", "rust"]
required_arg :path, complete: :file_system

Completions are somewhat related to acceptors, and it is a common pattern to set both in concert. But they perform distinct functions. Acceptors affect argument parsing, whereas completions affect tab completion in the shell.

Flags

Tools can also recognize flags on the command line. In our "greet" example, we declared a flag named :shout:

flag :shout, "-s", "--shout", desc: "Greet loudly."

Like a positional argument, a flag sets an option based on the command line arguments passed to the tool. In the case above, the :shout option is set to true if either -s or --shout is provided on the command line; otherwise it remains falsy. The two flags -s and --shout are effectively synonyms and have the same effect. A flag declaration can include any number of synonyms.

As with arguments, Toys can provide a method that you can call to retrieve the option value set by a flag. This method is provided if the flag name is a symbol representing a valid method name that does not collide with a method already present. In our example, Toys provides a method called shout that returns either true or false. If the option name cannot be made into a method, you can retrieve the value by calling Toys::Context#get.

Flag types

Toys recognizes the same syntax used by the standard OptionParser library. This means you can also declare a flag that can be set either to true or false:

flag :shout, "--[no-]shout"

You can declare that a short or long flag takes a value:

flag :whom, "--whom=VALUE"
flag :whom, "--whom VALUE"
flag :whom, "-wVALUE"
flag :whom, "-w VALUE"

You can also declare the value to be optional:

flag :whom, "--whom[=VALUE]"
flag :whom, "--whom [VALUE]"
flag :whom, "-w[VALUE]"
flag :whom, "-w [VALUE]"

Note that if you define multiple flags together, they will all be coerced to the same "type". That is, if one takes a value, they all will implicitly take a value. (This is the same behavior as OptionParser.) In this example:

flag :whom, "-w", "--whom=VALUE"

The -w flag will also implicitly take a value, because it is defined as a synonym of another flag that takes a value.

Note also that Toys will raise an error if those flags are incompatible. For example:

flag :whom, "-w[VALUE]", "--whom=VALUE"

Raises an error because one flag's value is optional while the other is required. (Again, this is consistent with OptionParser's behavior.)

Inferred flags

If you do not provide any actual flags, Toys will attempt to infer one from the name of the option. A one-character name will yield a short flag, and a longer name a long flag. Hence, the following two definitions are equivalent:

flag :shout
flag :shout, "--shout"

And the following two are equivalent:

flag :S
flag :S, "-S"

Inferred flags will convert underscores to hyphens. So the following two definitions are also equivalent:

flag :call_out
flag :call_out, "--call-out"

Handling optional values

There are some subtleties in how the Ruby OptionParser library treats flags with optional values. Although Toys does not use OptionParser interally, it does, for the most part, replicate OptionParser's behavior. It is thus important to understand that behavior if you use optional values.

First, if a flag has an optional value that is not provided on the command line, then the option is set to true, as if it were a normal boolean flag that didn't take a value. Consider this example:

tool "flags-demo" do
  flag :output, "--output [DIRECTORY]", default: "."
  def run
    puts "output is #{output.inspect}"
  end
end

If a user executes this without passing the --output flag, the default will be printed as we expect.

$ toys flags-demo
output is "."

If a user executes this and provides a value for --output, it will show up:

$ toys flags-demo --output /etc
output is "/etc"

If a user provides --output but omits the value, it displays true:

$ toys flags-demo --output
output is true

Second, if the following argument looks like a flag (i.e. it begins with a hyphen), it is not treated as an optional value. In this example, the argument --verbose is not treated as the value of --output but as a separate flag. (If --output had a required value, then --verbose would have been treated as the value.)

$ toys flags-demo --output --verbose
output is true

Finally, there is an important difference between the syntax "--output [DIRECTORY]" and "--output=[DIRECTORY]". In the former case, the following argument (as long as it doesn't look like a flag) will be treated as the value. In the latter case, however, the following argument is never treated as the value. In that latter case, you must use the equals sign syntax to provide a value.

To illustrate, consider two flags with optional values, one using space and the other using equals.

tool "flags-demo-space" do
  flag :output, "--output [DIRECTORY]", default: "."
  set_remaining_args :remaining
  def run
    puts "output is #{output.inspect}"
  end
end
tool "flags-demo-equals" do
  flag :output, "--output=[DIRECTORY]", default: "."
  set_remaining_args :remaining
  def run
    puts "output is #{output.inspect}"
  end
end

Here is the behavior:

$ toys flags-demo-space --output=/etc
output is "/etc"
$ toys flags-demo-space --output /etc
output is "/etc"
$ toys flags-demo-equals --output=/etc
output is "/etc"
$ toys flags-demo-equals --output /etc
output is true

Flag acceptors

Flags can use acceptors to define how to validate values and convert them to Ruby objects for your tool to consume. By default, Toys will accept a flag value string in any form, and expose it to your tool as a raw string. However, you can provide an acceptor to change this behavior.

tool "flags-demo" do
  flag :age, accept: Integer
  def run
    puts "Next year I will be #{age + 1}"  # Age is an integer
  end
end

If you pass a non-integer for this flag value, Toys will report a usage error.

You can use any of the ready-to-use coercion types provided by OptionParser, including the special types such as OptionParser::DecimalInteger and OptionParser::OctalInteger.

You can also create custom acceptors. See the section below on Custom Acceptors for more information.

Defaults and handlers

Flags are usually optional; a flag can appear in a command line zero, one, or any number of times.

If a flag is not passed in the command line arguments for a tool, by default its corresponding option value will be nil. You can change this by providing a default value for a flag:

flag :age, accept: Integer, default: 21

If you pass a flag multiple times on the command line, by default the last appearance of the flag will take effect. That is, suppose you define this flag:

flag :shout, "--[no-]shout"

Now if you pass --shout --no-shout, then the value of the :shout option will be false, i.e. the last value set on the command line. This is because a flag normally sets its option value, replacing any previously set value.

You can, however, change this behavior by providing a handler. A handler is a Ruby Proc that defines what a flag does to its option value. It takes two arguments, the new value given, and the previously set value (which might be the default value if this is the first appearance of the flag), and returns the new value that should be set.

Effectively, the default behavior (setting the value and ignoring the previous value) is equivalent to the following handler:

flag :shout, "--[no-]shout", handler: proc { | val, _prev| val }

Toys gives the default handler the special name :set. So the above is also equivalent to:

flag :shout, "--[no-]shout", handler: :set

The --verbose flag, provided automatically by Toys for most tools, shows an example of an alternate handler. Verbosity is represented by an integer value, defaulting to 0. The --verbose flag can appear any number of times, and each appearance increases the verbosity. Its implementation is internal to Toys, but looks something like this:

flag Toys::Context::Key::VERBOSITY, "-v", "--verbose",
     default: 0,
     handler: proc { |_val, prev| prev + 1 }

Similarly, the "--quiet" flag, which decreases the verbosity, is implemented like this:

flag Toys::Context::Key::VERBOSITY, "-q", "--quiet",
     default: 0,
     handler: proc { |_val, prev| prev - 1 }

Note that both flags affect the same option name, Toys::Context::Key::VERBOSITY. The first increments it each time it appears, and the second decrements it. A tool can query this option and get an integer telling the requested verbosity level, as you will see below.

Toys provides a few built-in handlers that can be specified by name. We already discussed the default handler that can be specified by its name :set or by simply omitting the handler: option. Another named handler is :push. This handler is intended for flags that take values and can be provided more than once. The final value is then an array of values.

In the following example, an invocation can provide any number of --include flags, and the :include option will be set to an array of the given paths.

flag :include, "-I", "--include PATH", default: [], handler: :push

The :push handler is equivalent to proc { |val, array| array.nil? ? [val] : array << val }.

Descriptions and the flags DSL

Flags can also have short and long descriptions, which are displayed in online help. Set descriptions via the desc: and long_desc: arguments to the flag directive. The desc: argument takes a single string description, while the long_desc: argument takes an array of strings. Here is an example:

flag :my_flag, "--my-flag",
     desc: "This is a short description for the arg",
     long_desc: ["Long descriptions can have multiple lines.",
                 "This is the second line."]

See the above section on Descriptions for more information on how descriptions are rendered and word wrapped.

Because long descriptions may be unwieldly to write as a hash argument in this way, Toys provides an alternate syntax for defining flags using a block.

flag :my_flag do
  flags "--my-flag"
  desc "This is a short description for the flag"
  long_desc "Long desc can be set as multiple lines together,",
            "like this second line."
  long_desc "Or you can call long_desc again to add more lines."
end

For detailed info on configuring an flag using a block, see the Toys::DSL::Flag class.

Flag completions

Shell tab completion supports flag values, and flags can be configured to present a set of completion candidates for themselves.

By default, a flag does not provide any completions for itself. To change that, set the completion option. Currently there are three ways to set the completion:

Provide a static set of possible values, as an array of strings.
Specify that values should be paths in the file system by setting the symbol :file_system.
Provide a Proc that returns possible values.

The following are two example flags, one that supports a static set of completions and the other that supports file paths.

flag :language, "--lang=VAL", complete_values: ["ruby", "elixir", "rust"]
flag :path, "--path=VAL", complete_values: :file_system

Completions are somewhat related to acceptors, and it is a common pattern to set both in concert. But they perform distinct functions. Acceptors affect option parsing, whereas completions affect tab completion in the shell.

Flag groups

Flags can be organized into groups. This serves two functions:

Grouping related flags in the help and usage screens
Implementing required flags and other constraints

To create a simple flag group, use the flag_group directive, and provide a block that defines the group's flags. You can also provide a group description that appears in the help screen.

flag_group desc: "Debug flags" do
  flag :debug, "-D", desc: "Enable debugger"
  flag :warnings, "-W[VAL]", desc: "Enable warnings"
end

Flag groups can have a "type" that specifies constraints on the flags contained in the group. Flags in a simple group like the above are ordinary optional flags. However, you can specify that flags in the group are required using the all_required directive:

all_required desc: "Login flags (all required)" do
  flag :username, "--username=VAL", desc: "Set the username (required)"
  flag :password, "--password=VAL", desc: "Set the password (required)"
end

If the tool is invoked without providing each of these required flags, it will display an option parsing error.

The all_required directive is actually just shorthand for passing type: :required to the flag_group directive. So the above is the same as:

flag_group type: :required, desc: "Login flags (all required)" do
  flag :username, "--username=VAL", desc: "Set the username (required)"
  flag :password, "--password=VAL", desc: "Set the password (required)"
end

The following are the supported types of flag groups:

The :required type, which you can create using the directive all_required. All flags from the group are required and must be provided on the command line to avoid an error.
The :exactly_one type, which you can create using the directive exactly_one_required. Exactly one, and no more than one, flag from the group must be provided on the command line to avoid an error.
The :at_most_one type, which you can create using the directive at_most_one_required. At most one flag from the group must be provided on the command line to avoid an error.
The :at_least_one type, which you can create using the directive at_least_one_required. At least one flag from the group must be provided on the command line to avoid an error.
The :optional type is the default created using the directive flag_group when no type is specified. Flags in the group are ordinary optional flags.

Flag group types are useful for a variety of tools. For example, suppose you are writing a tool that deploys an app to one of several different kinds of targets---say, a server, a VM, or a container. You could provide this configuration for your tool with a flag group:

tool "deploy" do
  exactly_one_required desc: "Deployment targets" do
    flag :server, "--server=IP_ADDR"
    flag :vm, "--vm=VM_ID"
    flag :container, "--container=CONTAINER_ID"
  end

  def run
    # Now exactly one of server, vm, or container will be set. The other
    # two options will be their default value, nil.
  end
end

Tool execution basics

When you run a tool from the command line, Toys will build the tool based on its definition in a Toys file, and then it will attempt to execute it by calling the run method. Normally, you should define this method in each of your tools.

Note: If you do not define the run method for a tool, Toys provides a default implementation that displays the tool's help screen. This is typically used for namespaces, as we shall see below. Most tools, however, should define run.

Let's revisit the "greet" example we covered earlier.

tool "greet" do
  optional_arg :whom, default: "world"
  flag :shout, "-s", "--shout"

  def run
    greeting = "Hello, #{whom}!"
    greeting = greeting.upcase if shout
    puts greeting
  end
end

Note that you can produce output or interact with the console using the normal Ruby $stdout, $stderr, and $stdin streams.

Note also how the run method can access values that were assigned by flags or positional arguments by just calling a method with that flag or argument name. When you declare a flag or argument, if the option name is a symbol that is a valid Ruby method name, Toys will provide a method that you can call to get the value. In the above example, whom and shout are such methods.

If you create a flag or argument whose option name is not a symbol or is not a valid method name, you can still get the value by calling Toys::Context#get. For example:

tool "greet" do
  # The name "whom-to-greet" is not a valid method name.
  optional_arg "whom-to-greet", default: "world"
  flag :shout, "-s", "--shout"

  def run
    # We can access the "whom-to-greet" option using the "get" method.
    greeting = "Hello, #{get('whom-to-greet')}!"
    greeting = greeting.upcase if shout
    puts greeting
  end
end

If a tool's run method finishes normally, Toys will exit with a result code of 0, indicating success. You can exit immediately and/or provide a nonzero result by calling Toys::Context#exit:

def run
  puts "Exiting with an error..."
  exit(1)
  puts "Will never get here."
end

If your run method raises an exception, Toys will display the exception and exit with a nonzero code.

Finally, you can also define additional methods within the tool. These are available to be called by your run method, and can be used to decompose your tool implementation. Indeed, a tool is actually a class under the hood, and you can define methods as with any other class. Here's a contrived example:

tool "greet-many" do
  # Support any number of arguments on the command line
  remaining_args :whom, default: ["world"]
  flag :shout, "-s", "--shout"

  # You can define helper methods like this.
  def greet(name)
    greeting = "Hello, #{name}!"
    greeting = greeting.upcase if shout
    puts greeting
  end

  def run
    whom.each do |name|
      greet(name)
    end
  end
end

This should be enough to get you started implementing tools. A variety of additional features are available for your tool implementation and will be discussed further below. But first we will cover a few important topics.

Namespaces and subtools

Like many command line frameworks, Toys supports subtools. You can, for example create a tool called "test" that runs your tests for a particular project, but you might also want "test unit" and "test integration" tools to run specific subsets of the test suite. One way to do this, of course, is for the "test" tool to parse "unit" or "integration" as arguments. However, it's often easier to define them as separate tools, subtools of "test".

To define a subtool, create nested tool directives. Here's a simple example:

tool "test" do
  tool "unit" do
    def run
      puts "run unit tests here..."
    end
  end

  tool "integration" do
    def run
      puts "run integration tests here..."
    end
  end
end

You can now invoke them like this:

$ toys test unit
run unit tests here...
$ toys test integration
run integration tests here...

Notice in this case, the parent "test" tool itself has no run method. This is a common pattern: "test" is just a "container" for tools, a way of organizing your tools. In Toys terminology, it is called a namespace. But it is still a tool, and it can still be run:

$ toys test

As discussed earlier, Toys provides a default implementation that displays the help screen, which includes a list of the subtools and their descriptions.

As another example, the "root" tool is also normally a namespace. If you just run Toys with no arguments:

$ toys

The root tool will display the overall help screen for Toys.

Although it is a less common pattern, it is possible for a tool to have its own run method even if it has subtools:

tool "test" do
  def run
    puts "run all tests here..."
  end

  tool "unit" do
    def run
      puts "run only unit tests here..."
    end
  end

  tool "integration" do
    def run
      puts "run only integration tests here..."
    end
  end
end

Now running toys test will run its own implementation.

(Yes, it is even possible to write a run method for the root tool. I don't recommend doing so, because you would lose the root tool's useful default implementation that lists all your tools.)

Toys allows subtools to be nested arbitrarily deep. In practice, however, more than two or three levels of hierarchy can be confusing to use.

Defining subtools using classes

It is also possible to define subtools by subclassing Toys::Tool. The class will support the same DSL as a tool block would. For example, this is what our greet tool would look like as a class:

class Greet < Toys::Tool
  optional_arg :whom, default: "world"
  flag :shout, "-s", "--shout"

  def run
    greeting = "Hello, #{whom}!"
    greeting = greeting.upcase if shout
    puts greeting
  end
end

Defining tools as clases is useful if you want your Toys files to look a bit more like normal Ruby or you want to avoid long blocks. Additionally, they can be useful to scope constants, which otherwise would be visible throughout the entire file, as noted in the section on using constants.

When you define a tool as a class, Toys infers the tool name by converting the class name to "kebab-case". For example, the Greet class above would define a tool called greet. If the class were called GreetMany, the tool would be called greet-many. If you need to override this behavior or set a tool name that is not possible from legal class names, pass the desired tool name to the Toys.Tool method like this:

class Greet < Toys.Tool("_my_greet")
  optional_arg :whom, default: "world"
  flag :shout, "-s", "--shout"

  def run
    greeting = "Hello, #{whom}!"
    greeting = greeting.upcase if shout
    puts greeting
  end
end

You can create subtools by nesting classes:

class Test < Toys::Tool
  class Unit < Toys::Tool
    def run
      puts "run only unit tests here..."
    end
  end

  class Integration < Toys::Tool
    def run
      puts "run only integration tests here..."
    end
  end
end

The above defines test unit and test integration tools as expected.

Finally, you may not nest a class-based tool inside a block-based tool. (This is because Ruby's constant definition rules would put such a class in an unexpected namespace, leading to potential clashes that would be difficult to diagnose.) It is, however, permissible to nest a block-based tool inside a class-based tool.

Hence, this is not allowed, and will result in an exception:

tool "test" do
  # This will raise Toys::ToolDefinitionError...
  class Unit < Toys::Tool
    def run
      puts "run unit tests..."
    end
  end
end

But this is legal:

class Test < Toys::Tool
  tool "unit" do
    def run
      puts "run unit tests..."
    end
  end
end

In general, though, as a best practice, I recommend against mixing the two styles. Define tools either as classes or as blocks, not both.

Understanding Toys files

Toys commands are defined in Toys files. We covered the basic syntax for these files in the above section on defining tools. In this section, we will take a deeper look at what you can do with Toys files.

Toys directories

So far we have been defining tools by writing a Toys file named .toys.rb located in the current working directory. This works great if you have a small number of fairly simple tools, but if you are defining many tools or tools with long or complex implementations, you may find it better to split your tools in separate files. You can have Toys load tools from multiple files by creating a Toys directory.

A Toys directory is a directory called .toys located in the current working directory. (Again, note the leading period.) Ruby files inside a Toys directory (or any of its subdirectories) are loaded when Toys looks for tool definitions. Furthermore, the name of the Ruby file (and indeed its path relative to the Toys directory) determines which tool it defines.

For example, one way to create a "greet" tool, as we saw before, is to write a .toys.rb file in the current directory, and populate it like this:

tool "greet" do
  optional_arg :whom, default: "world"
  def run
    puts "Hello, #{whom}"
  end
end

You could also create the same tool by creating a .toys directory, and then creating a file greet.rb inside that directory.

(current directory)
|
+- .toys/
   |
   +- greet.rb

The contents of greet.rb would be:

optional_arg :whom, default: "world"
def run
  puts "Hello, #{whom}"
end

Notice that we did not use a tool "greet" block here. That is because the name of the file greet.rb already provides a naming context: Toys already knows that we are defining a "greet" tool.

If you do include a tool block inside the greet.rb file, it will create a subtool of greet. In other words, the path to the Ruby file defines a "starting point" for the names of tools defined in that file.

If you create subdirectories inside a Toys directory, their names also contribute to the namespace of created tools. For example, if you create a directory .toys/test and a file unit.rb under that directory, it will create the tool test unit.

(current directory)
|
+- .toys/
   |
   +- greet.rb   <-- defines "greet" (and subtools)
   |
   +- test/
      |
      +- unit.rb   <-- defines "test unit" (and its subtools)

Once again, test unit is the "starting point" for tools defined in the .toys/test/unit.rb file. Declarations and methods at the top level of that file will define the test unit tool. Any tool blocks you add to that file will define subtools of test unit.

Index files

The file name .toys.rb can also be used inside Toys directories and subdirectories. Such files are called index files, and they create tools with the directory as the "starting point" namespace. For example, if you create an index file directly underneath a .toys directory, it will define top level tools (just like a .toys.rb file in the current directory.) An index file located inside .toys/test will define tools with test as the "starting point" namespace.

(current directory)
|
+- .toys/
   |
   +- .toys.rb   <-- index file, defines tools at the top level
   |
   +- greet.rb   <-- defines "greet" (and subtools)
   |
   +- test/
      |
      +- .toys.rb   <-- index file, defines "test" (and its subtools)
      |
      +- unit.rb   <-- defines "test unit" (and its subtools)

Index files give you some flexibility for organizing your tools. For example, if you have a number of subtools of test, including a lot of small tools and one big complex subtool called unit, you might define all the simple tools in the index file .toys/test/.toys.rb, while defining the large test unit tool in the separate file .toys/test/unit.rb.

Toys also loads the index file in a directory before any other files in that directory. This means they are convenient places to define shared code that can be used by all the subtools defined in that directory, as we shall see later in the section on sharing code.

The Toys search path

So far we have seen how to define tools by writing a .toys.rb file in the current directory, or by writing files inside a .toys directory in the current directory. These tools are "scoped" to the current directory. If you move to a different directory, they may not be available.

When Toys runs, it looks for tools in a search path. Specifically:

It looks for a .toys.rb file and/or a .toys directory in the current working directory.
It does the same in the parent directory of the current directory, and then its parent, and so on until it hits either the root of the file system or one of the global directories described in (3).
It looks in a list of global directories, specified in the environment variable TOYS_PATH. This variable can contain a colon-delimited list of directory paths. If the variable is not set, the current user's home directory, and the system configuration directory (/etc on unix systems) are used by default. Toys does not search parents of global directories.

It uses the first implementation that it finds for the requested tool. For example, if the tool greet is defined in the .toys.rb file in the current working directory, and also in the .toys/greet.rb file of the parent directory, it will use the version in the current directory.

This means you could write a default implementation for a tool in your home directory, and override it in the current directory. For example, you could define a tool get-credentials in your home directory that gets credentials you need for most of your projects. But maybe on particular project requires different credentials, so you could define a different get-credentials tool in that project's directory.

While a tool can be overridden when it is defined at different points in the search path, it is not allowed to provide multiple definitions of a tool at the same point in the search path. For example, if you define the greet tool twice in the same .toys.rb file, Toys will report an error. Perhaps less obviously, if you define greet in the .toys.rb file in the current directory, and you also define it in the .toys/greet.rb file in the same current directory, Toys will also report an error, since both are defined at the same point (the current directory) in the search path.

Note that in the search path above, steps (1) and (2) are context-dependent. That is, they may be different depending on what directory you are in. However, step (3) is not context-dependent, and is searched regardless of where you are located. Tools defined here are global, available everywhere.

Stopping search

Though it is uncommon practice, it is possible to stop the search process and prevent Toys from loading tools further down in the search path (e.g. prevent tools from being defined from parent directories or global directories). To do so, use the directive Toys::DSL::Tool#truncate_load_path!. This directive removes all directories further down the search path. It can be used, for example, to disable global tools when you run Toys from the current directory. It can also be useful if you are using Bundler integration to prevent bundle conflicts with parent directories, by disabling tools from parent directories.

The truncate_load_path! directive works only if no tools from further down the search path have been loaded yet. It will raise Toys::ToolDefinitionError if it fails to truncate the load path. In most cases, Toys is very smart about loading tools only when needed, but there are exceptions. To minimize the chance of problems, if you need to use truncate_load_path!, locate it as early as possible in your Toys files, typically at the top of the index file.

Loading remote files

(TODO)

The execution environment

This section describes the context and resources available to your tool when it is running; that is, what you can call from your tool's run method.

Each tool is defined as a class that subclasses Toys::Context. The base class defines helper methods, and provides access to a variety of data and objects relevant to your tool. We have already seen earlier how to use the Toys::Context#get method to retrieve option values, and how to use the Toys::Context#exit method to exit immediately and return an exit code. Now we will cover other resources available to your tool.

Built-in context

In addition to the options set by your tool's flags and command line arguments, a variety of other data and objects are also accessible using Toys::Context#get. For example, you can get the full name of the tool being executed like this:

def run
  puts "Current tool is #{get(TOOL_NAME)}"
end

The TOOL_NAME constant above is a well-known key that corresponds to the full name (as an array of strings) of the running tool. A variety of well-known keys are defined in the Toys::Context::Key module. They include information about the current execution, such as the tool name and the original command line arguments passed to it (before they were parsed). They also include some internal Toys objects, which can be used to do things like write to the logger or look up and call other tools.

Most of the important context also can be accessed from convenience methods. For example, the TOOL_NAME is also available from Toys::Context#tool_name:

def run
  puts "Current tool is #{tool_name}"
end

Let's take a look at a few things your tool can do with the objects you can access from built-in context.

Logging and verbosity

Toys provides a Logger (a simple instance of the Ruby standard library logger that writes to standard error) for your tool to use to report status information. You can access this logger via the LOGGER context key, or Toys::Context#logger. For example:

def run
  logger.warn "Danger Will Robinson!"
end

The current logger level is controlled by the verbosity. Verbosity is an integer context value that you can retrieve using the VERBOSITY context key or Toys::Context#verbosity. The verbosity is set to 0 by default. This corresponds to a logger level of WARN. That is, warnings, errors, and fatals are displayed, while infos and debugs are not. However, as we saw earlier, most tools automatically respond to the --verbose and --quiet flags, (or -v and -q), which increment and decrement the verbosity value, respectively. If you pass -v to a tool, the verbosity is incremented to 1, and the logger level is set to INFO. If you pass -q, the verbosity is decremented to -1, and the logger level is set to ERROR. So by using the provided logger, a tool can easily provide command line based control of the output verbosity.

Running tools from tools

A common operation a tool might want to do is "call" another tool. This can be done via the CLI object, which you can retrieve using the CLI key or Toys::Context#cli. These return the current instance of Toys::CLI, which is the "main" interface to Toys. In particular, it provides the Toys::CLI#run method, which can be used to call another tool:

def run
  status = cli.run("greet", "rubyists", "-v")
  exit(status) unless status.zero?
end

Pass the tool name and arguments as arguments to the run method. It will execute, and return a process status code (i.e. 0 for success, and nonzero for error). Make sure you handle the exit status. For example, in most cases, you should probably exit if the tool you are calling returns a nonzero code.

You can also use the exec mixin described below to run a tool in a separate process. This is particularly useful if you need to capture or manipulate that tool's input or output stream.

Helper methods and mixins

The methods of Toys::Context are not the only methods available for your tool to call. We saw earlier that a tool can define additional methods that you can use as helpers.

You can also include mixins, which are modules that bring in a whole set of helper methods. Include a mixin using the include directive:

tool "greet" do
  include :terminal
  def run
    puts "This is a bold line.", :bold
  end
end

A mixin can be specified by providing a module itself, or by providing a mixin name. In the above example, we used :terminal, which is the name of a built-in mixin that Toys provides. Among other things, it defines a special puts method that lets you include style information such as :bold, which affects the display on ANSI-capable terminals.

For details on the built-in mixins provided by Toys, see the modules under Toys::StandardMixins. We will look at a few examples of the use of these mixins below. Built-in mixins have names that are symbols.

You can also define your own mixins, as we will see in the upcoming section on defining mixins.

Executing subprocesses

Another common operation you might do in a tool is to execute other binaries. For example, you might write a tool that shells out to scp to copy files to a remote server.

Ruby itself provides a few convenient methods for simple execution, such as the Kernel#system method. However, these typically provide limited ability to control or interact with subprocess streams, and you also need to remember to handle the exit status yourself. If you do want to exert more control over subprocesses, you can use Process.spawn, or a higher-level wrapper such as the open3 library.

Another alternative is to use the :exec built-in mixin. This mixin provides convenient methods for the common cases of executing subprocesses and capturing their output, and a powerful block-based interface for controlling streams. The exec mixin also lets you set a useful default option that causes the tool to exit automatically if one of its subprocesses exits abnormally.

The exec mixin provides methods for running several different kinds of subprocesses:

Normal processes started by the operating system.
Another Ruby process.
A shell script.
Another tool run in a separate (forked) process.
A block run in a separate (forked) process.

For more information, see the Toys::StandardMixins::Exec mixin module and the underyling library Toys::Utils::Exec.

Formatting output

Interacting with the user is a very common function of a command line tool, and many modern tools include intricately designed and styled output, and terminal effects such as progress bars and spinners. Toys provides several mixins that can help create nicer interfaces.

First, there is :terminal, which provides some basic terminal features such as styled output and simple spinners. For information, see the Toys::StandardMixins::Terminal mixin module and the underyling library Toys::Utils::Terminal.

If you prefer the venerable Highline library interface, Toys provides a mixin called :highline that automatically installs the highline gem (version 2.x) if it is not already available, and makes a highline object available to the tool. For more information, see the Toys::StandardMixins::Highline mixin module.

The :pager mixin provides a convenient interface to pager utilities such as less, which let your users interactively page through long output. For more information, see the Toys::StandardMixins::Pager mixin module and the underlying library Toys::Utils::Pager.

You can also use other third-party gems such as tty. The section below on useful gems provides some examples.

Standard base directories

Command line tools often need access to common user-specific files such as caches and state data. The XDG Base Directory Specification specifies standard locations for these files and standard environment variables that can be used to override them. Toys provides easy access to these paths via the :xdg mixin.

Here's a simple example:

tool "my-tool" do
  include :xdg

  def run
    # Get config file paths, in order from most to least inportant
    config_files = xdg.lookup_config("my-tool/my-config.toml")
    config_files.each do |absolute_path|
      # Read the file...
    end
  end
end

The lookup_config method and many others are documented under the Toys::Utils::XDG class.

As you accumulate additional and more complex tools, you may find that some of your tools need to share some common configuration, data, or logic. You might, for example, have a set of admin scripts that need to do some common authentication. This section describes several techniques for sharing code between tools, and describes the scope of Ruby structures, such as methods, classes, and constants, that you might define in your tools.

Defining mixins

We saw earlier that you can mix a module (with all its methods) into your tool using the include directive. You can specify a module itself, or the name of a built-in mixin such as :exec or :terminal. But you can also define your own mixin using the mixin directive. A mixin defined in a tool can be included in that tool or any of its subtools or their subtools, recursively, so it's a useful way to share code. Here's how that works.

Define a mixin using the mixin directive, and give it a name and a block. The mixin name must be a string. (Symbols are reserved for built-in mixins.) In the block, you can define methods that will be made available to any tool that includes the mixin, in the same way that you can include a Ruby module.

(Note that, unlike full modules, mixins allow only methods to be shared. Mixins do not support constants. See the next section on using constants to learn how Toys handles constants.)

Here's an example. Suppose you had common setup code that you wanted to share among your testing tools.

tool "test" do
  # Define a mixin, which is just a collection of methods.
  mixin "common_test_code" do
    def setup
      # Do setup here
    end
  end

  tool "unit" do
    # Include the mixin by name
    include "common_test_code"
    def run
      setup  # Mixin methods are made available
      puts "run only unit tests here..."
    end
  end

  tool "integration" do
    include "common_test_code"
    def run
      setup
      puts "run only integration tests here..."
    end
  end
end

A mixin is available to the tool in which it is defined, and any subtools and descendants defined at the same point in the Toys search path, but not from tools defined in a different point in the search path. For example, if you define a mixin in a file located in a .toys directory, it will be visible to descendant tools defined in that same directory, but not in a different .toys directory.

A common technique, for example, would be to define a mixin in the index file in a Toys directory. You can then include it from any subtools defined in other files in that same directory.

Mixin initializers

Sometimes a mixin will want to initialize some state before the tool executes. For example, the :highline mixin creates an instance of Highline during tool initialization. To do so, provide an on_initialize block in the mixin block. The initializer block is called within the context of the tool after arguments are parsed, so it has access to the tool's built-in context and options.

If you provide extra arguments when you include a mixin, those are passed to the initializer block.

For example, suppose the "common_test_code" mixin needs to behave differently depending on the type of tests (unit vs integration). Let's have the subtools pass a value to the mixin's initializer:

tool "test" do
  mixin "common_test_code" do
    # Initialize the mixin, and receive the argument passed to the
    # include directive
    on_initialize do |type|
      # Initializers are called in the context of the tool, and so can
      # affect the tool's state.
      set(:test_type, type)
    end

    def setup
      puts "Setting up #{get(:test_type)}..."
    end
  end

  tool "unit" do
    # Pass an extra argument to include
    include "common_test_code", "unit"
    def run
      setup
      puts "run only unit tests here..."
    end
  end

  tool "integration" do
    include "common_test_code", "integration"
    def run
      setup
      puts "run only integration tests here..."
    end
  end
end

Mixin inclusion hooks

A mixin can also optionally provide directives to run when the mixin is included, by defining an on_include block. (This is functionally similar to defining an included method on a Ruby module.) The on_include block is called within the context of the tool DSL, so it can invoke any DSL directives.

If you provide extra arguments when you include a mixin, those are passed to the inclusion block.

Using constants

You can define and use Ruby constants, i.e. names beginning with a capital letter, in a Toys file. However, they are subject to Ruby's rules regarding constant scope and lookup, which can be confusing, especially in a DSL. Toys tries to simplify those rules and make constant behavior somewhat tractable, but if you do use constants (which includes modules and classes defined in a Toys file), it is important to understand how they work.

Constants in Toys are visible only within the Toys file in which they are defined. They normally behave as though they are defined at the "top level" of the file. Even if you define a constant lexically "inside" a tool or a mixin, the constant does not end up connected to that tool or mixin; it is defined at the file level.

tool "test" do
  tool "unit" do
    # This constant is now usable for the rest of the file
    API_KEY_FOR_TESTING = "12345"
    def run
      # It is visible here
      puts API_KEY_FOR_TESTING
    end
  end

  tool "integration" do
    def run
      # And it is still visible here
      puts API_KEY_FOR_TESTING
    end
  end
end

(Note it is still possible to attach constants to a tool or mixin by defining them with self::. However, this is uncommon Ruby practice and is mildly discouraged.)

Because of this, it is highly recommended that you define constants only at the top level of a Toys file, so it doesn't "look" like it is scoped to something smaller. In particular, do not attempt to define constants in a mixin, unless you scope them with self::.

Modules and classes defined using the module or class keyword, are also constants, and thus follow the same rules. So you could, for example, define a "mixin" module like this:

module CommonTestCode
  include Toys::Mixin
  def setup
    # Do setup here
  end
end

tool "test" do
  tool "unit" do
    # Include the modules as a mixin
    include CommonTestCode
    def run
      setup  # Module methods are made available
      puts "run only unit tests here..."
    end
  end

  tool "integration" do
    include CommonTestCode
    def run
      setup
      puts "run only integration tests here..."
    end
  end
end

The difference between this technique and using the mixin directive we saw earlier, is the scope. The module here is accessed via a constant, and so, like any constant, it is visible only in the same file it is defined in. The mixin directive creates mixins that are visible from all files at the same point in the search path.

Not also, when you define a mixin in this way, you should include Toys::Mixin in the module, as illustrated above. This makes on_initialize available in the module.

Templates

Another way to share code is to expand a template.

A template is a class that inserts a bunch of lines into a Toys file. It is often used to "instantiate" prefabricated tools. For instance, Toys comes with a template called "minitest" that can generate a test tool for you. You instantiate it using the expand directive in your Toys file, like this:

expand :minitest

And it will generate a tool called "test" that runs your test suite.

Most templates generate one or more complete tools. However, it is possible for a template to generate just part of a tool, such as one or more description directives. In general, expanding a template simply adds directives to your Toys file.

Many templates can be configured with options such as the name of the tool to generate, or details of the tool's behavior. This is done by passing additional arguments to the expand directive, such as:

expand :minitest, name: "unit-test", warnings: true

Alternatively, you can provide a block to expand. It will yield the template to your block, letting you modify its properties:

expand :minitest do |t|
  t.name = "unit-test"
  t.warnings = true
end

Toys provides several built-in templates that are useful for project and gem development, including templates that generate build, test, and documentation tools. The :minitest template illustrated above is one of these built-in templates. Like built-in mixins, built-in template names are always symbols. You can read more about them in the next section on using Toys as a Rake replacement.

You can also write your own templates. Here's how...

Defining templates

One way to define a template is to use the template directive. Like the mixin directive, this creates a named template that you can access inside the current tool and any of its subtools. Also, like mixins, your template name must be a string.

Following is a simple template example:

template "greet" do
  def initialize(name: "greet", whom: "world")
    @name = name
    @whom = whom
  end
  attr_accessor :name
  attr_accessor :whom

  on_expand do |template|
    tool template.name do
      desc "A greeting tool generated from a template"

      static :whom, template.whom

      def run
        puts "Hello, #{whom}!"
      end
    end
  end
end

expand "greet"

expand "greet", name: "greet-ruby", whom: "ruby"

Above we created a template called "greet". A template is simply a class. It will typically have a constructor, and methods to access configuration properties. When the template is expanded, the class gets instantiated, and you can set those properties.

Next, a template has an on_expand block. This block contains the Toys file directives that should be generated by the template. The template object is passed to the block, so it can access the template configuration when generating directives. The "greet" template in the above example generates a tool whose name is set by the template's name property.

Notice in the above example, we used the Toys::DSL::Tool#static directive to create a "static" option for template.whom. A "static" option is an option whose value is set directly in code rather than via a flag or argument. We use it here because we can't access the template local variable from within the run method. Static options are a common technique for accessing template configuration properties from inside tool methods.

By convention, it is a good idea for configuration options for your template to be settable both as arguments to the constructor, and as attr_accessor properties. In this way, when you expand the template, options can be provided either as arguments to the expand directive, or in a block passed to the directive by setting properties on the template object.

Template classes

Finally, templates are classes, and you can create a template directly as a class by including the Toys::Template module in your class definition.

class GreetTemplate
  include Toys::Template

  def initialize(name: "greet", whom: "world")
    @name = name
    @whom = whom
  end
  attr_accessor :name
  attr_accessor :whom

  on_expand do |template|
    tool template.name do
      desc "A greeting tool generated from a template"
      static :whom, template.whom
      def run
        puts "Hello, #{whom}!"
      end
    end
  end
end

expand GreetTemplate, name: "greet-ruby", whom: "ruby"

Remember that classes created this way are constants, and so the name GreetTemplate is available only inside the Toys file where it was defined.

You must include Toys::Template if you define a template directly as a class, but you can omit it if you use the template directive to define the template in a block.

Defining templates as classes is also a useful way for third-party gems to provide Toys integration. For example, suppose you are writing a code analysis gem, and you want to make it easy for your users to create a Toys tool that invokes your analysis. Just write a template class in your gem, maybe named MyAnalysis::ToysTemplate. Now, just instruct your users to include the following in their Toys file:

require "my_analysis"
expand MyAnalysis::ToysTemplate

Loading from a lib directory

For more complicated tools, you might want to write normal Ruby modules and classes as helpers. Toys provides a way to write Ruby code outside of its DSL and require the code from your tool, using .lib directories.

To use .lib directories, you must define your tools inside a Toys directory. When a tool is executed, it looks for directories called .lib in the Toys directory, and adds them to the Ruby load path. Your tool can thus call require to load helpers from any Ruby files in a .lib directory.

For example, take the following directory structure:

(current directory)
|
+- .toys/
   |
   +- .lib/   <-- available when a tool is executed
   |  |
   |  +- greeting_helper.rb
   |
   +- greet.rb

The greeting_helper.rb file can contain any Ruby code.

# .toys/.lib/greeting_helper.rb

module GreetingHelper
  def self.make_greeting(whom)
    "Hello, #{whom}!"
  end
end

Now you can require "greeting_helper" in your greet tool.

# .toys/greet.rb

tool "greet" do
  optional_arg :whom, default: "world", desc: "Whom to greet."
  def run
    require "greeting_helper"
    puts GreetingHelper.make_greeting(whom)
  end
end

Note that .lib directories are available only when your tool is being run, not when it is being defined. So any require statements should be located inside your run method.

tool "greet" do
  # Do not try to require the file here. Toys will not find it because
  # the tool is not yet being run.
  # require "greeting_helper"  # ERRORS!

  optional_arg :whom, default: "world", desc: "Whom to greet."
  def run
    # Require a helper file here, so it is loaded during tool execution.
    require "greeting_helper"
    # Now you can use classes defined in the helper
    puts GreetingHelper.make_greeting(whom)
  end
end

If your Toys directory has subdirectories, lib directories will be prioritized by how close they are to the tool being executed. For example:

(current directory)
|
+- .toys/
   |
   +- .lib/   <-- available when any tool defined in this directory
   |  |           is executed
   |  |
   |  +- helper.rb   <-- visible to "greet" but not "test unit"
   |  |
   |  +- helper2.rb   <-- visible to both "greet" and "test unit"
   |
   +- greet.rb
   |
   +- test/
      |
      +- .lib/   <-- available only when tools under "test" are executed
      |  |
      |  +- helper.rb   <-- overrides the other helper.rb when
      |                     "test unit" is executed
      |
      +- unit.rb

Preloading Ruby files

You can also provide Ruby files that are "preloaded" before tools are defined. This is useful if those Ruby files are required by the tool definitions themselves. Like files in the .lib directory, preloaded files can define Ruby classes, modules, and other code. But preloaded files automatically loaded (i.e. you do not require them explicitly) before your tools are defined.

To use preloaded files, you must define your tools inside a Toys directory. Before any tools inside a directory are loaded, any file named .preload.rb in the directory is automatically required. Next, any Ruby files inside a subdirectory called .preload are also automatically required.

For example, take the following directory structure:

(current directory)
|
+- .toys/
   |
   +- .preload.rb   <-- required first
   |
   +- greet.rb   <-- defines "greet" (and subtools)
   |
   +- test/
      |
      +- .preload/
      |  |
      |  +- my_classes.rb  <-- required before unit.rb
      |  |
      |  +- my_modules.rb  <-- also required before unit.rb
      |
      +- unit.rb   <-- defines "test unit" (and its subtools)

Toys will execute

require ".toys/.preload.rb"

first before loading any of the tools in the .toys directory (or any of its subdirectories). Thus, you can define classes used by both the greet and the test unit tool in this file.

Furthermore, Toys will also execute

require ".toys/test/.preload/my_classes.rb"
require ".toys/test/.preload/my_modules.rb"

first before loading any of the tools in the test subdirectory. Thus, any additional classes needed by test unit can be defined in these files.

Either a single .preload.rb file or a .preload directory, or both, can be used. If both are present in the same directory, the .preload.rb file is loaded first before the .preload directory contents.

Testing your tools

Tests play a critical part in the maintainability of any Ruby app. However, it can be difficult to set up tests for command line tools, Rake tasks, and similar code. Toys embraces testability by providing a dedicated place for tests, and a set of helpers that make it easy to test tools and parts of tools.

Writing and running tests

Toys integrates with Minitest to provide a testing framework for your tools. To write tests, you must define your tools inside a Toys directory. Create a directory called .test inside that directory. You can then write Minitest tests in files within that directory with names matching test_*.rb. You can also provide other files that do not match that name pattern, as fixture data or common test tools to load using require_relative. For example:

(current directory)
|
+- .toys/
   |
   +- greet.rb   <-- defines "greet" (and subtools)
   |
   +- .test/
      |
      +- test_greet.rb   <-- includes tests
      |
      +- helper.rb   <-- includes helper code but not tests

Test files must be written against Minitest, and can use either unit test or spec syntax. (RSpec support is not currently available.) The framework will require minitest/autorun for you. Here is trivial example content for the test_greet.rb file:

# test_greet.rb

require_relative "helper"

class MyTest < Minitest::Test
  def test_hello
    assert_equal "hello", MyHelper.hello
  end
end

And the helper.rb file:

# helper.rb

module MyHelper
  def self.hello
    "hello"
  end
end

You can run tests using:

toys system test

The built-in system test tool searches for the closest .toys directory, and runs any tests it finds. In the example above, toys system test will find the "hello" test in the file .toys/.test/test_greet.rb, run it, and display the results.

Now, the example above defines a file called test_greet.rb but it doesn't actually test our greet tool. So let's cover how to invoke tools from tests so you can test their functionality.

Testing tool functionality

Included in the Toys gem is the helper module Toys::Testing, which provides methods that load and execute tools for testing purposes. You can access these methods by including this module in your test class or describe block:

class MyTest < Minitest::Test
  include Toys::Testing

  # Write your tests..
end

The simplest way to test a tool's functionality is by running it and observing the results. To run a tool, invoke the Toys::Testing#toys_run_tool method, passing in the name of the tool and its arguments as if you were invoking the tool from the command line. For example, recall our original greet tool that takes an optional argument and a --shout flag:

# greet.rb

desc "Print a friendly greeting."
long_desc "Prints a friendly greeting. You can customize whom to" \
            " greet, and how friendly it should be.",
          "",
          "Example:",
          ["    toys greet --shout ruby"]

optional_arg :whom, default: "world", desc: "Whom to greet."
flag :shout, "-s", "--shout", desc: "Greet loudly."

def run
  greeting = "Hello, #{whom}!"
  greeting = greeting.upcase if shout
  puts greeting
end

To execute the tool from a test, call toys_run_tool and pass it ["greet"] as the command:

class GreetTest < Minitest::Test
  include Toys::Testing

  def test_greet_with_no_arguents
    exit_code = toys_run_tool(["greet"])
    assert_equal(0, exit_code)
  end
end

The toys_run_tool method executes the tool and returns the numeric exit code, which we can assert is 0 for a successful run. However, when we run this test, notice that "Hello, world!" is printed to the console in the middle of the test run. That is, of course, what our "greet" tool does. But normally, if you are testing a tool that writes to standard out, you'll want to assert against that output rather than have it pollute your test output. So typically, you'll wrap a call to toys_run_tool in a capture_subprocess_io block, to capture the output. So here's a complete test for our greet tool:

class GreetTest < Minitest::Test
  include Toys::Testing

  def test_greet_with_no_arguents
    out, err = capture_subprocess_io do
      exit_code = toys_run_tool(["greet"])
      assert_equal(0, exit_code)
    end
    assert_equal("Hello, world!\n", out)
    assert_empty(err)
  end
end

Generally, you should test various use cases of your tool, including the effect of passing different arguments and flags. For example, here's a test of the --shout flag:

class GreetTest < Minitest::Test
  include Toys::Testing

  def test_greet_with_shout_flag
    out, err = capture_subprocess_io do
      exit_code = toys_run_tool(["greet", "--shout])
      assert_equal(0, exit_code)
    end
    assert_equal("HELLO, WORLD!\n", out)
    assert_empty(err)
  end
end

It is important to note that toys_run_tool executes the tool in-process in your tests. This may be appropriate for simple tools, but may not work for other cases such as tools that replace the current process using Kernel#exec, or tools that require user interaction. For these cases, you can run a tool in a separate forked process using the Toys::Testing#toys_exec_tool method, which we shall look at next.

Running tools out of process

The Toys::Testing#toys_exec_tool method, like Toys::Testing#toys_run_tool, executes a tool and lets you assert on its behavior. However, while toys_run_tool executes the tool within the test process, toys_exec_tool forks a separate process to run the tool. This insulates your tests from anything disruptive the tool might do (such as replacing the current process). It also provides access to the powerful subprocess control functionality provided by Toys, letting you redirect, capture, or interact with the tool's streams while it is running.

There are two ways to run toys_exec_tool: in "capture" or "controller" mode. If you do not pass a block to toys_exec_tool, you are running in "capture" mode. It will close the input stream, and capture the output and error streams and return an object that provides the exit code and the stream contents. This object is actually an instance of Toys::Utils::Exec::Result, the same class returned by methods from the :exec mixin. Thus, we could use "capture" mode to rewrite our greeting test as follows:

class GreetTest < Minitest::Test
  include Toys::Testing

  def test_greet_with_no_arguents
    result = toys_exec_tool(["greet"])
    assert_equal(0, result.exit_code)
    assert_equal("Hello, world!\n", result.captured_out)
    assert_empty(result.captured_err)
  end
end

If you pass a block to toys_exec_tool, you will be running in "controller" mode. In this mode, all the tool's streams are redirected to a controller object that is passed to your block. See the controller's interface at Toys::Utils::Exec::Controller for more information.

In either mode, you can still override many aspects of the subprocess's behavior. For example, "capture" closes the input stream by default, but you can override this and pass data into the tool's input stream by providing the :in keyword argument. The various arguments available are documented in the Toys::Utils::Exec class. Here is an example, passing a string into our tool's input stream:

class GreetTest < Minitest::Test
  include Toys::Testing

  def test_greet_ignores_the_input_stream
    result = toys_exec_tool(["greet"], in: [:string, "I am ignored"])
    assert_equal(0, result.exit_code)
    assert_equal("Hello, world!\n", result.captured_out)
    assert_empty(result.captured_err)
  end
end

It is important to note that toys_exec_tool requires the Ruby runtime and underlying operating system to support fork. Standard "MRI" Ruby running on a Mac or Linux system will work; but JRuby, TruffleRuby, and any Ruby running on Windows, will not support fork and will not be able to run tests that use toys_exec_tool.

Testing helper methods

The Toys::Testing module also provides a way to load tools and test individual methods without running the entire tool. This can be useful for writing unit tests for tools that are large, complex, or otherwise should not be executed in their entirety during testing.

To test individual methods, pass a block to Toys::Testing#toys_load_tool. This method loads the tool associated with the given command line, parses the command line arguments, and prepares the tool for execution, but does not invoke the run method. The tool's context is passed to the block, and you can then execute any of the tool's helper methods and check assertions.

Let's consider an example tool with a helper method:

tool "hello" do
  flag :shout

  # The normal entrypoint "run" method
  def run
    puts message
  end

  # A helper method
  def message
    shout ? "HELLO" : "hello"
  end
end

Normally, testing this tool would involve capturing and asserting against the output printed by the run method. However, we can also test the message method in isolation without capturing any streams by loading the tool and calling the method directly.

class MyTest < Minitest::Test
  include Toys::Testing

  # A test of the tool when passed no command-line arguments
  def test_message_without_shout
    # Load the tool
    toys_load_tool(["hello"]) do |tool|
      # The tool's context is passed to the block, and you can use it to
      # call the tool's methods.
      assert_equal("hello", tool.message)
    end
  end

  # Another test that includes a command-line flag
  def test_message_with_shout
    toys_load_tool(["hello", "--shout"]) do |tool|
      assert_equal("HELLO", tool.message)
    end
  end
end

Like toys_run_tool, toys_load_tool loads the tool in-process in your tests. This means it would be difficult to use it to test methods that replace the current process or depend on user interaction. It is not currently possible to fork toys_load_tool into a separate process because Minitest would not be able to track assertions in that separate process. It is therefore appropriate mainly for true unit tests of isolated methods.

Organizing and selecting tests

So far we learned how to define tests in a .test directory located directly within a .toys directory. However, it is also possible to locate .test directories within subdirectories. Just like tools and tool namespaces can be defined within a directory structure, so corresponding tests can be associated with those tools by putting them in the same subdirectories.

Consider the following .toys directory that includes build and deploy tools:

(current directory)
|
+- .toys/
   |
   +- build/
   |  |
   |  +- .toys.rb   <-- defines "build" (and subtools)
   |  |
   |  +- .test/
   |     |
   |     +- test_build.rb  <-- defines tests for "build"
   |
   +- deploy/
      |
      +- .toys.rb   <-- defines "deploy" (and subtools)
      |
      +- .test/
         |
         +- test_staging.rb  <-- defines tests for "deploy"
         |
         +- test_prod.rb  <-- defines more tests for "deploy"

We've now created separate sets of tests under the build/ and deploy/ subdirectories. These are associated with those tools, and we can now run those sets of tests independently.

To run just the tests under build/:

toys system test build

To run just the tests under deploy/:

toys system test deploy

Now, if you do not provide a specific tool:

toys system test

...Toys will recursively search for .test directories, and run all the tests, including both the build and deploy tests. To disable this recursive search, pass --no-recursive. In the above case, that will result in no tests running because there are no tests defined at the top level.

It is also possible to focus on specific tests using the minitest-focus gem. Pass the --minitest-focus argument to toys system test to tell it to load the gem, and you will be able to use the focus keyword in your tests to temporarily focus on a few tests.

Using third-party gems

The toys executable itself uses only two gems: toys and toys-core. It has no other gem dependencies. However, the Ruby community has developed many resources for building command line tools, including a variety of gems that provide alternate command line parsing, control of the ANSI terminal, formatted output such as trees and tables, and effects such as hidden input, progress bars, various ways to spawn and control subprocesses, and so forth. You may find some of these gems useful when writing your tools. Additionally, if you are using Toys for your project's build scripts, it might be necessary to install your bundle when running some tools.

This section describes how to manage and use external gems with Toys. Note that running Toys with bundle exec is generally not recommended. We'll discuss the reasons for this, and what you can do instead.

Why not "bundle exec toys"

Bundler is often used when a command-line program depends on external gems. You specify the gem dependencies in a Gemfile, use bundler to resolve and install those dependencies, and then run the program prefixed by bundle exec to ensure those dependencies are in the Ruby load path. When running a Rake task, for example, it is almost automatic for many Ruby developers to run bundle exec rake my-task.

So why not simply run bundle exec toys my-tool?

In simple cases, this will work just fine. However, Toys is a much more flexible tool than Rake, and it covers two cases that are not well served by bundle exec.

First, Toys lets you define global tools that are defined in your home directory or system config directory. (See the previous section on the Toys search path.) These tools are global, and can be called from anywhere. But if they have gem dependencies, it might not be feasible for their Gemfiles to be present in every directory from which you might want to run them.

Second, it's possible for a variety of tools to be available together, including both locally and globally defined tools, with potentially different sets of dependencies. With bundle exec, you must choose beforehand which bundle to use.

Although traditional bundle exec doesn't always work, Toys provides ways for individual tools to manage their own gem dependencies.

Using bundler with a tool

The recommended way for a Toys tool to depend on third-party gems is for the tool to set up Bundler when it runs. The tool can load a bundle from an appropriate Gemfile at runtime, by including the :bundler mixin.

Here's an example. Suppose you are writing a tool in a Rails app. It might, for example, load the Rails environment and populate some data into the database. Hence, it needs to run with your app's bundle, represented by your app's Gemfile.

Simply include :bundler in your tool definition:

tool "populate-data" do
  include :bundler

  def run
    # The bundle will be set up before the tool is run,
    # so you can now run code that depends on rails:
    require "./config/environment.rb"
    # ... etc.
  end
end

When the :bundler mixin is included in a tool, it installs a mixin initializer that calls Bundler.setup when the tool is executed. This assumes the bundle is already installed, and brings the appropriate gems into the Ruby load path. That is, it's basically the same as bundle exec, but it applies only to the running tool.

Applying bundler to all subtools

In many cases, you might find that bundler is needed for many or most of the tools you write for a particular project. In this case, you might find it convenient to use Toys::DSL::Tool#subtool_apply to include the bundle in all your tools. For example:

# Include bundler in every tool under this one
subtool_apply do
  include :bundler
end

tool "one-tool" do
  # This tool will run with the bundle
  # ...
end

tool "another-tool" do
  # So will this tool
  # ...
end

See the section on applying directives to multiple tools for more information on subtool_apply.

Bundler options

By default, the :bundler mixin will look for a Gemfile within the .toys directory (if your tool is defined in one), and if one is not found there, within the context directory (the directory containing your .toys directory or .toys.rb file), and if one still is not found, in the current working directory. You can change this behavior by passing an option to the :bundler mixin. For example, you can search only the current working directory by passing search_dirs: :current as such:

tool "populate-data" do
  include :bundler, search_dirs: :current
  # etc...
end

The :search_dirs option takes a either directory path (as a string) or a symbol indicating a "semantic" directory. You can also pass an array of directories that will be searched in order. For each directory, Toys will look for a file called .gems.rb, gems.rb, or Gemfile (in that order) and use the first one that it finds.

The supported "semantic directory" symbols are :current indicating the current working directory, :context indicating the context directory, and :toys indicating the Toys directory in which the tool is defined. Furthermore, the semantic directory :toys is treated specially in that it looks up the .toys directory hierarchy. So if your tool is defined in .toys/foo/bar.rb, it will look for a Gemfile first in .toys/foo/ and then in .toys/. Additionally, when looking for a Gemfile in :toys, it searches only for .gems.rb and Gemfile. A file called gems.rb is not treated as a Gemfile under the :toys directory, because it could be a tool.

The default gemfile search path, if you do not provide the search_dirs: option, is equivalent to [:toys, :context, :current].

If the bundle is not installed, or is out of date, Toys will ask you whether you want it to install the bundle first before running the tool. A tool can also choose to install the bundle without prompting, or simply to raise an error, by passing another option to the :bundler mixin. For example, to simply install the bundle without asking for confirmation:

tool "populate-data" do
  include :bundler, on_missing: :install
  # etc...
end

See the documentation for Toys::StandardMixins::Bundler for more information about bundler options.

Solving bundle conflicts

It is important to understand that the :bundler mixin installs the bundle when the tool executes, rather than when the tool is defined. Gems in the bundle will not be available during tool definition, so for example you cannot reference bundled gems when you are setting up the tool's flags, description, or other directives. This is so that Toys can define tools with competing bundles. Your Rails app's tools can use that app's bundle, while your global tools can use a different bundle. They will not conflict because Toys will not actually load a bundle until one or the other tool is executed. (This is of course different from using bundle exec, which chooses and loads a bundle before even starting Toys.)

If a different bundle (i.e. a different Gemfile) is already in effect when a tool is run, then the :bundler mixin will raise an error. Ruby will not let you set up two different bundles at the same time. This might happen, for example, if you use bundle exec to run Toys, but the tool you are running asks for a different bundle---one more reason not to use bundle exec with Toys.

It might also happen if one tool that uses one bundle, calls a tool that uses a different bundle. If you need to do this, use the Toys::StandardMixins::Exec#exec_separate_tool method from the :exec mixin, to call the tool. This method spawns a separate process with a clean Bundler setup for running the tool.

When a bundle is needed to define a tool

Usually, the :bundler mixin sets up your bundle when the tool is executed. However, occasionally, you need the gems in the bundle to define a tool. This might happen, for instance, if your bundle includes gems that define mixins or templates used by your tool.

If you need the bundle set up immediately because its gems are needed by the tool definition, pass the static: true option when including the :bundler mixin. For example, if you are using the flame_server_toys gem, which provides a template that generates tools for the Flame web framework, you could include the flame_server_toys gem in your Gemfile, and make it available for defining tools:

# Set up the bundle immediately.
include :bundler, static: true

# Now you can use the gems in the bundle when defining tools.
require "flame_server_toys"
expand FlameServerToys::Template

There is a big caveat to using static: true, which is that you are setting up a bundle immediately, and as a result any subsequent attempt to set up or use a different bundle will fail. (See the section on bundle conflicts for a discussion of other reasons this can happen.) As a result, it's best not to use static: true unless you really need it to define tools. If you do run into this problem, here are two things you could try:

"Scope" the bundle to the tool or the namespace where you need it. Toys makes an effort not to define a tool unless you actually need to execute it or one of its subtools, so if you can locate include :bundler inside just the tool or namespace that needs it, you might be able to avoid conflicts.
Failing that, if you need a particular gem in order to define a tool, you could consider activating the gem directly rather than as part of a bundle. See the following section on Activating gems directly for details on this technique.

Activating gems directly

Although we recommend the :bundler mixin for most cases, it is also possible for a tool to install individual gems, using the :gems mixin. This mixin provides a way for a tool to install individual gems without using Bundler.

Here's an example tool that just runs rake. Because it requires rake to be installed in order to run the tool, we call the Toys::StandardMixins::Gems#gem method (which is provided by the :gems mixin) at the beginning of the run method:

tool "rake" do
  include :gems
  remaining_args :rake_args
  def run
    gem "rake", "~> 12.0"
    Kernel.exec(["rake"] + rake_args)
  end
end

The gem method takes the name of the gem, and an optional set of version requirements. If a gem matching the given version requirements is installed, it is activated. If not, the gem is installed (which the user can confirm or abort). Or, if Toys is being run in a bundle, a message is printed informing the user that they need to add the gem to their Gemfile.

If a gem satisfying the given version constraints is already activated, it remains active. If a gem with a conflicting version is already activated, an exception is raised.

The :gems mixin also provides a gem directive that ensures a gem is installed while the tool is being defined. In general, we recommend avoiding doing this, because it could make your tool incompatible with another tool that might need a competing gem during its definition. Toys would not be able to define both tools together. However, occasionally it might be useful.

Here's an example tool with flags for each of the HighLine styles. Because highline is needed to decide what flags to define, we use the gem directive to ensure highline is installed while the tool is being defined.

tool "highline-styles-demo" do
  include :gems
  gem "highline", "~> 2.0"
  require "highline"
  HighLine::BuiltinStyles::STYLES.each do |style|
    style = style.downcase
    flag style.to_sym, "--#{style}", "Apply #{style} to the text"
  end
  def run
    # ...
  end
end

Note these methods are a bit different from the gem method provided by Rubygems. The Toys version attempts to install a missing gem for you, whereas Rubygems will just throw an exception.

Activating gems outside the DSL

The above techniques for installing a bundle or activating a gem directly, are all part of the tool definition DSL. However, the functionality is also available outside the DSL---for example, from a class-based mixin.

To set up a bundle, call Toys::Utils::Gems#bundle. (Note that you must require "toys/utils/gems" explicitly before invoking the Toys::Utils::Gems class because, like all classes under Toys::Utils, Toys does not load it automatically.) For example:

require "toys/utils/gems"
gem_utils = Toys::Utils::Gems.new
gem_utils.bundle(search_dirs: Dir.getwd)

To activate single gems explicitly, call Toys::Utils::Gems#activate. For example:

require "toys/utils/gems"
gem_utils = Toys::Utils::Gems.new
gem_utils.activate("highline", "~> 2.0")

Useful gems

Now that you know how to ensure a gem is installed, either individually or as part of a bundle, let's look at some third-party gems that you might find useful when writing tools.

We already saw how to use the highline gem. Highline generally provides two features: terminal styling, and prompts. For these capabilities and many more, you might also consider TTY. It comprises a suite of gems that you can use separately or in tandem. Here are a few examples.

To produce styled output, consider Pastel.

tool "fancy-output" do
  def run
    require "pastel"
    pastel = Pastel.new
    puts pastel.red("Rubies!")
  end
end

To create rich user prompts, consider tty-prompt.

tool "favorite-language" do
  def run
    require "tty-prompt"
    prompt = TTY::Prompt.new
    lang = prompt.select("What is your favorite language?",
                         %w[Elixir Java Python Ruby Rust Other])
    prompt.say("#{lang} is awesome!")
  end
end

To create tabular output, consider tty-table.

tool "matrix" do
  def run
    require "tty-table"
    table = TTY::Table.new(["Language", "Creator"],
                           [["Ruby", "Matz"],
                            ["Python", "Guido"],
                            ["Elixir", "Jose"]])
    puts table.render(:ascii)
  end
end

To show progress, consider tty-progressbar for deterministic processes, or tty-spinner for non-deterministic.

tool "waiting" do
  def run
    require "tty-progressbar"
    bar = TTY::ProgressBar.new("Waiting [:bar]", total: 30)
    30.times do
      sleep(0.1)
      bar.advance(1)
    end
  end
end

A variety of other useful gems can also be found in this article.

Toys as a Rake replacement

Toys was designed to organize scripts that can be "scoped" to a project or directory. Rake is also commonly used for this purpose: you can write a "Rakefile" that defines rake tasks scoped to a directory. In many cases, Toys can be used as a replacement for Rake. Indeed, the Toys repository itself contains a .toys.rb file instead of a Rakefile, for running tests, builds, and so forth.

This section will explore the differences between Toys and Rake, and describe how to use Toys for some of the things traditionally done with Rake.

Comparing Toys and Rake

Although Toys and Rake serve many of the same use cases, they have very different design goals, and it is useful to understand the differences.

Rake's design is based on the classic "make" tool often provided in unix development environments. This design focuses on targets and dependencies, and is meant for a world where you invoke an external compiler tool whenever changes are made to an individual source file or any of its dependencies. This "declarative" approach expresses very well the build process for programs written in C and similar compiled languages.

Ruby, however, does not have an external compiler, and certainly not one that requires separate invocation for each source file as does the C compiler. So although Rake does support file dependencies, they are much less commonly used than in their Makefile cousins. Instead, in practice, most Rake tasks are not connected to a dependency at all; they are simply standalone scripts, what would be called "phony" targets in a Makefile. Such tasks are more imperative than declarative.

The Toys approach to build tools simply embraces the fact that our build processes already tend to be imperative. So unlike Rake, Toys does not provide syntax for describing targets and dependencies, since we generally don't have them in Ruby programs. Instead, it is optimized for writing imperative tools.

For example, Rake provides a primitive mechanism for passing arguments to a task, but it is clumsy and quite different from most unix programs. However, to do otherwise would clash with Rake's design goal of treating tasks as targets and dependencies. Toys does not have those design goals, so it is able to embrace the familiar unix conventions for command line arguments.

Toys actually borrows some of its design from the "mix" build tool used for Elixir and Erlang programs. Unlike C, the Erlang and Elixir compilers do their own dependency management, so mix does not require those capabilities. Instead, it focuses on making it easy to define imperative tasks.

All told, this boils down to the principle of using the best tool for the job. There will be times when you need to express file-based dependencies in some of your build tasks. Rake will continue to be your friend in those cases. However, for imperative tasks such as "run my tests", "build my YARD documentation", or "release my gem", you may find Toys easier to use.

Using Toys to invoke Rake tasks

If you've already written a Rakefile for your project, Toys provides a convenient way to invoke your existing Rake tasks using Toys. The built-in :rake template reads a Rakefile and automatically generates corresponding tools.

In the same directory as your Rakefile, create a .toys.rb file with the following contents:

# In .toys.rb
expand :rake

Now within that directory, if you had a Rake task called test, you can invoke it with:

$ toys test

Similarly, a Rake task named test:integration can be invoked with either of the following:

$ toys test integration
$ toys test:integration

Rake tasks with arguments are mapped to tool arguments, making it easier to invoke those tasks using Toys. For example, consider a Rake task with two arguments, defined as follows:

# In Rakefile
task :my_task, [:first, :second] do |task, args|
  do_something_with args[:first]
  do_something_else_with args[:second]
end

would have to be invoked as follows using rake:

$ rake my_task[value1,value2]

You may even need to escape the brackets if you are using a shell that treats them specially. Toys will let you pass them as normal command line arguments:

$ toys my_task value1 value2

The :rake template provides several options. If your Rakefile is named something other than Rakefile or isn't in the current directory, you can pass an explicit path to it when expanding the template:

# In .toys.rb
expand :rake, rakefile_path: "path/to/my_rakefile"

You can also choose to pass arguments as named flags rather than command line arguments. Set :use_flags when expanding the template:

# In .toys.rb
expand :rake, use_flags: true

Now with this option, to pass arguments to the tool, use the argument names as flags:

$ toys my_task --first=value1 --second=value2

From Rakefiles to Toys files

Invoking Rake tasks using Toys is an easy first step, but eventually you will likely want to migrate some of your project's build tasks from Rake to Toys. The remainder of this section describes the common patterns and features Toys provides for writing build tasks that are traditionally done with Rake.

Many common Rake tasks can be generated using code provided by either Rake or the third party library. Different libraries provide different mechanisms for task generation. For example, a test task might be defined like this:

require "rake/testtask"
Rake::TestTask.new do |t|
  t.test_files = FileList["test/test*.rb"]
end

In Toys, templates are the standard mechanism for generating tools.

expand :minitest do |t|
  t.files = ["test/test*.rb"]
end

The following sections will describe some of the built-in templates provided by Toys to generate common build tools.

Note that Rakefiles and Toys files can coexist in the same directory, so you can use either or both tools, depending on your needs.

Running tests

Toys provides a built-in template called :minitest for running unit tests with minitest. The following example directive uses the minitest template to create a tool called test:

expand :minitest, files: ["test/test*.rb"], libs: ["lib", "ext"]

See the Toys::Templates::Minitest documentation for details on the available options.

Toys also provides a built-in template called :rspec for running BDD examples using RSpec. The following example directive uses this template to create a tool called spec:

expand :rspec, pattern: "spec/**/*_spec.rb", libs: ["lib, "ext"]

See the Toys::Templates::Rspec documentation for details on the available options.

If you want to enforce code style using the rubocop gem, you can use the built-in :rubocop template. The following directive uses this template to create a tool called rubocop:

expand :rubocop

See the Toys::Templates::Rubocop documentation for details on the available options.

Building and releasing gems

The :gem_build built-in template can generate a variety of build and release tools for gems, and is a useful alternative to the Rake tasks provided by bundler. It is implemented by Toys::Templates::GemBuild. The following directive uses this template to create a tool called build:

expand :gem_build

The :gem_build template by default looks for a gemspec file in the current directory, and builds that gem into a pkg directory. You can also build a specific gem if you have multiple gemspec files.

You can also configure the template so it also releases the gem to Rubygems (using your stored Rubygems credentials), by setting the push_gem option. For example, here is how to generate a "release" tool that builds and releases your gem:

expand :gem_build, name: "release", push_gem: true

See the Toys::Templates::GemBuild documentation for details on the various options for build tools.

To create a "clean" tool, you can use the :clean built-in template. For example:

expand :clean, paths: ["pkg", "doc", "tmp"]

See the Toys::Templates::Clean documentation for details on the various options for clean.

Building documentation

Toys provides an :rdoc template for creating tools that generate RDoc documentation, and a :yardoc template for creating tools that generate YARD. Both templates provide a variety of options for controlling documentation generation. See Toys::Templates::Rdoc and Toys::Templates::Yardoc for detailed information.

Here's an example for YARD, creating a tool called yardoc:

expand :yardoc, protected: true, markup: "markdown"

Gem example

Let's look at a complete example that combines the techniques above to provide all the basic tools for a Ruby gem. It includes:

A testing tool that can be invoked using toys test
Code style checking using Rubocop, invoked using toys rubocop
Documentation building using Yardoc, invoked using toys yardoc
Gem building, invoked using toys build
Gem build and release to Rubygems.org, invoked using toys release
A full CI tool, invoked using toys ci, that can be run from your favorite CI system. It runs the tests and style checks, and checks (but does not actually build) the documentation for warnings and completeness.

Below is the full annotated .toys.rb file. For many gems, you could drop this into the gem source repo with minimal or no modifications. Indeed, it is very similar to the Toys files provided for the toys and toys-core gems themselves.

# This file is .toys.rb

# A "clean" tool that cleans out anything matched by the gitignore. You can
# also configure it to clean specific files and directories.
expand :clean, paths: :gitignore

# This is the "test" tool.
expand :minitest, libs: ["lib", "test"]

# This is the "rubocop" tool.
expand :rubocop

# This is the "yardoc" tool. We cause it to fail on warnings and if there
# are any undocumented objects, which is useful for CI. We also configure
# the tool so it recognizes the "--no-output" flag. The CI tool will use
# this flag to invoke yardoc but suppress output, because it just wants to
# check for warnings.
expand :yardoc do |t|
  t.generate_output_flag = true
  t.fail_on_warning = true
  t.fail_on_undocumented_objects = true
end

# The normal "build" tool that just builds a gem into the pkg directory.
expand :gem_build

# An "install" tool that builds the gem and installs it locally.
expand :gem_build, name: "install", install_gem: true

# A full gem "release" tool that builds the gem, and pushes it to rubygems.
# This assumes your local rubygems configuration is set up with the proper
# credentials.
expand :gem_build, name: "release", push_gem: true

# Now we create a full CI tool. It runs the test, rubocop, and yardoc tools
# and checks for errors. This tool could be invoked from a CI system.
tool "ci" do
  # The :exec mixin provides the exec_tool() method that we will use to run
  # other tools and check their exit status.
  include :exec
  # The :terminal mixin provides an enhanced "puts" method that lets you
  # write styled text to the terminal.
  include :terminal

  # A helper method, that runs a tool and outputs the result. It also
  # terminates if the tool reported an error.
  def run_stage(name, tool)
    if exec_tool(tool).success?
      puts("** #{name} passed", :green, :bold)
      puts
    else
      puts("** CI terminated: #{name} failed!", :red, :bold)
      exit(1)
    end
  end

  # The main run method. It just calls the above helper method for the
  # three tools we want to run for CI
  def run
    run_stage("Tests", ["test"])
    run_stage("Style checker", ["rubocop"])
    run_stage("Docs generation", ["yardoc", "--no-output"])
  end
end

Advanced tool definition techniques

This section covers some additional features that are often useful for writing tools. I've labeled them "advanced", but all that really means is that this user's guide didn't happen to have covered them until this section. Each of these features is very useful for certain types of tools, and it is good at least to know that you can do these things, even if you don't use them regularly.

Delegating tools

A tool can delegate to another tool, which means it uses the other tool's flags, arguments, and execution. Effectively, it becomes an "alias"---that is, an alternate name---for the target tool.

For example, suppose you have a tool called test that can be invoked with toys test. You could define a tool t that delegates to test. Then, running toys t will have the same effect as toys test.

To delegate a tool, pass the :delegate_to keyword argument to the tool directive. For example:

tool "test" do
  # Define test tool here...
end

tool "t", delegate_to: "test"

Tools can delegate to tools or namespaces. For example, you can delegate sys to the built-in namespace system:

tool "sys", delegate_to: "system"

That will let you run toys sys version (which will be the equivalent of toys system version).

To delegate to a subtool, pass an array, or a string delimited by ":" or "." characters, as the target:

tool "gem" do
  tool "test" do
    # Define the tool here
  end
end

tool "test", delegate_to: ["gem", "test"]

In most cases, if a tool delegates to another tool, you should not do anything else with it. For example, it should not have its own implementation or contain any subtools. However, there are a few exceptions. You might, for example, want a namespace to delegate to one of its subtools:

tool "test", delegate_to: ["test", "unit"] do
  tool "unit" do
    # Run unit tests
  end
  tool "integration" do
    # Run integration tests
  end
end

Now toys test delegates to, and thus has the same effect as toys test unit.

It is also possible to specify the delegate via relative name, using the :delegate_relative keyword argument. This can be helpful if you have a set of tools and delegates that are meant to be reused under different namespaces.

Applying directives to multiple tools

Sometimes a group of tools are set up similarly or share a set of flags, mixins, or other directives. You can apply a set of directives to all subtools (recursively) of the current tool, using the Toys::DSL::Tool#subtool_apply directive.

For example, it is common for tools to use the :exec built-in mixin to invoke external programs. You can use subtool_apply to ensure that the mixin is included in all subtools, so that you do not need to repeat the include directive in every tool.

subtool_apply do
  # Include the mixin only if the tool hasn't already done so
  unless include?(:exec)
    include :exec, exit_on_nonzero_status: true
  end
end

tool "my-tool" do
  def run
    # This tool has access to methods defined by the :exec mixin
    # because the above block is applied to the tool
    sh "echo hello"
  end
end

Importantly, subtool_apply blocks are "applied" at the end of a tool's definition. Therefore, when using subtool_apply, you have the ability to look at the current definition of the tool to decide whether to apply further changes. The subtool_apply block in the above example uses this technique; it checks whether the :exec mixin has already been included before attempting to include it. Thus, it is possible for a tool to "override" the inclusion, say, to use a different configuration:

tool "another-tool" do
  # Use a different configuration for the :exec mixin.
  # This "overrides" the subtool_apply block above.
  include :exec, exit_on_nonzero_status: false
  def run
    # This is run with exit_on_nonzero_status: false
    sh "echo hello"
  end
end

Custom acceptors

We saw earlier that flags and positional arguments can have acceptors, which control the allowed format, and can also convert the string argument to a Ruby object. By default, Toys supports the same acceptors recognized by Ruby's OptionParser library. And like OptionParser, Toys also lets you define your own acceptors.

Define an acceptor using the acceptor directive. You provide a name for the acceptor, and specify how to validate input strings and how to convert input strings to Ruby objects. You can then reference the acceptor in that tool or any of its subtools or their subtools, recursively.

There are several ways to define an acceptor.

You can validate input strings against a regular expression, by passing the regex to the acceptor directive. You can also optionally provide a block to convert input strings to objects (or omit the block to use the original string as the option value.) For example, a simple hexadecimal input acceptor might look like this:

acceptor("hex", /^[0-9a-fA-F]+$/) { |input| input.to_i(16) }

You can also accept enum values by passing an array of valid values to the acceptor directive. Inputs will be matched against the to_s form of the given values, and will be converted to the value itself. For example, one way to accept integers from 1 to 5 is:

acceptor("1to5", [1, 2, 3, 4, 5])

There are various other options. See the reference documentation for Toys::DSL::Tool#acceptor.

An acceptor is available to the tool in which it is defined, and any subtools and descendants defined at the same point in the Toys search path, but not from tools defined in a different point in the search path. For example, if you define an acceptor in a file located in a .toys directory, it will be visible to descendant tools defined in that same directory, but not in a different .toys directory.

A common technique, for example, would be to define an acceptor in the index file in a Toys directory. You can then include it from any subtools defined in other files in that same directory.

Controlling built-in flags

Earlier we saw that certain flags are added automatically to every tool: --verbose, --quiet, --help, and so forth. You may occasionally want to disable some of these "built-in" flags. There are two ways to do so:

If you want to use one of the built-in flags for another purpose, simply define the flag as you choose. Flags explicitly defined by your tool take precedence over the built-ins.

For example, normally two built-in flags are provided to decrease the verbosity level: -q and --quiet. If you define -q yourself (for example to activate a "quick" mode) then -q will be repurposed for your flag, but --quiet will still be present to decrease verbosity.

# Repurposes -q to set the "quick" option instead of "quiet"
flag :quick, "-q"

You can also completely disable a flag, and not repurpose it, using the disable_flag directive. It lets you mark one or more flags as "never use".

For example, if you disable the -q flag, then -q will no longer be a built-in flag that decreases the verbosity, but --quiet will remain. To completely disable decreasing the verbosity, disable both -q and --quiet.

# Disables -q but leaves --quiet
disable_flag "-q"

# Completely disables decreasing verbosity
disable_flag "-q", "--quiet"

Enforcing flags before args

By default, tools allow flags and positional arguments to be interspersed when command line arguments are parsed. This matches the behavior of most common command line binaries.

However, some tools prefer to follow the convention that all flags must appear first, followed by positional arguments. In such a tool, once a non-flag argument appears on the command line, all remaining arguments are treated as positional, even if they look like a flag and start with a hyphen.

You can configure a tool to follow this alternate parsing strategy using the enforce_flags_before_args directive.

The built-in tool toys do is an example of a tool that does this. It recognizes its own flags (such as --help and --delim) but once positional arguments start appearing, it wants further flags to be treated as positional so it can pass them down to the different steps it is executing. Here is a simplified excerpt from the implementation that tool:

tool "do" do
  flag :delim, default: ","
  remaining_args :commands  # the commands to execute
  enforce_flags_before_args
  def run
    # Now commands includes both the commands to run and
    # the "flags" to pass to them.
    commands.each do
      # ...
    end
  end
end

Requiring exact flag matches

By default, tools will recognized "shortened" forms of long flags. For example, most suppose you are defining a tool with long flags:

tool "my-tool" do
  flag :long_flag_name, "--long-flag-name"
  flag :another_long_flag, "--another-long-flag"
  def run
    # ...
  end
end

When you invoke this tool, you do not need to type the entire flag names. Abbreviations will also work:

$ toys my-tool --long --an

As long as the abbreviation is unambiguous (i.e. there is no other flag that begins with the same string), the Toys argument parser will recognize the flag. This is consistent with the behavior of most command line tools (and is also the behavior of Ruby's OptionParser library.)

However, it is possible to disable this behavior and require that flags be presented in their entirety, using the require_exact_flag_match directive.

tool "my-tool" do
  require_exact_flag_match
  flag :long_flag_name, "--long-flag-name"
  flag :another_long_flag, "--another-long-flag"
  def run
    # ...
  end
end

Now, all flags for this tool must be presented in their entirety. Abbreviations are not allowed.

$ toys my-tool --long-flag-name --another-long-flag

Currently you can require exact flag matches only at the tool level, applied to all flags for that tool. You cannot set this option for individual flags.

Disabling argument parsing

Normally Toys handles parsing command line arguments for you. This makes writing tools easier, and also allows Toys to generate documentation automatically for flags and arguments. However, occasionally you'll want Toys not to perform any parsing, but just to give you the command line arguments raw. One common case is if your tool turns around and passes its arguments verbatim to a subprocess.

To disable argument parsing, use the disable_argument_parsing directive. This directive disables parsing and validation of flags and positional arguments. (Thus, it is incompatible with specifying any flags or arguments for the tool.) Instead, you can retrieve the raw arguments using the Toys::Context#args method.

Here is an example that wraps calls to git:

tool "my-git" do
  desc "Prints a message, and then calls git normally"
  disable_argument_parsing
  def run
    puts "Calling my-git!"
    Kernel.exec(["git"] + args)
  end
end

Handling signals

If you interrupt a running tool by hitting CTRL-C or sending it a signal, Toys will normally terminate execution and display the message INTERRUPTED on the standard error stream.

If your tool needs to handle signals or inerrupts itself, you have several options. You can rescue the SignalException or call Signal.trap. Or you can provide a signal handler in your tool using the on_signal or on_interrupt directive. These directives either provide a block or designate a named method to handle a given signal received by the process. A separate handler must be provided for each signal type. (The on_interrupt directive is simply shorthand for registering a handler for SIGINT.)

If a signal or interrupt is received and is not caught via Signal.trap, the following takes place:

Ruby will terminate the tool's run method by raising a SignalException Any ensure blocks in the tool will be called.
Toys will call the signal handler, either a method or a block. If the handler takes an argument, Toys will pass it the SignalException object.
The signal handler is then responsible for tool execution from that point. It can terminate execution by returning or calling exit, or it can restart or resume processing (perhaps by calling the run method again). Or it can invoke the normal Toys signal handling (i.e. terminating execution and displaying the message INTERRUPTED or SIGNAL RECEIVED) by re-raising the same SignalException object.
If another signal is received or interrupt takes place during the execution of the handler, Toys will terminate the handler by raising a second SignalException (again calling any ensure blocks). Then, any matching signal handler will be called again for the new signal and passed the new exception. Any further signals will be handled similarly.

It is possible for a signal handler itself to receive signals. For example, if you have a long-running CTRL-C interrupt handler, it itself could get interrupted. You can tell how many signals have taken place by looking at the Exception#cause property of the SignalException. The first signal will have a cause of nil. The second signal (i.e. the first time a signal handler itself receives a signal) will have a cause pointing to the first SignalException (which in turn has a nil cause). The third signal's cause points at the second, and so forth. Hence, you can determine the signal "depth" by counting the length of the cause chain, which could be important to prevent "infinite" signals.

Here is an example that performs a long-running task. The first two times the task is interrupted, it is restarted. The third time, it is terminated.

tool "long-running" do
  def long_task(is_restart)
    puts "task #{is_restart ? 're' : ''}starting..."
    sleep 10
    puts "task finished!"
  end

  def run
    long_task(false)
  end

  on_interrupt do |ex|
    # The third interrupt will have a non-nil ex.cause.cause.
    # At that time, just give up and re-raise the exception, which causes
    # it to propagate out and invoke the standard Toys interrupt handler.
    raise ex if ex.cause&.cause
    # Otherwise, restart the long task.
    long_task(true)
  end
end

Handling usage errors

Normally, if Toys detects a usage error (such as an unrecognized flag) while parsing arguments, it will respond by aborting the tool and displaying the usage error. It is possible to override this behavior by providing your own usage error handler using the on_usage_error directive. This directive either provides a block to handle usage errors, or designates a named method as the handler.

If your handler block or method takes a parameter, Toys will pass it the array of usage errors. Otherwise, you can get the array by calling Toys::Context#usage_errors. This array will provide you with a list of the usage errors encountered.

You can also get information about the arguments that could not be parsed from the context. For example, the list of unrecognized flags is available from the context key UNMATCHED_FLAGS.

One common technique is to redirect usage errors back to the run method. In this way, run is called regardless of whether argument parsing succeeded or failed.

tool "lenient-parser" do
  flag :abc

  on_usage_error :run

  def run
    if usage_errors.empty?
      puts "Usage was correct"
    else
      puts "Usage was not correct"
    end
  end
end

Changing the entrypoint

The normal entrypoint for a tool is the run method. However, you can change it using the Toys::DSL::Tool#to_run directive, providing either a different method name as a symbol, or a block that will be called instead of a method to run the tool.

One reason to change which method is called, is if you need to modify a tool that was already defined by some other means (like a template expansion). For example, the tests for the Toys gems include a number of longer-running "integration" tests that run only when the TOYS_TEST_INTEGRATION environment variable is set. The Toys gems provide standard test tools using the :minitest template, but then modify those tools to add an --integration flag. When this flag is set, the TOYS_TEST_INTEGRATION environment variable gets set, causing the integration tests to run. Here's what that looks like:

# Use the minitest template to generate the normal test tool with the
# normal "run" method as the entrypoint
expand :minitest, libs: ["lib", "test"], bundler: true

# Reopen the "test" tool generated above
tool "test" do
  # Add a flag to the tool
  flag :integration

  # Create a new run method that "wraps" the existing one
  def run_with_integration
    # First set the environment variable to activate integration tests
    # if requested
    ENV["TOYS_TEST_INTEGRATION"] = "true" if integration

    # Invoke the original run method that was defined in the template
    run
  end

  # Now set the entrypoint to our new method
  to_run :run_with_integration
end

Passing a block to Toys::DSL::Tool#to_run is less common. One reason you might do it is if you need to access variables from an outer scope. (Normally, defining a method isolates your variables from any enclosing scopes.) Here is an example that prints out the time that the tool was defined (rather than executed).

tool "def-time" do
  # Grab the time during tool definition
  time = Time.now

  # Define the entrypoint using a block rather than a method.
  # The block is still called in the same object context, but now has
  # access to variables from enclosing scopes.
  to_run do
    puts "Defined at #{time}"
  end
end

The downside of this technique is that the entrypoint is not a method, and can't be called like one. Which means, among other things, you can't "wrap" it like we did with the test tool above.

If you need to access information from enclosing scopes, consider instead setting options using the Toys::DSL::Tool#static directive, as illustrated earlier when we discussed defining templates.

tool "def-time" do
  # Grab the time during tool definition and set it as a static option. 
  static :time, Time.now

  # Now you can access it from a run method like any other option
  def run
    puts "Defined at #{time}"
  end
end

Data files

If your tools require images, archives, keys, or other such static data, Toys provides a convenient place to put data files that can be looked up by tools either during definition or runtime.

To use data files, you must define your tools inside a Toys directory. Within the Toys directory, create a directory named .data and copy your data files there.

You mcanay then "find" a data file by providing the relative path to the file from the .data directory. When defining a tool, use the Toys::DSL::Tool#find_data directive in a Toys file. Or, at tool execution time, call Toys::Context#find_data (which is a convenience method for getting the tool source object using the TOOL_SOURCE key, and calling Toys::SourceInfo#find_data on it). In either case, find_data locates a matching file (or directory) among the data files, and returns the full path to that file system object. You can then read the file or perform any other operation on it.

For example, take the following directory structure:

(current directory)
|
+- .toys/
   |
   +- .data/
   |  |
   |  +- greeting.txt
   |  |
   |  +- desc/
   |     |
   |     +- short.txt
   |
   +- greet.rb   <-- defines "greet" (and subtools)

The data files in .toys/.data are available to any tool in the .toys directory or any of its subdirectories. For example, suppose we want our "greet" tool to use the contents of greeting.txt. We can call find_data to read those contents when the tool is executed:

# greet.rb
desc "Print a friendly greeting."
optional_arg :whom, default: "world", desc: "Whom to greet."
def run
  greeting = IO.read(find_data("greeting.txt")).strip
  puts "#{greeting}, #{whom}!"
end

You can include directories in the argument to find_data. For example, here is how to use the find_data directive to read the short description from the file "desc/short.txt":

# greet.rb
desc IO.read(find_data("desc/short.txt")).strip
optional_arg :whom, default: "world", desc: "Whom to greet."
def run
  greeting = IO.read(find_data("greeting.txt")).strip
  puts "#{greeting}, #{whom}!"
end

The find_data mechanism will return the "closest" file or directory found. In the example below, there is a desc/short.txt file in the .data directory at the top level, but there is also a desc/short.txt file in the .data directory under test. Tools under the test directory will find the more specific data file, while other tools will find the more general file.

(current directory)
|
+- .toys/
   |
   +- .data/
   |  |
   |  +- greeting.txt
   |  |
   |  +- desc/
   |     |
   |     +- short.txt  <-- default description for all tools
   |
   +- greet.rb   <-- defines "greet" (and subtools)
   |
   +- test/
      |
      +- .data/
      |  |
      |  +- desc/
      |     |
      |     +- short.txt  <-- override description for test tools
      |
      +- unit.rb   <-- defines "test unit" (and its subtools)

If, however, you find greeting.txt from a tool under test, it will still find the more general .toys/.data/greeting.txt file because there is no overriding file under .toys/test/.data.

The context directory

The context directory for a tool is the directory containing the toplevel .toys.rb file or the .toys directory within which the tool is defined. It is sometimes useful for tools that expect to be run from a specific working directory.

For example, suppose you have a Ruby project directory:

my-project/
|
+- .toys.rb  <-- project tools defined here
|
+- lib/
|
+- test/
|
etc...

Now suppose you defined a tool that lists the tests:

tool "list-tests" do
  def run
    puts Dir.glob("test/**/*.rb").join("\n")
  end
end

This tool assumes it will be run from the main project directory (my-project in the above case). However, Toys lets you invoke tools even if you are in a subdirectory:

$ cd lib
$ toys list-tests  # Does not work

Rake handles this by actually changing the current working directory to the directory containing the active Rakefile. Toys, however, does not change the working directory unless you tell it to. You can make the list-tests tool work correctly by changing the directory to the context directory (which is the directory containing the .toys.rb file, i.e. the my-project directory.)

tool "list-tests" do
  def run
    Dir.chdir context_directory do
      puts Dir.glob("test/**/*.rb").join("\n")
    end
  end
end

Note the context directory is different from __dir__. It is not necessarily the directory containing the file being executed, but the directory containing the entire toys directory structure. So if your tool definition is inside a .toys directory, it will still work:

my-project/   <-- context_directory still points here
|
+- .toys/
|  |
|  +- list-tests.rb   <-- tool defined here
|
+- lib/
|
+- test/
|
etc...

This technique is particularly useful for build tools. Indeed, all the build tools described in the section on Toys as a Rake Replacement automatically move into the context directory when they execute.

Changing the context directory

It is even possible to modify the context directory, causing tools that use the context directory (such as the standard build tools) to run in a different directory. Here is an example:

Suppose you have a repository with multiple gems, each in its own directory:

my-repo/
|
+- .toys.rb  <-- all project tools defined here
|
+- gem1/
|  |
|  +- lib/
|  |
|  +- test/
|
+- gem2/
|  |
|  +- lib/
|  |
|  +- test/
|
etc...

Assuming all the gems use the same set of build tools, it is possible to define those tools once in a single .toys.rb file and have it run in a particular gem directory depending on your current location. For example, you can cd into gem1 or even gem1/lib to have the tools run on gem1. Because the standard build tools execute within the context directory, you can accomplish this by setting the context directory to the gem directory corresponding to the current location. That is, if the working directory is my-repo/gem1/lib, set the context directory to my-repo/gem1. Here's what that could look like:

# .toys.rb content

require "pathname"
base_dir = Pathname.new context_directory
cur_dir = Pathname.new Dir.getwd

# The gem name is the first segment of the relative path from the context
# directory to the current directory.
relative_path = cur_dir.relative_path_from(base_dir).to_s
gem_name = relative_path.split("/").first

# Only proceed if we're truly in a subdirectory
if gem_name && gem_name != "." && gem_name != ".."

  # Now set the context directory to the gem directory.
  set_context_directory base_dir.join(gem_name).to_s

  # Define the build tools. Each of these uses the custom context directory
  # set above, and thus runs for the selected gem.
  expand :minitest
  expand :gem_build
  # etc.
end

Hidden tools

Tools whose name begins with an underscore (e.g. _foo) are called "hidden" tools. They can be executed the same as any other tool, but are normally omitted from the subtool list displayed in help and usage screens. You can use hidden tools as "internal" tools that are meant to be called only as part of the implementation of other tools.

If you pass the --all flag when displaying help, the help screen will include hidden tools in the subtools list.

Toys administration using the system tools

Toys comes with a few built-in tools, including some that let you administer Toys itself. These tools live in the system namespace.

Getting the Toys version

You can get the current version of Toys by running:

toys system version

Note that the same output can be obtained by passing the --version flag to the root tool:

toys --version

Upgrading Toys

To update Toys to the latest released version, run:

toys system update

This will determine the latest version from Rubygems, and update your Toys installation if it is not already current.

Normall it asks you for confirmation before downloading. To disable interactive confirmation, pass the --yes flag.

A similar effect can of course be obtained by running gem install toys.

Installing tab completion for Bash

Toys provides tab completion for the bash shell, and lets tools customize the completions for their arguments. However, you need to install the Toys completion tool into your shell. The following command sets up tab completion the current shell:

$(toys system bash-completion install)

Typically, you will want to include the above in your .bashrc or other bash initialization file.

By default, this associates the Toys tab completion logic with the toys executable. If you have other names or aliases for the executable, pass them as arguments. For example, I use t as an alias for toys, and I therefore install Toys's completion logic for t:

$(toys system bash-completion install t)

You can also remove the completion logic from the current shell:

$(toys system bash-completion remove)
$(toys system bash-completion remove t)

At this time, bash is the only shell that is supported directly. If you are using zsh, however, you can use the bashcompinit function to load the toys bash completion (as well as other bash-based completions). This mostly works, with a few caveats. Native zsh completion is on the future roadmap.

Getting information about tools

Sometimes you need to introspect the list of defined tools. For example, you might have a CI job that runs a Toys tool if available, but falls back to a Rake task if no Toys tools are defined. To determine whether a tool exists, you could use the status code returned from system tools show.

Two operations are available under the system tools namespace, one to search and list tools, and another to get information about a specific tool. In both cases, you can view results as either YAML or JSON. For more information, see the help screens:

toys system tools list --help
toys system tools show --help

Managing the Git cache

Toys manages a cache of files downloaded from remote Git repositories. This is used, for example, to share tools via Git. You can manage the contents of the Toys Git Cache using subtools under the system git-cache namespace. This includes querying the cache and using it to download files from git, gathering information about cache status, and deleting old data.

To display general information and a list of tools, run:

toys system git-cache --help

Writing your own CLI using Toys

Although Toys is not primarily designed to help you write a custom command-line executable, you can use it in that way. Toys is factored into two gems: toys-core, which includes all the underlying machinery for creating command-line executables, and toys, which is really just a wrapper that provides the toys executable itself and its built-in commands and behavior. To write your own command line executable based on the Toys system, just require the toys-core gem and configure your executable the way you want.

Toys-Core is modular and lets you customize much of the behavior of a command line executable, simply by setting options or adding plugins. For example:

Toys itself automatically adds a number of flags, such as --verbose and --help, to each tool. Toys-Core lets you customize what flags (if any) are automatically added for your own command line executable.
Toys itself provides a default way to run tools that have no run method. It assumes such tools are namespaces, and displays the online help screen. Toys-Core lets you provide an alternate default run method for your own command line executable.
Toys itself provides several built-in tools, such as do, and system. Toys-Core lets you write your own command line executable with its own built-in tools.
Toys itself implements a particular search path for user-provided Toys files, and looks for specific file and directory names such as .toys.rb. Toys-Core lets you change the search path, the file/directory names, or disable user-provided Toys files altogether for your own command line executable. Indeed, most command line executables do not need user-customizable tools, and can ship with only built-in tools.
Toys itself has a particular way of displaying online help and reporting errors. Toys-Core lets your own command line executable customize these and many other features.

For more information, see the Toys-Core documentation.

Toys User Guide

Conceptual overview

The Toys command line

Tools

Flags

Standard flags

Positional arguments

Tab completion

Defining tools

Basic Toys syntax

Tool descriptions

Positional arguments

Parsing required and optional arguments

Remaining arguments

Descriptions and the args DSL

Argument acceptors

Argument completions

Flags

Flag types

Inferred flags

Handling optional values

Flag acceptors

Defaults and handlers

Descriptions and the flags DSL

Flag completions

Flag groups

Tool execution basics

Namespaces and subtools

Defining subtools using classes

Understanding Toys files

Toys directories

Index files

The Toys search path

Stopping search

Loading remote files

The execution environment

Built-in context

Logging and verbosity

Running tools from tools

Helper methods and mixins

Executing subprocesses

Formatting output

Standard base directories

Sharing code

Defining mixins

Mixin initializers

Mixin inclusion hooks

Using constants

Templates

Defining templates

Template classes

Loading from a lib directory

Preloading Ruby files

Testing your tools

Writing and running tests

Testing tool functionality

Running tools out of process

Testing helper methods

Organizing and selecting tests

Using third-party gems

Why not "bundle exec toys"

Using bundler with a tool

Applying bundler to all subtools

Bundler options

Solving bundle conflicts

When a bundle is needed to define a tool

Activating gems directly

Activating gems outside the DSL

Useful gems

Toys as a Rake replacement

Comparing Toys and Rake

Using Toys to invoke Rake tasks

From Rakefiles to Toys files

Running tests

Building and releasing gems

Building documentation

Gem example

Advanced tool definition techniques

Delegating tools

Applying directives to multiple tools