Rubu

Rubu (Re-Usable Build Utility) is a library for building programs. Rubu is in practice a replacement for Make and Rake type of tools. Rubu is targeted to provide means for creating flexible build environments.

Make and Rake are rule and recipe based, and they are very effective when the build process can be captured to these rules and is fairly fixed. Rubu includes declarations which correspond to rules, but the declarations are lower level and provide more control over the behavior. Rubu also provides direct control over when the rules are executed.

Make and Rake are more compact for the simplest build environments, but there is a break-even point where Rubu becomes more convenient. This is likely to happen when the user needs many exceptions to basic rules, also when other tasks than just build tasks are needed.

Rubu library can be used from any Ruby program, since it is just a library. It can also be part of a larger program as less significant part of the overall functionality. From maintenance point of view this can be a win in many cases.

The easiest way to get started with Rubu is to study examples (See: “Example” chapter below). However for longer term the main concepts are important as well (See: “Concepts” chapter below).

Example

Example of Rubu build program is placed in (relative to your installation directory), which you should copy over somewhere to execute:

example/bin/rubu_example

It builds a “hello world” type program which is split into two C source files. The main file is hand written and the other file is generated with a script. File generation is part of the build program.

Different execution variations of the build program is captured in:

example/runme

Please see example executions within the script and run the script in the “example” directory:

cd example
runme

Also take a look into the build program itself:

cat example/bin/rubu_example

There are comments which highlight purpose of the build program content.

Sometimes it is useful to run Rubu automatically whenever a file changes. Linux provides a facility called inotify, which can be used in Ruby through “rb-inotify” gem. Automatic update example program is in:

example/up-to-date

This program runs Rubu each time there is a change in “src” or in “bin” directory. It compiles and generates C files if necessary. If there are new compilation requirements coming, before all previous have been completed, it will continue compilation. When all compilations are done (i.e. none pending), it will call Rubu to perform linking. “up-to-date” displays status messages in red and green on the screen. If it is started with “-v” option the compile commands are visible, and if it is started with “-d” option, there is extra delay in compilation command.

cd example
up-to-date -v -d

This is useful if you want to play around with the pending compilation processes when new changes are coming in. You can make (dummy) changes to “hello_world.c” (for example), and follow the updates in the terminal where you started “up-to-date”.

Concepts

Rubu Build Program, RBP, is a Ruby program that uses the Rubu library. Typically command line options are accepted to give options for the build process.

Build programs are about collecting source files and transforming them into targets. This process is explicit in Rubu. Each file, whether it’s source or target, is collected within the Rubu build program. The files are converted into Mark objects. Basically Mark corresponds to a file, but it includes methods that make the build process convenient.

Step is build action that transforms source file(s) to target file(s). For example, a C source file compilation is a Step where one source file is converted to a corresponding object file. Step is given source Mark and target Mark as arguments. Hence this type of Step a is one-to-one mapping.

Linking multiple objects to an executable is a many-to-one mapping. This is performed by a Step where there are multiple sources and one target, i.e. objects and the executable respectively.

Step class (or object) has “step” method which includes the Move that is needed to create the target. Move is either a Shell or a Ruby command. Move is executed only if necessary. Typically the Step is sensitive to source being newer that the target, just as with Make and Rake. However, it is possible to have a Step that is sensitive to file content, for example. Timestamp might change, but if the content is the same as previously, no Move is made.

Step is typically a simple command. In some cases Step can include multiple Moves. These can be performed in sequence or in parallel, with a multicore CPU.

Trail is a collection of Steps. Trail has a name, and it can be references by another Trail. Trail can be sequential or parallel. For example, C source to object compilation can be done in parallel, but linking of objects to executable must occur after all objects have been created.

Rubu Build Program includes all of the above, and the last step is to select which Trail is executed. There should be at least “default” Trail, and typically there is also a “clean” Trail.

Rubu supports the configuration spaces: Order, Var, and Info. Order is meant to be used for controlling Rubu itself. For example if user wants to see all executed command, then “verbose” option should be selected.

Var is used to control how Steps behave and also details of Move internals. Info is meant for runtime generated information. The division is semantic and user might choose to combine everything to Var space, for example.

To summarize and highlight the relationships between concepts, see the diagram below:

RBP => Trail+ => Step+ [s/p] => Move+ [s/p] => SourceMark*, TargetMark*
    => Order+
    => Var*
    => Info*

* = Zero or more
+ = One or more
s = Serial
p = Parallel

RBP includes Trails, Orders, and possibly Vars and Infos. Trail includes one or more Steps, which can be executed in sequence or in parallel.

Step includes one or more Moves, and they can also be sequential or parallel. Move has typically at least one SourceMark and at least one TargetMark. However, there are exceptions as well.

The suggested ordering within Rubu Build Program is:

configuration

Process command line arguments and assign options that are used to control the build process.

file collection

Collect source and target files either using Ruby with glob pattern, direct references, or through manifest files. Convert file names to Mark objects.

step definition

Define Steps. Inherit one of the Step types (classes) and define at least the “step” method.

trail definition

Define Trails. Create all possible flows that user wants to execute. Use hierarchy for your benefit in order to make RBP maintenance as light as possible.

trail selection

Find the selected Trail or Trails and execute.

Configuration

Order space default are set by the Rubu library and user can override the default based on command line parameters if needed.

Set default values for Var space. After default setting, override the defaults with possible control from command line arguments.

Finally setup Trails by calling:

Trail.setup

File collection

If the build environment has clear and clean directory structure, it is typically easiest to collect files programmatically. For example, to get all C source files:

cee_files = Mark.glob( "#{Var[:source_dir]}/*.c" )

This creates a list of Mark objects which are source files. The corresponding object files can be created with “peer” method:

obj_files = cee_files.peer( Var[ :build_dir ], '.o' )

This method invocation takes an Array of Mark objects and calls “peer” method for them. “peer” method changes the directory and extension, and returns a new Mark object. In some cases also the “basename” of the file is changed. This is achieved by giving the “peer” method a third argument specifying the new “basename”.

Step definition

Step is defined by inheriting of the Rubu Step classes and defining the custom “step” method. If a specific setup for the Step is required, then an additional “setup” method can be defined as well.

class CleanUp < StepAlways
    def step
        shrun "rm -f #{Var[ :build_dir ]}/*.o"
    end
end

“setup” method is typically used to change the command according to user configuration. “step” method is the action for Step. Rubu executes it, if necessary. For example, if Step is a StepAged class, then the “step” method is only called if the source Mark is newer than the target Mark.

“step” method may include only one command or a set of commands. If one command is needed, it can be simply executed:

def step
    shrun "rm -f #{Var[ :build_dir ]}/*.o"
end

If there are multiple commands, then the commands can be grouped for sequential (“walk”) or parallel execution (“fork”).

def step
    fork do
        shuse "rm -f #{Info[ :gen_files ].rpath}"
        shuse "rm -f #{Var[ :build_dir ]}/*.o"
    end
end

Note that the commands are defined with “shuse”. The logic is that we don’t execute the commands right away as in “shrun”. Instead we just register the commands to the group. When the group definition is closed (“end” keyword), it is complete, and we execute it either sequentially or in parallel.

Trail definition

Trail is a collection of Steps or other Trails, or a mix of both. For example, in order to compile a set of C source files to object files, we might define:

Fork.form 'compile-cee' do
    GccCompileFile.usezip( cee_files, obj_files )
end

This creates a parallel (“Fork”) Trail called “compile-cee”. Collection of Steps is created with “usezip” method. “usezip” performs two operations: it creates a zip type mix of “cee_files” and “obj_files”. The “zip” method of Ruby Array is used. “zip” creates an Array of Mark pairs. Each pair is passed to a “GccCompileFile” Step object. Finally we have a collection of Steps which are executed in parallel when “compile-cee” is invoked.

Here the defined Trail from above is referenced in order to create a Trail for the complete compile flow:

Walk.form 'default' do
    pick 'compile-cee'
    GccLinkExe.use( obj_files, exe_file )
end

This Trail definition defines that we want to first execute “compile-cee” and then a Step which will create an executable from object file collection. “GccLinkExe” Step is called with “use” method, which will register the Step to the enclosing Trail, “default”.

Trail selection

The final stage in RBP execution is to use command line control for selecting the executed Trails and then execute them. The above definition (from previous section) is executed with:

Trail.run( 'default' )

Final notes

Rubu treats single Mark objects and Array of Mark objects the “same” way. In practice this means that the most common methods for Mark objects are implemented for Array as well. Hence, you can call a Mark method for an Array, and they will actually be applied for each individual Mark in the collection.

Step has sources and targets. Even if a step has only one source and one target, the actual container is an Array. If you want to refer to the source, i.e. the only existing source, you can use the “source” method. Same applies to targets (i.e. use the “target” method).

Testing

For this version, example is the testcase.