litbuild – a literate build system

The idea of literate programming was invented by Donald Knuth. The basic notion – at least, the way I think of it – is that the primary users of the source code of a program are the people who are reading the source code to acquire an understanding of how it works (often, because they want to modify it), and that the transformation of the source code into an executable program that can be run, or the actual use of that program, is a secondary purpose of the code.

Because of this prioritization, most of the attention and energy invested in writing the program is in a narrative discussion of how the program works and what the purpose of each component (function declaration, subroutine or coroutine, or whatever) is within that narrative.

Automated build tools – such as make, maven, and jam – are great, because they allow developers on a project to focus their attention on the project codebase _per se,_ and let the build system worry about how to compile and test it. There are lots of build tools, reflecting a wide variety of design goals and value systems. Regardless of which tool is used, though, I have noticed that once a software project reaches a certain level of complexity, the automated build becomes similarly complex – to the point that it is a major undertaking to figure out how it actually works.

That makes it difficult to diagnose and correct problems with the build system when they arise, for the same reason that any other incomprehensible software code is hard to work with effectively… and since nobody pays any attention to the build machinery until it stops working for some reason, I’ve also noticed that it’s common for everyone on a team or project to forget how it works in between those times. At best, there’s one person who knows how it all works, and he or she gets saddled with the job of fixing it or tweaking it when that’s necessary.

Litbuild is an attempt to bring together the ideas of literate programming and automated build tools. My goal is to make it easy to write build scripts in such a way that they _tell a story_ about how to build a project (regardless of how big or complex that project is), focusing primarily on describing that build process with enough clarity that other people can understand it – while still providing enough structure to that story that the litbuild system can produce scripts that will perform an automated build of the project.

The original literate programming systems, WEB and CWEB, have two modes of operation: you can run the literate source code through a ‘tangle` program to produce source code in a traditional programming language like Pascal or C (which you can then feed to a compiler to produce a program), or you can run it through a `weave` program that produces document source (which you can send through a typesetting system like TeX or Docbook).

Litbuild is essentially a processor for a domain-specific language. It takes, as input, human-readable files – referred to as “blueprints” – that consist of narrative text written with AsciiDoc markup, interspersed with command blocks and other directives, just like the literate source in WEB and CWEB. Litbuild has a simpler interface, though: when it is run, it always generates both a set of bash scripts that will execute the actual build process and also AsciiDoc-formatted documentation files that can be run through a document production toolchain to produce HTML or PDF or whatever other documentation format you wish.

The vision of litbuild is that, from a set of blueprints, you should be able to say:

“Right now, on this system, supposing that I want to have thus-and-so package installed…

* "Tell me a story about how I can set that up and have it working
  properly", or
* "Produce a set of scripts that I can run to have that package set up
  and working properly,"

Or both!

Using Litbuild

The litbuild gem provides a simple driver script, ‘lb`, that loads all the blueprint files it finds in a directory tree and then generates output files for the blueprint target specified on the command line. You can only specify one blueprint as a target; if you want more than one thing to be built, create a Section blueprint that talks about all the blueprints you want to build and how they fit together. If the target you want to build is a phase within a blueprint, specify it as `blueprint::phase_name`; if the phase has a space in it, you can of course quote the command line argument like `“blueprint::phase name”`.

Blueprints can refer to parameterized values. It is expected that default values for all these parameters will be provided (by convention, I generally put all the default values into a “configuration” narrative blueprint). To override any parameter value from the default, set an environment variable with its name before running the ‘lb` driver program.

To use the lb driver to build a complete cross-toolchain as part of the Cross-Building Linux process, for example, you might use something like:

lb gnu-cross-toolchain

It’s not strictly necessary to use Section blueprints to perform a complete build for a complex target, because blueprints can specify what other bleuprints they depend on and litbuild will pull in whatever other blueprints are needed to resolve those dependencies. If all dependencies are correctly specified in all blueprints, all you really need to do is request the final target! But it’s still a good idea to write a Section for any situation where the various components of a build story need to have an overall framing narrative or discussion to make them comprehensible, or when all the parts of a build need to share a common configuration or environment – which is the case for most complicated processes.

If any commands in the generated scripts appear to be run under ‘sudo`, litbuild has a feature that can help to run those scripts non-interactively: it will produce sudoers entries (which can be included in `/etc/sudoers` or in a file in `/etc/sudoers.d`) that permit exactly those commands to be run under sudo without a password. For this feature to work, all programs run under `sudo` must either be specified with a full absolute path, or must exist in a specific set of default directories: /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, and /usr/local/sbin.

Debugging Blueprints

If litbuild is rejecting your blueprints with an error message and you can’t figure out why, you can produce a great deal of debugging information by setting the environment variable RUBYOPT to ‘–debug`.

Cross-Building Linux

The litbuild program is developed in conjunction with a set of blueprints called “Cross-Building Linux,” or CBL for short; this defines the build process used for Little Blue Linux. A lot of the features in litbuild are present for the convenience of CBL – like the s6-rc service directory and pipeline logic, the ability to produce package-user scripts, and the invocation of configuration file repository scripts. If you’re using litbuild for some other purpose, and don’t want to make use of those features, you might want to eliminate them.

We’re thinking of eventually rewriting litbuild as a C program to eliminate its dependency on ruby; if we do that, we’ll probably provide a compile-time configuration option that allows features like these to be compiled out of the program.