logo logo

forthebadge forthebadge Gem Version

RBT - The "Ruby Build Tools" Project

The main goal for the RBT project is to allow the user to install software, primarily on Linux systems, and primarily from source - that is, to compile the program at hand.

While the focus is predominantly on the Linux platform, whenever possible support for other platforms will be added as well; Linux is the main target OS, though.

While this is the main goal for the RBT project, the full scope of the RBT project is a wider one: the RBT project also tries to help as a build toolset, that is, as means to help people manage software on their respective computer systems in general. This definition also includes the tasks done by package managers, although support for this within RBT itself is presently (2019) fairly minimal.

Additionally, the RBT project may aid in the process of autogenerating files and code related to installing and compiling, such as generating standalone shell scripts or archive formats, such as .rpm or .deb. For example, one part of RBT can be used to autogenerate completion files for bash and zsh. Another script can modify Makefiles and change the prefix in that file.

In some ways RBT attempts to be a swiss army knife for compilation/installation.

Even if you do not intend to install or compile anything through RBT, many different programs are registered in this project, which may be of help in other ways, such as if you wish to determine which particular binary on a given Linux system belongs to which particular program, or if you intend to implement a package manager yourself. (The dataset for the standalone programs is available under the namespace RBT::Cookbooks. For a list of all registered programs, you can simply query RBT.available_programs?).

The primary core functionality of the RBT project, however had, lies in compiling programs from source, as this is what I use RBT for on an almost daily basis.

Installing the rbt gem itself

The canonical way to install the rbt gem is:

gem install rbt

However had, I recommend the --user-install option; this makes it easier to use the bin/ scripts that are part of this gem, as they will reside in your home directory, at bin/ there:

gem install rbt --user-install

Then you can use them easily if you have the bin/ to your home directory in your $PATH.

While the /usr/ hierarchy as prefix is fine, in principle, some distributions, in particular debian, have a habit to split up gems a lot. This is also another reason why the --user-install option is better, in my opinion - you don't have to guess where the executables will reside; they are just in the home directory. And you can move them from there to any other location anyway.

Since gems are also downloaded locally, and then available as a .gem file, other users can easily install these gems into their home directory as well, without depending on a global installation. This may be especially useful in environments where you are restricted, such as at a campus/university environment.

The history and philosophy of the RBT Project - philosophic considerations

This subsection in this README-document contains some information as to how the RBT project was initially started, and how it has changed over the course of time. Additionally some of the philosophy behind the RBT project will be explained.

The RBT project in itself started out initially as means to re-create the shell scripts' logic, and functionality, used in GoboLinux, several years ago.

I was already using Ruby at that time and I did not understand the shell scripts that were used in GoboLinux. Shell scripts can be quite confusing to read; parameters do not go in normally into a regular function via a function/method signature, but instead are accessed as special semi-global variables such as $1 or $2 within the function body. I find this extremely unintuitive, confusing and quite ugly.

So the seemingly logical thing to do here was to re-create the functionality in Ruby, as I find well-written ruby code much simpler and easier to read and understand than shell scripts. Of course people can write ugly and confusing code in ruby too; and there is a lot of somewhat decently written shell code spread over the www as well. But by and large, I found it to be true that well-written ruby code tends to be a lot better to read/maintain/change than well-written shell code, for equivalent functionality.

At the least for me personally, I have a hard time trying to understand what shell scripts do in general, even though the shell scripts used in GoboLinux are one of the easiest and nicest to read, in my opinion. The GNU Sorcery distribution also had some good shell scripts, but the GoboLinux scripts were better written, in my opinion.

Still, shell scripts are difficult to understand, and not very elegant, and elegance/beauty, which is a subjective metric, was one important reason as to why I did want to use ruby rather than shell scripts. I prefer using elegant systems. Both GoboLinux, from a conceptual point of view, and Ruby, are elegant systems/technologies, so it made sense to want to combine different ideas.

Elegance/beauty/symmetry in style helps ease the flow of mind in some ways. Why make something complicated if you can keep it simple?

People who may have been using other languages, such as PHP or perl, may agree with this e. g. when comparing these languages to, say, ruby or python.

Anyway. This was sort of the initial impetus for creating the RBT project. Lots of years have gone by since then ... well over a decade, really. I can not even pinpoint to a single year when the RBT project was started, since it started out as a set of semi-random .rb files/scripts, but I think the rbt project was created, through the old cookbooks project, in the year 2005, give or take. The file called COOKBOOK_STATISTICS.md keeps some old entries. For example, in late October in the year 2005, the old Cookbooks project, which was eventually changed into RBT, already had 274 programs registered, so it had to have been started prior to October 2005. For comparison - in the ~middle of May 2019, 3593 programs were already integrated into the RBT project.

Naturally if you have a project that is that old, a lot of things change. This was also the case for the RBT project, in particular the scope of the project has expanded since then.

The primary focus for the RBT project still focuses on compile-related activities. I use the RBT project almost daily in order to get something to compile/install. At a later point, RBT became more of a toolset project.

In some ways, this implies that one secondary goal of the RBT project is to be able to build a LFS ("Linux from Scratch") system from scratch, but not every functionality within the RBT project is equally well polished or battle-tested, so manual intervention is still necessary for the time being. The long-term goal is to be able to build a fully automated LFS, though.

Without buzzword talk, RBT itself is in many ways just a large, happy hackish project, rather than something that has a great, perfect design. While a good design is immensely useful, it is not quite how I approached the problem domain, nor was it possible for me to do so, since RBT is ultimately a hobby project; I more try to solve a given problem at hand, and then move on doing other things. Of course I do try to keep the code clean and well documented, but you know how it goes for any hobby project. Sometimes you have more time (and motivation) and sometimes you lack both.

As a rule of thumb, though - the simpler the task that you may want to achieve through RBT, in an automatic, scripted way, the easier it should be for RBT to be of help here.

As it was already mentioned, in the past there used to be a separate project called Cookbooks, the predecessor to RBT::Cookbooks. That project specifically handled the recipes for the programs, that is, how to install or compile something specifically. In the year 2018 the Cookbooks project has been fully merged into the RBT project, primarily to make syncing the code easier. The RBT::Cookbooks namespace handles the source files' logic, whereas RBT itself handles how to interprete this information, in particular through the main class to use for compiling something, which is the class RBT::Compile.

When you have a look at the RBT project, including the documentation on the homepage of the RBT gem, you may notice that the documentation is quite detailed and will explain quite a bit of the internals, that is, of the internal code used in RBT. This code base is predominantly written in ruby. As such, the whole RBT project caters heavily towards people who already use ruby as their main language of choice.

Why did I decide to also document and explain some internals? After all this will lead to a lengthier documentation. Now, there are two main reasons for this:

The first reason is that it actually helps me remember when I made a specific decision. When looking back at it, perhaps even years lateron, I need to understand why I did choose to go to a particular route.

The second reason, and more important one, is that I want to treat users as intelligent people, so I will try to explain to them how some of the RBT internals work. Naturally not all of RBT will be explained that way, but more important parts, such as e. g. RBT::Compile in particular, will have several entries that will explain what is done, and why.

I am aware that not everyone using RBT may use ruby. For example, you may use python as your primary language rather than ruby, but would still like to make use of the yaml files in the RBT project.

I will try to keep the yaml files and format well-documented and well-written; this is an ongoing effort, though. Keep in mind that some entries are literally very very old by now. (You can also compare this to the old GoboLinux recipes, which you can find on Github. Several of these projects no longer exist.)

You can generate a sanitized variant of these yaml files from ruby, and then just keep on using these .yml files in other projects - after all that is one advantage of yaml, that we are freed from any single programming language (such as e. g. hardcoding information into a .rb file or a .py file). If you wish to generate these expanded yaml files, you can use rbt --expand-cookbooks, but you may wish to set to another temporary directory before doing this - see other parts of this document for how to do so.

That still leaves some room to explain some of the philosophy of the RBT project.

There are different philosophies, but to name a few key ones:

  • Try best to solve the use cases that the user at hand has, without getting into the way or preventing specific actions if a user really wants to do so.

  • Try to be "apolitical", "agnostic" and as flexible as possible. For example, while I personally do not use systemd, the RBT project in itself has to be as flexible to allow those who use/depend on systemd the use of RBT.

  • Try to provide the most common functionality through RBT, as much as that is useful, including acting as a (surrogate) package manager or as a set of tools to auto-generate files. For instance, RBT includes one class that can modify the PREFIX variable in a Makefile. That way people can easily change the target prefix of a Makefile without having to use any other commandline options.

  • Focus on Linux as a first-class citizen but do not exclude any other operating system, if possible. Primary testing will happen on Linux but other operating systems should be supported whenever we can (if ruby works on these systems that is; but I will also keep in mind resource-constrained systems, e. g. small/embedded systems; for example, the RBT project can also autogenerate shell scripts, so this could in theory be extended to work on systems that only have shell scripts rather than ruby installed. Do note, though, that for small systems, it is often better to compile on a larger desktop machine than on such a small constrained device; compiling on a raspberry may take even days.)

Do note that in early 2020 I decided to partially abandon the GoboLinux scheme. The RBT project will retain the focus on GoboLinux, if only for compatibility - but other than that, the project attempts to be standalone and general-purpose, not coupled to any single distribution as such. This also means that alternative approaches will be explored.

Why a new project rather than extending existing code?

This is a valid question. There are a few problems with the existing projects out there, though.

The scope of these projects is often limited. For example, mac homebrew is a package manager, but other than that it isn't doing that much; and it does not work that well on linux or windows - I tried it in 2019 on linux and it did not work for me. Since I felt that Linux is a second-class citizen, I did not invest more time into the issues I have had.

I also did want to have a project that works as well as ruby does on any given operating system, as long as ruby is supported.

But there are also API-differences and the way how these systems are used. For example, the package managers do not allow you to easily use another prefix. Most of them, at the least on linux, assume that /usr/ is the only prefix that is to be used (or /usr/cellar or something like this). I do not like this restriction - I want more flexibility, and complete control over the prefix in use at all times.

This list can be extended, but ultimately it was less of an issue to create a project that has these goals in mind, rather than try to retrofit existing code bases out there towards my use cases.

@source_base_directory

The very important instance variable @source_base_directory, defined on the toplevel module RBT, can be used to denote the directory where archives of different programs are to be kept (or are already kept).

For example, on my home system I store the source code to many different programs in the directory /home/x/src/. That name is mostly due to historic reasons, as the user "x" was just a generic name I would use for various different tasks, including making regular backups.

As other users may wish to use another directory rather than the one listed above, such as /Files/Archives/ or /Depot/Archive/ or something like that, there are essentially three different ways how to define your own target directory for keeping such source archives:

(1) Simply set a new target altogether, via the method called RBT.set_source_base_directory(). A shorter API exists as well, an alias to this method, called RBT.set_source_directory().

You can also use RBT.source_directory = path, if you would like to.

As argument to this method, simply pass in the path to the directory that you wish to use, such as /opt/ or /home/src/ or any other location that you may wish to use.

Explicit examples for the API that was mentioned above, will be shown next:

RBT.set_source_directory('/opt')
RBT.set_source_base_directory('/opt')
RBT.source_directory = '/opt'

Note that this method will ensure that a trailing '/' token will be the last character here. (Directories should end with a trailing / - it makes the subsequent code easier when we can assume that this is always the case. I am not entirely sure why Linux/UNIX does not seem to share the same notion.)

The method call to RBT.set_source_directory() will not make the above directory persistent, mind you, so you may want to read on for option (2).

(2) Another option is to use a setting in the main configuration file, typically a yaml file called source_base_directory.yml, the path to the source directory can be set as well. This is the method that I use primarily.

In the directory of the RBT project where the configuration files are stored, you can simply use the target directory, by putting the path to the source archives into that file.

You can also use environment variables in this file - simply use ALL_CAPS to make use of such environment variables. (This won't work in all settings, e. g. for .cgi files it may not work since .cgi files may use fewer environment variables. But for regular commandline-work, this works fine. I may use an ALL CAPS constant such as MY_SRC_DIR there in this file.)

(3) Not everyone will be able to modify the directory where the configuration is kept, so a third way exists, using the environment variable called RBT_SOURCE_DIRECTORY.

In general, RBT may use environment variables that have a leading RBT_ as prefix. Simply define this variable for your shell pointing to the directory that you wish to use and it should work fine:

export RBT_SOURCE_DIRECTORY=/Files/Archives

The last option may work for you if option (2) is not available or not possible.

Return an Array of programs from a remote URL

In the RBT project there is code that allows us to query the remote programs listed on an URL.

This was a necessary addition because there are some websites that have made available several programs on the same URL - think of the GNOME project or the KDE project or the LFS/BLFS project page. These all list several programs there.

The basic API for obtaining an array of programs from a remote URL is the following:

RBT.return_programs_from_this_url
RBT.return_program_from_this_url('https://download.kde.org/stable/plasma/5.10.5/')

It will return an Array of programs at that site, if it can be found.

psych or syck

You can determine which yaml-engine to use.

In the past, the RBT project would try to use the syck engine, but since psych has become the default, RBT will try to make use of psych by default. This can be changed via the configuration file use_psych_or_syck.yml that resides at rbt/yaml/configuration/use_psych_or_syck.yml.

A top-level API also exists to query which yaml-engine is currently used:

RBT.yaml_engine?

If, for some reason, you have to or want to modify this, you can use the following API to toggle any other yaml engine:

RBT.yaml_engine=
RBT.set_yaml_engine

Configuration of the RBT project

Most of the configuration is done through yaml files (.yml). These individual yaml files are normally kept under the subdirectory rbt/yaml/configuration/.

For example, there is a file called be_verbose.yml. This determines as to whether the scripts are verbose by default or as to whether they are not. You can use shortcuts here, such as "t" which stands for true, or "f" which stands for false. Of course you can also use true and false, as content of said file. Additionally you can use "yes" or "no".

So if you put the string no, without quotes, into the file called be_verbose.yml then the RBT scripts will not be verbose by default.

Similar possibilities exist for the other files that accept a Boolean value (true/false aka yes/no). Obviously this would not work when you are expected to input a file path, such as the path to a certain directory. In this case you simply have to provide a string to the path in question - but you can also use environment variables if you have defined them. For example, I keep an environment variable called $MY_PROGRAMS, so I can input that String ($MY_PROGRAMS) into the file called programs_directory.yml and all RBT scripts will refer to that as the main /Programs/ directory (it is indeed usually /Programs/ but it could also point to my home directory or any other target).

How does the format of a cookbook recipe, the respective .yml file, look like?

Each cookbook entry is just an individual yaml file, with the extension name being .yml. For example, information about the programming language ruby will be stored in the file called ruby.yml; information about the programming language python will be stored in the file python.yml; information about the mate-desktop will be stored in the file matedesktop.yml. If there are any "-" tokens as part of the name, then these will be removed when rbt tries to determine the filename.

This yaml file should contain all the required information to get that particular program to install on a given system.

Additional information can be added as well, such as maintainer(s), extra information of the program at hand, tips and hints in how to install or use the program, sed-modifications, pkgconfig files (aka .pc files) that are installed alongside with, patchsets that are to be applied, binaries, headers and libraries and so on and so forth.

It can be a quite detailed file, mostly due to being able to allow some flexibility during installation.

Many programs "out in the wild" are well-behaving and will honour a --prefix variable given (or -DCMAKE_INSTALL_PREFIX=, such as in use by programs that depend on cmake, for compilation). Such programs usually require little to no modification.

Unfortunately, some other programs come with flawed installation scripts, hardcoded paths or workarounds that have to be employed to get these to work - very often sed-operations, such as done in the LFS project. Some programs may also have already become obsoleted, due to changes e. g. in glibc/gcc and other programs. If there is no maintainer of the downstream package then this normally means that other people have to either maintain that package; or provide patches to get this projects to work again.

As a reference project, the LFS project provides a lot of useful information in how to get many programs to work. I consider the LFS project to be hugely important for the whole Linux ecosystem, much more important than the various different distributions.

At any rate, let's now have a look at the fish.yml recipe, to give a specific example and explain that.

This fish.yml file will look like this, as of August 2018:

fish:
 binaries:
 - fish
 - fish_indent
 - fish_key_reader
 short_description: |
  FISH is an interactive (commandline) shell.
 description: |
  FISH is the Friendly Interactive Shell, a smart and user-friendly
  command line shell for macOS, Linux, and the rest of the family.
 url1: https://github.com/fish-shell/fish-shell/releases/download/2.7.1/fish-2.7.1.tar.gz
 url2: http://fishshell.com/files/2.1.0
 url3: http://sourceforge.net/projects/fish/
 homepage: http://fishshell.com/
 tags:
 - shell
 prefix: f
 keep_extracted: t
 last_update: 08 Jan 2018

The very first entry, on top, is the name of the program itself, in this case, fish.

While this part could in theory be omitted (since the name of the program is already stored as the first part of the very filename), I found that it is useful to have this entry within the file, primarily as a visual identifier if you work with it in an editor. It helps me a lot, for instance.

Note that no - or _characters are allowed there in this entry - it is really a stripped down name of the program at hand only.

If the name would be e. g. "gtk-doc", with an URL such as http://ftp.gnome.org/pub/gnome/sources/gtk-doc/1.26/gtk-doc-1.26.tar.xz, then the name there for the gtkdoc.yml file would have to be gtkdoc: - thus, all '-' were removed. The name that appears there is also the name of the .yml file at hand. The .yml files may also not include any '-' or '' characters. Do also note that while I am following this convention since many years, it may one day be changed to allow for '-' and ''. But for the time being, the strict requirement is that we may not have '-' or '_' characters there.

Next, let's look at the entry binaries:. This entry tells us which binaries/executables this particular program will install. This also allows us to remove binaries in the /usr/bin/ hierarchy, if you compiled into an AppDir prefix - for more information here, see class RBT::PurgeBinariesOfThisProgram.

The next entry is short_description, which is e. g. equivalent to slackware's pkg-format, as introduction to the specific program at hand, in a short notation.

Then we have the entry main description, followed by several url entries. The first url entry, aptly named url1, is the most important entry. It tells us where to find the archive to the program at hand. This is ideal the URL to an existing archive, but it could also be a github repository, in principle.

Then we also have the homepage entry, which denotes the homepage of the program (or any other webpage that has relevant information, in the event that there no homepage entry exists for that particular program).

Then we have tags: which just groups the program into popular tag-classifiers, such as "shell" in this case. Then the prefix in use, which is f for false, meaning "non-traditional" aka "appdir" prefix such as GoboLinux makes use of. Then there is the keep_extracted setting, which tells us whether to keep the archive extracted or not after extracting/compiling it. Last but not least, the last_update tag tells us when this program was last updated. The default format here is dd.mm.yyy, but shortcuts are allowed. For example, Oct, for October, can be used rather than 10, and so on.

The next entry is prefix. This one is mostly synonymous for the --prefix= option given to GNU autoconfigure based projects. A prefix value of t means true, which stands for /usr/. A prefix value of f means false, which stands for the particular AppDir-like prefix. In the case here, for fish, this would expand to /Programs/Fish/2.7.1. GoboLinux uses a very similar AppDir layout by default, although with slightly different program names that can also be upcased, e. g. "KDE-Libs" and so forth. (RBT will offer both variants here; the GoboLinux naming scheme, and the simpler scheme where only the first character is upcased, and the other characters are all downcased.)

There are many more entries, most of which should be documented under the doc/ subdirectory of this gem. What follows next are some more mentions of important (and less important) entries.

The entry configure_base_dir: unix determines which base directory is to be used. In the above example, RBT would cd into the directory called unix/. This is important for some programs such as tk or xvid - have a look at their directory structure to understand this setting.

Since as of December 2018 it is possible to use github URL that are truncated. Take the program conky as an example. It has an url like the following:

https://github.com/brndnmtthws/conky/archive/v1.11.0.tar.gz

This will now be automatically expanded into the correct naming scheme, aka conky-1.11.0. No idea why GitHub is not doing so on their own automatically; the tarball is even wrongly built, since it extracts to precisely that name after download. But anyway, ask GitHub about this, not me.

The content of these individual .yml files is essentially just a Hash that describes the various attributes of the given project at hand (and some additional information when it is required for installing/compiling this program).

Note that since as of January 2020, you can also just provide a new URL and register a program this way anew.

Commandline Example for this:

rbt http://ftp.gnu.org/pub/gnu/kawa/kawa-3.1.tar.gz

Cookbooks - Recipes to install programs from source

The Cookbooks module, part of module RBT and thus residing under the RBT::Cookbooks namespace, attempts to collect recipes and instructions that allow you to install programs from source (primarily).

This project has well over 3350 programs registered so far - and that number continues to grow slowly.

If you are interested in seeing the specific programs and the associated programs versions and URLs, have a look at the side pane on the right hand side, at Available program versions.

What is the use case for this project? How could this project help you?

The use case for the Cookbooks component, part of the RBT project, is to provide the necessary instructions for compilation/installation of programs. These instructions will be primarily stored in yaml files.

For example, the text-editor called nano will have some information that details how it should be compiled, in the file called nano.yml.

This is valid for all programs that are registered, by the way - for each program, there will be a single .yml file that describes that program in question.

Now, how could this be of help to you or anyone else?

This of course depends on your use case at hand, but say that you may want to compile a linux from scratch system and then a beyond linux from scratch system on top of it. You can - and should - follow the official instructions of course, but once you understand the basic concepts, perhaps you may want to have more control over the programs at hand or wish to compile/install more programs, including their dependencies. That's exactly one use case for the RBT project, through its Cookbooks subcomponent.

There are more use cases of course, such as updating your programs from source quickly. Keep in mind that a lot of information is stored in all the yaml files together - describing the program at hand, where to find the source, how to install it, and so forth. This information may always be helpful to some people. Many yaml files include the homepage link, so rather than having to use google, one can quickly jump to the registered homepage at hand (the entry in the .yml file will be called homepage:).

The Cookbooks project is a component of the RBT (Ruby Build Tools) project. The latter project, aka rbt, is the one that formally integrates the dataset provided by the module Cookbooks part, into general compile/install related activities. I use rbt to compile programs from source or otherwise install them.

If the cookbooks project and the RBT project are used in combination, then they are in some ways similar to the MacOSX homebrew project or the fpm project - sort of.

There may be some philosophical differences between these projects, but the cookbooks project and the RBT project will attempt very hard to provide similar functionality as fpm and mac homebrew do, too, in the long run. Please do keep in mind that this is a hobby project here, and as a consequence, it may not move as fast as homebrew does.

The similarity towards fpm is that the cookbooks project will also attempt to allow you to interconvert the registered dataset into .deb or .rpm files or gobolinux files or snapcraft files - or even homebrew formulas. But that functionality is very limited (as of in 2018). It will definitely become better over time. The slackware support is a bit better since I myself mostly use slackware and GoboLinux, but the slackware support in cookbooks/rbt still requires quite some more polish and clean-up before it can be easily used.

The RBT project is in some ways similar to the LFS/BLFS project in that it attempts to put together information that would be useful for when a user wishes to compile some program from source, in particular when these programs are requiring modifications to their build system, such as when "sed" is used - but the RBT project could also be used when installing a binary program. In doing so, the RBT project can also be used as a basic building block for creating package managers or other compile-tools in Ruby. If you have a project like this and may want to make use RBT, and require some modification, let me know - I will try to adapt the project to make them useful for these cases as well.

Now - let us have a look as to how information is stored in the RBT project.

The information for the respective programs is stored in yaml files by default, largely due to convenience and "historic reasons". Taken together, we could call the collection of yaml files a "yaml database" - even though it is not a real database, per se.

These individual .yml files, by default, reside in the subdirectory called yaml/individual_cookbooks/ of this gem. The .yml files usually also make use of shortcuts and abbreviations, which will be sanitized when the specific yaml file is loaded by the Cookbooks project.

For example, a common entry point may be like so:

"keep_extracted: t"

The "t" at the end is a shortcut for "true". The entry keep_extracted means that, after an archive is extracted, it will be kept rather than removed, after compilation has finished.

The Cookbooks project operates under the assumption that all archives that the user will use, are put into a certain base directory, which, for the purpose of this file here, will be called the archive directory.

This is the directory where your archives should reside. You can ultimately decide which directory that shall be.

On my home system, this archive directory is at /home/x/src/, mostly due to historic reasons how I kept making backups. On GoboLinux, you may want to use perhaps /Files/Archives/ or /Depot/Backup/ or something similar. You can query the current archive directory in use by issuing this command:

cookbooks --archive-directory?

There, source files can be stored, such as htop-2.0.2.tar.xz or ruby-2.4.1.tar.gz and so forth.

Since many other users may wish to use another base directory for their archives, you can define another archive directory on the commandline like so:

cookbooks --permanently-set-base-directory-to=/opt

The above would assume that all source archives will be stored within the /opt/ hierarchy.

Or use the more prevalent rbt name:

rbt --permanently-set-base-directory-to=/Mount/USB1/last_backup/src/
rbt --permanently-set-source-archive=/Mount/USB1/last_backup/src/
rbt --set-source-dir=/Mount/USB1/last_backup/src/

The above would assume that all source archives will be stored within the /Mount/USB1/last_backup/src/ hierarchy.

Since as of April 2019, the following shorter variant can also be used instead:

rbt --source-dir=/SRC

I added this primarily because typing the longer variants above, on a freshly set-up computer, is somewhat cumbersome. So the shorter variant should do fine in these cases.

Within that archive directory, subdirectories have to be created for the respective program at hand.

Actually, ruby may already do this for you, via the rbt project; but I recommend that you first start slowly and manually create directories for now.

So, pick just some program, anyone will do fine as long as it is registered in the RBT project. Create that archive directory if it does not yet exist, cd into it, create the directory of the program which is the program name (downcased, and without any '-' or '_' tokens).

Let's break this down and use specific examples, to illustrate what is meant - we wil pick the program "htop" as example.

First, create a subdirectory called "htop/", within the source archive directory that you picked.

On any linux/unix system, "mkdir" should be able to allow you to do that - but you could also use Ruby's fileutils class to do so, via FileUtils.mkdir_p().

Next, download the remote source for this program locally (you can also use the rbt project to do so for you: "rbt htop --download-source" but for the first steps, again, I recommend to do these steps manually instead).

wget http://hisham.hm/htop/releases/2.0.2/htop-2.0.2.tar.gz

More specifically, on my home system, I'd do the following things manually (but keep in mind that ruby does this for me automatically):

mkdir /home/x/src/htop
cd /home/x/src/htop
wget http://hisham.hm/htop/releases/2.0.2/htop-2.0.2.tar.gz

If you, for instance, want to store multiple versions of GLib, then a downcased directory called glib/ has to exist within that source archive directory - just as the example above shows.

Different versions of GLib can then be stored in that directory.

The path would, in my case, be:

/home/x/src/glib/

That's it, essentially!

Past this point, the cookbooks project can feedback some information about htop via "scookie".

For example, to use scookie (bin/scookie) and thus display information about a program in a colourized manner, on the commandline, do:

scookie glib
scookie htop

By default this will truncate too many entries (e. g. more than 5 entries for libraries, headers, and binaries). If you would rather not want scookie to truncate, you can pass the commandline flag --do-not-truncate, like so:

 scookie glibc --do-not-truncate
 scookie htop --do-not-truncate

Take note that you can enable this functionality also through bin/rbt (aka rbt on the commandline) as well, such as in the following examples:

rbt --scookie=glib
rbt --scookie=htop

The associated .yml file is always available anyway, with or without the archive being available locally as well. So you can always query the information stored in these .yml files the moment you install the rbt project; it can thus function as a weak "database". (I will at some point add SQL instructions in how to create a SQL database and use it with the RBT project. But note that the .yml way will most likely remain the default way; primary reason being out of convenience, since it is so trivial to edit a single text file such as .yml files.)

If you are wondering why the name "scookie" was picked - it originated from the sanitize_cookbook_dataset functionality, which ensures that the dataset is correct. I was using the abbreviation "scook" to refer to sanitize_cookbook_dataset, but I found that "scookie" is easier to remember. Everyone likes cookies, right?

The cookbooks project should now be usable to display information about this project - provided that it has been registered of course. If it is not yet registered, then someone has to register it; there are some helper scripts that can aid you there, but as a rule of thumb, I'll add new program entries when needed. They are essentially just a single .yml file.

Currently, there are more than 3000 programs that are registered in the Cookbooks project; almost 3200 already and this number is very slowly continuing to grow.

Cookbooks may also keep track of information such as whether a given progam at hand will install a .pc file (pkgconfig file). This may help if you are on a non-GoboLinux system and wish to find out which particular program installed which specific .pc file, too. (On GoboLinux this is trivial to find out since it keeps AppDir-related and versioned programs, whereas on the traditional FHS preferring /usr prefix, the individual .pc files would usually reside under /usr/lib/pkgconfig/, without always giving you a simple way to ind out as to where they came from.)

You can query this by using "scookie", which will show this, and more, in a colourized manner.

An example follows:

scookie libvorbis

If you wish to support this project, for now please just test it and report any bugs or report inadequate or confusing documentation.

The project is still in a beta stage and it will take quite a long while before it can leave that beta stage.

Obtaining the number of registered programs

You can obtain the number of registered programs from the commandline by issuing any of the following:

cookbooks --registered_programs?
cookbooks --n_programs?
cookbooks registered_programs?

The number of registered programs can be queried via the commandline as well, like in this way:

rbt --n_programs?

Although the output may be a little bit different.

This would then output something such as this:

→ 3330 programs registered as of 30 January 2018.
→ 3414 programs registered as of 18 September 2018.
→ 3635 programs registered as of 01 December 2019.
→ 3656 programs registered as of 12 January 2020.

(The format may be a bit different to the above, but the information content is the same.)

Optional dependencies for the RBT gem

The wget gem is recommended, but you can also use the RBT project without the wget gem. In that case, if the wget gem is not available, the RBT project will attempt to use the wget binary via a system()-call, hence an "external" program, to ruby.

This requires the installation of wget e. g. from this URL (or from your package manager's repository), unless wget is already available on your local computer system:

http://ftp.gnu.org/pub/gnu/wget/?C=M;O=D

Of course, most Linux systems will have "wget" available by default, so the above more applies towards operating systems such as windows (although more recent windows versions also include a linux-subsystem including bash, the WSL).

Increment Program Version

class RBT::Cookbooks::IncrementProgramVersion, stored in the file rbt/utility_scripts/increment_program_version.rb, can be used to automatically update a local program version to a newer version, provided that the new version can be "guessed" by the script.

Let's use a specific example, because it wil be easier to explain this when we can be specific.

Take the program called wget. Let's say that we wish to increment the program version of wget to a more recent one.

An older version, as a tarball release, may exist at a remote location such as this one here:

ftp://ftp.gnu.org/gnu/wget/wget-1.19.1.tar.gz

The (more recent) version in use may be for version 1.19.2.

class IncrementProgramVersion will first test whether a remote URL such as:

ftp://ftp.gnu.org/gnu/wget/wget-1.19.2.tar.gz

exists. If it does exist, it will be downloaded and the new program version will be changed towards wget 1.19.2.

This can be done for all programs, and it allows a simple update-mechanism for individual programs.

If you have to or want to batch-update several programs, this is also possible for some of the *nix stack. For example, there are some classes that can download the whole KDE5 stack; have a look at the KDE entry or the mate-desktop entry, in this (autogenerated) README.md file here.

Commandline ways to manipulate the dataset

There are some classes that can be used to manipulate the dataset stored in the individual cookbooks files (the corresponding .yml file). This is presently quite limited, though.

The only commandline-manipulation that is currently available, is done by class RBT::Cookbooks::ToggleKeepExtractedValue.

This acts as a toggle-switch. If you invoke the class, and pass the name of the program at hand, then the keep_extracted: value will be toggled, from false to true and from true to false.

Additional information

Consider having a look at the doc/ subdirectory of this project's gem, for some more information. Another useful subdirectory may be the lib/yaml/ subdirectory, which also contains some specifications such as specification_of_registered_cookbook_entries.yml.

Stable software versus Unstable software

This subsection here explains how the RBT project currently (in January 2019) views stable and unstable software releases upstream - that is, by other developers, companies and so forth.

From my own experience, unstable releases may bring in new features, but this often comes at a cost. Compilation errors may happen; bugs may happen while running the software. What is even worse is that other software may depend on functionality that may be changed during an unstable release. (While it may also be changed in a stable release eventually, stable releases in general often have problems fixed whereas unstable releases often do not.)

In particular the latter is one reason why I focus on stable releases - there will be fewer issues in the long run, thus easening the maintenance burden. That also means that the Cookbook projects tries to be moderately conservative - stable releases are favoured, unstable releases are often skipped. For example, I skip odd-versioned releases of the GNOME project and also of the KDE project. Waiting for stable releases just a little bit longer is worth the time, IMO.

This is a loose philosophy, not a strict policy, so expect some deviations - but in general, as a rule of thumb, there will be stable releases preferentially in the RBT project. (In the future we may have to change the cookbook entries to denote whether a release is stable or unstable, but for now, in the beginning of the year 2019, this is how the RBT project will handle updates of any its registered programs.)

The difference between "build tools" and a "package manager"

A package manager allows you to handle packages in one way or another; installing, removing, tweaking, changing, activating, selecting software. Things like that. As such, a package manager is very useful in itself - it can help keep your system up-to-date with new, or patched, packages.

Build tools in this context, though, are much more general. The above description of a package manager can easily fit into a set of build tools as well, but the scope of "build tools" is overall simply larger and more inclusive.

Build tools may also include helper code that can fix build-related problems, problems in Makefiles, act as sed-wrapper, and so on and so forth. A package manager on the other hand rarely needs to tamper with sed instructions, since that should have already been handled by the developers who maintain the packages for a particular distribution.

So the scope of the build tools is larger and not as narrowly defined as just a "mere package manager" alone. Another good comparison here is the ublock origin extension or the adblock plus extension - the latter wants to block only or primarily ads. The former allows you to act as a "general content blocker" under your control, which CAN include ads but can also block generic content, which makes it a lot more powerful and flexible in its approach.

I consider the approach taking up by ublock origin option to be far more useful to users in general, so this is in some ways how one could look at the difference between a "package manager" per se, and "build tools" on the other hand, too.

Build tools could easily include a package manager but a package manager will probably never include build-tools related helper scripts as its primary means to go about handling anything. The RBT project is going the "build tools" route - it attempts to collect code that can help with building, installing, compiling packages in general, but also automatic patching and automatic generation of files, and so forth.

If you wish to install the rbt project, via RubyGems, you can issue the following command:

gem install rbt

Do note that some functionality can work outside of the RBT::Cookbooks namespace, such as RBT::Libtool related actions.

Now, when the above installation method has worked - and if it has not, please do consider reporting this - you should be able to compile something via the RBT scripts.

Let's start with "htop" as example, a useful program written by one of the original creators of the GoboLinux project:

rbt htop

This should, if everything works fine, download htop, extract and compile it.

If you wish to use another prefix, such as:

/Programs/Htop/2.0.1

you can do this:

rbt htop --appdir
rbt htop non-traditional
rbt htop ntrad

ntrad stands for "non-traditional". Note that the entry is NOT hardcoded towards /Programs/ - you can define your own app-dir prefix and even make use of $SHELL variables. For example, I point to this directory via a custom variable called $MY_PROGRAMS, which is stored in the file programs_dir.yml that is part of the RBT project. More about this elsewhere though - for now just keep in mind that this will install into a versioned AppDir directory, but you can of course use any other variant such as "/pkg" or "/Programme" or "/opt" - whatever. Note that you can omit the trailing '/' there, so "/Programs/" is the very same as "/Programs" - the RBT scripts will internally make sure that this will be a directory anyway, and thus have a trailing / character.

AppDirs, for the purpose of this document, means that what belongs to one program, goes into the same, versioned directory.

You can also use another prefix, in particular /usr. This is called the "traditional" prefix. In the respective yaml file, the cookbook file at hand, this entry is called "prefix: ".

If you see a "t" there then this is short for "traditional", and it will be expanded towards "/usr" when used. Whereas the character "f" will be expanded to mean "non-traditional", thus AppDir prefix. Do note that you can also use other shortcuts there and absolute targets as well - the reason why single characters are used is largely to save me, but also others, some typing work. If you want to use the longer variants, you can do so just fine, but I assume that after some typing work, you may be bored of having to make use of long words and consider using abbreviations instead. :)

Anyway, here is the example to use /usr as prefix:

rbt htop --traditional
rbt htop --trad
rbt htop trad

Use any variant there. I prefer the last one, "trad", because it is shorter to type.

If you wish to look at the available help options that are available for rbt, do:

rbt --help

Note that this will showcase mostly just the more common ones. There are many more options that can be used.

There is a fairly large tutorial available as well. I have to publish it in .pdf format eventually.

Colourizing output

In general, --disable-colours or --disable-colours or --nocol will disable colour output. The rest of this subsection here explains some of the colour-settings used by RBT.

By default, RBT will attempt to colourize output from some external programs, in particular "make" and "make install".

The class RBT::ColourizeParser will handle most of this. It can also be used in standalone scripts such as ColourMake or ColourMakeInstall, which will run a coloured variant of "make" and "make install", respectively.

The idea behind the colours is to mostly make it easier to detect what is going on; and in particular where errors may occur. I personally found the colours to be helpful, but it is indeed VERY colourful, which is why a commandline option exists to disable the colours.

The colours are presently hardcoded, but in the future we could use different profiles and then let the user decide which (other) profile to use. But I'll extend this part only when other people may want to have this - for now, the default colour "profile" has to suffice.

If you wish to permanently disable the colours for the RBT project, then you may use the following command:

rbt --permanently-disable-colours

And to permanently enable the colours again, issue:

rbt --permanently-enable-colours

Note that there are also a few classes that can handle colours or colourizing. In particular, class ColourizeParser will take any external output (e. g. output that has been taken via system()-like calls by ruby) and colourizes it.

The general idea here is to make it visually easier to see which problem may have been encountered while compiling a program.

Users of RBT should be able to fully disable coloured output at any moment in time.

(Auto)Registering problems

The RBT suite attempts to identify errors, register them and sometimes report these problems. The main idea behind this is twofold:

(a) To be able to auto-correct reported (and registered)
    problems.

(b) To notify the user and optionally provide information
    as to why the error happens and how it could be fixed.

Ideally, autocorrecting would work to the extent that all problems that occur, could be automatically solved. But this is not trivial; while some problems can be quite easily autocorrect, others can not. And sometimes there is ambiguity so only the user can decide which variant is the correct one.

This is why the notification part is more important than the auto-correct part. See the next subsection for some autocorrection that can be done.

Autocorrecting some errors

The RBT scripts can attempt to auto-correct some errors. This is presently in testing. The few errors that RBT can try to auto-correct are mostly related to libtool.

If the file use_autofixing.yml contains a String called t or true, meaning "enable autofixing", then auto-correcting will be used. There is another option that has to be set to true though, which is the "is_on_roebe?" settings, which indicates my home directory use. Presently (Sep 2017) this option does not work too reliably, so I am still testing it. You can test it too though if you enable "is_on_roebe".

Finding all programs that start with a specific letter

If you wish to find all registered programs that start with a specific letter, such as k, do this:

rbt k*

This will output all programs that start with k and also tell you how many programs with that letter exist.

Cmake

Cmake support in the RBT scripts exists, to some extent.

If you want to compile specifically via cmake, then you can use the following instruction on the commandline:

--use-cmake

Note that the target build system has to support cmake. Auto-generating cmake files does not yet work for the RBT gem alone.

Using meson

You can use the build-system meson but keep in mind that this requires python version 3.x and the build tool called "ninja".

You can then add a dependency on meson in the individual cookbook file, at required_deps_on:, or you can use meson via the commandline switch --use-meson.

Usage example:

rbt glib --use-meson

Note that presently (2018) this does not work perfectly well, which may be understandable since meson/ninja is fairly new, compared to GNU autoconfigure-based systems.

Base subclass

The RBT project has a base subclass, residing at rbt/base/base.rb.

Most classes in the RBT namespace will inherit from this base class.

There is another class, called prototype.rb, which is pulled in by base.rb. The reason why prototype.rb was needed is to avoid circular dependency warnings - prototype.rb is much smaller than base.rb and provides mostly just the core functionality that is sufficiently useful for several classes.

Aliases for program names

The cookbooks project makes use of several aliases to program names. This is mostly a convenience feature.

Sometimes this may lead to unexpected or unwanted results, such as when you want to compile program a, but the rbt project renames the program to b instead.

In order to avoid this, a commandline-option exists for the RBT project.

Invoke it like so:

rbt sdlttf --do-not-use-cookbook-aliases
rbt sdlttf --do-not-use-cookbook-alias  

This will specifically disable making use of cookbook aliases. (That functionality only works if the Cookbooks gem is used. It resides within the source of the Cookbooks project rather than the RBT project.)

Show the last configure option

Since as of January 2018, a way exists to show the last configure option used for a given program. This depends on the file configure_command_database.yml.

If this file exists, and if the sought program has been compiled before (meaning that RBT has stored the configure command used in that .yml ile) - say, the program attica was compiled, then you can query this information from the commandline, via:

rbt attica --show-last-configure-option

This could then be used to copy/paste, or modify if you experience some problems related to compiling a program from source, for instance.

Of course you can also always peek into that .yml file on your own. Thankfully yaml is sufficiently readable, more so than big, clunky XML files.

Timing of the compilation (duration)

Since as of January 2018, code exists to tell us how long it took to compile a given program at hand.

This requires the configuration option time_compilation set to true it the corresponding yaml file (within the configuration/ subdirectory).

If this setting is set to true, then RBT::Compile will show, on the commandline, how long it took to compile a program, in seconds (and also in minutes). The default for this setting is true as of January 2018.

Note that the time it took to compile the given program at hand will only be shown if there was no proble/error during the compilation process. This may be fine-tuned at a later time, but for now, the functionality at the least exists as part of RBT::Compile.

The time it took, in seconds, will also be stored in the file called database_storing_compile_times.yml, in a simple key-value hash. (This file will be generated into the rbt_logs/ subdirectory.)

If you compiled several programs via rbt, you can also show how long it took to compile these programs.

The commandline option for this is:

rbt --show-compile-time-statistics

systemd

I personally do not like systemd in the slightest nor can I align with the, in my opinion, absolutely unnecessary and constantly changing goals of the ever-expanding systemd project or the attitude of the main systemd developers. It seems as if systemd's primary job is not to solve any user-related problems, but instead to be of the main benefit to those who wrote, created and maintain it, primarily.

This subsection here is, however had, not about projecting my personal opinions about systemd onto others - this subsection is how systemd relates to RBT, if it does so at all.

The RBT project is "apolitical", that is - it is for the people who use it, the users. If users want to use systemd then this is fine. And if users do not want to use systemd then this is fine as well.

Note that the LFS (linux from scratch) project follows a similar philosophy. It has a non-systemd and a systemd variant, so you can pick either variant at your own discretion.

Compiling and installing programs should work at every moment in time, as far as the RBT project itself is concerned. At the least RBT will attempt to make things work and aid a user in precisely that way.

However had, the RBT scripts are predominantly tested on non-systemd linux systems, so it may be that parts of RBT have to be changed/adjusted if you are using the RBT project on a systemd-dependent setup.

I will accept bug fixes and behaviour changes in regards to compliance with systemd systems IF non-systemd systems can continue to work just fine as-is. That is, changes may not happen in an exclusive manner - changes that do will be rejected. They will of course be accepted if it does not break this assumption and if they are in the spirit of the RBT project itself.

The primary reason for this here does not relate to my personal opinion either, but the simple question of the amount of time that is available to me, which I invest into various projects already. (RBT is only one project among many more; and that's just computer-related projects. I have many non-computer related projects, hobbies and things that require time investment.)

Creating a .yml yaml file for a specific program

If you modify a .yml file, from the rbt project, and then wish to autogenerate a new expanded cookbook dataset, then you can use this commandline switch from within rbt to do so, too:

rbt --create-yaml-file-for=subversion

This requires the cookbooks gem to be installed.

You can do so from the rbt project already as-is, so why was this functionality added to rbt?

Mostly out of convenience. I feel that it may be simpler to make available functionality that is somewhat common between the two projects (cookbooks and rbt) from rbt itself too. After all, rbt is the project that sort of builds upon - and thus extends - the cookbooks project.

Issues and Problems with Compiling in general

This subsection is not extensive; it only provides some information for somewhat advanced users. In other words, for those people who already have compiled some programs more or less successfully.

Some programs may fail due to various reasons, some of which may be hard to debug.

For example, perhaps you may run into a "bad file descriptor" problem one day. I did so in March 2018. The fix for me here was to change the $PATH setting.

You can modify the $PATH setting for rbt too at runtime.

For example, to prioritize on /usr/bin/, you can use this commandline switch:

rbt --traditional-path

Of course you can also change the $PATH variable on your own permanently - environment variables are in general available in ENV, in ruby files.

RBT_SOURCE_DIRECTORY

Since as of April 2018, there is a way to specify another target for the SRC_DIRECTORY, that is - the directory under which all other local source-archives reside.

On my home system, the source directory resides under /home/x/src/.

If there is an environment variable called RBT_SOURCE_DIRECTORY defined on your system, then this is the variable that we will use across RBT for the source directory.

If there is no such variable then, by default, we will use the setting specified by the Cookbooks project, if it is available.

Do note that the variable RBT_SOURCE_DIRECTORY will overrule every other setting, except for values set from the commandline to RBT::Compile, such as via --use-this-source-directory= - so use this variable with care. If you do not need it, I recommend to simply not set it at all; it really only exists in the event that there are no better alternative ways, or because the alternative ways may be too cumbersome to use.

If you wish to find out the current source directory used by RBT then you can use either one of the following commandline flags:

rbt --source-dir?
rbt --source-archive?

Prefixes to use

The most common (default) prefix on most linux systems will be /usr/ or /usr/local/.

That is, if a program is compiled from source, and no explicit --prefix is used, then most programs will default towards /usr/local/ as prefix.

Sometimes /opt/ is also used as prefix; and sometimes we can use versioned AppDirs (Application Directories) or compile into the home directory. The latter two ways were what the GoboLinux creators used on university clusters.

Some convenient shortcuts exist here, in order to make working with prefixes simpler:

ry m4 ULOCAL

This will use the --prefix=/usr/local/ setting, so it is exactly the same as if you would have been issuing this instruction:

ry m4 --prefix=/usr/local/

So the only real benefit of using upcased ULOCAL is that you save a few keystrokes - which is a real benefit for people who like to type as little as possible. :-)

Do note that if the user provides a specific --prefix= setting on the commandline then this setting will overrule every other means of setting the target prefix at hand. The reason for this is that RBT assumes of the user to know what he or she wants to do - thus, the commandline must be useful for overruling other settings. User decisions have a higher priority than other settings.

Let's look at another example:

ry m4 --opt_prefix

This will use --prefix=/opt/.

If you want to, you can also compile into your $HOME directory and build up an AppDir layout; this is how GoboLinux has been created actually, a long time ago:

ry htop --appdir-into-home-dir

If you only wish to compile into the home directory, without using the AppDir approach, then you can use the following invocation:

ry htop --compile-into-home-dir

Do note that since as of the 26.01.2019 (26th January 2019) you can also use RBT::CompileIntoHomeDir.new if you wish to compile into your home directory. There is also a bin/ executable available, called home_dir, so if you wish to compile htop into your home dir, via proper prefix, you could use:

home_dir htop

This is functionally equivalent to:

ry htop --prefix=/root/

if you are the superuser; and if you are a regular user then of course it will use your default HOME_DIR setting (usually somewhere under /home/).

Note that the last prefix that was used for any given compilation, is always stored on the module RBT "namespace", that is, the toplevel RBT.

You can query its value through the following ruby code:

RBT.last_prefix_used?

This will either return nil, if no prefix was used yet (which is the case on startup of the RBT project, for example), or it will be a String that points to the last prefix that was in use. That way, if other projects want to find out what prefix was used in the RBT project, they can do so through using the above method invocation.

Do note that using a --prefix setting is not mandatory, and it can be disabled, for the current run, via --no-prefix or --noprefix.

Example:

ry htop --noprefix

Determine how many core/processors the computer has

If you wish to find out how many processors/core a target machine has, you can use any of the following two invocation examples:

 rbt --n-processors?
 rbt --n-cores?

The code for this resides in the method try_to_output_how_many_processors_this_computer_has, which is part of class RBT::Compile. (This statement is correct at the least for October 2018).

Compile/Installation shell

Since as of January 2018, code exists that allows us to batch-compile programs, via a shell-input. It also allows us to query installation options and scan for what is installed, where these programs are installed exactly as well, and so forth.

This is experimental - do not expect for this to work very well as of yet.

The future goal is that we can use this special shell to easily install/remove any given software on any given computer system at hand.

The namespace for this code currently resides under RBT::Compile::Shell.

For the available options of the compile-shell, input "help" for help (or just "h").

The general idea behind the compile-shell is that we can not only batch-compile, or easily compile one program after the other - but that we can also modify and control the compilation process via a shell-like interface, sort of like an interface to customize the target computer system at hand.

If you wish to change the prefix for compilation, this can also be done, e. g. via:

gmp trad

This would compile GMP with the traditional prefix aka /usr/ prefix.

If you want to find out which were the last n programs that were updated, say n = 35, then you can do this in the compile-shell interface:

last_update? 35

The idea is to also have the compile-shell as some kind of guide or tutor that can properly explain some cave-eats or problems and make some "semi-seducated guesses", in order to fix this. But this is a long-term goal - for now, the primary purpose of the compile-shell is to aid in batch-compiling programs.

Since as of March 2018, an autocompile-option exists in the shell. This is currently unfinished but in the future the idea is to allow us to automatically compile to the most recent program versions, without necessiting user input. This would then allow us to update a linux-system to the most current program versions - stay tuned for more improvements here in the future.

You can also edit the cookbooks, if you have the cookbooks gem installed, from within the compile-shell, via:

edit bluefish
edit vim
edit ruby

And so forth. This is only useful if you have the cookbooks gem, since all modifications will take place on the Cookbooks namespace there, rather than RBT.

You can query the main prefix in use via:

prefix?

To set a new global prefix, do something like this:

set_prefix=/usr/local

Updating all registered gems

You can batch-update all registered rubygems via:

rbt --update-all-gems

Note that this presently will only work if you use the Cookbooks project. I am open to extend this functionality to allow other projects to use another dataset, though.

The boolean constant ALSO_AUTOMATICALLY_INSTALL_THE_UPDATED_GEM, defined in class RBT::Compile, controls whether the updated gems will automatically be installed by the RBT project. If set to true then RBT will not only download the respective gem, but also install it. Since not everyone may want this, the constant handles this case for the time being - I personally like this constant set to true, since it saves me some keystrokes in the long run.

If you do not want to, or can not, update all registered .gem entries, yet you still want to install the .gem files that are registered, then you can use this commandline:

rbt --install-all-rubygems

This will install all registered .gem files, in an alphabetical manner. Dependencies will not be checked that way, but in principle if all .gem have been properly registered in the Cookbooks project, then this batch-gem installation should work just fine.

build_systems_priorities.yml

The file build_systems_priorities.yml simply determines which build systems are to be prioritized over others. The entry on top is the main one; then comes the entry at second position, then at third and so forth.

Some programs may come with more than one way to be compiled, such as some gnome-related software (they switched or are switching from GNU autoconfigure to meson).

The reason why the file build_systems_priorities.yml had to be created was so that we can favour one build tool over the other. Some configure-flags are not valid in every build tool or may have to be changed, which is why this yaml file had to be added.

ccache

There is a configuration setting, use_ccache, which, when enabled, attempts to make use of ccache for compilation.

This may speed up repeated compilation cycles of the same program quite efficiently.

If you do not need ccache or do not have it, you can disable the use of ccache permanently, as far as RBT is concerned, via:

rbt --permanently-disable-ccache

Alternatively you can also modify the file rbt/yaml/configuration/use_ccache.yml, which is where the configuration setting as to whether to use ccache or not, resides. The allowed values are true or false, yes and no (for all configuration values, by the way).

If you only want to disable ccache for the current run, then you can use any of the following variants (here we assume that the target program you wish to compile is fluxbox):

rbt fluxbox --disable-ccache 
rbt fluxbox --do-not-use-ccache
rbt fluxbox --no-ccache
rbt fluxbox --ccache-

Auto-resolve compile-related problems

Note that since as of September 2017, the RBT scripts can also give some advice how to resolve some compile-related problems.

This is currently fairly minimal, but it will be extended subsequently in the future. It was also a reason why the namespace Libtool has been integrated into RBT, as RBT::Libtool. This allows us to auto-correct some libtool-related problems and keep all libtool-related code within that namespace.

Available programs and the corresponding program version

Every time the cookbooks gem is updated, a .html file is created and uploaded at this location:

http://shevegen.square7.ch/programs_version.html

This remote .html file will show which programs are registered, part of the RBT project, and what the corresponding program versions are.

Note that you can also generate a .html page for some select programs, via the commandline. For instance, if you wish to show ruby, python and php, then you can use this commandline invocation:

cookbook --create-html-page-for-these-programs=ruby,python,php

The input program names should be separated via , and not include any '-' or '_' character. There are also some ways to display the full, e. g. KDE stack. More on this in another section of this README file.

Checking for binary duplicates

If you ever want to find out whether you have binary duplicates at two different locations, such as at /System/Index/bin/ and at the more traditional /usr/bin/ location, then you can use the following method:

RBT.check_for_binary_duplicates()

The code to this method resides in the file rbt/toplevel_methods/check_for_binary_duplicates.rb.

You can also invoke this from the commandline, of course.

Usage examples for the commandline invocation:

rbt --check-for-binary-duplicates
rbt --binary-duplicates
rbt --report-binary-duplicates
rbt --report-duplicates

Future Goals

The RBT Cookbooks project may not necessarily have truly grand goals as the scope is quite limited.

The primary project's goal is to gather information about source code, and to present and make use of that data to other programs or suites, be it on the commandline or via a GUI or via the WWW. Thus, the project has to be a practical project rather than a visionnairy one.

But there are some future goals nonetheless.

  • One goal is to translate the instructions stored in the various yaml files, into specific SQL instructions which can be used to populate a SQL table - this is currently (October 2017) not working, but stay tuned; it will eventually work and be properly detailed and documented. The idea behind this is to be able to store all the data stored in the various .yml files, into a single database instead.

  • Another goal is to use the project in order to create a full LFS/BLFS build, in combination with the RBT project.

  • And yet another goal is, similar to the point above, to turn an existing Linux-system into a Linux-system based on an AppDir approach. Note that this is also covered by the RBT project primarily, and it does not work yet (Jan 2018). That means that rather than having files spread all over the place on a system, they will be contained in the respective directory of the given program at hand.

Stay tuned until these goals have been reached.

Compile hooks

Compile hooks are defined in the rbt gem, in the file compile_hooks.yml

In that file you can hook the compilation of some programs to the compilation of other programs.

This is somewhat similar to the chained programs.

For example, if you compile ffmpeg, and have mpv installed, then a new compilation of ffmpeg will trigger the re-compilation of mpv. This will only be done if you have mpv installed at $PROGRAMS/Mpv though.

Be careful with this, though - it can lead to problems, in particular when compilation may fail. I do not know whether I will keep this functionality (as of the year 2018); the future will tell.

For now I recommend to use this feature only sparingly, if at all. Manual control over the process may be simpler in the long run.

Advanced usage of the RBT project - helpful information

This subsection here will explain how to use rbt on a linux system - at the least how I use it.

The subsection will also give some advice and it is intended largely for somewhat more advanced users that may already know a thing or two about ruby and linux systems.

The whole RBT project is somewhat inspired by the Linux from scratch project.

That is, people who may want to maintain their own system, the way they want to. This is also one focus that the RBT project has - to be able to compile any program and have things work as-is. The fpm project, for example, also follows that general philosophy to some extent as well, by the way; you can use fpm to create packages specific for a particular distribution.

So what can - or should - be done on a Linux system?

Well - this of course depends on what YOU want to have and need, and what kind of base system you may want to use. Perhaps you have full superuser access - but perhaps you may be in a more restricted cluster-environment where you only have full access to your home directory; the latter may be the case if you work on an university campus site. Either way, the RBT project attempts to be useful in both cases.

Let's first start with a linux desktop system, since that is the one that I am using most of the time.

I would recommend that you first compile your own version of a core linux software stack. This will help in bootstrapping immensely.

The following command should suffice - but review this list on your own before you use it; as a first step, it may be simpler to compile step-by-step, rather than all in one giant step.

rbt make sed awk coreutils utillinux gawk m4

(Note that the above may no longer work; you may have to be more explicit like so:)

rbt --compile-these-programs=make,sed,awk,coreutils,utillinux,gawk,m4

I recommend that, if possible, compile as many of these programs as a static binary. That way, if you somehow break a dependent shared library the tools will still continue to work.

Having the application called busybox installed, in a static manner, helps too - more on this on a separate section.

At any rate - once you have that set going, you should try to compile your own version of GCC, ideally a recent GCC if that is possible.

This probably takes longer than the steps above and may require a few more programs on top of that. Make sure to have enabled C and C++ as languages for GCC, and perhaps a few more languages, should you need them (the programming language R requires fortran, I believe, as example).

Invocation examples:

rbt mpfr
rbt gmp
rbt mpcr
rbt gcc

Keep in mind that you can also use the LFS/BLFS instructions. They are also available via the "blfs" entry in the respective .yml cookbook file.

Glibc is obviously also a target candidate that could/should be upgraded eventually, but this is a rather delicate step - many programs have to be recompiled if your glibc version changed.

My best advice here is to compile glibc into a versioned AppDir (directory, such as /Programs/Glibc/2.30/) but to NOT symlink it yet.

An invocation example follows here:

rbt glibc ntrad --do-not-symlink

Or use another --prefix, such as /opt/glibc/2.30 or just /opt/glibc or your home directory.

Since different people may want to tell RBT to compile a subsystem for linux, a commandline option exists to allow us to compile the core programs here.

Invoke it like so:

rbt --compile-base-system
rbt --compile-basic-system

The exact programs that will be compiled may change from time to time. If you need a commandline result of the programs that will be compiled that way, simply append a '?' character to the above way to compile the base system, such as:

 rbt --compile-base-system?
 rbt --compile-basic-system?

If everything works here then you should have several "linux" programs under the /Programs/ hierarchy installed.

Of course /Programs/ can be changed to any other prefix that you may want to use.

Build directories

Some progams insist on using a build directory that is separate from the directory where the source code is kept.

For example, the KDE program called kirigami2 requires an "out-of-source" build directory.

There is a relevant entry in the cookbooks .yml files that can be used, called use_build_directory. If this entry is set to true then the RBT project will make use of a build directory for compiling a particular program.

You can also get an overview over the programs that make use of a build directory, by using any of the following APIs on the commandline:

ry --show-all-programs-that-make-use-of-a-build-directory
ry --all-build-directory
ry --all-build-directories
ry --build-directories?
ry --build-directories

RBT project will install these executables

The RBT project will install the following executables in the bin/ directory:

autosymlink
autoupdate_this_program
binary_of
blfs
build_detector
colour_make
colour_make_install
compile_these_programs
compile_via_appdir_prefix_without_symlinking
cookbooks
create_app_dir_skeleton
create_new_cookbook
create_pkgconfig_file
extra_information
find_alternative_archive
home_dir
home_dir_without_symlinking
homepage
ntrad
parse_help
query_file_association
rbt
rbt_config
rbt_download
rbt_make
rbt_test_alias
remove_symlinks
report_total_size_of_all_archives
ruby_libtool
scookie
search_for_tags
show_all_about
show_compile_chain
show_configuration_options
suggest_cookbook_for
symlink_this_program
to_current
toolchain
update_entry
update_kde_applications
url_action

Mate-desktop

You can try to update some of the mate-desktop components.

In order to do this, call either of the following variants.

cookbooks --update-mate
cookbooks --update-mate-desktop
cookbooks --check-for-mate-updates
cookbooks --check-for-mate-desktop-updates
cookbooks --check-for-mate-desktop
cookbooks --check-for-matedesktop

The first variant is the shortest, whereas the second one is the main entry point in the case/when setup.

This instruction will check for new updates to the mate-desktop releases and download them when found. Note that presently, any more recent version is preferred, so unstable versions will often be downloaded (since e. g. 1.19.1 is a higher version than e. g. 1.18.0).

Similar to the various KDE components (look at the KDE entry in this file), if you wish to compile the mate-desktop components then you can also use a simplified scheme, using "mate1", "mate2" and so forth, in linear order. This will be replaced with the registered name of the mate-desktop component.

Invocation example:

rbt mate1
rbt mate2
rbt mate3
rbt mate4

This is the very same as doing e. g.:

rbt mate-desktop
rbt libmatemixer

and so on. Using numbers may be simpler, though, as you do not have to remember the names as-is, while still being able to use the correct compilation order.

GNOME

The gnome-project tarball archives can be found here:

http://ftp.gnome.org/pub/gnome/sources/?C=M;O=D

The releases usually have even numbers for stable releases; and odd numbers for unstable releases.

I myself usually stayed on the "bleeding edge", that is to use unstable releases - but after some time, I now think that the stable releases are better, since you will have less problems or errors in the long run. Every time I encountered a problem related to an unstable release, I ended up investing time trying to fix the problem, which often was not possible. In the end I did waste too much time for too little gain, so I decided to stop making use of unstable gnome releases.

I added some code to the class that automatically downloads gnome-packages, to only consider even numbers. But this behaviour can be toggled via a commandline switch on the commandline, anyway - so the subsection here just explains as to why I personally favour stable gnome releases.

You can also compile the gnome components in an ordered fashion, by making use of numbers. (This also works for kde, xfce, xorg, mate-desktop, games and the like.)

Examples:

rbt gnome1
rbt gnome2
rbt gnome3
rbt gnome4

And so forth.

To batch compile most of these components, try:

rbt gnome1..200

Module programs (the extended cookbook dataset)

Module programs refers to the ability to access all registered programs at the toplevel "RBT." namespace. In other words, we can "copy" the program names onto the RBT top-level module "namespace".

For instance, htop would be available at

RBT.htop

if it is called as a "module program".

Note that this is NOT the default. By default the above way to call the method .htop() on RBT would not work.

The program binutils would, as module program, be available at:

RBT.binutils

And so on, and so forth. No "-" characters are allowed there, as these are not allowed in methods written in ruby.

Note that this will return a sanitized Hash that holds all the information required to compile a program from source - this dataset thus describes these programs. That information may still have to be interpreted by a program, of course.

Simply require the necessary file for this:

require 'rbt/module_programs'

After that, the above methods will work. Let's see a few more examples for this:

RBT.wayland
RBT.ruby
RBT.php
RBT.gnomemimedata
RBT.evince

Note that reader methods via '?' character are available too, although this could be toggled. The '?' will access the datastructure as a Hash rather than a specialized object.

Invoke it like so:

RBT.htop? # => {"archive_size"=>323560, "short_description"=>"ncurses-based interactive process viewer.", "github"=>nil, "set_env_variables"=>{"LDFLAGS"=>"-ltinfow"}, "homepage"=>"http://hisham.hm/htop/", "binaries"=>["htop"], "headers"=>[], "libraries"=>[], "postinstall"=>[], "pre_make_commands"=>[], "required_deps_on"=>["glibc => 2.3.2", "ncurses => 5.6"], "tags"=>[], "pkgconfig_files"=>[], "program_path"=>"/home/x/SRC/htop/htop-2.0.2.tar.xz", "program_full_name"=>"htop-2.0.2.tar.xz" ... and so forth }

This functionality exists primarily because it may be easier for you to use the above API calls. Internally though, the RBT project will prefer API calls such as RBT::Cookbooks::Cookbook.new :htop instead.

Note that the above works without any '-' as part of the name, so there will not be RBT.gnome-mime-data() - it would be RBT.gnomemimedata().

When may you want to use this? Well, it could be used if you do want to use a simpler API. It is very easy to remember, too: just use "RBT." followed by the name of the program, without any '-' or '_' characters there.

Showing the last 750 updated programs

You can show the last updated programs, currently, by default, being 750 programs, like this:

rbt --show-last-updated-programs
rbt --slu

Auto-removing .la files

If you do not like .la files (libtool files), you can set the configure option called delete_libtool_files to true. Then, if a program has been successfully compiled into an AppDir prefix, such as /Programs/Libcanberra/0.30/ and at the least one .la file has been found under the lib/ subdirectory, then these .la files will be removed. That is, deleted.

That way .la files do not taint the given host system any longer (or at the least will be removed after having been created).

If, for some reason, you rather want to NOT remove .la files for the current compilation run, you can always use the following commandline flag:

--keep-la-files

You can also purge all .la files manually, via the commandline.

class RBT::Libtool::RemoveLibtoolFiles is able to do. Simply pass to it the path of the directory that contains these .la files, and that class will eliminate all these .la files. (Be sure to know exactly what you are doing before doing so on a directory such as /usr/lib/ - if in doubt, always keep full backups of directories like this.)

During normal compilation, class RBT::Libtool::RemoveLibtoolFiles is invoked if you compile a program with an AppDir prefix into something such as the /Programs/ hierarchy, as value to --prefix.

Informally I myself call this class a standalone libtool killer.

If you want to use a top-level API instead, consider the following variant:

RBT.remove_libtool_files_from(this_directory = '')
RBT.remove_libtool_files_from()

Again, pass in the path to the local directory that contains these .la files.

Since as of September 2018, the last faulty .la file will also be registered in a local .md file. You can then let the RBT::Libtool namespace fix the last faulty .la file via:

 rubylibtool --remove-from-file
 rubylibtool --remove-from-stored-file

The reason why this exists is because I sometimes found myself without a running xorg-server. Copy/pasting then was difficult with the mouse (and I dislike gpm). So I wanted a way how to fix these faulty .la files without requiring the mouse - thus the idea was to store the last faulty .la file into a local file). The usual location for this would be at /Depot/Temp/rbt/libtool/ but this of course depends on where your temp-directory is set at.

You can also use rbt directly to remove a faulty .la file, by passing in the path, such as:

rbt /usr/lib/libpango-1.0.la

Working with (debian) .deb files

This subsection has been added mostly because one goal of the RBT project is to handle .deb files in one way or another - so I wanted to collect useful information in one place.

If you wish to create .deb files on your own, you may need the dpkg-dev package. The following syntax should allow for this:

apt-get install dpkg-dev

apt will handle the necessary dependencies.

Sed operations

The binary called sed is well known - and often used - on *nix operating systems. The primary reason for this is, most likely, that sed is so useful and versatile. You can replace text very efficiently via sed.

If you look at the LFS/BLFS homepage (the "Linux from Scratch" or the "Beyond Linux from Scratch" homepage), sed is often used to change settings in various programs, before compilation begins.

The RBT project also supports sed-related changes to programs. The sed-operations have to be defined in the individual yaml file that stores the instructions in how to compile a program from source, which is part of the Cookbooks gem.

So for example, the yaml file for a given program simply has a tag called sed:, which specifies an Array, and also it specifies which sed operations are to be run, in a linear order. The first entry is processed, then the second entry, and so forth (if there are several entries that is).

RBT will then apply these sed-modifications by default.

If you do not need this for a given program then you can disable this behaviour on the commandline, via:

rbt htop --disable-sed
rbt htop --disable-sed-modifications

Note that there are essentially two ways as to how to use sed-operations, as far as RBT is concerned. One is by simply using the binary called "sed"; and the other is to use class SedWrapper. That class is unfortunately a little buggy as of February 2018 (and beyond) for some sed-operations.

So for the time being, RBT will default to the system binary called "sed". If this is not available, then RBT may default to class SedWrapper, which is a pure-ruby variant.

Most Linux systems will have a sed binary, so this should work fine on these systems; should also work fine on Windows under the WSL subsystem these days.

If you want to query whether the internal sed wrapper is used in general, as-is, then use the following commandline option:

rbt --use-the-internal-sed-wrapper?

The sed-operations are run by class RBT::ApplySedOperations. This class can also be used standalone, that is, from the commandline. Or you can invoke them from within rbt itself.

Example:

rbt htop --only-sed # only apply the sed-operations, then exit.

If you wish to simply run the sed-operations registered in a cookbook .yml file, then you can invoke:

RBT::ApplySedOperations.new
RBT::ApplySedOperations.new :glibc
RBT::ApplySedOperations.new :htop
RBT::ApplySedOperations.new 'coreutils'

Compiling a random program

If, for whatever reason, want to compile a random program, then you can use a commandline switch such as the following:

rbt --random-program

Upgrading programs through a file

You can read in input from a file, aka programs that have to be compiled, stored in a file.

The default name for this file will be these_programs_can_be_upgraded.yml, but you can also use another name. The file has to exist, obviously.

Invocation examples:

rbt --upgrade-from-this-file=these_programs_can_be_upgraded.yml
rbt --upgrade-from-this-file=:default
rbt --upgrade-from-this-file=:def

:default or :def will refer to the file these_programs_can_be_upgraded.yml.

The reason why a default filename exists is so that other projects can auto-generate such a file, which we can then use for RBT.

Compiling the dependencies of a program as well

You can compile the dependencies of a program if you want to, via a commandline such as:

ry kbreakout --compile-dependencies-as-well
ry kbreakout --with-deps

Note that this will try to compile all the dependencies AND the program as well, with the dependencies first, and the program that was given as argument coming last.

This presently (October 2018) works in a brute-force manner, meaning that it will compile the dependencies even if they are already installed.

In the future, this behaviour may change, but for the time being, that has to suffice (the feature is thus somewhat experimental right now; it has to be tested further).

Internals of class RBT::Compile

class RBT::Compile is the main class of the RBT project, as it allows us to compile or install a program from source. It is, more or less, the central entry point to RBT - even though you can invoke the other .rb files directly, RBT::Compile often provides a simpler convenience access to the disparate functions of the RBT project.

This subsection here will explain a few key decisions made, in the long run; and will also explain a few internals.

Do not worry if this may be a bit confusing at first - it is meant primarily for me, and for others who may want to extend the RBT project one day.

The method set_configure_base_directory(), defined in the file misc.rb, is used to determine the directory where the configure script ought to reside. That way we can enter a separate build directory yet still be able to invoke configure (or cmake) at the right directory target.

Games support

Similar as to how it has been done for KDE and mate-desktop, you can use numbers to compile a particular game too.

For example:

rbt game3
rbt game5
rbt game6

And so forth.

You can also compile all games, e. g. by doing this:

rbt --compile-this-tag=games

Or, even simpler:

rbt --compile-all-games

(And I also did set an alias at home, to invoke this via compile_all_games).

Compiling all programs that belong to a particular tag

Tags are registered in the individual .yml file of a program.

You can also batch-compile these programs, such as the instruction of "compile all programs that belong to the tag called game":

rbt --compile-this-tag=game
rbt --compile-this-tag=gnome

The tag that you wish to compile has to exist of course, as otherwise no program would be compiled.

The functionality exists mostly as a convenience feature, say if you wish to compile all registered games.

All binaries of every registered program

If you wish to determine the binaries of every registered program then you can use the following top-level API:

RBT.all_binaries?

This will return a Hash, where the keys constitute all binaries that belong to a particular program at hand.

For example, the two binaries xzcmp and xzcat both point to the main program xz (such as is available on an URL like https://tukaani.org/xz/xz-5.2.4.tar.xz). The Hash will store this association. Another example is the binary xtrapproto*, which will point towards the program called **xtra ( "xtrapproto"=>"xtrap")

How may this functionality be useful?

This depends; if you ever want to find out to which particular program a given binary belongs too, you could try to use this API (or ask your package manager instead, of course).

You can also use the following top-level API to find out whether a binary is registered within the RBT project or whether it is not:

RBT.has_this_binary? 'xzcmp' # => true

This will return a boolean value - true if it is registered and false otherwise.

You can also gather some "statistical" information, such as through how many binaries are registered in the RBT project, via this method:

RBT.report_how_many_binaries_are_registered

If you only want to see which binaries are registered in all the registered programs of the RBT project, then you can use this commandline:

rbt --all-binaries?

The mate-desktop

If you wish to compile the components of the mate-desktop, you can try:

rbt --compile-mate-desktop

Note that this presently (July 2018) does not handle the dependencies correctly, so there is a very high chance of failure, unless you already have the dependencies installed on your system.

You can also compile all of the mate-desktop into a standalone directory, such as /Programs/Mate/1.12/.

Issue the following command for this:

ry --mate-desktop-into-standalone-dir

Purging traditional FHS binaries and FHS libraries

The RBT scripts support hybrid systems, that is, systems that both use the legacy/traditional prefix at /usr, and also GoboLinux-style AppDirs such as /Programs/PROGRAM_NAME/PROGRAM_VERSION or similar layouts. For example, php-7.2.1 would reside at /Programs/PHP/7.2.1/ (or /Programs/Php/7.2.1/ - all depending on which naming scheme is used precisely).

This may lead to having e. g. binaries duplicated, first at /usr/bin/, and then also in the respective AppDir bin/ directory.

If you then wish to get rid of the old /usr/bin/ binary of a specific program, you can issue the following commands:

 rbt --purge-traditional-binaries-from=xscreensaver
 rbt --purge-traditional-binaries-of=gnupg

The last entry is the program at hand, in these two examples xscreensaver or gnugp.

Be sure to really want to do so before invoking this functionality. It is mostly a convenience functionality, avoiding some manual removal of these binaries from the commandline.

A similar functionality exists for libraries. So if you want to purge libaries at /usr/lib/, you can do this:

 rbt --purge-traditional-libraries-from=libva
 rbt --purge-traditional-libraries-of=libva

The relevant entries for binaries and libraries can be found in the individual .yml cookbook - have a look at the binaries: and libraries: entries of the particular .yml file at hand.

Since as of November 2018, an additional way exists to remove binaries:

rbt poppler --purge-its-binaries

This will currently (November 2018) only remove binaries found under the /usr/bin/ hierarchy. In the future /usr/sbin/ may be scanned as well, but for the time being only /usr/bin/ is targeted.

Remember to do so only if you are sure that you want to do this.

Autoswitching python version

There is a configuration setting, stored in the file autoswitch_python.yml.

This can hold true or false (or yes or no). If set to true, then it means that RBT will attempt to autoswitch the python version, e. g. version 2 to version 3, if a program at compile-time depends on the other python version.

This allows us to try to recompile the program at hand.

This will be tried only once, though, in order to avoid infinite loops.

Do note that this functionality currently will only be used if the RBT scripts run on "roebe", which is my home system. This may also be enabled for all users one day in the future, but for now it is mostly experimental on my home system. (You can of course also enable "is on roebe"; I use an environment setting for this.)

The directory /System/Tags/ and registered tags in the RBT project

The directory at /System/Tags/ can be used to store tag-related binaries. Other entries, such as in a lib/ subdirectory, will be ignored.

For example, take the game called wesnoth, which may, as a binary, reside at a path such as /Programs/Wesnoth/Current/bin/wesnoth.

This binary may then be symlinked into a path such as /System/Tags/Games/wesnoth. Note that this resides under the /System/Tags/ hierarchy. This functionality thus effectively allows you to quickly look at which application types you may have installed on your computer system. All compiled/installed games that have been registered with the tag "game" will then reside in the subdirectory called Games/. It provides an overview over the system and I think that this is of some use, since different tags are used to describe the various programs at hands.

If you need an overview over all registered tags, from the commandline, try to issue this command:

rbt --available-tags?

This only works for programs that are installed into the /Programs/ hierarchy for the time being. (Remember that /Programs/ is not hardcoded as such but can be changed by the user; it just happens to be /Programs/ on my system, but you could take any other name too, such as /pkg/ or /home/programs/ and so forth.)

class RBT::CleanupSystemTags can eliminate stray (non-existing) symlinks, and it is invoked from class RBT::Compile, so the directory at /System/Tags/ will be guaranteed to not have any stray symlinks whenever RBT::Compile is run. (This is checked in the post-installation part of class RBT::Compile.)

The class RegisteredTags can be used to register all available tags in a .yml "database".

Simply run this file to register every available tag into a yaml file, which we can then load if we are to search for registered tags, rather than relying on "grep" itself. Thus, searching for tags will now be available as well without grep - you have to run it once to register the available programs before you can use it.

The method try_to_find_an_alias_to_this_input_unless_the_program_exists()

The method try_to_find_an_alias_to_this_input_unless_the_program_exists() can be used to find an alias name to a given input.

For example, the input xorgser will be converted to xorgserver. This will help us compile programs, since abbreviations are available for all the registered programs in the RBT project.

Copying all headers to the current working directory

If you want to copy all .h files from, say, glibc, to the current working directory, then you could do it via this commandline flag:

rbt glibc --copy-headers-from=/usr/include

I needed this functionality so that I could copy a glibc version installed into the /usr or / prefix, into /Programs/Glibc/VERSION_NUMBER/include/.

There also exists a slightly simply API:

ry glibc --copy-headers
ry glibc --copy-headers-into-pwd
ry glibc --copy-headers-to-pwd

The two variants are not completely the same, though. The second variant will only copy .h files that reside in an AppDir-like prefix, so it is more likely to fail. It will, however had, never pick up any possibly incorrect .h files from /usr/include/.

Guessing the build-type of a source package

You can try to guess the build type of a source directory.

First, extract the source tarball, such as for zsh (the URL may be: http://www.zsh.org/pub/zsh-5.6.2.tar.xz).

Next, cd into that extracted directory and do:

rbt --guess-build-type

This will return a String (or Symbol), such as "configure" or "cmake" and so forth.

The functionality was needed so that RBT can make a guess when information is otherwise unavailable about a given program at hand (e. g. not yet registered in a .yml file).

Compile chain

Several programs require the compilation of prior, other programs. For example, KDE or KDE Plasma or mate-desktop.

If you want to find out which programs are part of a particular chain, you can try any of the following:

ry --show-chain=plasma
ry --show-compile-chain-of=mate

The first variant would show the proper compile chain that ought to be usable for KDE plasma; the second variant would show the proper compile-chain for the *mate-desktop.

Simply pass in the argument of the compile-chain that you seek. Note that the main class that handles this on the commandline, class RBT::ShowCompileChain, will do a little bit sanitizing in order to find proper and plausible compile-chains.

Note that you can also batch-compile various components that are listed that way, as they are registered in the file chained_programs.yml.

For example, to try to batch-compile the KDE5 applications framework, you can try the following command:

rbt --batch-compile=kde_apps
rbt --batch-compile=kde-plasma

Or, simpler:

rbt --kde-framework
rbt --compile-kde-framework

Keep in mind that the prior dependencies ought to have been installed already for this to work (that is, to have compiled successfully before).

Since as of July 2018, a few simpler shortcuts exist as well.

For example, to compile all programs of the KDE5 foundation, you can simply use the following command:

rbt --compile-kde-foundation

You can also manually specify which programs you may wish to batch-compile (that is, to compile them one after the other). The following example shows how this can be done:

rbt --compile-these-programs=make,sed,awk,coreutils,utillinux,gawk

This would specifically compile these programs in the given order, e. g. starting with make, over to sed, awk, coreutils, utillinux and finally gawk.

Since as of 25.12.2019 it is also possible to compile the first n programs of, for example, KDE.

For example:

ry --kde1-first=10
ry --kde1-first=20

This would compile the first 10 or first 20 programs of the KDE5 foundation stack. Why has this functionality been added?

I did not want to have to remember the names; instead, I just wanted to tell to RBT that I want to compile the first 10 or 20 etc.. programs of said stack.

The first part here, "kde1", is an alias to "KDE5 foundation".

More such aliases may be added in the future, but for now this has to suffice.

Everything about symlinking

This subsection contains some information about how symlinking is handled by the RBT project.

In general, the main functionality resides in the file rbt/toplevel_methods/symlink.rb. When it comes to class RBT::Compile then the file is rbt/compile/symlink.rb.

Certain settings in the cookbook .yml files can be used to do some trivial symlinking jobs, such as symlinking lib64/ to lib/. The boolean setting autosymlink_lib64 handles this. If you need such a symlink-job during post-installation, then set this entry to true in the respective .yml file, such as for the program called poppler and the associated poppler.yml file:

autosymlink_lib64: t

Note that presently (November 2018) this is only honoured if we compile in an AppDir-like manner.

If you wish to remove all symlinks under the /Programs/ hierarchy, or wherever you have your own particular equivalent of this main Application directory set to, then you can use the following commandline switch:

rbt --remove-all-symlinks
ry --remove-all-symlinks

(ry is an alias on my system towards rbt)

Why was this commandline flag added?

I needed to test the re-creation of symlinks, so I had to add the functionality to also remove all symlinks prior to testing this functionality.

Determine to which program a particular .h file in /usr/include/ or an executable in /usr/bin/ may belong to

If you ever do want to find out to which program a particular .h file (header file) belongs to, you can make use of the class RBT::QueryHeaderToPackage for this task.

A specific usage example in ruby follows:

RBT::QueryHeaderToPackage.new('/usr/include/')

You can also do so from the commandline, if your target is the /usr/include/ hierarchy.

Example:

rbt --query-headers

You can designate another target directory to be checked for .h files, via a commandline flag such as:

rbt --query-headers=/usr/local/include/

This functionality also works for binaries found under /usr/bin/, by issuing something like:

rbt --query-programs
rbt --query-programs?
rbt --attribute?

This is equivalent to the following ruby code:

RBT::QueryBinaryToPackage.new('/usr/bin/')

Exiting at the "make" or "make install" stage

If you wish to instantly exit when reaching the "make install" stage, you can use the following commandline for this:

ry htop --stop-at-make-install-stage

Something similar exists for the "make" stage:

ry htop --stop-at-make-stage

class RBT::SymlinkIntoUsrLibDirectory

class RBT::SymlinkIntoUsrLibDirectory can be used to symlink content into the /usr/lib/ directory. The name for this class is fairly unwieldy, but I needed the functionality in a specific .rb file that I could call from the commandline.

By default, RBT::Compile does not make use of this class, but you can enable it via any of the following commandline flags:

ry gtk+ --symlink-into-usr-lib-dir
ry gtk+ --symlink-into-lib-dir
ry gtk+ --symlink-lib-dir
ry gtk+ --symlink-libs

Why may this functionality be useful to have?

For practical purposes it may be better to use a hybrid linux system, e. g. one that has versioned AppDirs under /Programs/, but also keeps symlinks at /usr/lib/. A similar approach is used by the program stow, for example. Once you understand how stow works, it is quite easy to realize that this functionality is useful to be had, so RBT also has the same or at the least a similar functionality.

Furthermore you can also call this class from the commandline, by simply calling the .rb file at hand.

Batch compiling all locally existing archives of a given program at hand

Since as of 23.11.2018 (23rd November 2018) it is possible to batch-compile programs that reside locally, if they differ in their versions, into an AppDir-like prefix.

This sounds difficult, so let's show an example to make this easier to understand.

Say that you have a directory called /home/x/SRC/libsigc++/. In this directory you store source archives such as the following ones:

libsigc++-2.10.1.tar.xz
libsigc++-2.6.2.tar.xz
libsigc++-2.8.0.tar.xz
libsigc++-2.9.3.tar.xz
libsigc++-2.99.11.tar.xz

Now you may wish to compile all these, into their proper prefix, such as that libsigc++-2.10.1 will end up in /Programs/Libsigc++/2.10.1/ and so forth. In fact, this is precisely why I added this functionality. I needed to be able to compile different versions, including old versions, into the corresponding hierarchy under /Programs/.

The commandline for this is something like:

ry libsigc++ --compile-all-available-local-versions
ry libsigc++ --all-available-local-versions

This is a convenience feature.

Check for duplicate binaries

class RBT::CheckForDuplicateBinaries can check for binaries that are a duplicate. By default, this class will scan for the /usr/bin/ hierarchy, but you can specify other target directories. If a duplicate was found, the user is notified.

That way you, as the end user, can decide how to proceed - for example, I needed this class so I can find out which files I can remove on a hybrid system.

If you have aliased this class to check_for_duplicate_binaries then you can invoke it like so, with another target directory:

check_for_duplicate_binaries --dir=/usr/local/bin/
check_for_duplicate_binaries --pwd
check_for_duplicate_binaries PWD # Is the same as the above ^^^ variant.

If you only want to check for the current working directory, you can use the bin/rbt executable as well, via:

rbt --check-for-binary-duplicates
rbt --check-for-duplicates
rbt --duplicates-in-pwd?

Determine to which particular program a specific binary name belongs to

If you wish to find out to which program, e. g., the binary called "shar" belongs to, then you can query this from the commandline via:

ry --this-binary?=shar

Alternatively you can use this:

ry --binary=env

The output of these would be:

RBT::BinaryNameBelongsToWhichProgram: The binary called `shar` belongs to the program `sharutils`.
RBT::BinaryNameBelongsToWhichProgram: The binary called `env` belongs to the program `coreutils`.

That is the main purpose of class RBT::BinaryNameBelongsToWhichProgram.

Since as of June 2019, an executable exists under bin/ that can be used for this task, the name being binary_of - mostly for convenience.

Example equivalent to the two above commands:

binary_of shar
binary_of env

Binaries installed by a given program

Several programs (cookbooks) that are registered in this gem will install binaries/executables under a bin/ subdirectory. This will usually be at the prefix /usr/bin/ or at the prefix /usr/local/bin/, for most programs on a Linux system. How does the RBT project handle these executables?

In the rbt project, in the respective .yml files, there will be an (optional) entry called "binaries:", followed by an Array of binaries that this program will install.

So, for example, take the file called binutils.yml.

Binutils will, by default, install the following binaries/executables when you compile it from source:

addr2line
ar
as
c++filt
gprof
ld
nm
objcopy
objdump
ranlib
readelf
size
strings
strip
libiberty
libbfd
libopcodes

If you want to find out how many binaries are registered in the cookbooks project so far, in total, then you can invoke the cookbooks executable like in the following manner:

cookbooks --n-binaries?

The result will then be something like this:

RBT: There are 4102 registered binaries in all cookbook files.

Environment settings

Sometimes environment settings, stored in ENV as far as ruby itself is concerned, may interfere with compilation. For example, assigning to the env variable called TZ may cause a build to fail (TZ stands for TIMEZONE).

I consider this pretty awful for shells such as bash or zsh, but I did not design any of them.

What this means as far as the RBT project is concerned is that we may need a way to easily AVOID using any ENV variable that may be faulty.

The simplest way how to do this may be by not using any ENV variable, save for PATH (which has to work, as otherwise the compiler can not find anything).

Usage examples:

ry nss  --clear-envs ntrad
ry nspr --clear-envs ntrad

This would attempt to compile both nss and nspr without any environment variables (save for PATH).

Since as of December 2018, a new class called RBT::CompileViaEnvironmentVariableAsPrefix can be used to specifically compile into a prefix that is specified via an environment variable.

This allows me to then use something like this on the commandline:

env_prefix xfconf

And it will compile the XFCE xfconf component into the prefix specified by the environment variable called XFCE_PREFIX, which I happen to have set to a value such as /Programs/Xfce/4.12.0/.

How do I use the RBT project personally

How do I use the rbt project primarily, on my Linux systems?

When I compile a (new) program via RBT, such as by issuing "compile htop" or "rbt htop" or "ry htop" (I make use of aliases a lot), then the RBT-scripts will compile the program htop for me, as specified by the corresponding "htop.yml" file, which is part of the RBT project - within the Cookbooks namespace.

If I need to make a change to the compilation process, such as by using a different default prefix, then I simply edit the htop.yml file, change, add or remove the corresponding entry, and things will (should) work past this point - but I can also override the settings from the commandline, so I have all the flexibility required available.

In many ways I thus control most aspects via these individual .yml files. Other package managers do something similar if you look at them.

The "default settings" that I may use go into a specific .yml file, whereas the commandline switches allow me to specifically override or cherry-pick other settings, based on the machine that I am working with. For example, on GoboLinux the AppDir prefix makes a lot of sense. On slackware on the other hand I may decide to just use /usr as option given to --prefix=.

On traditional linux distributions, /usr as prefix is not the only possibility of course. Another popular way is to compile into the user's home directory. A commandline switch exists for the latter, called --home-directory. This is equivalent to whatever the user's home directory ought to be, such as /home/debug/.

This is useful in the sense that, ideally, I can get away with only passing the name of the program that is to be compiled, without needing to change anything. It allows me to be lazy - and just get things to "work", when they are compiled (if the compilation process itself will work).

Suggest the content of a cookbook .yml file

If you compile a program from source successfully, say glib, into an AppDir prefix such as /Programs/Glib/2.58.1/, then you may use class RBT::SuggestCookbookFor to display the content of a .yml file for glib. This is obviously not perfectly accurate, but may help to get things started when you wish to create your own .yml files.

Invocation examples from the commandline:

rbt --suggest-cookbook-for=glib
rbt --suggest-cookbook-for=htop
rbt --suggest-cookbook-for=pango

Note that this depends on these programs being installed in the programs_directory?, which on my system is equal to /Programs/. See elsewhere in this document how to set this value to some other target location.

Do note that class RBT::SuggestCookbookFor is also used to indicate (to me) if a specific .yml file needs to be updated, in regards to libraries. As of 28th December 2018 this is experimental; but I may need to extend this eventually, and also include headers and binaries in this. The win-win situation here would be that the .yml files that are part of the RBT project, may become better and more complete in the long run.

The clang compiler and LLVM support

LLVM offers the clang compiler. Compiling most programs should be possible via clang. The content of the configuration file use_this_compiler.yml tells RBT::Compile which compiler to favour. If clang is available, and the content of the file use_this_compiler.yml is clang, then clang will be used - otherwise RBT::Compile will default to gcc.

The environment variable RBT_USE_THIS_COMPILER can overrule the content of this yaml file if it is set, so you will always have a way to toggle the behaviour. (From within ruby code, you can modify it via ENV['RBT_USE_THIS_COMPILER'] = compiler_to_use_goes_in_here.)

To find out which compiler will be used you can use the following from the commandline:

rbt --compiler?

Colourizing configure --help related output

If you wish to use colours for whenever you do:

./configure --help

then this is available via the executable at bin/parse_help.

The class that is doing this is called RBT::ConfigureHelp.

Simply invoke it in a directory that can respond to a "./configure --help" output and you'll get some colourization.

This was added in January 2019, so expect this to not be quite perfect as of yet.

You can also give, as input, an archive, such as foobar.tar.xz; the class will extract this archive before doing a ./configure --help run.

Querying the host system

The host system can be queried from within RBT via:

rbt --host-system

Inferring the build system that a particular program uses

If you look at different programs, they may use GNU configure, meson, python setup.rb files, scons, cmake, waf and so forth - in short, a vast array of different build systems.

The toplevel method called RBT.infer_build_system() can be used to determine which build system is assumed, by RBT, to be used. It will return a one-word String such as meson or configure. Do note that this is NOT perfect - you should not absolutely rely on this method, but for most programs this should work fine.

The code for this resides in the file rbt/toplevel_methods/infer_build_system.rb.

Some projects may use more than one build system, e. g. harfbuzz would allow you to use either configure or meson. In this case, what should the method RBT.infer_build_system() return? It could return an Array of both build systems, or that method could just "pick" one variant and return it. The latter is presently (January 2019) the current default. A yaml file keeps track of which build system is prioritized by default.

The RBT project supports colours on the commandline.

By default, the scripts available within module RBT support colours on the commandline. Colours may help put proper focus on what is important and what is not.

If, for any reason, you do not want to use colours, you can disable them either via the configuration (in the .yml file), or use this toplevel API to disable colours:

RBT.disable_colours
RBT.disable_colors # ← This works as well, of course.

The above method-call will completely disable colours for the whole RBT namespace.

The configuration file that is used to determine whether to use colours is called use_colours.yml. Since not everyone may be able to modify the .yml file, depending on the host setup (such as in a restricted environment like at a university campus site), a commandline flag is also available:

rbt htop --disable-colours

Auto-updating libraries installed under a versioned AppDir

You can auto-update the "libraries:" entry in a given .yml file, but this is usually only useful when I do so. I update the libraries entry; and eventually I will upload a new version of RBT.

If you, for any reason, wish to try to do so too, you can do so via any of the following invokation ways:

rbt --update-libraries-of=poppler
rbt --update-libraries-for=poppler

This would then update the file called poppler.yml, after making a backup of the old poppler.yml file first. (Make sure to have set the environment variable called IS_ROEBE to a value of 1).

RBT::Cookbooks::CheckForInvalidEntriesInThisCookbook

class RBT::Cookbooks::CheckForInvalidEntriesInThisCookbook can be used to check that only valid entries are registered in a cookbook file.

It can be found under the subdirectory rbt/validation/.

The log directory

The log directory for RBT is the directory into which RBT can store various files. This directory will also be used for storing extract archives, so the name log directory is a bit of a misnomer - it simply is the main directory that RBT will use to keep stuff stored that may be useful for compilation, or created during a compilation/installation process.

On my home system this defaults to the directory /Depot/Temp/rbt/.

You can also generate this log directory from the commandline via either of the following:

rbt --create-log-dir
rbt --create-log-directory

Usefulness of this project

If you may wonder how this project may be useful to you, well - I think the main target audience will be fairly small, mostly only more experienced users, in particular linux users. These people often already know a lot about, not only linux, but also programming.

While I will continually improve the documentation so that novice users can understand what the RBT project is doing as well, to the point of being easily able to use the project, I think that there will not be many novice users who are even interested in such a project in the first place.

So the project should perhaps be seen as a "complementary" project to the LFS/BLFS project (Linux from Scratch etc..) - or for smaller linux distributions as an additional set of scripts that aid in managing a distribution.

At the very least, the programs found in this project have a description, so this alone can be of help to distribution-creators. And there are some classes that focus on validation the dataset, so that people who use the project can be reasonable sure that everything is in proper order.

Obtaining the main URL for a given program

The method RBT.return_url1_of_this_program() can be used to return, as String, the remote URL to a program, also called "url1". This entry usually denotes the path to a tarball/archive that can be downloaded from the commandline.

If you need to determine the URL from the commandline, you can try this:

rbt --show-url=readline
rbt --show-url=php
rbt --show-url=ruby
rbt --show-url=python

Report the status of the project

You can report the status of the project via:

rbt --report-status

ColourMake and ColourMakeInstall

If you wish to use some colours when doing "make" or "make install" then you can use bin/colour_make and bin/colour_make_install, like in this way:

colour_make
colour_make_install

You can also tap into this code from the commandline, through the bin/rbt interface, such as in this way:

rbt --colour_make
rbt --colour_make_install

Of course the slightly shorter variants work too:

rbt --colourmake
rbt --colourmakeinstall

And this as well:

rbt --colour-make
rbt --colour-make-install

RBT::ShowAllAbout

You can quickly show all about a given program at hand, via the class RBT::ShowAllAbout. I have aliased this class to show_all_about, so then I can do:

show_all_about ruby
show_all_about make

This will show the content of the .yml file at hand. In order for this to work, the Cookbooks project has to be available - and that particular program has to exist, too. Thus meaning, that a .yml file must exist, such as ruby.yml and so forth.

You can also invoke the above from the commandline, by issuing:

rbt --show-all-about=ruby
rbt --show-all-about=php

In these two cases, we would query the content of the two yaml files called ruby.yml and php.yml.

You can also use the current working directory as "input", via:

rbt --show-all-about-this-dir

For example, if the directory name is /Depot/Temp/xcb-util-xrm-1.0/, then calling rbt via the --show-all-about-this-dir flag will assume to be the same as if you would have done:

rbt --show-all-about=xcb-util-xrm

This allows you to type a little less (and not have to think about the name of the program at hand, since you are just using the working directory as the "input" / argument).

Extract to and the Temp directory

Archives such as foobar-1.0.tar.xz or barfoo-2.0.zip will first have to be extracted before they can be compiled. The RBT project thus needs to have this functionality; and you, as a prospective user, also need to have (some) control over this behaviour, in particular if you wish to extract to a specific directory.

The file temp_directory.yml, which is part of RBT's configuration, designates the main directory to which RBT will extract an archive.

For example, if the content of this yaml-file is /tmp/ then RBT would extract archives into that directory. (Actually, RBT will extract into the rbt/ subdirectory, in order to not make a mess of the target temp-directory; so a value of /tmp/ would really mean /tmp/rbt/ instead.)

It is permissible and possible to use $FOO variables inside of that yaml-file, to designate and make use of shell variables. For example, I may use the variable called $TEMP as the content of the file temp_directory.yml, which in turn points towards /Depot/Temp/ on my home system. This is the very same as if you would have used /Depot/Temp/ as the content of this yaml-file, so the only net-gain is that you may not have to hardcode the same path several times; instead, you can re-use shell variables that way.

If you want to extract into the home directory instead, then you can use any of the following commands to do so permanently:

ry --permanently-extract-to-home
ry --permanently-extract-to-home-dir
ry --extract-to-home

You can also specify a specific target directory that is used for the extraction step of a compilation that you are about to start.

Consider the following example:

ry htop ntrad --use-this-temp-dir=/opt/foobar_create_it 

This would use the directory /opt/foobar_create_it for extracting the source archive.

I needed this functionality in particular because I sometimes have to do two compilation runs at the same time, e. g. for different target prefixes - so it made sense to also use a different* **temp dir for extraction.

If you want to, or have to, you can also designate another temp-directory, permanently, via the following toplevel API:

RBT.permanently_set_temp_dir
RBT.permanently_set_temp_dir '/tmp/rbt/'

You can also quickly extract any local archive via this toplevel method:

RBT.extract()

The latter method can also be used on the commandline, through bin/rbt via:

rbt --extract htop

This will first download htop and then extract it, in the current working directory (cwd).

You can also quickly extract a locally existing archive, such as bison-3.2.4.tar.xz.

rbt --extract-this=bison-3.2.4.tar.xz

Another functionality is that you can change the directory as to where source archives will be extracted to, permanently, from the commandline. Normally a good directory for this may be /tmp/ but you can choose another directory, too. I tend to use /Depot/Temp/ as my temp-directory - to me it feels more clean than /tmp/ but your mileage may vary.

Examples:

rbt --permanently-set-temp-dir-to=/user/bioinf4/Temp/
rbt --permanently-set-temp-dir=/user/bioinf4/Temp/
rbt --permanently_extract_to=pwd
rbt --permanently_extract_to=/Depot
rbt --permanently_extract_to=/tmp
rbt --permanently-set-temp-dir=/user/bioinf4/Temp/
rbt --permanently-set-temp-dir-to=/user/bioinf4/Temp/
rbt --use-this-temp-dir=/Depot/
rbt --use-temp=/Depot

These are all equivalent.

If you wish to extract to, for example, the target location /opt/test just for the current run, then the following command should suffice:

ry htop --extract-to=/opt/test

Repackaging archives

You can quickly repackage archives, such as foobar-1.0.tar.gz into foobar-1.0.tar.xz, by issuing this command:

rbt --repackage=foobar-1.0.tar.gz

Running strip on binaries

You can invoke the command called "strip" on binaries (executables), from within rbt too.

A simple way to do so is to first navigate to a bin/ directory, and then invoke the following command:

rbt --strip-binaries

But be careful before running the above - this will really run "strip" on every file found in the current working directory.

Only do so when you are sure you want to invoke strip there.

You can also specifically strip only the binaries that belong to a certain program, such as through:

rbt --strip-binaries-of=htop

Batch-downloading all source archives

You can download all registered source archives by calling class RBT Cookbooks::DownloadAllSourceArchives.

I use an alias to call that .rb file. Then I can do, for example:

download_all_source_archives

Alternatively, you can also invoke the file from the commandline directly, by issuing either one of the following:

rbt --download-all-source-archives
rbt --downloadallsourcearchives
rbt --grab-source-archives
rbt --grabsourcearchives

From within Ruby code, you can use the following toplevel-API to download all source archives:

RBT.download_all_source_archives

You can also download only those files that have a certain tag registered. For example, if you want to download all programs that have the tag "plasma" attached, you can issue the following command (and argument) this:

download_all_source_archives --tags=plasma

In order for this to work, the tag has had to be registered (and thus, logically, exist).

If you need an overview over the existing tags, issue this command:

cookbooks --available-tags

Available utility scripts of the RBT project

There is a large collection of small scripts available in the subdirectory called rbt/utility_scripts/ - you can either go into that directory to have a look, or show a summary by calling:

RBT.show_utility_scripts

Each .rb file there should have a small header that explains what the respective .rb file will do. You may want to check these out.

Commandline examples for rbt

This subsection here has a few examples how I use rbt every now and then. I will try to expand this subsection with useful snippets in the long run, but they will not be explained in this subsection - see for other parts of the README.me file here for explanations, or use "rbt --help".

rbt gdkpixbuf --home-dir --use-meson

Show everything about a given program (show all)

There is a class called ShowAllAbout, which will simply show the whole .yml entry of a given program.

I aliased this on my system to "show_all", so then I can type:

show_all htop

And the content of the .yml file is shown. This is convenient because you can quickly view the content of the yaml file at hand.

Note that there exists another way as well, via scookie, but scookie does not necessarily show all information, whereas class ShowAllAbout will do so.

A commandline variant also exists via cookbooks itself:

cookbooks --show-all-about=htop
cookbooks --show-all=htop
cookbooks --show-file=htop

Of course this works through rbt as well:

rbt --show-all-about=htop
rbt --show-all=htop
rbt --show-file=htop

Postinstallation steps

If you compile a program from source, sometimes there may be post-installation steps that may have to occur after the program has been compiled. For example, if you wish to symlink cc to point to gcc, then we need to have this specified in the corresponding file called gcc.yml. The class that handles these postinstallation tasks is called class RBT::PostInstall.

You can invoke this class manually too, if you want to - I aliased the name "post_install" to point to the file in question. This I needed because I sometimes need to perform post-installation steps through the commandline, so it was simpler to have this a standalone, separate class.

Note that postinstallation steps can become quite complex, which is why RBT tries to simplify some of the steps by using shortcuts.

For example, say that you wish to copy several files to another location automatically after compilation has finished.

You can do so via the copy_files instruction. Let's show a specific example for doing so:

- copy_files { VERSION,assembly,common,eclipse,epub,epub3,extensions,fo,highlighting,html,htmlhelp,images,javahelp,lib,manpages,params,profiling,roundtrip,slides,template,tests,tools,webhelp,website,xhtml,xhtml-1_1,xhtml5 } /usr/share/xml/docbook/xsl-stylesheets-PROGRAM_VERSION

The leading ' - ' in the line above is for YAML and it denotes an entry that is to be kept as an Array. The rest of this Array is quite easy to understand. The instruction copy_files will simply copy several files; and these files are denoted within the square brackets { }. These entries can b a file or a directory and RBT::Compile will correctly handle these.

The very LAST entry is the target location that you wish to copy these files to, which in the above case is at /usr/share/xml/docbook/xsl-stylesheets-PROGRAM_VERSION.

Note that the upcased word PROGRAM_VERSION is considered a MACRO and will automatically be expanded to the current program version at hand, e. g. the one that is inferred or specified within the specific cookbook .yml file.

You can have as many postinstallation steps as you want to, in the specific .yml file. The entry that is important here is called postinstall.

Of course you can use more commands than "just" copy_files; another example would be the command create_pkgconfig_file, such as in this way.

postinstall:
- create_pkgconfig_file

This would automatically try to create a pkgconfig .pc file under the pkgconfig/ directory. The file libmad.yml needs this.

Some programs may require a post-install symlink step, such as vte. vte will install a file called bin/vte-2.91. It is easier to approach this file as bin/vte so a symlink option is used in the file vte.yml.

The corresponding entry in the postinstall section of that .yml file is any of the following two:

symlink_binary
symlink_this_binary

Have a look at the file called vte.yml for a complete example.

The $PATH variable

You can put /usr/bin/ at the beginning of $PATH, via:

rbt --traditional-path

This may be useful if you ever run into some PATH-related problems, such as when you mix different paths. I have had an issue with bash once, which is why I added the above flag to resolve this issue.

Of course other ways may be to simply change PATH in your shell, such as in bash:

export PATH="/usr/bin:$PATH"

This is probably the best, as in most reliable way. Remember that within ruby, PATH is available under:

ENV['PATH']

Autogen

Some programs require the program called "autogen". In the respective cookbook file, this is the corresponding entry called use_autogen.

If this entry is set to true, then RBT::Compile will invoke autogen for that particular program.

If you want to find out whether a program, such as "pacman", requires the use of autogen, you can use this query to find out on the commandline:

ry pacman -use-autogen?

The result may be output like this:

RBT::Compile: No, autogen will NOT be used for the program pacman.

Picking a specific program version

You can e. g. select a local version via:

ry python --2

This will only work if you have a python version available locally in the right directory, starting with the version number 2.

You can also use specific versions, such as in this way:

ry caja 1.21.4 --trad

This latter example would require a tarball archive, a file, such as caja-1.21.4.tar.xz, to exist in the main directory for caja/. I tend to have multiple versions for many programs stored locally, for added flexibility.

Snapcraft files

Since as of June 2018, the RBT project can also autogenerate the necessary information for an (ubuntu) snapcraft file.

The class that handles this file-creation is called RBT::Cookbooks::CreateSnapcraftFile. It can be found at the path rbt/utility_scripts/create_snapcraft_file.rb within the RBT gem.

Note that the functionality is currently very limited; not every aspect of a snapcraft file is presently supported. But if you only need a very simple snapcraft file, or want to auto-generate one and then improve on it manually, then this may be useful for you to make use of.

The argument to class RBT::Cookbooks::CreateSnapcraftFile should be the name of the program that you wish to generate a snapcraft.yml file for, such as "m4" or "php" or "ruby" and so forth.

There are essentially two different ways how to go about creating this snapcraft file:

(a) Either from within ruby code, you can do this:

require 'rbt/utility_scripts/create_snapcraft_file.rb'

RBT::Cookbooks::CreateSnapcraftFile.new('m4')

or, via a toplevel API:

RBT.create_snapcraft_file('m4')
RBT.snapcraft_file('m4')

or

(b)

from the commandline, do:

cookbooks --create-snapcraft-for=NAME_OF_THE_PROGRAM_HERE
cookbooks --create-snapcraft-for=ruby
cookbooks --create-snapcraft-for=m4
cookbooks --create-snapcraft-for=php
cookbooks --create_snapcraft_for=php,ruby,m4

The last variant allows you to use a ',' separated list for several programs.

The toplevel API exists mostly due to convenience; the commandline variant may be the simplest to do so, though.

If you wish to look at the official documentation for a snapcraft.yaml file, look here:

https://docs.snapcraft.io/snapcraft-yaml-reference/4276

Colours used by the RBT project

By default, the rbt project will use lots of different colours on the commandline.

These colours are usually used on a KDE Konsole or a mate-terminal, on my home system - and they work best with a black background in these terminals.

If the default colours are to your dislike or do not work with your given setup, you can always disable them via an option such as "--disable-colours" or "--disable-colors", if you prefer the US spelling.

Examples:

rbt htop --disable-colours
rbt htop --disable-colors

This would try to compile htop while disabling colour support for the current compilation-run.

More shortcuts exist as well, such as "nocolours" and so forth.

If you want to permanently disable the colours, use a commandline variant such as the following:

rbt --permanently-disable-colours

This will modify the file 'use_colours.yml' in the yaml/ subdirectory of the RBT project, which is the file that determines whether RBT will use any colours or not. By default, colours will be used.

In the future I may provide colour support for users who prefer a white or light background, but I would need a specific suggestion and a name for this colour "scheme". Then I can add more colours and even colour schemes for use here.

Specifying which build system to use

By default, most .yml files that are part of the RBT project will make use of the GNU configure build system - e. g. configure, make and make install. You can, on the commandline, determine which build system is to be used; and you can also instruct RBT::Compile to use another build system.

For example, if you wish to specifically use configure, then you can do this:

ry htop --use-this-build-system=configure

Or, if you want to use an even shorter variant, the following example is equivalent to the one issued above:

ry htop --use-configure

Similarly, for cmake or meson, you could use any of the following variants:

ry htop --use-this-build-system=cmake
ry htop --use-this-build-system=meson

The XFCE desktop environment

If you wish to compile all of XFCE into a single directory, such as /Programs/Xfce/4.12.0/, then you can issue the following command:

rbt --xfce-into-standalone-dir

Some aliases exist to the above, such as:

rbt --compile-xfce-components-into-one-standalone-directory
rbt --compile-xfce-components-into-a-standalone-directory
rbt --compile-xfce-components-into-standalone-directory
rbt --use-xfce-prefix

This functionality is experimental as of October 2018, and relies on a hardcoded path; but in the future, this functionality may be extended to more components than just XFCE, and not depend on a hardcoded path. When it is changed, this section will also be changed.

If you wish to compile all of XFCE into a standalone directory, but wish to do so in a step-like fashion e. g. in order to reduce errors, have a look at the subsection here in this page called Environment settings.

You can also batch-compile the XFCE components via:

ry --compile-xfce

Configure options saved

The configure options that were used in order to compile a particular program at hand, will be stored in the file called configure_command_database.yml, which can typically be found at a location such as:

/Depot/Temp/rbt/configure_command_database.yml

The entries in this yaml file may look like this:

minuet: cmake -DCMAKE_INSTALL_PREFIX=/usr/ /Depot/Temp/rbt/minuet-18.12.3/ 2>&1

So the format is - first comes the name of the program; followed by the particular command that was invoked.

Note that this will only be stored in that .yml file if no problem was encountered during compilation - thus, assumingly, would mean that the compilation was successful or at the least continued without major problems/errors encountered.

The primary reason as to why this functionality was added was so that the user can quickly look at that .yml file and see which configure option was used to compile a program. This may be useful at some later time when you wish to find out what went wrong (if something went wrong) or perhaps if you can not access the xorg-server but need to re-compile something.

LXQt

LXQt is a lightweight desktop environment based on Qt. It may be an alternative if KDE5 plasma is too heavy for a given computer system.

You can compile individual LXQt components, such as in this convenient way, through rbt:

rbt --lxqt1 # ← This refers to "libqtxdg"
rbt --lxqt2 # ← This refers to "lxqtbuildtools"
rbt --lxqt3 # ← This refers to "libsysstat"
rbt --lxqt4 # ← This refers to "liblxqt"
rbt --lxqt5 # ← This refers to "lxqtabout"
rbt --lxqt6 # ← This refers to "lxqtadmin"
rbt --lxqt7 # ← This refers to "lxqtconfig"
rbt --lxqt8 # ← This refers to "lxqtglobalkeys"
rbt --lxqt9 # ← This refers to "lxqtl10n"

and so forth.

Of course passing in the name, rather than these numbers, will also work just fine. The number-based input exists purely due to convenience.

Examples for the long names corresponding to --lxqt1, --lxqt2, --lxqt3, --lxqt4, would be:

rbt libqtxdg        # lxqt1
rbt lxqtbuildtools  # lxqt2
rbt libsysstat      # lxqt3
rbt liblxqt         # lxqt4

(See the content of the file called chained_programs.yml for the complete listing of all LXQt components.)

If you wish to compile all LXQt components, in the order specified by the file chained_programs.yml, then you can use this commandline instruction:

rbt --compile-lxqt

You can also manually update lxqt via the following top-level API from within ruby:

RBT.update_lxqt

(But normally this should not be needed, as long as I maintain the cookbooks-dataset, as I will already do so when a new lxqt release is published.)

Tab-Completion support for various shells

Tab completion support exists for some of the commands, for bash, zsh and fish. These are completions that are invoked when the user hits the tab-key on the keyboard.

For RBT, this can simplify compilation. Simply type "rbt", followed by a space, and then the first character of the program that you wish to install, followed by hitting the TAB key.

Let us next look at how generate these (tab) completions:

rbt --generate-completions
ry --generate-completions
rbt --completion
ry --completion

The code for this functionality resides in these two files:

rbt/compile/shell.rb
rbt/utility_scripts/generate_shell_completion.rb

Now - after you have sourced this file into your shell, e. g. via source /path/to/the/file.sh, you should be able to use the tab-completions, like:

rbt gnomede**TAB**

TAB in the context above means to press the tab key on the keyboard.

The tab-completion shell script can be generated anew via the above .rb file, but since as of 27th March 2019 the shell code for bash and zsh is also distributed with rbt itself. This may simplify making use of it. I source that .sh file in my .bashrc file since that same day.

If you want to find out where that .sh file exists on your system, try this commandline invocation:

rbt --shell-completion?

Installing the registered python addons

The RBT project has some python add-ons registered. These can be installed via either:

rbt --enable-python-addons
rbt --python-addons

You will most likely not need this, since this is catered to my own use case (since I have to batch-install things on a new computer, for example). But the option exists nonetheless, since as of March 2019.

Generate a .html file for several programs

A .html file can be generated via the method RBT.create_html_page_for_these_programs().

The input to this method should be an Array listing the programs that you want to view in that .html file. Alternatively, a String that contains these programs separated via a ',' token can also be used.

Example:

"php,ruby,elixir,enchant,m4"

The code that is part of the above method will generate a .html file that shows some information about these programs, in the given input-output order.

Commandline invocation example:

cookbooks --create_html_page_for_these_programs="php,ruby,elixir,enchant,m4"

Why was this functionality added?

Mostly because I needed a way to autogenerate a .html file that contains information about several programs, such as the whole kde5-set of archives. This then generates a .html file with about ~170 kde5-related programs, and can then be uploaded somewhere for other people to have a look at, not unlike the LFS/BLFS project is doing already so.

In fact, if you also want to generate a .html file for the whole KDE toolchain, try this:

cookbooks --create_html_page_for_these_programs=:kde5_toolchain

You can also autogenerate a set of "homepages" for all programs, via class Cookbooks::GenerateHomepage.

Either invoke the file directly (cookbooks/utility_scripts/generate_homepage/generate_homepage.rb) or simply use this commandline instruction:

cookbooks --generate-homepage

This will generate .html files that are somewhat similar to the LFS/BLFS files.

Downloading in general and downloading a remote URL into a specific local directory

The rbt project has some support for downloading a remote archive into the local filesystem/directory. This functionality was added primarily to aid in the setup stage of the RBT project, in particular when you are before a freshly setup computer system.

If you would like to download an archive from a remote URL into a specific (local) directory, you can use an API such as this here, from the commandline:

ry htop --download-into-this-directory=/opt/bla/

The first argument is the registered program that ought to be used; in this case the program called htop. The second argument specifies the local directory to be used, in this case /opt/bla/. Obviously you could also download first, and then mv the file anyway, but sometimes you may want to do it in one go.

If you wish to quickly download e. g. htop or another program, you can also use the following simpler toplevel API (from within ruby):

RBT.download :htop

This is treated as the very same way as if you would have used the following API with the full URL instead:

RBT.download 'http://hisham.hm/htop/releases/2.2.0/htop-2.2.0.tar.gz'

It thus is simply a convenience feature. Obviously "wget http://hisham.hm/htop/releases/2.2.0/htop-2.2.0.tar.gz" is simpler, but since RBT makes use of ruby, functionality such as this has to be included.

You can also use the current directory as input, e. g. if you would first create the directory htop, then cd into it, and then invoke that method:

RBT.download :pwd

This may save a little bit of typing.

The functionality can also be used from the commandline, through the bin/rbt script, such as in the following examples:

rbt --download htop
rbt --download ruby
rbt --download python

This would download htop, or ruby, or python, respectively, into the current working directory.

Note that the following syntax works as well:

rbt --download=htop
rbt --download=ruby

Use whatever you prefer here, ' ' or '=' work just fine.

This commandline option thus allows you to quickly download remote source archives.

If you do not want to type "rbt --download" then you could also use bin/rbt_download, aka:

rbt_download
rbt_download htop
rbt_download python

Or put an alias to this to make this even shorter.

If you wish to find out which programs can be downloaded that way, you could try to issue this command:

rbt --show-downloadable-programs

This will display all the names of the registered programs, more than 3500 (so it is quite a long output; you may want to pipe the output into more or less on Linux).

You can also download remote programs into the proper local directory. The "proper" directory is the one where RBT expects source archives to reside at.

Examples:

rbt --download-into-proper-directory htop
rbt --download-proper ruby # ← This variant is significantly shorter

On my home system, this would download http://hisham.hm/htop/releases/2.2.0/htop-2.2.0.tar.gz into the local directory /home/x/src/htop/. The exact local position depends on your SRC_DIR (source directory). See elsewhere in this documentation how the source directory can be set for the RBT project.

Do note that since as of April 2019 we can also attempt to download different versions of a given program. As an example, let's say that we wish to download glibc version 2.28. You could use this commandline way in order to achieve this:

rbt --download glibc-2.28
rbt --download glibc-2.27
rbt --download glibc-2.26
# and so forth

This has a few constraints, though. Firstly, the remote file must exist at the assumed position. Second, not all file extensions may be checked. The functionality can be improved in the long run but for now, it is like that. (The above should work reliably well, though).

Certain github-based URLs may also not work.

Feedback wanted

Please consider providing feedback about rbt, whether you found anything useful in it, whether there are some bugs or other problems, and suggestions to improve it.

Presently oldschool email should work best ( [email protected] ) - I may not reply to every email, but I will most assuredly read every email and consider making changes to RBT in the long run.

It is a hobby project, thus the amount of time that I invest into it is limited, but I also use RBT on my own home (linux) systems and it works fairly well on these systems. So I am interested in the use case of other people here.

RBT::Make

RBT::Make is the main class that may be invoked for make-related actions.

It also supports some toplevel methods such as:

RBT::Make.run

Additionally it can also be used to run "make install" or even "ninja", the alternative build system.

This class exists primarily to make it easier to deal with the "make" and "make install" step. Some instructions can be given to make, such as the -j option, and it is simpler to abstract this away through RBT directly, so that we can use it when compiling any program from source.

The IS_ROEBE environment variable

The IS_ROEBE environment variable is set to 1, aka true, on my home systems. When this variable exists and is set, then RBT will behave slightly different from the default. For example, on my home system I tend to use a hybrid system between the /usr/ hierarchy and AppDirs under /Programs/. Sometimes it is useful to symlink from /Programs/Foobar/1.0.0/lib/* into /usr/lib/. If the IS_ROEBE environment variable is set to 1, then the RBT scripts will, after compilation has completed successfully, symlink into the /usr/lib/ hierarchy.

IS_ROEBE to a value of 1).

Perl

In order to compile perl, you could simply issue the following command:

ry perl

If you wish to install the various registered perl-addons, you can try the following command:

ry --compile-perl-addons

(Unfortunately there is some bug, where installation stops after the first installed perl program. I will have to investigate this at a later time; this notice has been written in 20.04.2019).

porg

Porg is a small utility that allows a user to track which files are installed. RBT supports the use of porg through a configuration file, called use_porg.yml.

To query from the commandline whether porg is available on the given computer system, try:

ry --is-porg-available?

Some options for the individual cookbook files explained

This section will explain some entries in a .yml file.

run_configure:

The option option run_configure is set to true by default. If it is set to false, then RBT will NOT attempt to invoke GNU ./configure.

This may be useful for programs that you are certain to not have and use any configure script. The man-pages archive at https://www.kernel.org/pub/linux/docs/man-pages/man-pages-4.16.tar.xz is one such an example. It does not use any configure file, so a configure step would fail. (Note that there may be other ways to do the same, but for the time being, run_configure seems simple and effective.)

Changing the prefix in any given Makefile

Since as of 03.05.2019 (3rd May of 2019) it is now possible to change the prefix in a given Makefile, via class RBT::ChangePrefix.

This class can modify the PREFIX variable in that Makefile. It was added so that we can change the PREFIX target of any Makefile on the fly.

Currently it does not have a lot of configurable options but this may change in the future. For the time being, the first argument should be the path to the Makefile that should be modified.

Xorg

The Xorg-Server is the main implementation for the graphical Linux system. It is a display server for the X Window System.

Some helper-code exists in the RBT project to make compiling xorg-related components somewhat easier. One aspect here is that the RBT project has code that will easily allow you to compile all of the xorg-proto programs.

To invoke that functionality, do:

rbt --compile-all-protos

Do note that since some time, at about the year 2018 or so, the individual protos have been bundled into the project called xorg-protos.

If you want to display all xorg-related entries on the commandline, do:

rbt --tags=xorg

You can also compile the xorg components in an ordered fashion, by making use of numbers.

Examples:

rbt xorg1
rbt xorg2
rbt xorg3
rbt xorg4

And so forth.

To batch compile most of these components, try:

rbt xorg1..200

If you wish to compile post-xorg-server components, that is components that ought to be installed after *xorg-server has been compiled/installed, then the following commandline may be used:

rbt --compile-xorg-server-post
rbt --post-xorg-server

If you wish to only compile some of the addons, in particular input-addons related to mouse, keyboard, and a few more entries, then you can issue the following command:

ry --compile-mouse-and-keyboard

If you wish to compile all xorg-libraries, also called as the xorg-base, you can issue any of the following commands:

ry --compile-xorg-libraries
ry --compile-xorg-base

Gentoo and ebuilds

Gentoo's portage software makes use of ebuilds.

In theory, RBT can (auto)generate ebuild files but this has not been tested sufficiently - mostly because I myself do not use Gentoo, so someone else has to do the testing.

You can give the current implementation a try though, via:

ry ruby --create_ebuild_recipe
ry php --create_ebuild_recipe
ry perl --create_ebuild_recipe
ry python --create_ebuild_recipe

And see which parts aren't working well yet.

YAML Engine

Ruby itself defaults to psych as its main yaml engine.

When the cookbooks project was started (before it was merged into the RBT project), syck was the main yaml engine.

Since as of January 2018, the RBT project will default to psych. If for some reason you prefer syck, then you can use a commandline switch to denote which yaml engine is to be used.

To enable syck, do:

cookbooks --use-syck

To enable psych, do:

cookbooks --use-psych

This will modify the yaml file called use_psych_or_syck.yml.

Note that in order to use syck, you may have to install the syck gem first, via:

gem install syck

If you want to find out which yaml engine is in use, you can do:

RBT.yaml_engine?

Both engines should work fine. The reason why I kept on using syck for such a long time was mostly because I used to have some german comments in the various yaml files, which had umlauts in non-UTF encoding. I lateron switched to pure english instead, since it is easier for other people, and since that time, psych works just fine for the RBT project.

Otherwise I recommend to default to psych as well, as this may make other things simpler to work with (such as working with UTF-8 / Unicode).

Permanently set to another programs directory

The default programs directory is at /Programs/. Programs will be installed into that directory.

If you wish to permanently set to another directory you can use this commandline flag:

rbt --permanently_set_programs_dir=/pkg
rbt --use-this-as-programs-dir=/pkg

Static compilation

Some programs can be compiled statically, but this may also lead to problems sometimes - such as when the program can NOT be compiled statically.

The optional entry build_static in a cookbook file allows us to determine whether compilation should be statically or whether it should not be static.

If you wish to overrule this on the commandline and specifically disable static compilation, you can use a commandline instructions such as:

ry htop --do-not-build-statically

Note that as of December 2018 you can also combine --enable-shared --disable-static in one variant via:

ry libxmu --shared-no-static ntrad

To determine whether a program can be compiled statically, you can issue a command such as the following:

ry tar --can-be-compiled-statically?

Do note that this requires a manual registration in the corresponding yaml file; by default this will return false (aka no).

Checking the validity of the yaml files

Within the rbt project code exists to ensure that the yaml files are (mostly) valid and correct.

Code for this functionality can in general be found under the validation/ subdirectory. Most users will not need to use the files in that directory, though - these scripts are mostly for developers or maintainers of the rbt gem.

The code in the validation/ subdirectory can be thought as some kind of test-code, with the primary focus on ensuring that the cookbook yaml files adhere to the given specification at hand - and are, in general, correct.

If you do happen to want to check the validity of the cookbook files, then the following command can be run:

cookbooks --validate

Cookbook entries that are invalid, will eventually be fixed - please be patient (or report the erroneous entry to the lead maintainer(s)).

Class Cookbooks::ScanForIncompleteLastUpdateEntries can be used to ensure that the last_update: entries have a valid date entry. The currently used scheme is for "DD MM YYYY" where DD and YYYY are numbers and MM is the three-letter abbreviation for the month at hand. For example, "22 Mar 2018" means the 22nd of March in the year 2018. In the future I may allow for other formats, but for the time being this is the variant that is to be used (I am open to add automatic display-conversion to any other date format).

pre_configure_steps

The entry called pre_configure_steps can be used in a cookbook .yml file. It should be an Array.

The entries listed there will be run one after the other.

A typical example for this may be like this:

- autoreconf -vfi

A few GNU configure programs require this step, such as hunspell. Have a look at the hunspell.yml file to find out how this is used.

Using the DESTDIR variable

This subsection is just an explanation of a feature of a Makefile.

If you wish to use a specific --prefix value for a Makefile as well then you can use DESTDIR such as in this way:

make DESTDIR=/Programs/Bzip2/1.0.7 install

class RBT::Cookbooks::ReportTheseProgramsCouldBeUpdated

class RBT::Cookbooks::ReportTheseProgramsCouldBeUpdated has been added in August (06.08.2019). It is assumed to report, on the commandline, all programs that could be updated.

In order for this to work, the class will have to check several different remote websites.

Currently (August 2019) the class will only check distrowatch.org, but in the future this will most likely be extended, so that we can use this to update programs from the commandline alone.

class RBT::SimplifyRootEntries

This very specialized class will replace all root:root entries in a file, such as a Makefile, with 0:0 entries.

The reason why this small class was added was primarily because sometimes the host system may be partially broken, and only display 0:0 rather than root. This may happen more frequently in a chrooted environment.

Since I was tired of makefiles then failing due to this reason, I created that class in August 2019.

Expanding the cookbook dataset within the RBT project

By default, the individual yaml files, such as ruby.yml, python.yml, php.yml and so forth, are kept in a minimal state, primarily because it is easier to modify the dataset then, via an editor.

A minimal state also means that the format is not terribly useful by default.

For example, the setting keep_extracted: t is an abbreviated form of the setting keep_extracted: true. (Note that this entry means that the archive will remain extracted after compilation has finished.)

The setting use_build_directory: can use a value of yes or y rather than t or true for true. The RBT project deliberately tries to remain flexible here, so multiple ways are supported for the end user's convenience.

When such a "minimal" yaml file is loaded up, the RBT project (submodule Cookbooks there) normally has to sanitize that yaml file, which takes a little time. For individual files this is not a huge deal, but sanitizing all ~3600 something .yml files can take quite a long time, so code had to be added to expand this dataset into its "final" variant. This final variant is the variant that has all the information in a ready-to-use format.

If you do not want to wait, you can speed up this process by running the expand functionality early on, such as after installation of RBT. The expand functionality will expand all yaml files and store them in the expanded (correct) format. Then, when such a yaml file is loaded up lateron, no change has to be made at all, thus no further time has to be spent sanitizing the information since it is already sanitized. This makes things a bit faster in the long run.

In order to do so, you can invoke the rbt project via any of the following commandline flag:

rbt --expand
rbt --expand-dataset 

You normally only have to do this once, but it is a purely optional step. It is recommended to do so, though.

If the expanded dataset is not available then the rbt project may (have to) sanitize the dataset.

(Also note that the expanded dataset could be converted into SQL instructions. I'll still have to add that step at a later time, though.)

Since as of June 2019, on my home system, rbt will expand individual entries when no expanded cookbook exists for that given program at hand as of yet.

Different datasets

The first, raw material for the dataset is a fairly short .yml file that may contain information specific to the program at hand. For example, ruby.yml or php.yml.

This variant has to be expanded and sanitized, though. If the end user wishes to skip this step then this short .yml file can be converted into a so-called expanded .yml file (via rbt --expand-dataset, for instance).

Since September 2019 an additional, SQL-based way exists. This variant is currently (re)written and requires more testing before it can be added completely. Consider this as work-in-progress for now. In the long run, RBT will optionally allow for the use of a SQLite database in order to allow for faster queries.

configure_command_to_use

This short subsection is to briefly explain the option called configure_command_to_use that is available for individual cookbook .yml files.

By default, this command can be omitted, and will then default to ./configure - aka GNU configure. This should cover most programs.

There are some programs that do not use GNU configure, though, in particular cmake-based projects, scons, waf, meson/ninja and so forth. These are handled separately for the most part.

There are, however had, also programs that use their own custom and unique build system, often defaulting to some shell script. The RBT project thus has to be flexible here and allow for custom shell scripts to be used as well.

An example for this can be seen in the file discount.yml for the program discount. If you look at that file you can see the following line:

configure_command_to_use: ./configure.sh

It thus specifies that RBT::Compile will make use of that configure.sh shell scripts rather than the more generic configure invocation run.

Some other programs use other configure_command_to_use settings; a recursive grep can find these, should you be interested in that.

Some commandline options explained

If you do not want to pick up the configure options from a specific .yml file, you can pass in the option --dont-use-configure-options to avoid doing so.

Examples:

rbt htop --dont-use-configure-options
rbt glib --user-home-prefix --use-meson --no-configure-options
rbt ruby --do-not-use-any-configure-options

This allows you to just use the "barebones" compilation mode; may be helpful if you encounter some errors due to the configure options used.

Show the last compiled programs

If you want to, for example, show the last 8 compiled programs, then the following commandline may be useful:

rbt --last-compiled=8

Note that you first have to have compiled some programs, as otherwise nothing, or less than 8 programs, would be shown.

As of October 2019, another way can also be used, if you wish to show all last compiled programs:

rbt --last-compilations?

To show only the last compiled program, try this:

rbt --last-compiled-program?

If you want to find out which were the last 20 programs that were compiled, then you can use this commandline option as shown above:

rbt --last-compiled=20

Copying archives to the current working directory

If you ever need to copy some archives into the current working directory (pwd), such as if you wish to copy them onto a USB stick, then the following API may be of help to you:

rbt --copy-these-archives=ruby,php,python
rbt --copy-archives=kde5_plasma

The latter in particular will copy all the different KDE5-plasma components onto a USB stick. This was the original use case as to why I have added this functionality - I needed to copy the plasma components to another computer located elsewhere (but accessible via USB stick only), in order to compile these programs on that machine.

Note that you could also, in principle, use abbreviations here:

rbt --copy-these-archives=xorgser

This would copy the local directory containing the xorg-server tarball, rather than the user being required to input "xorg-server".

In order for this functionality to work, the directory has to exist prior to invoking the above command(s).

Take note that in order for this functionality to work, you must have already downloaded these programs into local directories. Currently the RBT project does not automate these downloads for you (although it would be simple to add

  • we will see when this functionality is actually required).

If you only wish to copy a single source archive to the current working directory then you can use an API such as the following:

ry --copy-source=php

Using header files such as ao.h or udev.h as input for compilation

class RBT::Headers is a helper class that can map the given input at hand, if it is a header file such as ao.h, to the corresponding program at hand - which is libao in this case.

This also works if you wish to use a .h header file as compile input and don't care about the program name; or do not remember the name of the program.

Consider the following:

ry udev.h ntrad

This would replace udev.h with eudev, and compile it in a non-traditional (== AppDir) way.

This functionality has been added in November 2019, but it is quite experimental at this time. My personal recommendation is to use the real name of the program, because using the .h as name can lead to errors (such as when different programs use the same .h file, for example).

On the other hand, the functionality was specifically added because sometimes you just don't care about to which program a particular .h file belongs, yet still wish to compile/install it, so that is the rationale as to why that functionality has been added.

The constant called FILE_ABBREVIATIONS

FILE_ABBREVIATIONS is a constant pointing to a file that keeps the abbreviations to the registered programs. Abbreviations in this context are words such as "ph", which may stand for "php". So in short: an alias to a longer name.

The above example may be useful if you wish to compile PHP on the commandline: rather than typing "php", you can type "ph" and it should still work.

The net gain in this example is little, but consider a longer name such as xorg-server. You could compile it via:

ry xorgser

Convenient, right?

Do note that tab-completion support for bash and zsh also exists to enable a functionality like this, but I wanted to be able to find shorter aliases within the ruby code itself too, so that an API such as this one may also work:

RBT.compile 'php'     # For php.
RBT.compile 'ph'      # For php, too.
RBT.compile 'xorgser' # For xorg-server.

All variants should work. The longer variant, while it is more typing work, is the recommended way, though. This is primarily because that way you can avoid ambiguity.

The usual residency of the file pointed through the constant FILE_ABBREVIATIONS is at:

rbt/yaml/abbrevations_to_the_programs.yml

In short: the reason why this file exists is so that we can use abbreviations, if we wish to compile programs from source.

Speeding up the compilation

This subsection deals with speeding up the compilation.

make itself accepts a commandline option, through -jN, where N is the number of parallel builds to use.

For example:

make -j4

would make use of four parallel builds. This should speed up compilation significantly. Do not put any incorrect number there, such as 0 or some crazy huge number. The maximum number should be the maximum amount of processors that you have, which you can find out via this way, on Linux:

grep -c '^processor' /proc/cpuinfo

You can also obtain this from RBT itself, on the commandline, via:

rbt --n-processors?

Passing the option -j4 to make means to "spawn off 4 parallel compiling processes". There is actually not a limit for the number of job slots that can be used, but as a rule of thumb, one should not exceed more than twice the number of CPU cores that the local computer system has. So if we have 4 processors, we should never use a number higher than 8 to "make -j".

Whether this is really true or not, I have absolutely no idea. I think the main message here is to not use any arbitrary high number.

If you wish to append the default value to make -j, which is the number of processors that the local computer has, then do this, for example:

ry flac ntrad --speed-up-the-compilation
ry ruby --ntrad --speed-up-the-compilation

The important option here is --speed-up-the-compilation.

Note that you can also directly pass this -j option to rbt, such as exemplified through the following usage example:

rbt lame -j5

This is the same as doing "make -j5", by the way, during the make step.

You may also define this in a cookbook file, by adding a parameter called parameters_to_make. Be careful though, since this may not always be what you may want, in particular if you use different computers. An alternative to this may be to use a yaml file, part of rbt, called use_maximum_speed_for_compilation.yml.

This can also be disabled for the individual run, via:

 --do-not-use-maximum-speed
 --no-auto-speedup

More complete example how to disable this from the commandline:

ry lame --do-not-use-maximum-speed
ry xvidcore --ntrad --donot-use-maximum-speed

It was important to provide this option ont he commandline because the end user may not want to use maximum speed for compilation, or because some other problem exists on the given computer system that requires the end user to avoid this setting.

Installing libstdc++

If you wish to install libstdc++, according to http://www.linuxfromscratch.org/lfs/view/development/chapter05/gcc-libstdc++.html, then the method called RBT.install_libstdc_plus_plus() may be of help.

Usage example in ruby:

require 'rbt/toplevel_methods/install_libstdc_plus_plus.rb'

RBT.install_libstdc_plus_plus

Note that you may have to download gcc prior to invoking this method.

In November 2019, this functionality has been extended. It is now possible to actually compile a fake-libstdc++ program into the /Programs/ hierarchy (or wherever you designated your programs directory to reside on your machine).

A compile-profile has been created for libstdc++, which resides in the file called libstdc++.md, as part of the RBT project.

Libstdc++ can be compiled into the programs directory, such as:

/home/Programs/Libstdc++/9.2.0

It can be kept separate from e. g. /home/Programs/Gcc/9.2.0.

Presently the path for libstdc++ is mostly hardcoded; reason being that it is simpler here, and my use case is only to get it up and running quickly, anyway.

Invocation examples:

ry --compile-libstdc
ry --compile-libstdc++
ry --install-libstdc
ry --install-libstdc++

class RBT::ShowDescriptionAndExtraInformation

class RBT::ShowDescriptionAndExtraInformation is just a little helper class for commandline-use. It can be used to quickly show the description, and the extra_information entry from a cookbooks .yml file.

It also uses colours, which was one major reason why I wrote it - sometimes you need to quickly get some helpful information when you compile something; and this class achieves precisely that. (Keep in mind that the extra_information entry has been added for precisely that goal: to show useful additional information about a program, or how to get that program to compile.)

Comments about GCC, the GNU Compiler Suite

This subsection is mostly just for commenting on random parts of GCC. Evidently GCC is quite critical on a Linux system, together with glibc (or musl), and the binutils.

One question that people may have is: which configure settings should I use for GCC? The following subsection will explain some of these; and serve as a mnemonic.

For GCC 8.1.0 I would typically use this configure line:

../configure --prefix=/Programs/Gcc/8.1.0 --enable-languages=c,c++,lto,objc,fortran --enable-shared --disable-static --disable-multilib --target=x86_64-slackware-linux --build=x86_64-slackware-linux --host=x86_64-slackware-linux --disable-libunwind-exceptions --enable-bootstrap --enable-linker-build-id

Not all of these seem to be required, but evidently --enable-languages= is very important.

Flexbox

Until November 2019, the RBT project had class RBT::Infobox. This was moved into the AsciiParadise project in November 2019, and renamed to Flexbox.

The Flexbox can show some information about the program that is to be compiled/installed next - or a program that you may wish to install. It is thus only a visual aid, through colours, and Unicode symbols, on the commandline.

For this to work, the program has to be registered prior, as part of the RBT project.

You can specifically tell RBT to only show the Flebox, by making use of the commandline flag called --show-flexbox-only.

Usage examples for this:

rbt ruby --show-flexbox-only
rbt python --show-flexbox-only
rbt php --show-flexbox-only

If you do not wish to see this flexbox (as infobox) during compilation, you can also disable it for the current compilation/installation run, via a commandline flag such as --no-flexbox:

rbt ruby --no-flexbox
rbt python --no-flexbox
rbt php --no-flexbox
rbt sed --no-flexbox

You can also use the flexbox directly, through bin/flexbox if you have the AsciiParadise installed, such as in:

flexbox ruby
flexbox php
flexbox sed

Note that this may show slightly different results, as the variant that makes use of rbt will also honour specific configuration settings, whereas the flexbox variant will only try to make use of the dataset stored in the respective .yml file. So if you would like the information displayed to be more accurate, you should use the longer variant with rbt; if you just need some general information then flexbox should do just fine.

Since as of May 2019 the flexbox will also show which compiler will be used. Normally this will be GCC, but clang may also be an option here; this information serves to be indicative of that.

Since as of June 2019 the Flexbox will try to use Unicode symbols on the commandline, for the outside box. This should look better than using === and | (ASCII characters). Do note that this is work-in-progress and may be subject change until it is consolidated.

Status of the RBT project

The RBT project is currently (November 2019) in a slighty advanced testing stage, somewhat "semi-complete". Or at the least, "sufficiently useful" for my use cases. Presently (in November 2019) I am working through the existing dataset and clean it up mostly, improving on it.

Nonetheless I do not recommend anyone to use this project as of yet, but fearless testers could test it and it may even work to some extent.

Please be wary when using the RBT project for the time being, as there may be some bugs, but also behaviour that may be surprising to you. The project is not yet on the same level as e. g. homebrew - things will be "cleaned up" more rigorously at a later stage.

The RBT project makes certain assumptions, a certain layout of the filesystem, and so forth. While this can be tweaked by the user, for the time being, it is best to just follow these assumptions for now.

Busybox

Having the application called busybox installed, in a static manner, is quite useful.

I recommend to have a working version of busybox available on the local computer system, as that may help in recovering in the event that something has gone awry.

To compile busybox, try this:

rbt busybox

Make sure to compile busybox statically. That way you become less dependent on .so files being available.

In November 2019, a default configuration was added for busybox, allowing static linking. The name of that file is static_busybox_config.md and it can be found under the profiles/ directory of the rbt gem.

This should allow you to quickly install a static version of busybox.

I needed this because I did not want to manually run "make menuconfig" or "make config", since I only needed the static version; and the rest is fine either way for me.

For a static compilation of busybox, this should work fine by default:

ry busybox ntrad

At a later time more flexibility may be added here, but for the time being (November 2019) this has to suffice.

Show all statically compiled programs on the given computer system

If you need to show all statically compiled programs on the given computer system then the following may be of help:

rbt --static-overview?

Note that the checks used here are not perfect, so expect some false positives. But for a quick overview this should do fine. The class was added in November 2019 to the rbt project.

Registering errors during compilation/installation

Some problems and errors may happen during compilation or installation of programs.

These are handled internally by RBT::Compile, but they will also be registered in a module-method onto the RBT Namespace.

It is available via the following code:

RBT.error_message?

Note that this will be reset to nil, its default value, whenever a new program is compiled.

Sometimes it is not wanted to stop on errors, in particular if the error reported is not a real error. Thus, the user needs to have a way to overrule this behaviour, which can be done by the following commandline flag:

ry --dont-stop-on-errors
ry --ignore-errors

This would instruct RBT::Compile to not stop on errors and instead (try to) continue.

Pre-install actions

Pre-install actions are actions taken before RBT::Compile will invoke the configure script, or cmake invokation.

Typically the pre-install action(s) may involve calling a shell script, such as bootstrap, but certain sed-operations may also have to occur.

You can pass in which particular pre-install action should be done, via:

rbt htop --preinstall="sh bootstrap"
rbt htop --prerun="sh bootstrap"

rbt htop --preinstall="sh bootstrap, clear"
rbt htop --prerun="sh bootstrap; clear"

As can be seen above, you can separate multiple arguments via either ',' or ';'.

Additionally, you can use ruby classes in the preinstall setting, such as:

preinstall:
- ChangeLib64ToLib.new

In the cookbook file gcc.yml. This will call the respective class and invoke it (the class has to exist of course).

The reason why the support for .new was added here was because I was tired of the shell scripts used in the LFS/BLFS scripts. The classes-approach leads to somewhat more lines of code, but it also is easier to read, and it will properly inform the user as to what is done.

Configure options and passing additional configure options

The content of the various .yml files may have an entry called configure_options. This is usually the entry that keeps track of the arguments that you may normally pass towards ./configure, such as --enable-static or --disable-shared.

Different build systems, such as meson, use a slightly different syntax, such as -Dlibpsl=false. Additionally, I sometimes want to preserve the old configure options, so that I can compile different versions of a given program where the build system may have changed. That is why I added, in December 2018, the ability to use specific configure options for meson. These are called meson_configure_options. An example for this may be the file networkmanager.yml.

Some configure options can be passed to an instance of RBT::Compile as well.

For example, if you wish to use --enable-shared then you can simply pass it, like so:

rbt htop --enable-shared

This will simply pass the configure option --enable-shared.

In the future, more options will probably be added. Some options may be translated into the equivalent cmake-option, if cmake is used, but for the time being, this is presently (November 2018) not well supported. Stay tuned for later changes in this regard.

Since as of November 2019, the configure options that were used to compile a program, will also be stored into the target file at Resources/configure_options.md if it was installed via an AppDir prefix (e. g. /Programs/Ruby/2.6.5/ and similar), if the compilation was a success and if there was at the least one configure options - thus, empty Strings will not be stored.

This file will only store the actual configure_options that were used; for the full configure-like command, see the corresponding file in the same directory, whose name is full_configure_command.md.

class RBT::MapThisInputToThatRegisteredProgram

class RBT::MapThisInputToThatRegisteredProgram was added shortly after class RBT::QueryFileAssociation was written.

The core idea for class RBT::MapThisInputToThatRegisteredProgram is to translate (user) input into a registered program.

So for example:

x11  libx11
hto  htop
udev.h  eudev

Thus, the idea is to get any input, partial or complete, and find the corresponding program. In the above example, the header file udev.h is part of the program called eudev.

The idea here is to remain flexible when it comes to user input. The user would like to compile and install something? Then a partial input should work just fine.

Do note that this is NOT perfect. If in doubt, always input the full program name, as-is, because the class will have to make certain decisions. For example, some .h header files have the same name across different programs, yet class RBT::MapThisInputToThatRegisteredProgram will only pick one of them (always the .first one found in the Array).

That behaviour may also be subject to change in the future, so if in doubt, again - rely mostly on the real name. But if you forgot the name and don't want to look up or can not look up, you can try to leverage class RBT::MapThisInputToThatRegisteredProgram.

class RBT::Compile also makes use of that class if the input is not a registered program.

Configure profiles / decorators

This subsection is about configure profiles, also called decorators within the context of the RBT project.

These profiles (or decorators) allow you to use different configure "profiles", depending on the given context. For example, if you wish to build a LFS system from scratch then the configure option that you may want to use will be different for regular compilation of GCC.

Thus, a profile exists called gcc_lfs_pass1.md. This file contains the configure options that are normally used for the LFS build, excluding the --prefix switch (because you will still be able to decide which prefix is to be used here, on your own).

This functionality has been added to the RBT project in June 2019 and is still experimental - use it with care, if you decide to use it at all.

Specific examples follow next.

First, to display an overview of the available profiles, you can issue any of the following commands:

ry --available-profiles
ry --profiles?

In order to use the profile for GCC first pass, as described in the LFS project, use a commandline switch as the following:

ry gcc --profile=gcc_lfs_pass1

If you want to compile a minimal GCC, into its versioned AppDir prefix, then you can use the following call:

ry gcc --profile=gcc-minimal --ntrad

You can also use a slightly more explicit variant:

ry qt --use-profile=qt
ry qt --use-profile=qt --trad

For some profiles you can also use a shorter variant, such as --basic-gcc for --profile=gcc-minimal:

ry gcc trad --basic-gcc

In order to see individual profiles based on the program name at hand, such as gcc, you can use this sub-query search:

ry --show-this-profile=gcc
ry --show-these-profiles=gcc
ry --profiles-of?=gcc

Note that in order to display such a profile, it has to exist. If it does not exist for the given program at hand then RBT will notify the user about this fact.

Validating the cookbook recipes and validation within the RBT project as a whole

You can invoke, from the commandline, the following:

rbt --validate-entries

This will validate the dataset stored in the various .yml files.

It will run the test-code in the validation/ subdirectory.

This is not so useful for casual users of the RBT project, though, but more interesting for people who may want to contribute recipes and such. That way they can ensure that the recipe that they have created is valid.

If you wish to validate all cookbook entries then you use the following class:

RBT::Cookbooks::ValidateAllCookbookEntries.new

This class can validate all cookbook-entries. It will analyse them in detail and report the findings on the commandline.

Again - this is not so useful for the average user, but it helps me respectively anyone else maintaining the code and cookbook recipes if we seek to maintain a correct, and consistent, dataset.

CreateAppDirSkeleton (RBT::CreateAppDirSkeleton)

The class RBT::CreateAppDirSkeleton, residing at rbt/create_app_dir_skeleton/create_app_dir_skeleton.rb, prepares a program for installation under the /Programs/ hierarchy.

Keep in mind that the target at /Programs/ can be changed by the user - so the rest of this paragraph should be kept in this context. The main programs directory could be /opt/ or /pkg/ or any other target too.

The behaviour by class RBT::CreateAppDirSkeleton is loosely modelled after the GoboLinux PrepareProgram script. Differences exist mostly in regards to the usage pattern and scope between these two programs; and what is reported to the user.

Here is a link to the GoboLinux program called PrepareProgram:

https://github.com/gobolinux/Documentation/wiki/PrepareProgram

In order to remain somewhat compatible to the GoboLinux PrepareProgram script, options such as the following will work just fine:

create_program --tree Foo 1.0

(Here we assume that create_program is an alias to the file rbt/bin/create_app_dir_skeleton).

The much simpler variant would be to do either one of the following instead, as far as the RBT project is concerned:

create_program Foo 1.0
create_program Foo-1.0
create_program foo-1.0

I personally prefer the last variant, since it is the simplest one for me to remember - but use whichever variant you prefer. The last one is simple for me because I can sort of copy/paste the directory structure through just one argument, to that class.

You can also use this from the commandline, if you would like to do so, via rbt like in the following examples:

rbt --create-program=foo-1.0
rbt --create-program=php-7.4.3
rbt --create-program=python-3.3.3

Do take note that CreateAppDirSkeleton may be called from some RBT scripts, in particular RBT::Compile. Some directories will not see that their default symlink (called Current) is removed, such as gcc or binutils, as otherwise subsequent compilation runs would not work to the Current not being set properly.

You can also use the current working directory as input.

For example, say that you are in a directory called yelp-3.30.2/.

If you wish to make use of that directory, as the input, as if you would have passed the above as String into RBT::CreateAppDirSkeleton.new(), you can use either of the following two variants:

rbt --create-program
rbt --rcp

Compile statistics

This subsection shows just a few random programs that were compiled in December 2019, to show how long it took my main computer to compile them. This machine is a fairly cheap one (x86_64, AMD A8-7600 Radeon R7, 10 Compute Cores 4C+6G, 4 CPU cores, RAM (max): 14954 MB) but it is quite fast considering its low cost. For faster machines you will obviously see better results.

--------------------------------------------------------------------------------
Name of the program         Compile time in seconds Compile time in minutes File size of the archive
--------------------------------------------------------------------------------
 python:                          316.62 seconds  (5.28 minutes)   17836412 Kb
 rhythmbox:                       248.61 seconds  (4.14 minutes)    6410600 Kb
 ruby:                            203.06 seconds  (3.38 minutes)   11553580 Kb
 mesa:                            128.94 seconds  (2.15 minutes)   11460812 Kb
 postgresql:                       72.63 seconds  (1.21 minutes)   16050068 Kb
 php:                              60.81 seconds  (1.01 minutes)   10232208 Kb
 libogg:                           22.09 seconds  (0.37 minutes)     428696 Kb
 esound:                           20.51 seconds  (0.34 minutes)     327352 Kb
 geoclue:                          19.62 seconds  (0.33 minutes)      86376 Kb
 ncftp:                            18.14 seconds  (0.30 minutes)     420760 Kb
 audiofile:                        16.15 seconds  (0.27 minutes)     530760 Kb
 madplay:                          14.76 seconds  (0.25 minutes)     364040 Kb

class RBT::SystemCompilePossibilities

class RBT::SystemCompilePossibilities is on my todo list. It essentially attempts to inform the user of the compile-possibilities of the given system at hand; and will also compile "missing" or outdated programs, optionally. So it works as some sort of information and auto-compile tool.

It is presently (middle of May 2018) in a very early beta stage.

Do not consider it to be complete as of yet - I will have to test it extensively yet. The idea is to use this as a basis for an "automatic linux from scratch" system - or to complement the latter.

BLFS - Beyond Linux From Scratch (and also LFS)

class RBT::Blfs has "support" for BLFS in that it will ... simply show a URL to the remote BLFS page of a given package, if it exists.

If you only need to know the URL to the BLFS homepage, to embed it for display into a webpage perhaps or to display it onto the commandline, you can use the following simple toplevel-API for this:

RBT.blfs(:openssl) # => "http://www.linuxfromscratch.org/blfs/view/8.1/postlfs/openssl.html"

This will return the remote BLFS page, as URL (a String, actually), if it exists, and if it has been registered in the corresponding .yml file (cookbook file for the program at hand; in this example the file openssl.yml).

You can also use class RBT::LFS to build/create a LFS from scratch. This is presently (2019) quite incomplete, though. Lots of different things have to be done in the correct sequence before a LFS can be built from scratch successfully, and possible errors have to be checked before the build can continue. In the long run, for the future, this will eventually work - but for now it is an incomplete stub and not recommended for testing.

If you wish to quickly paste the BLFS page of a given program onto the commandline, for display, then you can do something like this:

ry php --parseblfs

Add some later point in the future (August 2019), this may even become its own bin/ entry - we'll see about that.

Since as of December 2019, it is now also possible to display, on the commandline, any registered BLFS entry. This will help in the event that you can not access the xorg-server at the moment, yet still require information for different packages. In order for this to work, you evidently need to have an internet connection available - but in principle, the information stored in the BLFS project could be zipped up and distributed too. For the time being, though, only pasting is supported - there is no plan to bundle the information into the RBT project at this stage.

Usage example:

rbt --parse-blfs-page-for=llvm

In December 2019 another feature was added - the ability to make use of the BLFS instructions directly, and run them.

The idea here is that you may wish to compile some program from source, but may not access a working xorg-server, as you are working on the commandline only.

Invocation example:

ry kerberos --use-blfs-instructions # Compile kerberos in the BLFS way.

class RBT::PurgeIncorrectYamlDirectoriesInTheProgramsHierarchy

class RBT::PurgeIncorrectYamlDirectoriesInTheProgramsHierarchy can be used to purge incorrect yaml/ directories on the system, under the Programs hierarchy (/home/Programs/ on my system).

To invoke this from the commandline, try either of the following:

rbt --purge-incorrect-yaml-directories-in-the-programs-hierarchy
rbt --purge-flawed-yaml-directories-in-the-programs-hierarchy

Do note that this class is largely unnecessary these days - it was created as a quick ad-hoc solution, to work around the buggy behaviour of another class, until that class was fixed.

class RBT::CreatePkgconfigFile

class RBT::CreatePkgconfigFile was added in December 2019.

The idea is to be able to generate, on an ad-hoc basis, .pc files (pkgconfig-files).

We can tap into the already available existing dataset of the RBT project, so autogenerating the .pc files is trivial - and sometimes necessary, for programs that do not come with their own .pc files.

On my home system, from the commandline, I tend to invoke this like so:

create_pkgconfig :htop

This means that RBT::Compile will be used internally, and then generate the file htop.pc. (In this case create_pkgconfig is an alias to bin/create_pkgconfig_file).

Checking a remote git repository

If you wish to quickly download the git-source of a program, say binutils, you could use any of the following options:

ry binutils --check-source
ry binutils --git
ry mpv --git-download

This would, in principle, work for all programs that are hosted via git. A requirement is that the .yml file at hand has the corresponding git_url: setting set to a valid value - a remote URL. See the file binutils.yml for an example of this.

Do note that this will also rename the downloaded directory and create a .tar.xz file from that. This is mostly a convenience feature, since I tend to store those .tar.xz files anyway.

class RBT::Linux::Slackware::InstallThisSlackwarePackage

class RBT::Linux::Slackware::InstallThisSlackwarePackage can be used to quickly install a slackware package.

installpkg works on slackware by default, so why was this class created?

It was specifically created because I wanted to install via an AppDir prefix, rather than onto /usr/ by default. The AppDir installation can be triggered like so:

islackware binutils-2.26-x86_64-3.txz --appdir

Note that this was added on 18.12.2019 and has not been tested extensively yet.

More commandline examples including a brief tutorial

This subsection includes different commandline examples, to see how the RBT project can be used. A short explanation will be given for each invocation.

You can compile a program based on a .pc (pkg-config file) as an input.

Example:

ry xau.pc # ← This is equivalent to "ry libxau"

You can also compile through registered .h header files such as:

ry  SMlib.h --ntrad

If you wish to see a brief introduction on the commandline, issue the following command:

cookbooks --tutorial

This tutorial is really just very short though, and not always up-to-date - the other, more verbose, and lengthy documentation that can be found in this file (README.md) may be sufficiently more useful to users of the RBT project.

Installing the Linux Kernel Header files

The LFS project has a step where the kernel headers are copied to the /usr/include/ target. I wanted this on a versioned AppDir layout as well, so the following commands were added:

rbt --install-kernel-headers
rbt --install-linux-headers

This will essentially copy the important .h files from the linux kernel to the correct target AppDir location.

Using a different set of configure-options

Sometimes you may want to overrule, on the commandline, the configure options that are to be used by a program.

You can do so via the switch --use-these-configure-options=.

Usage examples:

--use-these-configure-options="--enable-shared --enable-bootstrap --enable-languages=c,c++,lto,objc,fortran --enable-threads=posix --enable-checking=release --enable-objc-gc --with-system-zlib --enable-libstdcxx-dual-abi --with-default-libstdcxx-abi=gcc4-compatible --disable-libunwind-exceptions --enable-__cxa_atexit --enable-libssp --enable-lto --disable-install-libiberty --with-gnu-ld --verbose --with-arch-directory=amd64 --disable-gtktest --disable-multilib --target=x86_64-slackware-linux --build=x86_64-slackware-linux --host=x86_64-slackware-linux"

Complete usage example:

rbt gcc --use-these-configure-options="--enable-shared --enable-bootstrap --enable-languages=c,c++,lto,objc,fortran --enable-threads=posix --enable-checking=release --enable-objc-gc --with-system-zlib --enable-libstdcxx-dual-abi --with-default-libstdcxx-abi=gcc4-compatible --disable-libunwind-exceptions --enable-__cxa_atexit --enable-libssp --enable-lto --disable-install-libiberty --with-gnu-ld --verbose --with-arch-directory=amd64 --disable-gtktest --disable-multilib --target=x86_64-slackware-linux --build=x86_64-slackware-linux --host=x86_64-slackware-linux"

rbt poppler --use-these-configure-options="--enable-xpdf-headers"

File simple_appdir_configure.rb

File simple_appdir_configure.rb can be invoked to create an AppDir layout program, for compilation.

I aliased this file to appconf. Then I can navigate to a directory, and simply invoke appconf - the program will be compiled with the default prefix as-is.

Since as of December 2019, support for cmake is possible as well, via:

appconf --cmake

Do note that this is just a method right now; in the future this may be changed to become a class, though.

The full path to the file is at:

rbt/toplevel_methods/simple_appdir_configure.rb

If you need to use a build directory for this method, you can use this:

appconf --down

Statistics (and class RBT::CompileStrategies)

The RBT project includes code that is related to statistics.

For example, class RBT::CompileStrategies can be used to query how many programs use meson, python, ruby or cmake, in order to be compiled or installed. class RBT::ShowHowManyFilesAreTracked can show how many .h header files, binaries, and library files (.so and .a) files are tracked by the project.

Since the latter may be interesting from a historic perspective, this is the result I got after running the latter class on 15.09.2019 (15th september 2019):

There are 4512 programs registered in total in the RBT project.
There are 3616 binaries registered in the RBT project.
There are 17221 .so / .a library files registered in the RBT project.
There are 7278 .h header files registered in the RBT project.

A module-method allows you to query for the compile strategies used in the registered cookbook files of the RBT project, via:

RBT.compile_strategies?

And from the commandline, you can invoke it via the following invocation:

rbt --compile_strategies?

If you would like to see a more comprehensive overview over the whole RBT project, and associated statistics, you can do so via either of the following variants:

rbt --stats?
rbt --stats?

This will essentially invoke class RBT::ShowStatistics, which in turn calls the other classes. (Take note that this may take a while, since lots of .yml files may be processed.)

The cookbook-submodule of the RBT project also includes some statistics, which can be next seen here, displayed (and stored) in the file called rbt/doc/COOKBOOK_STATISTICS.md:


# =========================================================================== #
# === Statistics for the Cookbooks found in the RBT project
#
# This file here (COOKBOOK_STATISTICS.md) was probably created in
# September 2005. It was later renamed from source_files.yml to 
# cookbook.yml (on the 04.05.2006), before it was then eventually
# renamed to COOKBOOK_STATISTICS.md years lateron.
#
# The main purpose of this file is mostly to outline the history of
# the RBT project, how many programs were registered when.
# =========================================================================== #
# - How many programs do we have registered in the RBT project?
#
#   →  274 programs registered as of 25.10.2005
#   →  340 programs registered as of 31.12.2005
#
#   →  851 programs registered as of 28.04.2006
#   → 1114 programs registered as of 11.07.2006
#   → 1168 programs registered as of 01.08.2006
#   → 1195 programs registered as of 07.08.2006
#   → 1230 programs registered as of 24.08.2006
#   → 1245 programs registered as of 31.08.2006
#   → 1270 programs registered as of 16.09.2006
#   → 1315 programs registered as of 15.10.2006
#   → 1352 programs registered as of 13.11.2006
#   → 1402 programs registered as of 27.12.2006
#
#   → 1432 programs registered as of 28.01.2007
#   → 1452 programs registered as of 07.02.2007
#   → 1506 programs registered as of 17.03.2007
#   → 1547 programs registered as of 19.04.2007
#   → 1584 programs registered as of 24.05.2007
#   → 1598 programs registered as of 04.06.2007
#   → 1649 programs registered as of 01.07.2007
#   → 1704 programs registered as of 05.08.2007
#   → 1736 programs registered as of 01.09.2007
#   → 1822 programs registered as of 27.09.2007
#   → 1835 programs registered as of 03.10.2007
#   → 1887 programs registered as of 27.10.2007
#   → 1931 programs registered as of 30.11.2007
#   → 1994 programs registered as of 30.01.2007
#
#   → 2107 programs registered as of 05.04.2008
#   → 2172 programs registered as of 02.06.2008
#   → 2202 programs registered as of 29.07.2008
#   → 2256 programs registered as of 12.12.2008
#
#   → 2269 programs registered as of 31.01.2009
#   → 2288 programs registered as of 03.05.2009
#   → 2318 programs registered as of 01.09.2009
#   → 2325 programs registered as of 24.11.2009
#   → 2328 programs registered as of 08.12.2009
#
#   → 2347 programs registered as of 31.03.2010
#
#   → 2373 programs registered as of 28.02.2011
#   → 2380 programs registered as of 19.03.2011
#   → 2393 programs registered as of 25.04.2011
#   → 2482 programs registered as of 29.08.2011
#   → 2491 programs registered as of 13.10.2011
#
#   → 2516 programs registered as of 04.01.2012
#   → 2270 programs registered as of 28.07.2012
#
#   → 2518 programs registered as of 25.09.2013
#   → 2524 programs registered as of 25.10.2013
#   → 2540 programs registered as of 20.11.2013
#
#   → 2600 programs registered as of 26.01.2014
#   → 2609 programs registered as of 26.02.2014
#   → 2628 programs registered as of 22.06.2014
#   → 2642 programs registered as of 12.07.2014
#   → 2644 programs registered as of 14.09.2014
#
#   → 2652 programs registered as of 10.01.2015
#   → 2742 programs registered as of 04.06.2015
#   → 2780 programs registered as of 26.07.2015
#   → 2781 programs registered as of 02.08.2015.
#   → 2795 programs registered as of 06.09.2015. 
#   → 2800 programs registered as of 24.09.2015.
#   → 2803 programs registered as of 28.09.2015. 
#   → 2817 programs registered as of 26.11.2015.
#
#   → 2903 programs registered as of 01.04.2017.
#   → 2976 programs registered as of 09.08.2017.
#   → 3231 programs registered as of 23.10.2017.
#   → 3264 programs registered as of 01.12.2017.
#
#   → 3314 programs registered as of 04.01.2018.
#   → 3405 programs registered as of 25.08.2018.
#   → 3444 programs registered as of 14.10.2018.
#
#   → 3582 programs registered as of 14.04.2019.
#   → 3594 programs registered as of 12.05.2019.
#   → 3616 programs registered as of 17.09.2019.
#   → 3635 programs registered as of 30.11.2019.
#
# =========================================================================== #

Keeping empty directories in the /Programs/ hierarchy

By default, class RBT::ToCurrent and class RBT::SymlinkThisProgram will remove empty subdirectories of programs that are installed in an AppDir-like fashion.

This is not always wanted, though, so a commandline flag exists to disable the auto-removal of empty directories:

ry feh --ntrad --do-not-remove-empty-directories

Ensuring that all relevant directories exist

To ensure that all important directories exist you can invoke the following method to do so:

rbt --ensure-all-directories-exist

class RBT::QueryFileAssociation

Sometimes you may wish to find out to which particular program any file belongs to - for example, the header called stdio.h.

class RBT::QueryFileAssociation does answer this.

Usage example:

require 'rbt/utility_scripts/query_file_association.rb'

RBT::QueryFileAssociation.new(of: 'libc.so')
RBT::QueryFileAssociation.new(of_this_file: 'vlc')
RBT::QueryFileAssociation.new(of_this_file: 'stdio.h')

Any registered input will do fine, be it a binary, a library or a .h header file. The file must be registered as part of the RBT project, though. If that condition is met then this class will report, on the commandline, more information as to which program this input-file belongs to.

Do note that before January 2020, multiple associations were not reported properly. Since as of January 2020, if for example a .h header file belongs to several programs then this information will be properly displayed on the commandline. That way if a header file, such as stdio.h, belongs to different programs, all of them will be reported as well.

Removing outdated (local) archives

I tend to batch-update the RBT project quite frequently. As a result of this, I end up having lots of local archives, usually in the .tar.xz format. Since removing these manually is annoying, a class has been added on 07.01.2020, called RBT::RemoveOutdatedArchives.

This class is presently not very flexible, since I only used it to remove a specific set of programs.

For example:

RBT::RemoveOutdatedArchives.new(:kde_plasma)

This would remove all outdated archives belonging to the KDE Plasma stack.

Another example:

RBT::RemoveOutdatedArchives.new(:kde_applications)

The above would remove all outdated KDE5 applications (if all goes well).

In the future this class may be extended, but for now this has to suffice.

Also note that whenever this paragraph refers to KDE, it refers to KDE5. This may change in the future (e. g. KDE6 release and similar), but for now (January 2020) it is up-to-date and thus correct.

class RBT::SymlinkThisProgram

class RBT::SymlinkThisProgram can be used to symlink an AppDir program into the /usr/ prefix (or another target main prefix in use).

You can also decide to only symlink certain subdirectories, such as bin/.

Example:

cd /home/Programs/Geany/Current # first we cd to this directory
rnsymc --bin
# rnsymc --bin/ ← This variant works as well.

Do note that on my computer system rnsymc is an alias to the executable called symlink_this_program.

In January 2020 the --force commandline argument was added. This allows us to remove the target files at /usr/bin/, before the symlink-operation is done. Only use this when you are absolutely certain that you wish to remove the target file(s) from there.

class RBT::BuildDetector

class RBT::BuildDetector is used to determine the build type of a given directory. This should be the extracted archive, such as for htop-1.2.0.tar.xz, the class will expect of the user to have changed the directory into htop-1.2.0/. Then it will try to determine the build type, e. g. configure or cmake or meson/ninja. Note that this may fail, so do not rely too much on it. For most regular software projects out there, this class should work fine, though.

KDE

Many KDE components (== programs) are registered in the rbt project.

These programs usually have the registered tag kde or kde5, in the respective .yml file. You can also search for these via class Cookbooks::SearchForTags; just pass in kde or kde5 as argument to that file/class. (I have the file aliased to "stag", so I can just do "stag kde5" quickly.)

You can also batch-download KDE components - that is, to download them all to the local filesystem (aka to your local harddisc).

There are several ways how to achieve this. Either call the respective class directly. These .rb files will reside in the subdirectory check_for_updates/.

Alternatively, you can invoke them directly from the commandline like so:

rbt --update-kde-plasma

This will download all of the KDE5 plasma components (unless they are already available locally).

If you wish to download the KDE5 applications, then you can try this command:

rbt --update-kde-applications

Aliases exist to the above:

rbt --kde-apps
rbt --compile-kde-apps

In general, the rbt project attempts to allow aliases, for sake of convenience. The "official" entry points are usually the longer ones though, as they are less likely to conflict with other instructions given on the commandline. ("update kde applications" is more explicit than "kde apps", but the latter is shorter, and thus may be more convenient when used on the commandline).

To update the KDE5 Frameworks, do:

rbt --update-kde-framework

To update the KDE5 porting aids, do:

rbt --update-kde-porting-aids

Note that you could also use '_' tokens in the above invocation examples, or omit these, so the following would work just as well:

rbt --update_kde_porting_aids
rbt --updatekdeportingaids

You can also query the KDE "status", that is, which variant is currently the most recent one. Use something like this:

rbt --kde-status

The output of the above command, on the commandline, would be something like this:

KDE Frameworks version:      5.50
KDE Applications version: 18.08.1
KDE Plasma version:        5.13.5

Or, in December 2019 on my computer system:

KDE Frameworks version:      5.65
KDE Applications version: 19.12.0
KDE Plasma version:        5.17.4

This information may be useful if you go to the official KDE homepage and want to find out whether there have been any updates recently. Then you can compare your local version to the remote one and see whether you may want to update the KDE components that you are presently using.

Since as of the year 2018, I am able to compile just about all of the KDE components, through the rbt project (excluding a few troublemakers, in particular of the qt-chromium web-stack)).

My primary target among these KDE components is usually the KDE konsole ( e. g. at https://download.kde.org/stable/release-service/19.12.0/src/konsole-19.12.0.tar.xz ).

If I am able to compile the KDE konsole then I can usually compile the rest of KDE as well. It is a fine konsole; give it a try if you haven't yet.

You can also use a number to compile a particular KDE component.

For example:

ry --kde1
ry --kde2
ry --kde3

Will compile these registered kde components. The reason as to why this functionality exists is that it makes batch-compiling the KDE components easier. I simply have to increment a number and compile the KDE suite en suite, without having to remember the name (my memory is quite bad, so using numbers is so much easier).

You can drop the -- too, of course:

ry kde1
ry kde2
ry kde3

You can chain these via a range-like syntax:

ry kde1..100

This would compile the first 100 KDE components (if all goes well and no errors have occurred).

You can also be more verbose, if you'd like to. The following demonstrates this:

ry --kde1..kde250

The linear order of these numbers can be seen in the file rbt/yaml/chained_programs.yml, under the respective KDE component (kde-plasma, kde-applications and so forth).

If you wish to compile all of the KDE5 plasma components, you can use either one of the following variants:

rbt --compile-all-kde5-plasma-components
rbt --compile-plasma

This is a convenience-shortcut. Of course the option with --batch= also works.

To update all kde-applications you could do the following:

rbt --update-kde-apps

To compile all of KDE, try this:

rbt --compile-all-of-kde

Installing (compiling) the dependencies of a given program at hand

The required dependencies of a given program at hand can be compiled or installed, respectively, via the following means:

rbt rails --install-its-dependencies

This will NOT install the program itself, though - in this case rails. It is just a quick-and-dirty way to compile dependencies, without any further checking. All entries specified in required_deps_on: will be processed.

Available program versions

The following list, as html-pre tag, shows which versions are currently tracked, including the URL to these programs.

The four entries are:

(1) program name

(2) program version

(3) last update of this program (in regards to the cookbooks project)

(4) remote URL

The most useful entry of this list for users of the RBT project is probably the remote URL entry there - the most right entry.

Note that some old programs are no longer available, which explains why they have not been updated in many years. Can't update what is no longer there now, can you. :)

(Very outdated program entries will eventually be removed from the Cookbooks listing, in particular if the program is no longer available.)

Currently this project does not accept donations, but in the future this may be subject to change, including more information in how you could support the project.

Of course documentation, patches, bug fixes and so forth are always appreciated.

Contact information

If you have specific suggestions to make this gem more useful for others, please drop me an email at:

shevegen@gmail.com

(Please keep in mind that responding to emails may take a while depending on the amount of work I may have at that moment in time due to reallife.)

Thank you.