forthebadge forthebadge Gem Version

This gem was last updated on the 20.01.2024 (dd.mm.yyyy notation), at 18:58:06 o'clock.

The Backup Paradise project

This project attempts to assist you - and me - in backing up data to an external USB device specifically.

The project is mostly catering to my own use cases, so it may not be very useful to other ruby users. I will try to make adjustments so that other ruby users can use it (see the file called config.yml), but this is an ongoing effort that may take some time.

The project is primarily just a fancy recursive copy operation, but it will do some minor additional notifications on the commandline, make use of colours (if the colours gem has been installed), automatically create some log files and even provide ruby-gtk3 bindings, as well as tk-bindings (although the latter ones are currently incomplete).

So it is a recursive copy script with some add-ons really, nothing really sophisticated aside from this on top.

The project has one major goal and a few minor goals.

The major goal is to be usable as a backup-assistant, on both linux, but also on windows systems.

The minor goals include:

  • Flexibility, to also allow us to back up only individual files or individual directories. The user should be able to specify what has to be backed up.

  • Notify the user about the status of the backup-operation and prior backup-operations.

  • Provide some simple GUI bindings to use this project, in particular for both ruby-gtk and ruby-tk. The latter so that we can use this on windows.

General usage of the BackupParadise project - backing up data via ruby

For the time being, I have aliased the main executable at bin/backup_paradise onto:

backup_paradise

To then use this, I can issue the following commands:

backup_paradise usb1
backup_paradise usb2
backup_paradise tousb1
backup_paradise tousb2
backup_paradise tousb3
backup_paradise tousb4

Depending on where the external USB device is mounted.

What does an argument, such as usb1, imply?

This used to refer to the hardcoded path at /Mount/USB1/ on my home system. usb2 would then point at /Mount/USB2/ and so forth.

These 'main aliases' can also be accessed via some shortcuts, such as "tousb1", without the quotes. See the examples listed above.

I adopted this scheme after having used GoboLinux. Note that this used to be valid from December 2018 up to May 2021.

In May 2021 this approach was changed. While I still use the above instructions, such as via backup_paradise usb1, a .yml file now decides the target directory. So, usb1, usb2 and so forth, are now simply aliases to whatever the .yml file specifies. The advantage of this approach is that other users can re-define the targets there - simply have a look at the config.yml file and change it if you would like to. (Of course overruling this via the commandline still works, so you don't really need to have to modify the config.yml file.)

Other mount points can also be used, although not with a convenience shortcut. Have a look at the option --backup-to=/opt/ documented elsewhere in this file, to specify another base-directory to use.

The configuration file called config.yml

The BackupParadise project comes with a small configuration file, called config.yml, in the yaml/ subdirectory.

This is an optional file, though - if it does not exist then the project will ignore this file completely and use internal defaults as-is. But if it is available, and it can be found by ruby, then it will be used as the default configuration.

Since as of September 2021* the **config.yml file has the following five main configuration settings:

use_this_program_to_rename_tabs: String
show_popup_notification:         Boolean
use_system_cp:                   Boolean
backup_these_directories:        Array
local_audio_directory:           String

The first three entries are not so important, but the fourth entry is quite important, because this Array specifies which directories we want to back up and thus copy. The fifth entry can be used to denote your local collection of audio files, if these are kept in a directory. I use /home/x/songs/ for this.

More options may be added in the future, but I will try to only add options that make sense, rather than overwhelm users with lots of configuration options.

The individual aliases, such as usb1, usb2, and so forth, are also specified now:

usb1: /Mount/USB1/
usb2: /Mount/USB2/
usb3: /Mount/USB3/
usb4: /Mount/USB4/
usb5: /Mount/USB5/

The rationale for these default to my home setup, where I mount devices to these directories (on Linux).

Log files

The BackupParadise project will try to log as to what has happened, since as of May 2018. Several different log files may be created.

If there is more than one log file stored at the target device then the current behaviour is to delete all but one log file.

Another log file that may commonly be used is the one that targets the file /Depot/Temp/backup.log. This will, however had, only work if the directory /Depot/Temp/ exists, which is the case on my home system. The whole gem here is heavily tailored to my own needs - I can adapt it to meet other people's wishes if necessary, of course.

The format of the log file going into backup.log is simply how long it took to make the backup, in seconds, and the exact date when that particular log file was created. This is equal to when the backup-operation happened, too.

Backing up individual directories

If you have a directory called FOOBAR/, then you can back it up simply by issuing:

backup_paradise --FOOBAR
backup_paradise FOOBAR

Whichever variant you prefer. Note that this requires that the external device has been mounted already. You can set where the device should reside via:

BackupParadise.set_mounted_path()

Simply pass the directory that you may wish to use for backup purpose there.

Note that class AdvancedBackup will assume a certain target device automatically, tailored to my own system. This is usually at /Mount/USB1 for the first mounted USB device on my system.

If you wish to overrule that and use another target, which you may most likely want to do, then you can use any of the following commandline flags for this:

backup_paradise --use-this-as-target-for-backup=/opt/
backup_paradise --use-this-as-target=/opt/
backup_paradise --backup-to=/opt/

In this case /opt/ would be the target that you use as source directory for the backup-operation.

If you wish to backup the /Programs/ hierarchy, should you keep an AppDir layout on your system, then this commandline invocation may be of help:

backup_paradise --programs-dir

To backup the /home/x/studium/ directory, the following commands are possible:

backup_paradise --studium-dir
backup_paradise --studium-directory
backup_paradise --studium

To backup, on my home system, the /AUTOGENERATED/ directory, I do:

backup_paradise --autogenerated

Miscellaneous comments

The BackupParadise project has been rewritten a few times already over the years.

In the past there was an instance variable in class AdvancedBackup called @last_backup_directory, which was pointing to an absolute path such as /Mount/USB1/last_backup-13.08.2018-21:11:15/.

It is no longer in use, but I keep it as a reference hint to myself - who knows what may be changed in the future.

Specific examples

This subsection details some invocation examples in a short way.

Some of these will only work on my home system for the time being, until I have sufficiently changed the project to allow more flexibility in this regard.

Backup all audio-files to /opt/:

backup_paradise --backup-to=/opt/ --audio-dir

Backup all relevant entries into the default chroot target (on my home system):

backup_paradise --chroot

How do I make use of this project

I make use of the backup_paradise project to back up relevant data on my home system. For example, all source-archives that are registered within the RBT project I also keep available locally. Then there is my ruby code, other files, and possibly books, papers and tutorials stored in .pdf format.

In September 2020, the regular backup size was 54G per backup-action, mostly owing to the source archives (krita alone occupies more than 167 MB, for example). This is way too much, in my opinion, because it takes about half an hour or so to back up on USB 2.0 already.

I will have to keep reducing file size and removing stuff that I no longer need; aka do some clean-up. Or perhaps hope that future USB variants will improve things here.

At any rate, this subsection shows just a bit how I may use the project.

My usual go-to alias is:

rbackup usb1

to backup to the mount point /Mount/USB1.

This is the command invocation I tend to use most frequently so.

class BackupParadise::Backup

This class is the main interface for the BackupParadise project.

For a list of documented options, pass in --help.

A few examples will be mentioned here as well, in a succinct manner:

rbackup --pwd # backup into the current working directory

BackupParadise.simple_backup

BackupParadise.simple_backup can be used as a simpler alternative to class BackupParadise::Backup. While the latter class is very feature-rich, sometimes you may want to use BackupParadise.simple_backup() instead.

Usage examples:

BackupParadise.simple_backup(:audio, '/Mount/HDD1/')
BackupParadise.simple_backup(:audio, :usb1)

Backup time (statistics)

This subsection is just a remainder for me how long backing up all my dataset (which is a LOT) takes. These times are gathered on Linux.

Ideally this time should be as small as possible, but evidently, the more time that has to be transferred, the longer it will take. This refers to about 100 GB or more right now. I am open for any suggestions to cut the time down.

The measurements happen when there is a low CPU load. 

04.06.2021: 39.17 minutes

GUI components

Right now the GTK3 version (ruby-gtk3) is the most advanced GUI component of this project - but it is still rather limited. The primary point of the gtk-widgets for the project is to demonstrate what could be done: we want to show what is possible, in the event that others may want to build upon these ideas.

Additionally I am working on ruby-tk code as well, just so that this project can be used on windows (addendum in August 2021: this has been superceded by ruby-libui, which is a LOT simpler to use on windows). The priority for ruby-tk is lower than that of the ruby-gtk3 code. (In the future, one day, we may try to use the same, unified code base for GUIs, but this is a minor, additional goal.)

Yet another use case is ruby-libui, so in the long run it is planned to add at the least three different widget sets, as well as "GUIs" for the world wide web, such as via sinatra.

You can try to invoke the GUI components from the commandline, possibly via any of the following flags:

backup_paradise --gui
backup_paradise --gtk # This variant now defaults to ruby-gtk3.

Or more specifically:

backup_paradise --gtk3

There used to be gtk2-bindings, but they currently (May 2021) do not work very well. In August 2021 I abandoned the ruby-gtk2 code completely - new GUI code will be written either in ruby-gtk3 and/or in ruby-libui. ruby-gtk2 is still nice, but ruby-gtk3 allows for more flexibility, in particular in regards to CSS.

In September 2021 I rewrote the old ruby-gtk3 code. The current iteration now looks like this:

The code base for that class is now largely a module, which I use for libui-bindings too. In the long run I would like to have ruby-tk and fxruby support as well as for the www (via sinatra and .cgi files). Stay tuned in this regard, but don't expect this to come too soon.

Automounting

Since as of March 2023 it is possible to automount a second harddisc (should it exist). This is currently only the case when the underlying system is a roebe-system, and when the default mount target /Mount/HDD, is NOT mounted yet. If this is the case then automounting will occur, using a hardcoded command.

In the future this may be improved, including automounting of USB-devices. This has to be revisited eventually; for now the code that was added has to suffice.

Using the backup_paradise gem on Windows

On Linux the code I write in ruby tends to work just fine. On Windows, this is a bit more difficult. For instance, Windows does not know symlinks in the way that Linux does. I tried this recently in July 2023, and had awkward error messages - for instance, if you have a directory called C++ or Cpp and then symlink to it via C++ or Cpp. The C++ symlink in particular led to some oddities on windows where I could not copy the whole directory. This was very confusing. Removing the symlink suddenly allowd the copy-operation. Windows is strange.

At any rate, since symlinks are not so useful on windows, and may lead to errors, since as of August 2023 the backup_paradise gem will remove all symlinks found in the backup-target IF a NTFS file system is used. Not everyone may want this behaviour by default, but for my use case, I found it to work better. I document this behaviour here in the event that other users may want to change this for their own use cases.

Licence

Since as of August 2021 the MIT licence is used for new releases of this gem. While the GPL is a fine licence in and by itself, I simply do not have the time or willingness to want to enforce it for such a simple project like backup_paradise. MIT is much simpler to adhere to as well, so there we go.

Contact information and mandatory 2FA (no longer) coming up in 2022 / 2023

If your creative mind has ideas and specific suggestions to make this gem more useful in general, feel free to drop me an email at any time, via:

shevy@inbox.lt

Before that email I used an email account at Google gmail, but in 2021 I decided to slowly abandon gmail, for various reasons. In order to limit the explanation here, allow me to just briefly state that I do not feel as if I want to promote any Google service anymore when the user becomes the end product (such as via data collection by upstream services, including other proxy-services). My feeling is that this is a hugely flawed business model to begin with, and I no longer wish to support this in any way, even if only indirectly so, such as by using services of companies that try to promote this flawed model.

In regards to responding to emails: please keep in mind that responding may take some time, depending on the amount of work I may have at that moment. So it is not that emails are ignored; it is more that I have not (yet) found the time to read and reply. This means there may be a delay of days, weeks and in some instances also months. There is, unfortunately, not much I can do when I need to prioritise my time investment, but I try to consider all feedback as an opportunity to improve my projects nonetheless.

In 2022 rubygems.org decided to make 2FA mandatory for every gem owner eventually:

see https://blog.rubygems.org/2022/06/13/making-packages-more-secure.html

Mandatory 2FA will eventually be extended to all rubygems.org developers and maintainers. As I can not use 2FA, for reasons I will skip explaining here, this means that my projects will eventually be removed, as I no longer have any control over my projects hosted on rubygems.org (because I can not use 2FA).

At that point, I no longer have any control what is done to my projects since whoever is controlling the gems ecosystem took away our control here. I am not sure at which point ruby became corporate-controlled - that was not the case several years ago, so something has changed.

Ruby also only allows 2FA users to participate on the issue tracker these days:

https://bugs.ruby-lang.org/issues/18800

But this has been reverted some months ago, so it is no longer applicable. Suffice to say that I do not think that we should only be allowed to interact on the world wide web when some 'authority' authenticated us, such as via mandatory 2FA, so I hope this won't come back again.

Fighting spam is a noble goal, but when it also means you lock out real human people then this is definitely NOT a good situation to be had.