oswitch - virtual environments for flexible and reproducible bioinformatics analyses

More and more bioinformatics software are being "containerised" with docker, making them installable on personal computers and compute clusters alike with a single command in a reproducible manner. However, using containerised software can still be a challenge as the containers are effectively a different operating system altogether. oswitch enhances usability of docker containers by a) automatically making available the local file-system and user profile inside the container, b) using the host user's login shell and current working directory as the entry point. The net effect is similar to entering "virtual environment" on the host system containing specific versions of software of interest.

mymacbook:~/2015-02-01-myproject> abyss-pe k=25 reads.fastq.gz
    zsh: command not found: abyss-pe

# List available images.
mymacbook:~/2015-02-01-myproject> oswitch -l
    yeban/biolinux:8
    ubuntu:14.04
    ontouchstart/texlive-full
    ipython/ipython
    hlapp/rpopgen
    bioconductor/release_sequencing

# Enter the continaer and run commands interactively.
mymacbook:~/2015-02-01-myproject> oswitch biolinux
    ###### You are now running: biolinux in container biolinux-7187. ######
biolinux-7187:~/2015-02-01-myproject> abyss-pe k=25 reads.fastq.gz
    [... just works on your files where they are...]
biolinux-7187:~/2015-02-01-myproject> exit
mymacbook:~/2015-02-01-myproject>
    [... output is where you expect it to be ...]


# Use a container non-interactively.
pixel:~/test/ $ oswitch yeban/biolinux blastp -remote -query mygene.fa -db nr > mygene_blastp_nr.txt

Installation

oswitch first requires a working docker install.

On Ubuntu

A deb package of oswitch is available in BioLinux repository for Trusty, Vivid and Jessie. This will install oswitch systemwide:

$ sudo add-apt-repository ppa:nebc/bio-linux
$ sudo apt-get update
$ sudo apt-get install oswitch

Using homebrew (on Mac)

Depending on whether homebrew is installed systemwide or only for your user, this will install oswitch systemwide or only for your user:

brew tap homebrew/science
brew install oswitch

Using RubyGems (on Linux & Mac)

Requirements: Ruby 2.0 or higher (available by default on Mac and through package managers on Linux). This will install oswitch systemwide:

$ sudo gem install oswitch

Usage note

  • Volume mounting on Mac OS hosts is imperfect.
  • SELinux must be disabled on CentOS hosts for mounting volumes to work (check the SELinux documentation to see the implications of doing this).
  • We have tested oswitch on Debian, Ubuntu, CentOS based docker images on the following hosts:
    • Mac OS X Yosemite, El Captain
    • Ubuntu 14.04.1
    • CentOS 7

FAQ

Q. Directories mounted within container on Mac host are empty.

The problem is, on Mac boot2docker is the real host, not OS X. oswitch can mount only what's available to it from boot2docker. For example, /Applications.

Run boot2docker ssh ls /Applications and you will find it empty as well.

The workaround is to correctly mount the directories you want in boot2docker first.

boot2docker down
VBoxManage sharedfolder remove boot2docker-vm --name Applications
VBoxManage sharedfolder add boot2docker-vm --name Applications --hostpath /Applications
boot2docker up
boot2docker ssh "sudo mkdir -p /Applications && sudo mount -t vboxsf -o uid=1000,gid=50 Applications /Applications"
Q. cwd is empty in the container

This means the said directory was not mounted by oswitch, or was incorrectly mounted. On Linux host, directories that can conflict with paths within container are not mounted. On Mac, boot2docker can get in the way.

Please report this on our issue tracker. To help us debug, please include:

  1. the directory in question
  2. the operating system you are running
Q. oswitch does not work with my docker image

Please report this on our issue tracker with oswitch's output. If the image you are using is not available via docker hub or another public repository, please include the Dockerfile as well.

Q. How does all this work?

We create a new image on the fly that inherits from the given image. While creating the new image we execute a shell script that installs packages required for oswitch to work and creates a user in the image (almost) identical to that on the host.

Q. How can I connect to an existing container?

In another shell, use docker ps to see which containers are already running. Copy the identifier from the CONTAINER ID (column this looks something like 37e4e6ada6a4), and use it to run docker attach 37e4e6ada6a4 (replace with your container's id). This will create a new ssh connection to your existing container.

Contribute

$ git clone https://github.com/yeban/oswitch
$ cd oswitch
$ gem install bundler && bundle
$ bundle exec bin/oswitch biolinux

Contributors & Funding

Development funded as part of NERC Environmental Omics (EOS) Cloud at
Wurm Lab, Queen Mary University of London.