contained_mr

Map-Reduce where both the mappers and the reducer run inside Docker containers.

Prerequisites

contained-mr requires docker version 1.9 or above. If used, docker-swarm must be at vesion 1.0 or above.

Development Environment

contained-mr requires access to a Docker Swarm daemon. The easiest way to bring up a development setup is to install Docker Machine, Docker Swarm and VirtualBox.

The commands below install the prerequisites on OSX using Homebrew.

brew install docker docker-machine docker-swarm
brew tap Caskroom/cask
brew cask install virtualbox

Create a Docker Swarm cluster. This is a one-time setup.

TOKEN=$(docker-swarm create)
docker-machine create --driver virtualbox --engine-storage-driver overlay \
    --swarm --swarm-master --swarm-discovery "token://$TOKEN" swarm-master
docker-machine create --driver virtualbox --engine-storage-driver overlay \
    --swarm --swarm-discovery "token://$TOKEN" swarm-agent-1
docker-machine create --driver virtualbox --engine-storage-driver overlay \
    --swarm --swarm-discovery "token://$TOKEN" swarm-agent-2

Set up the local environment to point to the Swarm master. This must be executed in every shell where contained-mr is used.

eval "$(docker-machine env --swarm swarm-master)"

Cleanup

When tests go wrong, they often leave dead containers and images behind. Removing all stopped containers and all unused images is a big stick that works quite well in development environments.

docker ps --all --quiet --no-trunc | xargs docker rm
docker images --quiet --no-trunc | xargs docker rmi

Testing Against Experimental Docker

This cluster setup will use the development version of Docker.

TOKEN=$(docker-swarm create)
ISO_URL=$(curl \
    https://api.github.com/repos/ahbeng/boot2docker-experimental/releases/latest \
    | grep -o https://.*/boot2docker.iso)
docker-machine create --driver virtualbox --engine-storage-driver overlay \
    --swarm --swarm-master --swarm-discovery "token://$TOKEN" \
    --swarm-image "dockerswarm/swarm:master" \
    --virtualbox-boot2docker-url $ISO_URL swarm-master
docker-machine create --driver virtualbox --engine-storage-driver overlay \
    --swarm --swarm-discovery "token://$TOKEN" \
    --swarm-image "dockerswarm/swarm:master" \
    --virtualbox-boot2docker-url $ISO_URL swarm-agent-1
docker-machine create --driver virtualbox --engine-storage-driver overlay \
    --swarm --swarm-discovery "token://$TOKEN" \
    --swarm-image "dockerswarm/swarm:master" \
    --virtualbox-boot2docker-url $ISO_URL swarm-agent-2

Contributing to contained_mr