COMAP

Container Orchestrator Metrics Aggregator for Prometheus.

COMAP aggregates prometheus-formatted metrics from different instances of a service in a Docker containers cluster environment (swarm mode) using Docker API for service containers discovery.

By design, a single COMAP instance should monitor a single service.

It is best used within a container belonging to the same network as the services it follows.

A service and container labels is added to each metric with respectively service name and original container name.

Aggregated metrics intended to be fetched as a target by Prometheus are exposed on port 9397.

Installation

Add this line to your application's Gemfile:

gem 'comap'

And then execute:

$ bundle

Or install it yourself as:

$ gem install comap

Usage

Launch COMAP as a Swarm service, specifying the service (specified by service_name and its internal port) to follow.

It need to access the docker socket of a manager. For a small cluster containing managers only, it is possible to mount the docker socket as volume. In other (most) case, it is recommended to use tls connection.

sdir=/run/secrets docker service create \
  --network your_network \
  --publish '9397:9397'  \
  --secret cert.pem --secret key.pem --secret ca.pem \
  makeorg/comap          \
  -H manager -p 2376     \
  -s client_cert=$sdir/cert.pem,client_key=$sdir/key.pem,ca_file=$sdir/ca.pem \
  -n your_network service_name:port

If standalone or within your docker image:

Usage: comap [options] swarm_service
    -H, --docker_host HOST         Docker Engine API Host
                                   (default: http://127.0.0.1)
    -p, --docker_port PORT         Docker Engine API Port (default: 2375)
    -s, --docker_ssl k1=v1,k2=v2   Docker Engine API SSL (default: none)
    -n, --docker_network NETWORK   Docker swarm service network (default: none)
    -m, --metrics_path             Metrics path (default: none)
    -h, --help                     Display this screen

Quickstart

Kitchen is currently used during development for test purpose. However, the kitchen test suite can be set up if you just want to quickly see COMAP running :

$ kitchen converge

That should brings up a functional docker-in-docker swarm environment with a manager/worker and a stand-alone consul. Configuration is automatically handled with Chef and some Chef Platform cookbooks.

$ kitchen login comap-swarm-manager-centos-7

You can now play with the services running in the swarm :

A metrics service with two replicas exposing prometheus-formatted metrics on 5678 port

  [root@comap-swarm-manager-centos-7 /]# curl localhost:5678
  # TYPE metrics_sample_seconds gauge
  metrics_sample_seconds 42
  # TYPE metrics_sample_2_seconds gauge
  metrics_sample_2_seconds 24

A single COMAP service exposing aggregated metrics from previous service on 9397 port

  [root@comap-swarm-manager-centos-7 /]# curl localhost:9397
  # TYPE metrics_sample_seconds gauge
  metrics_sample_seconds{service="metrics",container="metrics.1"} 42
  metrics_sample_seconds{service="metrics",container="metrics.2"} 42
  # TYPE metrics_sample_2_seconds gauge
  metrics_sample_2_seconds{service="metrics",container="metrics.1"} 24
  metrics_sample_2_seconds{service="metrics",container="metrics.2"} 24

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Please read carefully CONTRIBUTING.md before making a merge request.

License and Author

Author:: Sylvain Arrambourg ([email protected])
Author:: Samuel Bernard ([email protected])

Copyright (c) 2017-2018 Make.org

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.