Logstash Output Monasca-Log-Api Plugin

This module is a logstash-output-plugin for the Monasca Log Api.

Get latest stable version

https://rubygems.org/gems/logstash-output-monasca_log_api

gem install logstash-output-monasca_log_api

Build from source

Requirements

Install requirements

RVM

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash
source /home/vagrant/.rvm/scripts/rvm

JRuby

rvm install jruby

Clone project

git clone https://github.com/FujitsuEnablingSoftwareTechnologyGmbH/logstash-output-monasca_api.git

Use jruby

rvm use jruby

Install bundler

gem install bundler

Fetch dependencies

bundle install

Unit tests

Run unit tests with:

bundle exec rspec

Run specified test

bundle exec rspec spec/outputs/monasca/monasca_log_api_client_spec.rb

Run coverage

JRUBY_OPTS="-Xcli.debug=true --debug" bundle exec rspec

Coverage report can be found in ./coverage

Code style check

rubocop lib/

Deploy Plugin to logstash

Source: https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_output_plugin.html#_building_and_testing_4

Build Gemfile

First we need to create a Gemfile.

What is a Gemfile? Gems are packages for the Ruby programming language. A Gemfile defines all dependencies which are neccessary to build the product.

How to build a Gemfile? Run this:

gem build logstash-output-monasca_log_api.gemspec

Deploy Gemfile to logstash

  • Download logstash (>=2.2.0)
  • Extract logstash and navigate into the folder
  • Add this line to the Gemfile
  gem "logstash-output-monasca_log_api", :path => "/vagrant_home/cloudmonitor/logstash-output-monasca_log_api"
  • logstash-output-monasca_log_api = name of the gem
  • /vagrant_home/cloudmonitor/logstash-output-monasca_log_api = Path to git workspace

Run this command from logstash folder to install the plugin

bin/plugin install --no-verify

Verify installed plugins: With bin/plugin list you can check installed plugins. There should be logstash-output-monasca_log_api.

Start logstash output plugin

Configuration

Plugin name: monasca_log_api

Save the configfile wherever you like. For example ~/logstash.conf

name description type required default example
monasca_log_api_url monasca log api url string true https://192.168.10.4:5607/v3.0
monasca_log_api_insecure set to true if monasca-log-api is using an insecure ssl certificate boolean false false
keystone_api_url keystone api url string true http://192.168.10.5:35357/v3
keystone_api_insecure set to true if keystone is using an insecure ssl certificate boolean false false
project_name Keystone user credentials: project name string true monasca
username Keystone user credentials: username string true admin-agent
password Keystone user credentials: password string true password
user_domain_name Keystone user credentials: user domain name string true default
project_domain_name Keystone user credentials: project domain name string true default
dimensions global array dimensions in form of key-value pairs to describe the monitored node array false ['app_type:kafka', 'priority:high']
num_of_logs maximum number of logs that are send by one request to monasca-log-api number false 125
elapsed_time_sec send logs if the maximum elapsed time in seconds is reached number false 30
delay delay time in seconds to wait before checking the elapsed_time_sec again number false 10
max_data_size_kb maximum size in kb of logs that are send by one request to monasca-log-api number false 5120

Example configuration files

Simple
output {
  monasca_log_api {
    monasca_log_api_url => "http://192.168.10.4:5607/v3.0"
    keystone_api_url => "http://192.168.10.5:35357/v3"
    project_name => "cmm"
    project_domain_name => "Default"
    username => "cmm-operator"
    user_domain_name => "Default"
    password => "admin"
  }
}
Complete
output {
  monasca_log_api {
    monasca_log_api_url => "https://192.168.10.4:5607/v3.0"
    monasca_log_api_insecure => true
    keystone_api_url => "http://192.168.10.5:35357/v3"
    keystone_api_insecure => false
    project_name => "cmm"
    project_domain_name => "Default"
    username => "cmm-operator"
    user_domain_name => "Default"
    password => "admin"
    dimensions => ["hostname: monasca", "ip:192.168.10.4"]
    num_of_logs => 125
    delay => 10
    elapsed_time_sec => 30
    max_data_size_kb => 5120
  }
}

Run

bin/logstash -f ~/logstash.conf

Run in debug mode

bin/logstash -f ~/logstash.conf --debug

Specify log output file

bin/logstash -f ~/logstash.conf -l /var/log/monasca/log/agent/test-log-agent.log

Logstash File Input plugin

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

Configuration

Local dimensions can be added with add_field setting

input {
  file {
    add_field => { "dimensions" => { "service" => "monasca-api" }}
    add_field => { "dimensions" => { "language" => "java" }}
    add_field => { "dimensions" => { "log_level" => "error" }}
    path => "/var/log/monasca/api/error.log"
    }
  }

Open tasks

  • Language translations (Replace hardcoded String messages with a configuration/language file)
  • Exception handling (monasca-api requests)