logstash-output-monasca_log_api

Travis Build Status

This module is a logstash-output-plugin for the Monasca Log Api.

Compatible logstash version: Logstash 7.x

Get latest stable version

https://rubygems.org/gems/logstash-output-monasca_log_api

gem install logstash-output-monasca_log_api

Build from source

Requirements

Install requirements

RVM

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash
source /home/vagrant/.rvm/scripts/rvm

JRuby

rvm install jruby

Clone project

git clone https://github.com/logstash-plugins/logstash-output-monasca_log_api.git

Use jruby

rvm use jruby

Install bundler

gem install bundler

Fetch dependencies

bundle install

Unit tests

Run unit tests with:

bundle exec rspec

Run specified test

bundle exec rspec spec/outputs/monasca/monasca_log_api_client_spec.rb

Run coverage

JRUBY_OPTS="-Xcli.debug=true --debug" bundle exec rspec

Coverage report can be found in ./coverage

Code style check

rubocop lib/

Deploy Plugin to logstash

Source: https://www.elastic.co/guide/en/logstash/current/_how_to_write_a_logstash_output_plugin.html#_building_and_testing_4

Build Gemfile

First we need to create a Gemfile.

What is a Gemfile? Gems are packages for the Ruby programming language. A Gemfile defines all dependencies which are neccessary to build the product.

How to build a Gemfile? Run this:

gem build logstash-output-monasca_log_api.gemspec

Deploy Gemfile to logstash

Run this command from logstash folder to install the plugin

bin/logstash-plugin install /my/logstash/plugins/logstash-output-monasca_log_api-1.0.0.gem

Verify installed plugins: With bin/plugin list --verbose you can check installed plugins. There should be logstash-output-monasca_log_api.

Start logstash output plugin

Configuration

Plugin name: monasca_log_api

Save the configfile wherever you like. For example ~/logstash.conf

name description type required default example
monasca_log_api_url monasca log api url string true https://192.168.10.4:5607/v3.0
monasca_log_api_insecure set to true if monasca-log-api is using an insecure ssl certificate boolean false false
keystone_api_url keystone api url string true http://192.168.10.5:35357/v3
keystone_api_insecure set to true if keystone is using an insecure ssl certificate boolean false false
project_name Keystone user credentials: project name string true monasca
username Keystone user credentials: username string true admin-agent
password Keystone user credentials: password string true password
user_domain_name Keystone user credentials: user domain name string true default
project_domain_name Keystone user credentials: project domain name string true default
dimensions global array dimensions in form of key-value pairs to describe the monitored node array false ['app_type:kafka', 'priority:high']
num_of_logs maximum number of logs that are send by one request to monasca-log-api number false 125
elapsed_time_sec send logs if the maximum elapsed time in seconds is reached number false 30
delay delay time in seconds to wait before checking the elapsed_time_sec again number false 10
max_data_size_kb maximum size in kb of logs that are send by one request to monasca-log-api number false 5120

Example configuration files

Simple
output {
  monasca_log_api {
    monasca_log_api_url => "http://192.168.10.4:5607/v3.0"
    keystone_api_url => "http://192.168.10.5:35357/v3"
    project_name => "cmm"
    project_domain_name => "Default"
    username => "cmm-operator"
    user_domain_name => "Default"
    password => "admin"
  }
}
Complete
output {
  monasca_log_api {
    monasca_log_api_url => "https://192.168.10.4:5607/v3.0"
    monasca_log_api_insecure => true
    keystone_api_url => "http://192.168.10.5:35357/v3"
    keystone_api_insecure => false
    project_name => "cmm"
    project_domain_name => "Default"
    username => "cmm-operator"
    user_domain_name => "Default"
    password => "admin"
    dimensions => ["hostname: monasca", "ip:192.168.10.4"]
    num_of_logs => 125
    delay => 10
    elapsed_time_sec => 30
    max_data_size_kb => 5120
  }
}

Run

bin/logstash -f ~/logstash.conf

Run in debug mode

bin/logstash -f ~/logstash.conf --debug

Specify log output file

bin/logstash -f ~/logstash.conf -l /var/log/monasca/log/agent/test-log-agent.log

Logstash File Input plugin

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

Configuration

Local dimensions can be added with add_field setting

input {
  file {
    add_field => { "dimensions" => { "service" => "monasca-api" }}
    add_field => { "dimensions" => { "language" => "java" }}
    add_field => { "dimensions" => { "log_level" => "error" }}
    path => "/var/log/monasca/api/error.log"
    }
  }

Cross tenant functionality

The output plugin is able to post data on behalf of another tenant to the monasca-log-api. For example you can use a user of tenant X to send logs to the monasca-log-api for another tenant Y. The logs will be stored in the index of tenant Y.

To use this functionality you need to pass through another field called cross_tenant to each incoming event. This can be done by using the add_field for file input plugin. Example configuration:

input {
  file {
    add_field => { "dimensions" => { "service" => "neutron" "component" => "firewall" }}
    add_field => { "cross_tenant" => "903ac629d8424dc39ae928a5fff338e7" }
    path => "/var/log/neutron/firewall.log"
  }
}

Important: The user which is sending the data on behalf of other tenants requires a specific role. This role is defined in the configuration file of the monasca-log-api as delegate_roles.

Exception handling

This output plugin will sleep for maximum 60 seconds after the request to Monasca API fails and will retry the request until successful.

Open tasks

  • Language translations (Replace hardcoded String messages with a configuration/language file)
  • Unit tests for failed requests handling