JSON Transform parser plugin for Fluentd

Overview

This is a parser plugin for fluentd. It is INCOMPATIBLE WITH FLUENTD v0.10.45 AND BELOW.

It was created for the purpose of modifying good.js logs before storing them in Elasticsearch. It may not be useful for any other purpose, but be creative.

Installation

git clone https://github.com/graysonc/fluent-plugin-json-transform
cd fluent-plugin-json-transform && gem build fluent-plugin-json-transform.gemspec
gem install fluent-plugin-json-transform

Configuration

<source>
  type [tail|tcp|uydp|syslog|http] # or a custom input type which accepts the "format" parameter
  format json_transform
  transform_script [nothing|flatten|custom]
  script_path "/home/grayson/transform_script.rb" # ignored if transform_script != custom
</source>

transform_script: nothing to do nothing, flatten to flatten JSON by concatenating nested keys (see below), or custom

script_path: ignored if not using custom script. Point this to a Ruby script which implements the JSONTransformer class.

Flatten script

Flattens nested JSON by concatenating nested keys with '.'. Example:

{
  "hello": {
    "world": true
  },
  "goodbye": {
    "for": {
      "now": true,
      "ever": false
    }
  }
}

Becomes

{
  "hello.world": true,
  "goodbye.for.now": true,
  "goodbye.for.ever": false
}

Implementing JSONTransformer

The JSONTransformer class should have an instance method transform which takes a Ruby hash and returns a Ruby hash:

# lib/transform/flatten.rb
class JSONTransformer
  def transform(json)
    return flatten(json, "")
  end

  def flatten(json, prefix)
    json.keys.each do |key|
      if prefix.empty?
        full_path = key
      else
        full_path = [prefix, key].join('.')
      end

      if json[key].is_a?(Hash)
        value = json[key]
        json.delete key
        json.merge! flatten(value, full_path)
      else
        value = json[key]
        json.delete key
        json[full_path] = value
      end
    end
    return json
  end
end