fluent-plugin-record-reformer

Build Status

Fluentd plugin to add or replace fields of a event record

Installation

Use RubyGems:

gem install fluent-plugin-record-reformer

Configuration

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false

  tag reformed.${tag_prefix[-2]}
  <record>
    hostname ${hostname}
    input_tag ${tag}
    last_tag ${tag_parts[-1]}
    message ${message}, yay!
  </record>
</match>

Assume following input is coming (indented):

foo.bar {
  "remove_me":"bar",
  "not_remove_me":"bar",
  "message":"Hello world!"
}

then output becomes as below (indented):

reformed.foo {
  "not_remove_me":"bar",
  "hostname":"YOUR_HOSTNAME",
  "input_tag":"foo.bar",
  "last_tag":"bar",
  "message":"Hello world!, yay!",
}

Configuration (Classic Style)

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  tag reformed.${tag_prefix[-2]}

  hostname ${hostname}
  input_tag ${tag}
  last_tag ${tag_parts[-1]}
  message ${message}, yay!
</match>

This results in same, but please note that following option parameters are reserved, so can not be used as a record key.

Option Parameters

  • output_tag (obsolete)

    The output tag name. This option is deprecated. Use tag option instead

  • tag

    The output tag name.

  • remove_keys

    Specify record keys to be removed by a string separated by , (comma) like

    remove_keys message,foo
    
  • renew_record bool

    renew_record true creates an output record newly without extending (merging) the input record fields. Default is false.

  • keep_keys

    You may want to remain some record fields although you specify renew_record true. Then, specify record keys to be kept by a string separated by , (comma) like

    keep_keys message,foo
    
  • enable_ruby bool

    Enable to use ruby codes in placeholders. See Placeholders section. Default is true (just for lower version compatibility).

Placeholders

The keys of input json are available as placeholders. In the above example,

  • $remove_me
  • $not_remove_me
  • $message

shall be available. In addition, following placeholders are reserved:

  • $hostname Hostname of the running machine
  • $tag Input tag
  • $time Time of the event
  • $tags[N] (Obsolete. Use tag_parts) Input tag splitted by '.'
  • $tag_parts[N] Input tag splitted by '.' indexed with N such as ${tag_parts[0]}, ${tag_parts[-1]}.
  • $tag_prefix[N] Tag parts before and on the index N. For example,

    Input tag: prefix.test.tag.suffix
    
    ${tag_prefix[0]}  => prefix
    ${tag_prefix[1]}  => prefix.test
    ${tag_prefix[-2]} => prefix.test.tag
    ${tag_prefix[-1]} => prefix.test.tag.suffix
    
  • $tag_suffix[N] Tag parts after and on the index N. For example,

    Input tag: prefix.test.tag.suffix
    
    ${tag_suffix[0]}  => prefix.test.tag.suffix
    ${tag_suffix[1]}  => test.tag.suffix
    ${tag_suffix[-2]} => tag.suffix
    ${tag_suffix[-1]} => suffix
    

It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as

  • $time.strftime('%Y-%m-%dT%H:%M:%S%z')
  • $tag_partstag_parts.last

but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.

Relatives

Following plugins look similar:

ChangeLog

See CHANGELOG.md for details.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.