fluent-plugin-buffer-lightening

Fluentd buffer plugin on memory to flush with many types of chunk limit methods:

  • events count limit in chunk

These options are to decrease latency from emit to write, and to control chunk sizes and flush sizes.

NOTICE: Lightening buffer plugin stores data on memory, so these data will be lost when process/server crashes.

And current version of this plugin adds try_flush_interval option to BufferedOutput plugins, to flush buffer chunk with high frequency. For this option, run fluentd with -r fluent/plugin/output_try_flush_interval_patch.

Installation

Do gem install fluent-plugin-buffer-lightening or fluent-gem ....

Configuration

Lightening buffer plugin can be enabled with all of buffered output plugins.

To flush chunks per 100 records, configure like this:

<match data.**>
  type any_buffered_output_plugin
  buffer_type lightening
  buffer_chunk_records_limit 100
  # other options...
</match>

Options of buffer_type memory are also available:

<match data.**>
  type any_buffered_output_plugin
  buffer_type lightening
  buffer_chunk_limit 10M
  buffer_chunk_records_limit 100
  # other options...
</match>

For less delay

For more frequently flushing, use flush_interval and try_flush_interval with floating point values:

<match data.**>
  type any_buffered_output_plugin
  buffer_type lightening
  buffer_chunk_records_limit 100
  # other options...
  flush_interval 0.5
  try_flush_interval 0.1 # 0.6sec delay for worst case
</match>

And, execute fluentd as fluentd -r fluent/plugin/output_try_flush_interval_patch -c fluentd.conf.

TODO

  • remove output_try_flush_interval_patch with incoming fluentd dependency
  • more limit patterns
  • patches welcome!
  • Copyright (c) 2013- TAGOMORI Satoshi (tagomoris)
  • License
    • Apache License, Version 2.0