fluent-plugin-bin

Fluentd input plugin for binary files based on in_tail.

Usage Example

This exmaple shows how to install and configure fluent-plugin-bin together with fluent-plugin wendelin output plugin to ingest binary files into Wendelin on Ubuntu 14.04.4 LTS.

sudo apt-get install supervisor
sudo service supervisor restart
sudo apt-get install python-software-properties
sudo apt-add-repository ppa:brightbox/ruby-ng
sudo apt-get update
sudo apt-get install ruby2.0 ruby2.0-⁠⁠dev make
sudo gem install fluentd --no-ri --no-rdoc
sudo mkdir -p /etc/fluent/plugin
cd /etc/fluent/plugin
sudo wget https://lab.nexedi.com/nexedi/fluent-plugin-bin/raw/master/lib/fluent/plugin/in_bin.rb
sudo wget https://lab.nexedi.cn/nexedi/fluent-plugin-wendelin/raw/master/lib/fluent/plugin/out_wendelin.rb
sudo wget https://lab.nexedi.cn/nexedi/fluent-plugin-wendelin/raw/master/lib/fluent/plugin/wendelin_client.rb

Create /etc/fluent/fluentd.conf:

<source>
  @type bin
  format none
  path /path/to/files/*.ext
  pos_file /mic/upload.pos
  enable_watch_timer false
  read_from_head true
  tag my_prefix.*
</source>

<match my_prefix.path.to.files.*.ext>
  @type wendelin
  @id wendelin_out

  streamtool_uri https://wendelin_url/portal_ingestion_policies/my_policy
  user      my_user_name
  password  my_password

  buffer_type       memory
  flush_interval    1s
  disable_retry_limit    true
</match>

Create /etc/supervisor/conf.d/fluentd.conf:

[program:fluentd]
command=/usr/local/bin/fluentd
autostart=true
autorestart=true
stderr_logfile=/var/log/fluentd.log
stdout_logfile=/var/log/fluentd.log

Start fluentd:

sudo supervisorctl start fluentd