Scribe input plugin for Fluent

Overview

This is a plugin for fluentd event collector. This plugin adds the Scribe compatible interface to fluentd.

What’s Scribe?

Scribe is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.

It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.

What’s Scribe plugin for fluent?

The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

The category field is used as fluentd ‘tag’.

How to use?

fluent-plugin-scribe contains both input and output.

Scribe Input

Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.

# Scribe input
<source>
  type scribe
  port 1463
</source>

These options are supported.

port: port number (default: 1463)
bind: bind address (default: 0.0.0.0)
server_type: server architecture either in ‘simple’, ‘threaded’, ‘thread_pool’, ‘nonblocking’ (default: nonblocking)
is_framed: use framed protocol or not (default: true)
add_prefix: prefix string, added to the tag (default: nil)
msg_format: format of the messages either in ‘text’, ‘json’, ‘url_param’ (default: text)

Scribe Output

Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, ‘field_ref’ parameter is prepared to specify which field will be transferred.

# Scribe output
<match *>
  type scribe
  host scribed-host.local
  port 1463
  field_ref message
</match>

These options are supported.

host: host name or address (default: localhost)
port: port number (default: 1463)
field_ref: field name which sent as scribe log message (default: message)
timeout: thrift protocol timeout (default: 30)
format_to_json: if true/yes, format entire record as json, and send as message (default: false)
remove_prefix: prefix string, removed from the tag (default: nil)

For Developers

To run fluentd with this plugin on chaging,

$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf

Then please execute the sample client.

$ bundle exec bin/fluent-scribe-remote

Contributors

Copyright

License: Apache License, Version 2.0