Flume input/output plugin for Fluent
Overview
This is a plugin for fluentd event collector. This plugin adds the Flume compatible interface to fluentd.
What’s Flume?
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
It uses Thrift, a cross-language RPC framework, to communicate between clients and servers.
What’s Flume plugin for fluent?
The Flume plugin for fluentd, which enables fluentd to talk the Flume protocol. Flume protocol is defined as follows, in Thrift-IDL format:
typedef i64 Timestamp
enum Priority {
FATAL = 0,
ERROR = 1,
WARN = 2,
INFO = 3,
DEBUG = 4,
TRACE = 5
}
enum EventStatus {
ACK = 0,
COMMITED = 1,
ERR = 2
}
struct ThriftFlumeEvent {
1: Timestamp timestamp,
2: Priority priority,
3: binary body,
4: i64 nanos,
5: string host,
6: map<string,binary> fields
}
# Instead of using thrift's serialization, we just assume the contents are serialized already.
struct RawEvent {
1: binary raw
}
service ThriftFlumeEventServer {
oneway void append( 1:ThriftFlumeEvent evt ),
oneway void rawAppend( 1:RawEvent evt),
EventStatus ackedAppend( 1: ThriftFlumeEvent evt ),
void close(),
}
A value that is stored in the ThriftFlumeEvent.fields map is used as fluentd ‘tag’. A key of the value enables be specified by users as configuration parameter.
How to use?
fluent-plugin-flume contains both input and output.
Flume Input
Please add the following configurations to fluent.conf.
# Flume input
<source>
type flume
port 56789
</source>
These options are supported.
-
port: port number (default: 56789)
-
bind: bind address (default: 0.0.0.0)
-
server_type: server architecture either in ‘simple’, ‘threaded’, ‘thread_pool’, (default: simple)
-
is_framed: use framed protocol or not (default: false)
-
tag_field: key name of fluentd ‘tag’ that is stored in ThriftFlumeEvent.fields (default: nil)
-
default_tag: default fluentd ‘tag’ (default: ‘category’)
-
add_prefix: prefix string, added to the tag (default: nil)
Flume Output
Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Flume server. Note that fluentd conveys semi-structured data while Flume conveys unstructured data. Thus the plugin translates semi-structured data into JSON data and conveys it to Flume.
# Flume output
<match *>
type flume
host flume-host.local
port 56789
</match>
These options are supported.
-
host: host name or address (default: localhost)
-
port: port number (default: 35863)
-
timeout: thrift protocol timeout (default: 30)
-
remove_prefix: prefix string, removed from the tag (default: nil)
Contributors
Copyright
- Copyright
-
Copyright © 2012 Treasure Data, Inc.
- License
-
Apache License, Version 2.0