fluent-plugin-webhdfs
Fluentd output plugin to write data into Hadoop HDFS over WebHDFS/HttpFs.
WebHDFSOutput slices data by time (specified unit), and store these data as hdfs file of plain text. You can specify to:
- format whole data as serialized JSON, single attribute or separated multi attributes
- or LTSV, labeled-TSV (see http://ltsv.org/ )
- include time as line header, or not
- include tag as line header, or not
- change field separator (default: TAB)
- add new line as termination, or not
And you can specify output file path as 'path /path/to/dir/access.%Y%m%d.log', then got '/path/to/dir/access.20120316.log' on HDFS.
Configuration
WebHDFSOutput
To store data by time,tag,json (same with 'type file') over WebHDFS:
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /path/on/hdfs/access.log.%Y%m%d_%H.log
</match>
To store data as LTSV without time and tag over WebHDFS:
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /path/on/hdfs/access.log.%Y%m%d_%H.log
output_data_type ltsv
</match>
With username of pseudo authentication:
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /path/on/hdfs/access.log.%Y%m%d_%H.log
username hdfsuser
</match>
Store data over HttpFs (instead of WebHDFS):
<match access.**>
type webhdfs
host httpfs.node.your.cluster.local
port 14000
path /path/on/hdfs/access.log.%Y%m%d_%H.log
httpfs true
</match>
Store data as TSV (TAB separated values) of specified keys, without time, with tag (removed prefix 'access'):
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /path/on/hdfs/access.log.%Y%m%d_%H.log
field_separator TAB # or 'SPACE', 'COMMA' or 'SOH'(Start Of Heading: \001)
output_include_time false
output_include_tag true
remove_prefix access
output_data_type attr:path,status,referer,agent,bytes
</match>
If message doesn't have specified attribute, fluent-plugin-webhdfs outputs 'NULL' instead of values.
Performance notifications
Writing data on HDFS single file from 2 or more fluentd nodes, makes many bad blocks of HDFS. If you want to run 2 or more fluentd nodes with fluent-plugin-webhdfs, you should configure 'path' for each node. You can use '$hostname' or '$uuid:random' placeholders in configuration for this purpose.
For hostname:
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /log/access/%Y%m%d/${hostname}.log
</match>
Or with random filename (to avoid duplicated file name only):
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /log/access/%Y%m%d/${uuid:random}.log
</match>
With configurations above, you can handle all of files of '/log/access/20120820/*' as specified timeslice access logs.
For unstable Namenodes
With default configuration, fluent-plugin-webhdfs checks HDFS filesystem status and raise error for inacive NameNodes.
If you were usging unstable NameNodes and have wanted to ignore NameNode errors on startup of fluentd, enable ignore_start_check_error
option like below:
<match access.**>
type webhdfs
host namenode.your.cluster.local
port 50070
path /log/access/%Y%m%d/${hostname}.log
ignore_start_check_error true
</match>
TODO
- patches welcome!
Copyright
- Copyright (c) 2012- TAGOMORI Satoshi (tagomoris)
- License
- Apache License, Version 2.0