fluent-plugin-qqwry
Fluentd Output plugin to add information about geographical location of IP addresses with QQWry databases.
fluent-plugin-geoip has bundled qqwry.dat
Dependency
before use, install dependent library as:
Installation
install with gem
or fluent-gem
command as:
# for fluentd
$ gem install fluent-plugin-qqwry
# for td-agent
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-qqwry
Usage
<match access.apache>
type qqwry
# Specify one or more qqwry lookup field which has ip address (default: host)
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
qqwry_lookup_key host
# Specify optional qqwry database (using bundled GeoLiteCity databse by default)
qqwry_database '/path/to/your/qqwry.dat'
# Set adding field with placeholder (more than one settings are required.)
<record>
city ${city['host']}
area ${area_code['host']}
country ${country['host']}
province ${province['host']}
</record>
# Settings for tag
remove_tag_prefix access.
tag qqwry.${tag}
# Set log_level for fluentd-v0.10.43 or earlier (default: warn)
log_level info
# Set buffering time (default: 0s)
flush_interval 1s
</match>
Tips: how to geolocate multiple key
<match access.apache>
type qqwry
qqwry_lookup_key user1_host, user2_host
<record>
user1_city ${city['user1_host']}
user2_city ${city['user2_host']}
</record>
remove_tag_prefix access.
tag qqwry.${tag}
</match>
Tutorial
configuration
<source>
type forward
</source>
<match test.qqwry>
type copy
<store>
type stdout
</store>
<store>
type qqwry
qqwry_lookup_key host
<record>
city ${city['host']}
lat ${latitude['host']}
lon ${longitude['host']}
</record>
remove_tag_prefix test.
tag debug.${tag}
</store>
</match>
<match debug.**>
type stdout
</match>
result
# forward record with Google's ip address.
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.qqwry
# check the result at stdout
$ tail /var/log/td-agent/td-agent.log
2013-08-04 16:21:32 +0900 test.qqwry: {"host":"66.102.9.80","message":"test"}
2013-08-04 16:21:32 +0900 debug.qqwry: {"host":"66.102.9.80","message":"test","city":"Mountain View","lat":37.4192008972168,"lon":-122.05740356445312}
Parameters
include_tag_key
(default: false)tag_key
Add original tag name into filtered record using SetTagKeyMixin.
Further details are written at http://docs.fluentd.org/articles/in_exec
remove_tag_prefix
remove_tag_suffix
add_tag_prefix
add_tag_suffix
Set one or more option are required unless using tag
option for editing tag name. (HandleTagNameMixin feature)
tag
On using this option with tag placeholder like tag qqwry.${tag}
(test code is available at test_out_qqwry.rb), it will be overwrite after these options affected. which are remove_tag_prefix, remove_tag_suffix, add_tag_prefix and add_tag_suffix.
flush_interval
(default: 0 sec)
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
Copyright
Copyright (c) 2014- Chris Song (@fakechris)
License
Apache License, Version 2.0