fluent-plugin-geoip
Fluentd Filter/Output plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.
fluent-plugin-geoip has bundled cost-free GeoLite2 Free Downloadable Databases and GeoLite City database by default.
Also you can use purchased GeoIP City database (lang:ja) which costs starting from $50.
The accuracy details for GeoLite City (free) and GeoIP City (purchased) has described at the page below.
- http://www.maxmind.com/en/geolite_city_accuracy (lang:ja)
- http://www.maxmind.com/en/city_accuracy (lang:ja)
Requirements
fluent-plugin-geoip | fluentd | ruby |
---|---|---|
>= 1.0.0 | >= v1.0.2 | >= 2.1 |
< 1.0.0 | >= v0.12.0 | >= 1.9 |
If you want to use this plugin with Fluentd v0.12.x or earlier use 0.8.x.
Compatibility notice
We've used Fluentd v1 API in this plugin since 1.0.0. So we have dropped some features from GeoipOutput.
See also official document
Fluent::Mixin::RewriteTagName
${tag}
,__TAG__
Alternative: Use
${tag}
placeholder${tag_parts[n]}
,__TAG_PARTS[n]__
Alternative: Use
${tag[n]}
placeholder${hostname}
,__HOSTNAME__
Alternative1: Use filter before this plugin and chunk keys:
<filter> @type record_transformer <record> hostname ${hostname} </record> </filter> <match **> @type geoip tag geoip.${tag[1]}.${hostname} <record> city ${city["host"]} </record> <buffer tag, hostname> flush_interval 1s </buffer> </match>
Alternative2: Just inject hostname into record you can use
<inject>
section instead:<match **> @type geoip tag geoip.${tag[1]}.${hostname} <record> city ${city["host"]} </record> <inject> hostname_key hostname </inject> </match>
Dependency
before use, install dependent library as:
# for RHEL/CentOS
$ sudo yum groupinstall "Development Tools"
$ sudo yum install geoip-devel --enablerepo=epel
# for Ubuntu/Debian
$ sudo apt-get install build-essential
$ sudo apt-get install libgeoip-dev
# for OS X
$ brew install geoip
$ bundle config build.geoip-c --with-geoip-dir=/usr/local/include/
Installation
install with gem
or td-agent provided command as:
# for fluentd
$ gem install fluent-plugin-geoip
# for td-agent
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-geoip
# for td-agent2
$ sudo td-agent-gem install fluent-plugin-geoip
Usage
For GeoipFilter
Note that filter version of geoip plugin does not have handling tag feature.
<filter access.apache>
@type geoip
# Specify one or more geoip lookup field which has ip address (default: host)
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
geoip_lookup_keys host
# Specify optional geoip database (using bundled GeoLiteCity databse by default)
# geoip_database "/path/to/your/GeoIPCity.dat"
# Specify optional geoip2 database
# geoip2_database "/path/to/your/GeoLite2-City.mmdb" (using bundled GeoLite2-City.mmdb by default)
# Specify backend library (geoip2_c, geoip, geoip2_compat)
backend_library geoip2_c
# Set adding field with placeholder (more than one settings are required.)
<record>
city ${city.names.en["host"]}
latitude ${location.latitude["host"]}
longitude ${location.longitude["host"]}
country ${country.iso_code["host"]}
country_name ${country.names.en["host"]}
postal_code ${postal.code["host"]}
</record>
# To avoid get stacktrace error with `[null, null]` array for elasticsearch.
skip_adding_null_record true
# Set @log_level (default: warn)
@log_level info
</filter>
Tips: how to geolocate multiple key
<filter access.apache>
@type geoip
geoip_lookup_keys user1_host, user2_host
<record>
user1_city ${city.names.en["user1_host"]}
user2_city ${city.names.en["user2_host"]}
</record>
</filter>
Advanced config samples
It is a sample to get friendly geo point recdords for elasticsearch with Yajl (JSON) parser.
<filter access.apache>
@type geoip
geoip_lookup_keys host
<record>
# lat lon as properties
# ex. {"lat" => 37.4192008972168, "lon" => -122.05740356445312 }
location_properties '{ "lat" : ${location.latitude["host"]}, "lon" : ${location.longitude["host"]} }'
# lat lon as string
# ex. "37.4192008972168,-122.05740356445312"
location_string ${location.latitude["host"]},${location.longitude["host"]}
# GeoJSON (lat lon as array) is useful for Kibana's bettermap.
# ex. [-122.05740356445312, 37.4192008972168]
location_array '[${location.longitude["host"]},${location.latitude["host"]}]'
</record>
# To avoid get stacktrace error with `[null, null]` array for elasticsearch.
skip_adding_null_record true
</filter>
On the case of using td-agent3 (v1-config), it have to quote { ... }
or [ ... ]
block with quotation like below.
<filter access.apache>
@type geoip
geoip_lookup_keys host
<record>
location_properties '{ "lat" : ${location.latitude["host"]}, "lon" : ${location.longitude["host"]} }'
location_string ${location.latitude["host"]},${location.longitude["host"]}
location_array '[${location.longitude["host"]},${location.latitude["host"]}]'
</record>
skip_adding_null_record true
</filter>
For GeoipOutput
<match access.apache>
@type geoip
# Specify one or more geoip lookup field which has ip address (default: host)
# in the case of accessing nested value, delimit keys by dot like 'host.ip'.
geoip_lookup_keys host
# Specify optional geoip database (using bundled GeoLiteCity databse by default)
geoip_database "/path/to/your/GeoIPCity.dat"
# Specify optional geoip2 database
# geoip2_database "/path/to/your/GeoLite2-City.mmdb"
# Specify backend library (geoip, geoip2_compat, geoip2_c)
backend_library geoip
# Set adding field with placeholder (more than one settings are required.)
<record>
latitude ${location.latitude["host"]}
longitude ${location.longitude["host"]}
country ${country.iso_code["host"]}
country_name ${country.names.en["host"]}
postal_code ${postal.code["host"]}
region ${subdivisions.0.iso_code["host"]}
region_name ${subdivisions.0.names.en["host"]}
city ${city.names.en["host"]}
</record>
# Settings for tag
tag geoip.${tag[1]}
# To avoid get stacktrace error with `[null, null]` array for elasticsearch.
skip_adding_null_record true
# Set @log_level (default: warn)
@log_level info
<buffer tag>
# Set buffering time (default: 0s)
flush_interval 1s
</buffer>
</match>
Tutorial
For GeoipFilter
configuration
<source>
@type forward
</source>
<filter test.geoip>
@type geoip
geoip_lookup_keys host
<record>
city ${city.names.en["host"]}
lat ${location.latitude["host"]}
lon ${location.longitude["host"]}
</record>
</filter>
<match test.**>
@type stdout
</match>
result
# forward record with Google's ip address.
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip
# check the result at stdout
$ tail /var/log/td-agent/td-agent.log
2016-02-01 12:04:37 +0900 test.geoip: {"host":"66.102.9.80","message":"test","city":"Mountain View","lat":37.4192008972168,"lon":-122.05740356445312}
You can check geoip data format using utils/dump.rb.
$ bundle exec ruby urils/dump.rb geoip2 66.102.3.80
$ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80
$ bundle exec ruby urils/dump.rb geoip 66.102.3.80
For GeoipOutput
configuration
<source>
@type forward
</source>
<match test.geoip>
@type copy
<store>
@type stdout
</store>
<store>
@type geoip
geoip_lookup_keys host
<record>
lat ${location.latitude["host"]}
lon ${location.longitude["host"]}
country ${country.iso_code["host"]}
</record>
tag debug.${tag[1]}
</store>
</match>
<match debug.**>
@type stdout
</match>
result
# forward record with Google's ip address.
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip
# check the result at stdout
$ tail /var/log/td-agent/td-agent.log
2013-08-04 16:21:32 +0900 test.geoip: {"host":"66.102.9.80","message":"test"}
2013-08-04 16:21:32 +0900 debug.geoip: {"host":"66.102.9.80","message":"test","lat":37.4192008972168,"lon":-122.05740356445312,"country":"US"}
You can check geoip data format using utils/dump.rb.
$ bundle exec ruby urils/dump.rb geoip2 66.102.3.80
$ bundle exec ruby urils/dump.rb geoip2_compat 66.102.3.80
$ bundle exec ruby urils/dump.rb geoip 66.102.3.80
Placeholders
GeoIP2
You can get any fields in the GeoLite2 database and GeoIP2 Downloadable Databases.
For example(geoip2_c backend):
placeholder attributes | output example | note |
---|---|---|
$citycity.namescity.names.en[lookup_field] | "Mountain View" | - |
$locationlocation.latitude[lookup_field] | 37.419200000000004 | - |
$locationlocation.longitude[lookup_field] | -122.0574 | - |
$countrycountry.iso_code[lookup_field] | "US" | - |
$countrycountry.namescountry.names.en[lookup_field] | "United States" | - |
$postalpostal.code[lookup_field] | "94043" | - |
$subdivisionssubdivisions.0subdivisions.0.iso_code[lookup_field] | "CA" | - |
$subdivisionssubdivisions.0subdivisions.0.namessubdivisions.0.names.en[lookup_field] | "California" | - |
For example(geoip2_compat backend):
placeholder attributes | output example | note |
---|---|---|
$city[lookup_field] | "Mountain View" | - |
$latitude[lookup_field] | 37.419200000000004 | - |
$longitude[lookup_field] | -122.0574 | - |
$country_code[lookup_field] | "US" | - |
$country_name[lookup_field] | "United States" | - |
$postal_code[lookup_field] | "94043" | |
$region[lookup_field] | "CA" | - |
$region_name[lookup_field] | "California" | - |
NOTE: geoip2_compat backend supports only above fields.
Related configurations:
backend_library
:geoip2_compat
orgeoip2_c
geoip2_database
: path to your GeoLite2-City.mmdb
GeoIP legacy
Provides these placeholders for adding field of geolocate results.
For more example of geolocating, you can try these websites like Geo IP Address View or View my IP information.
placeholder attributes | output example | type | note |
---|---|---|---|
$city[lookup_field] | "Ithaca" | varchar(255) | - |
$latitude[lookup_field] | 42.4277992248535 | decimal | - |
$longitude[lookup_field] | -76.4981994628906 | decimal | - |
$country_code3[lookup_field] | "USA" | varchar(3) | - |
$country_code[lookup_field] | "US" | varchar(2) | A two-character ISO 3166-1 country code |
$country_name[lookup_field] | "United States" | varchar(50) | - |
$dma_code[lookup_field] | 555 | unsigned int | only for US |
$area_code[lookup_field] | 607 | char(3) | only for US |
$region[lookup_field] | "NY" | char(2) | A two character ISO-3166-2 or FIPS 10-4 code |
Further more specification available at http://dev.maxmind.com/geoip/legacy/csv/#GeoIP_City_Edition_CSV_Database_Fields
Related configurations:
backend_library
:geoip
(default)geoip_database
: path to your GeoLiteCity.dat
Parameters
GeoipFilter
Note that filter version of geoip
plugin does not have handling tag
feature.
Plugin helpers
See also Filter Plugin Overview
Supported sections
Parameters
geoip_database (string) (optional)
- Default value: bundled database
GeoLiteCity.dat
Path to GeoIP database file.
geoip2_database (string) (optional)
- Default value: bundled database
GeoLite2-City.mmdb
.
Path to GeoIP2 database file.
geoip_lookup_keys (array) (optional)
- Default_value:
["host"]
Specify one or more geoip lookup field which has IP address.
geoip_lookup_key (string) (optional) (deprecated)
- Default value:
nil
.
Use geoip_lookup_keys instead.
skip_adding_null_record (bool) (optional)
- Default value:
nil
Skip adding geoip fields when this valaues to true
.
On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything.
backend_library (enum) (optional)
- Available values:
geoip
,geoip2_compat
,geoip2_c
- Default value:
geoip2_c
.
Set backend library.
GeoipOutput
Plugin helpers
See also Output Plugin Overview
Sections
Parameters
geoip_database (string) (optional)
- Default value: bundled database
GeoLiteCity.dat
Path to GeoIP database file.
geoip2_database (string) (optional)
- Default value: bundled database
GeoLite2-City.mmdb
.
Path to GeoIP2 database file.
geoip_lookup_keys (array) (optional)
- Default value:
["host"]
Specify one or more geoip lookup field which has IP address.
geoip_lookup_key (string) (optional) (deprecated)
- Default value:
nil
.
Use geoip_lookup_keys instead.
skip_adding_null_record (bool) (optional)
- Default value:
nil
Skip adding geoip fields when this valaues to true
.
On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything.
backend_library (enum) (optional)
- Available values:
geoip
,geoip2_compat
,geoip2_c
- Default value:
geoip2_c
.
Set backend library.
tag (string) (optional)
On using this option with tag placeholder like tag geoip.${tag}
(test code is available at test_out_geoip.rb).
Articles
IPアドレスを元に位置情報をリアルタイムに付与する fluent-plugin-geoip v0.0.1をリリースしました #fluentd - Y-Ken Studio
http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-has-released初の安定版 fluent-plugin-geoip v0.0.3 をリリースしました #fluentd- Y-Ken Studio
http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-v0.0.3fluent-plugin-geoip v0.0.4 をリリースしました。ElasticSearch+Kibanaの世界地図に位置情報をプロットするために必要なFluentdの設定サンプルも紹介します- Y-Ken Studio
http://y-ken.hatenablog.com/entry/fluent-plugin-geoip-v0.0.4Released GeoIP plugin to work together with ElasticSearch + Kibana v3
https://groups.google.com/d/topic/fluentd/OVIcH_SKBwM/discussionFluentd、Amazon RedshiftとTableauを用いたカジュアルなデータ可視化 | SmartNews開発者ブログ
http://developer.smartnews.be/blog/2013/10/03/easy-data-analysis-using-fluentd-redshift-and-tableau/
TODO
Pull requests are very welcome!!
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request
Copyright
Copyright (c) 2013- Kentaro Yoshida (@yoshi_ken)
License
Apache License, Version 2.0
This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com.