fluent-plugin-geoip-kk

Fluentd filter plugin to add geoip.

Requirements

fluent-plugin-geoip-kk	fluentd	ruby
>= 1.0.2	>= v0.14.0	>= 2.1
< 1.0.0	>= v0.12.0	>= 1.9

Installation

# for fluentd
$ gem install fluent-plugin-geoip-kk

# for td-agent2
$ sudo td-agent-gem install fluent-plugin-geoip-kk

Configuration Parameters

Parameter	Type	Default	Description
key_name	string	client_ip	Field name that contains the IP address
out_key	string	geo	Output field name to store GeoIP data
database_path	string	data/GeoLite2-City.mmdb	Path to the GeoIP database file
flatten	bool	false	Flatten the GeoIP data structure
cache_size	integer	8192	Size of the LRU cache
cache_ttl	integer	3600	TTL for cached items in seconds
skip_private_ip	bool	true	Skip adding GeoIP data for private IP addresses

Usage Examples

Basic Configuration

<filter access.nginx.**>
  @type geoip
  key_name client_ip
  out_key geo
</filter>

Advanced Configuration

<filter access.nginx.**>
  @type geoip

  # IP address field configuration
  key_name client_ip
  out_key geo

  # Database configuration
  database_path /path/to/GeoLite2-City.mmdb

  # Cache configuration
  cache_size 10000    # Cache up to 10000 IP addresses
  cache_ttl 3600      # Cache TTL: 1 hour

  # Output configuration
  flatten false       # Keep nested structure

  # IP processing configuration
  skip_private_ip true  # Skip private IP addresses
</filter>

Input Example

{
  "client_ip": "93.184.216.34",
  "scheme": "http",
  "method": "GET",
  "host": "example.com",
  "path": "/",
  "query": "-",
  "req_bytes": 200,
  "referer": "-",
  "status": 200,
  "res_bytes": 800,
  "res_body_bytes": 600,
  "taken_time": 0.001,
  "user_agent": "Mozilla/5.0"
}

Output Example (Default Structure)

{
  "client_ip": "93.184.216.34",
  "scheme": "http",
  "method": "GET",
  // ... other original fields ...
  "geo": {
    "coordinates": {
      "latitude": 42.150800000000004,
      "longitude": -70.8228,
      "accuracy_radius": 100
    },
    "country": {
      "code": "US",
      "name": "United States"
    },
    "city": {
      "name": "Norwell",
      "confidence": 90
    },
    "region": {
      "code": "MA",
      "name": "Massachusetts"
    },
    "postal": {
      "code": "02061",
      "confidence": 95
    },
    "timezone": "America/New_York"
  }
}

Output Example (Flattened Structure)

When flatten true is specified:

{
  "client_ip": "93.184.216.34",
  // ... other original fields ...
  "geo_coordinates_latitude": 42.150800000000004,
  "geo_coordinates_longitude": -70.8228,
  "geo_coordinates_accuracy_radius": 100,
  "geo_country_code": "US",
  "geo_country_name": "United States",
  "geo_city_name": "Norwell",
  "geo_city_confidence": 90,
  "geo_region_code": "MA",
  "geo_region_name": "Massachusetts",
  "geo_postal_code": "02061",
  "geo_postal_confidence": 95,
  "geo_timezone": "America/New_York"
}

Performance Optimization

The plugin includes several performance optimizations:

LRU Cache with TTL
- Caches GeoIP lookups to reduce database access
- Configurable cache size and TTL
- Automatic cache cleanup for expired entries
Skip Private IPs
- Optionally skip processing private IP addresses
- Reduces unnecessary database lookups
Efficient Record Access
- Uses Fluentd's record accessor for optimized field access
- Reduces memory allocations

VS.

fluent-plugin-geoip Fluentd output plugin to geolocate with geoip. It is able to customize fields with placeholder.

Easy to install.
- Not require to install Development Tools and geoip-dev library.
- ( fluent-plugin-geoip use geoip-c gem but our plugin use geoip. It's conflict. )
5-10 times faster by the LRU cache.
- See benchmark.

Development

After checking out the repo, run bundle install to install dependencies. Then, run rake test to run the tests.

To install this gem onto your local machine, run bundle exec rake install.

Publishing

This gem uses GitHub Actions for automated publishing. To publish a new version:

Update the version number in fluent-plugin-geoip-kk.gemspec
Commit the changes: bash git add fluent-plugin-geoip-kk.gemspec git commit -m "Bump version to x.x.x"
Create and push a new tag: bash git tag vx.x.x git push origin main vx.x.x
The GitHub Action will automatically build and publish the gem to RubyGems.org

Contributing

Fork it ( https://github.com/kevin197011/fluent-plugin-geoip-kk/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request

Copyright

License

Apache License, Version 2.0.

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com.