Kinesis Output Plugin

Build Status Gem info

This is a plugin for Logstash.

It will send log records to a Kinesis stream, using the KPL library.

Configuration

Minimum required configuration to get this plugin chugging along:

output {
  kinesis {
    stream_name => "logs-stream"
    region => "ap-southeast-2"
  }
}

This plugin accepts a wide range of configuration options, most of which come from the underlying KPL library itself. View the full list of KPL configuration options here.

Please note that configuration options are snake_cased instead of camelCased. So, where KinesisProducerConfiguration offers a setMetricsLevel option,this plugin accepts a metrics_level option.

AWS Credentials

There aren't many Kinesis streams out there that allow you to write to them without some AWS credentials.

This plugin does not allow you to specify credentials directly. Instead, the AWS SDK DefaultAWSCredentialsProviderChain is used. It's quite flexible,you can provide your credentials via any of the following mechanisms:

  • AWS_ACCESS_KEY_ID / AWS_SECRET_KEY environment variables
  • ~/.aws/credentials credentials file
  • Instance profile for your running EC2 instance

Building a partition key

Kinesis demands a partition key be provided for each record. By default, this plugin will provide a very boring partition key of -. However, you can configure it to compute a partition key from fields in your log events.

output {
  kinesis {
    # ...
    event_partition_keys => ["[field1]", "[field2]"]
  }
}

Known Issues

Noisy shutdown

During shutdown of Logstash, you might get noisy warnings like this:

[pool-1-thread-6] WARN com.amazonaws.services.kinesis.producer.Daemon - Exception during updateCredentials
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at com.amazonaws.services.kinesis.producer.Daemon$5.run(Daemon.java:316)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

This is caused by amazon-kinesis-producer#10

Developing

bundle install
bundle exec rake

Contributions

Are more than welcome. Raising an issue is great, raising a PR is better, raising a PR with tests is best.

License

Apache License 2.0