fluent-plugin-aws-sqs

General

This plugin can read data from an AWS SQS.

Installation

fluent-gem install fluent-plugin-aws-sqs --no-document --version=1.0.7

Config

<source>

  type sqs

  # following attribute is required

  sqs_url {queue_url}
  tag {tag_value}

  # following attributes are required if you don't use IAM Role nor export credentials to ENV

  aws_key_id {your_aws_key_id}
  aws_sec_key {your_aws_secret_key}

  # following attributes are optional

  parse_body_as_json {try to parse message.body field as json, if it fails the original field is used} default: false
  attribute_name_to_extract {message custom attribute name, will be extract the string value} default: nil
  add_receipt_handle {adds amazon receipt_handle field to the record} default: false
  add_message_id {adds amazon unique message id} default: false
  add_md5_of_body {adds the md5 of the message body} default: false
  add_queue_url {adds the url of the queue the message came from} default: false

  region {your_region} default: nil

  ### region list ###
  # Asia Pacific (Tokyo) [Default] : ap-northeast-1
  # Asia Pacific (Singapore)       : ap-southeast-1
  # US-East (Virginia)             : us-east-1
  # US-West (Oregon)               : us-west-2
  # US-West (N.California)         : us-west-1
  # EU-West (Ireland)              : eu-west-1
  # South America (São Paulo)      : sa-east-1

  receive_interval {receive interval in seconds} default: 0.1

  max_number_of_messages {the max number of messages per pull} default: 10

  wait_time_seconds {the receive message wait time in seconds} default: 10

  delete_message {delete the message from the queue after the plugin processing and before the 'router.emit' call} default: false

</source>

Pre-requisites

Daemon should be provided with proper credentials and (IAM) permissions to use the SQS queue provided in config. Also, since AWS encourage costumers to use KMS to encrypt data sent to SQS, fluentd agent should also recieve permissions to decrypt KMS keys.

Sample IAM Policy

{
   "Version": "2012-10-17",
      "Statement": [
        {
         "Effect": "Allow",
         "Action": [
            "kms:Decrypt"
         ],
         "Resource": "<<KMS_KEY_ARN>>"
         }, 
         {
         "Effect": "Allow",
         "Action": [
            "sqs:ReceiveMessage",
            "sqs:DeleteMessage",
            "sqs:DeleteMessageBatch"
         ],
         "Resource": "<<SQS_QUEUE_ARN>>"
      }
    ]
}

Note: DeleteMessage permission is relevant only if delete_message is set to true in configuration file.