fluent-plugin-aws-sqs

General

This plugin can read data from an AWS SQS.

Installation

fluent-gem install fluent-plugin-aws-sqs --no-document --version=1.0.7

Config

<source>

  type sqs

  # following attribute is required

  sqs_url {queue_url}
  tag {tag_value}

  # following attributes are required if you don't use IAM Role nor export credentials to ENV

  aws_key_id {your_aws_key_id}
  aws_sec_key {your_aws_secret_key}

  # following attributes are optional

  attribute_name_to_extract {message custom attribute name, will be extract the string value}
　
  region {your_region}

  ### region list ###
  # Asia Pacific (Tokyo) [Default] : ap-northeast-1
  # Asia Pacific (Singapore)       : ap-southeast-1
  # US-East (Virginia)             : us-east-1
  # US-West (Oregon)               : us-west-2
  # US-West (N.California)         : us-west-1
  # EU-West (Ireland)              : eu-west-1
  # South America (São Paulo)      : sa-east-1

  receive_interval {receive_interval_seconds}

  max_number_of_messages {max_number_of_messages}

  wait_time_seconds {receive_message_wait_time_seconds}

  delete_message {delete_message_after_receiving}

</source>

Pre-requisites

Daemon should be provided with proper credentials and (IAM) permissions to use the SQS queue provided in config. Also, since AWS encourage costumers to use KMS to encrypt data sent to SQS, fluentd agent should also recieve permissions to decrypt KMS keys.

Sample IAM Policy

{
   "Version": "2012-10-17",
      "Statement": [
        {
         "Effect": "Allow",
         "Action": [
            "kms:Decrypt"
         ],
         "Resource": "<<KMS_KEY_ARN>>"
         }, 
         {
         "Effect": "Allow",
         "Action": [
            "sqs:ReceiveMessage",
            "sqs:DeleteMessage",
            "sqs:DeleteMessageBatch"
         ],
         "Resource": "<<SQS_QUEUE_ARN>>"
      }
    ]
}

Note: DeleteMessage permission is relevant only if delete_message is set to true in configuration file.