Amazon ELB log input plugin for fluentd

Overview

• Amazon Web Services ELB log input plubin for fluentd

Requirements

fluent-plugin-elb-log	fluentd	ruby
>= 0.3.0	>= v0.14.0	>= 2.1
< 0.3.0	>= v0.12.0	>= 1.9

Installation

$ fluentd-gem fluent-plugin-elb-log

AWS ELB Settings

• settings see: Elastic Load Balancing
• developer guide:

Different from version 0.4.x

• Using version 3 of the AWS SDK for Ruby.

Support Application Load Balancer (ver 0.4.0 or later)

• Support Access Logs for Application Load Balancer
  • https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html
• Existing ELB is called Classic Load Balancer
  • http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/access-log-collection.html

When SSL certification error

log:

SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

Do env setting follows:

SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt (If you using amazon linux)

Configuration

<source>
  @type elb_log

  # following attibutes are required
  region            <region name>
  s3_bucketname     <bucketname>
  s3_prefix         <elb log's prefix>
  timestamp_file    <proc last file timestamp record filename>
  buf_file          <buffer file path>
  refresh_interval  <interval number by second>
  tag               <tag name(default: elb.access)>
  delete            <boolean delete processed log files from S3(default: false)>
  include_all_message <boolean (default:false)>

  # following attibutes are required if you don't use IAM Role
  access_key_id     <access_key>
  secret_access_key <secret_access_key>
</source>

Example setting

<source>
  @type elb_log
  region            us-east-1
  s3_bucketname     my-elblog-bucket
  s3_prefix         prodcution/web
  timestamp_file    /tmp/elb_last_at.dat
  buf_file          /tmp/fluentd-elblog.tmpfile
  refresh_interval  300
  tag               elb.access
  delete            false
  include_all_message false
  access_key_id     XXXXXXXXXXXXXXXXXXXX
  secret_access_key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</source>

<match **>
  @type stdout
</match>

json output example

{
    "account_id":"123456789012",
    "region":"ap-northeast-1",
    "logfile_date":"2015/06/15",
    "logfile_elb_name":"my-elb-name",
    "elb_ip_address":"52.0.0.0",
    "logfile_hash":"12squv5w",
    "elb_timestamp":"20150615T0400Z",
    "key":"TEST/AWSLogs/123456789012/elasticloadbalancing/ap-northeast-1/2015/06/15/123456789012_elasticloadbalancing_ap-northeast-1_my-elb-name_20150615T0400Z_52.68.215.138_69squv5w.log",
    "prefix":"TEST",
    "elb_timestamp_unixtime":1434340800,
    "time":"2015-06-15T03:47:12.728427+0000",
    "elb":"my-elb-name",
    "client":"54.1.1.1",
    "client_port":"43759",
    "backend":"10.0.0.1",
    "backend_port":"80",
    "request_processing_time":4.0e-05,
    "backend_processing_time":0.105048,
    "response_processing_time":2.4e-05,
    "elb_status_code":"200",
    "backend_status_code":"200",
    "received_bytes":0,
    "sent_bytes":4622,
    "request_method":"GET",
    "request_uri":"https://my-elb-test.example.com/",
    "request_protocol":"HTTP/1.1",
    "user_agent":"Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)",
    "ssl_cipher":"DHE-RSA-AES128-SHA",
    "ssl_protocol":"TLSv1.2",
    "type":"http",
    "target_group_arn": "arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/lbgrp1/605122a4e4ee9f2d",
    "trace_id": "\"Root=1-xxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx\"",
    "domain_name": "-",
    "chosen_cert_arn": "-",
    "matched_rule_priority": "0",
    "request_creation_time": "2099-10-26T06:10:03.050000Z",
    "actions_executed": "forward",
    "redirect_url": "-",
    "error_reason": "-",
    "option1": "\"192.168.0.1:443\"",
    "option2": "\"301\"",
    "option3": null
}