fluent-plugin-parser-cloudfoundry-syslog

A fluentd parser for CloudFoundry specific syslog drains - basically the opposite end of their output formatter.

This plugin should make CF metrics more accessible than the current syslog parser for fluent allows for.

Sample

An access log in the format

<14>1 2021-12-24T22:20:01.438069+00:00 some-hostname some-appname [RTR/0] - [tags@47450 __v1_type="LogMessage" app_id="some-app-id" app_name="some-appname" component="route-emitter" deployment="eu-gb-prod" index="some-index" instance_id="0" ip="some-ip" job="router" organization_id="some-org-id" organization_name="some-org-name" origin="gorouter" process_id="some-process-id" process_instance_id="some-process-instance-id" process_type="web" source_type="RTR" space_id="some-space-id" space_name="dev"] example.com - [2021-12-24T22:20:01.429164095Z] "GET /styles.css HTTP/1.1" 304 0 0 "https://example.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" "some-remote-host" "some-backend-host" x_forwarded_for:"a, b" x_forwarded_proto:"https" vcap_request_id:"some-request-id" response_time:0.008452 gorouter_time:0.000625 app_id:"some-app-id" app_index:"0" instance_id:"some-instance-id" x_cf_routererror:"-" x_global_transaction_id:"some-global-transaction-id" true_client_ip:"-" x_b3_traceid:"some-trace-id" x_b3_spanid:"some-span-id" x_b3_parentspanid:"-" b3:"some-b3"

.. is turned into ..

{
  "header" => {
    "pri" => {
      "facility" => 1,
      "severity" => "info",
    },
    "version" => 1,
    "timestamp" => "2021-12-24T22:20:01.438069+00:00",
    "hostname" => "some-hostname",
    "app_name" => "some-appname",
    "proc_id" => "[RTR/0]",
    "msg_id" => "-",
  },
  "sd" => {
    "tags@47450" => {
      "__v1_type" => "LogMessage",
      "app_id" => "some-app-id",
      "app_name" => "some-appname",
      "component" => "route-emitter",
      "deployment" => "eu-gb-prod",
      "index" => "some-index",
      "instance_id" => "0",
      "ip" => "some-ip",
      "job" => "router",
      "organization_id" => "some-org-id",
      "organization_name" => "some-org-name",
      "origin" => "gorouter",
      "process_id" => "some-process-id",
      "process_instance_id" => "some-process-instance-id",
      "process_type" => "web",
      "source_type" => "RTR",
      "space_id" => "some-space-id",
      "space_name" => "dev",
    },
  },
  "gorouter" => {
    "host" => "example.com",
    "timestamp" => "2021-12-24T22:20:01.429164095Z",
    "method" => "GET",
    "pathname" => "/styles.css",
    "protocol" => "HTTP/1.1",
    "status" => "304",
    "bytes_received" => "0",
    "bytes_sent" => "0",
    "referer" => "https://example.com/",
    "user_agent" => "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0",
    "remote_address" => "some-remote-host",
    "backend_address" => "some-backend-host",
    "x_forwarded_for" => "a, b",
    "x_forwarded_proto" => "https",
    "vcap_request_id" => "some-request-id",
    "response_time" => 0.008452,
    "gorouter_time" => 0.000625,
    "app_id" => "some-app-id",
    "app_index" => "0",
    "instance_id" => "some-instance-id",
    "x_cf_routererror" => "-",
    "x_global_transaction_id" => "some-global-transaction-id",
    "true_client_ip" => "-",
    "x_b3_traceid" => "some-trace-id",
    "x_b3_spanid" => "some-span-id",
    "x_b3_parentspanid" => "-",
    "b3" => "some-b3",
  },
  "message" => 'example.com - [2021-12-24T22:20:01.429164095Z] "GET /styles.css HTTP/1.1" 304 0 0 "https://example.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" "some-remote-host" "some-backend-host" x_forwarded_for:"a, b" x_forwarded_proto:"https" vcap_request_id:"some-request-id" response_time:0.008452 gorouter_time:0.000625 app_id:"some-app-id" app_index:"0" instance_id:"some-instance-id" x_cf_routererror:"-" x_global_transaction_id:"some-global-transaction-id" true_client_ip:"-" x_b3_traceid:"some-trace-id" x_b3_spanid:"some-span-id" x_b3_parentspanid:"-" b3:"some-b3"',
}

Usage

Install the plugin:

# See https://github.com/BitPatty/fluent-plugin-parser-cloudfoundry-syslog/releases for a list of valid versions
gem install fluent-plugin-parser-cloudfoundry-syslog --version "<desired version>"

Create a logdrain and update your fluent configuration:

<source>
  # Use TCP or HTTP, depending on what your logdrain is
  # configured to use
  @type http

  # Your source configuration...

  <parse>
    @type cloudfoundry_syslog

    # Set this to true if access log messages should be parsed.
    # Defaults to false
    parse_gorouter_access_log true

    # Set this to true if you want the raw message to be available
    # under the key `raw`. Defaults to false
    include_raw_message true
  </parse>
</source>

Limitations

Credit