Fluentd output plugin to Rackspace cloud feeds

This plugin mirrors functionality in the cf-flume-sink except for fluentd instead of flume.

Configuration

It is highly recommended to use this plugin with the bufferize plugin. This plugin has been designed to fail in a way that allows the bufferize plugin to properly retry.

Fluentd configuration for this plugin

type rackspace_cloud_feeds
identity_endpoint http://identity.com/post/for/token
identity_username username
identity_password teh_password
feeds_endpoint http://feed.com/endpoint/of/feed

• type Needs to be rackspace_cloud_feeds for fluentd to recognize the plugin
• identity_endpoint The url to authenticate with identity
• identity_username The username to use during posting to get a token
• identity_password The password component of getting a token
• feeds_endpoint The endpoint of the authenticated feed to post events to

log4j2 logging configuration

This example configuration assumes that you're using the Repose HERP filter, as this is the primary purpose for building this output plugin.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.openrepose.commons.utils.xslt">
    <Appenders>
        <RollingFile name="UserAccessEventLog" fileName="/var/log/repose-uae.log"
                     filePattern="logs/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
            <PatternLayout pattern="%m%n"/>
            <Policies>
                <TimeBasedTriggeringPolicy />
                <SizeBasedTriggeringPolicy size="250 MB"/>
            </Policies>
        </RollingFile>
    </Appenders>
    <Loggers>
        <Logger name="highly-efficient-record-processor-post-Logger" level="trace">
            <AppenderRef ref="UserAccessEventLog"/>
        </Logger>
    </Loggers>
</Configuration>

This will ensure that the output log lines from Repose are one JSON per line so that the fluentd can consume them.

Naieve sample configuration for fluentd

This is a very naieve configuration for fluentd that can be extended upon to configure fluentd to pull in entries from the repose log and send them out the cloud-feeds fluent plugin through the bufferize plugin.

<source>
 type tail
 path /var/log/repose-uae.log
 pos_file /var/log/repose-uae.log
 format none
 tag repose
</source>

<match repose*>
  type bufferize
  buffer_type file
  buffer_path /tmp/repose.*.buffer
  <config>
    type rackspace_cloud_feeds
    identity_endpoint http://localhost:8080/tokens
    identity_username lolol
    identity_password butts
    feeds_endpoint http://localhost:8081/feed
  </config>
</match>

There are many options with the bufferize plugin, be sure to read the documentation for bufferize and fluentd to ensure that the stuff is set up properly for your environment.