fluent-plugin-gcloud-pubsub-custom

This plugin is forked from https://github.com/mdoi/fluent-plugin-gcloud-pubsub

Overview

Cloud Pub/Sub Input/Output(BufferedOutput) plugin for Fluentd with gcloud gem

• Publish messages to Cloud Pub/Sub
• Pull messages from Cloud Pub/Sub

Preparation

• Create a project on Google Developer Console
• Add a topic of Cloud Pub/Sub to the project
• Add a pull style subscription to the topic
• Download your credential (json) or set scope on GCE instance

Installation

Install by gem:

$ gem install fluent-plugin-gcloud-pubsub-custom

Configuration

Publish messages

Use gcloud_pubsub output plugin.

<match example.publish>
  @type gcloud_pubsub
  project <YOUR PROJECT>
  key <YOUR KEY>
  topic <YOUR TOPIC>
  autocreate_topic false
  max_messages 1000
  max_total_size 10000000
  buffer_type file
  buffer_path /path/to/your/buffer
  flush_interval 1s
  try_flush_interval 0.1
  format json
</match>

• project (optional)
  • Set your GCP project
  • Running fluentd on GCP, you don't have to specify.
  • You can also use environment variable such as GCLOUD_PROJECT.
• key (optional)
  • Set your credential file path.
  • Running fluentd on GCP, you can use scope instead of specifying this.
  • You can also use environment variable such as GCLOUD_KEYFILE.
• topic (required)
  • Set topic name to publish.
• autocreate_topic (optional, default: false)
  • If set to true, specified topic will be created when it doesn't exist.
• max_messages (optional, default: 1000)
  • Publishing messages count per request to Cloud Pub/Sub.
  • See https://cloud.google.com/pubsub/quotas#other_limits
• max_total_size (optional, default: 10000000 = 10MB)
  • Publishing messages bytesize per request to Cloud Pub/Sub.
  • See https://cloud.google.com/pubsub/quotas#other_limits
• buffer_type, buffer_path, flush_interval, try_flush_interval
  • These are fluentd buffer configuration. See http://docs.fluentd.org/articles/buffer-plugin-overview
• format (optional, default: json)
  • Set output format. See http://docs.fluentd.org/articles/out_file#format

Pull messages

Use gcloud_pubsub input plugin.

<source>
  @type gcloud_pubsub
  tag example.pull
  project <YOUR PROJECT>
  key <YOUR KEY>
  topic <YOUR TOPIC>
  subscription <YOUR SUBSCRIPTION>
  max_messages 1000
  return_immediately true
  pull_interval 2
  format json
</source>

• tag (required)
  • Set tag of messages.
• project (optional)
  • Set your GCP project
  • Running fluentd on GCP, you don't have to specify.
  • You can also use environment variable such as GCLOUD_PROJECT.
• key (optional)
  • Set your credential file path.
  • Running fluentd on GCP, you can use scope instead of specifying this.
  • You can also use environment variable such as GCLOUD_KEYFILE.
• topic (optional)
  • Set topic name to pull.
• subscription (required)
  • Set subscription name to pull.
• max_messages (optional, default: 100)
  • See maxMessages on https://cloud.google.com/pubsub/subscriber#receiving-pull-messages
• return_immediately (optional, default: true)
  • See returnImmediately on https://cloud.google.com/pubsub/subscriber#receiving-pull-messages
  • If return_immediately is true, this plugin ignore pull_interval.
• pull_interval (optional, default: 5)
  • Pulling messages by intervals of specified seconds.
• format (optional, default: json)
  • Set input format. See format section in http://docs.fluentd.org/articles/in_tail

ChangeLog

• Release 0.1.0 - 2016/08/22
  • Use formatter / parser plugin and add format configuration
  • Bump up gcloud-ruby to 0.12
  • Remove dependency on lightening buffer
  • Fix error caused by Pub/Sub quotas

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

TODO

• Add tag attribute in output plugin and use tag attribute as tag in input plugin.
• Send ack after other output plugin committed (if possible).

Authors

• @mdoi : First author
• @mia-0032 : This version author