Treasure Data output plugin for Fluentd

This Fluentd output plugin is used to upload logs to Treasure Data using Treasure Data’s REST APIs.

This plugin relies on the Treasure Data’s Ruby client library to communicate with the Treasure Data’s REST APIs.

For more information, please visit the official Fluentd TD Output plugin page at docs.fluentd.org.

Configuration

Please refer to the example.conf example configuration file contained in the root of this repository for some examples. The configuration file allows the user to specify various settings for this output plugin.

The configuration options currently supported are:

apikey

Sets the configuration parameter to specify the Treasure Data API key.

Requires the API key as argument.

This parameter is required.

auto_create_table

Specifies whether to create a database and/or table if the destination database and/or table do not exist.

Requires no argument to enable; one optional boolean (true or false) argument accepted. When this option is enabled, the destination database and table will be determined based on the event tag: e.g. an even with tag ‘td.mydb.mytable’ will require the plugin to upload data to the ‘mydb’ database and ‘mytable’ table within it.

If this option is not used or set to false, the user is required to specify the database and table configuration parameters as explained below.

NOTE

depending on the access control permissions associated to the API key, a database can or cannot be created if not already available. See the Treasure Data Access Control documentation page for details.

database

Specifies the destination database in the Treasure Data cloud.

Requires the database name as argument.

This parameter is required unless the auto_create_table option is used.

table

Specifies the destination table in the above mentioned database within the Treasure Data cloud.

Requires the table name as argument.

This parameter is required unless the auto_create_table option is used.

endpoint

Specifies the Treasure Data’s REST API endpoint for import requests.

Requires the endpoint as argument.

If not specified, a default for import requests is used; please refer to the td-client-ruby’s :endpoint option for more details.

api_endpoint

Specifies the Treasure Data’s REST API endpoint.

Requires the endpoint as argument.

If not specified, a default is used; please refer to the td-client-ruby’s :endpoint option for more details.

use_ssl

Specifies whether to communicate using SSL encryption over HTTPS.

Requires no argument to enable; one optional boolean (true or false) argument accepted.

This option corresponds to td-client-ruby’s :ssl option: please refer to that page for important information concerning it.

http_proxy

If communication with the Treasure Data API endpoint has to occur through a proxy, you can use this option to configure it.

Requires the proxy URL as argument. Please refer to the td-client-ruby’s :http_proxy option for details.

connect_timeout read_timeout send_timeout

All three options specify the timeout used for every connection, when reading data from the Treasure Data API endpoints, and when sending data to it.

They require the number of seconds for the timeout as argument.

Please refer to the td-client-ruby’s :connect_timeout, :read_timeout, :send_timeout options for details.

tmpdir

Specifies the temporary folder to use for storing the temporarily converted and compressed inputs.

Requires the path to the temporary folder as argument. If not specified, the system default as specified by Ruby’s Dir.tmpdir is used (the TMPDIR environment variable is honored).

use_gzip_command

Use external gzip command instead of Ruby’s Gzip library. It uses another CPU core so it may improve compression and IO performance.

Copyright

License: Apache License, Version 2.0