Fluent::Plugin::SolrOutput
This is a Fluentd output plugin for send data to Apache Solr. It support SolrCloud not only Standalone Solr.
Requirements
| fluent-plugin-output-solr | fluentd | td-agent | ruby |
|---|---|---|---|
| 1.x.x | >= 0.14.0, < 2 | 3 | >= 2.1 |
| 0.x.x | ~> 0.12.0 | 2 | >= 1.9 |
- The 1.x.x series is developed from this branch (master)
- The 0.x.x series (compatible with fluentd v0.12, and td-agent 2) is developed on the v0.x.x branch
Installation
Install it yourself as:
$ gem install fluent-plugin-output-solr
How to build
$ gem install bundler
$ bundle install
$ rake test
$ rake build
$ rake install
Config parameters
url
The Solr server url (for example http://localhost:8983/solr/collection1).
url http://localhost:8983/solr/collection1
zk_host
The ZooKeeper connection string that SolrCloud refers to (for example localhost:2181/solr).
zk_host localhost:2181/solr
collection
The SolrCloud collection name (default collection1).
collection collection1
defined_fields
The defined fields in the Solr schema.xml. If omitted, it will get fields via Solr Schema API.
defined_fields ["id", "title"]
ignore_undefined_fields
Ignore undefined fields in the Solr schema.xml.
ignore_undefined_fields false
unique_key_field
A field name of unique key in the Solr schema.xml. If omitted, it will get unique key via Solr Schema API.
unique_key_field id
string_field_value_max_length
A string field value max length. If set -1, it means unlimited (default -1). However, there is a limit of Solr.
string_field_value_max_length -1
timestamp_field
A field name of event timestamp in the Solr schema.xml (default event_timestamp).
flush_size
A number of events to queue up before writing to Solr (default 100).
flush_size 100
commit_with_flush
Send commit command to Solr with flush (default true).
commit_with_flush true
Plugin setup examples
Sent to standalone Solr using data-driven schemaless mode.
<match something.logs>
@type solr
# The Solr server url (for example http://localhost:8983/solr/collection1).
url http://localhost:8983/solr/collection1
</match>
Sent to SolrCloud using data-driven schemaless mode.
<match something.logs>
@type solr
# The ZooKeeper connection string that SolrCloud refers to (for example localhost:2181/solr).
zk_host localhost:2181/solr
# The SolrCloud collection name (default collection1).
collection collection1
</match>
Solr setup examples
How to setup Standalone Solr using data-driven schemaless mode.
1.Download and install Solr
$ mkdir $HOME/solr
$ cd $HOME/solr
$ wget https://archive.apache.org/dist/lucene/solr/5.4.0/solr-5.4.0.tgz
$ tar zxvf solr-5.4.0.tgz
$ cd solr-5.4.0
2.Start standalone Solr
$ ./bin/solr start -p 8983 -s server/solr
3.Create core
$ ./bin/solr create -c collection1 -d server/solr/configsets/data_driven_schema_configs -n collection1_configs
How to setup SolrCloud using data-driven schemaless mode (shards=1 and replicationfactor=2).
1.Download and install ZooKeeper
$ mkdir $HOME/zookeeper
$ cd $HOME/zookeeper
$ wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
$ tar zxvf zookeeper-3.4.6.tar.gz
$ cd zookeeper-3.4.6
$ cp -p ./conf/zoo_sample.cfg ./conf/zoo.cfg
2.Start standalone ZooKeeper
$ ./bin/zkServer.sh start
3.Download an install Solr
$ mkdir $HOME/solr
$ cd $HOME/solr
$ wget https://archive.apache.org/dist/lucene/solr/5.4.0/solr-5.4.0.tgz
$ tar zxvf solr-5.4.0.tgz
$ cd solr-5.4.0
$ ./server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd clear /solr
$ ./server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd makepath /solr
$ cp -pr server/solr server/solr1
$ cp -pr server/solr server/solr2
4.Start SolrCloud
$ ./bin/solr start -h localhost -p 8983 -z localhost:2181/solr -s server/solr1
$ ./bin/solr start -h localhost -p 8985 -z localhost:2181/solr -s server/solr2
5.Create collection
$ ./bin/solr create -c collection1 -d server/solr1/configsets/data_driven_schema_configs -n collection1_configs -shards 1 -replicationFactor 2
Development
After checking out the repo, run bundle install to install dependencies. Then, run rake test to run the tests.
Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/mosuka/fluent-plugin-output-solr.