fluent-plugin-feedly

Overview

Fluentd input plugin to fetch RSS/ATOM feed via feedly Cloud API.

Dependencies

Ruby 1.9.3+
Fluentd 0.10.55+

Installation

install with gem or fluent-gem command as:

# for system installed fluentd
$ gem install fluent-plugin-feedly

# for td-agent
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-feedly

# for td-agent2
$ sudo td-agent-gem install fluent-plugin-feedly

Configuration

<source>
  type             feedly

  # Set feedly access token
  # ref. https://feedly.com/v3/auth/dev
  access_token     ACCESS_TOKEN    # Required

  # Set file-path to store last fetched article position
  state_file       /var/log/td-agent/feedly.state  # Required

  # Set output tag
  tag              input.feedly    # Required

  # List subscribe categories in your feedly account with JSON Array
  subscribe_categories  ["global.all"]  # Optional (default: global.all)

  # Set update checking frequency
  run_interval     30m             # Optional (default: 10m)

  # Set bulk read size
  fetch_count      20              # Optional (default: 20)

  # fetching range of time within 30d
  fetch_time_range 3d              # Optional (default: 3d)

  # fetching range of time for initial startup within 30d
  fetch_time_range_on_startup 2w   # Optional (default: 2w)

  # Using sandbox account
  enable_sandbox   false           # Optional (default: false)

  # Set log level for this plugin. To hide info level, set 'warn' for this value.
  # it can see at stdout as like `$ tail -f /var/log/td-agent/td-agent.log`
  log_level        info 　　　　　　 # Optional (default: info)
</source>

note : The subscribe_categories is also supported with single or multi line configuration like below.

# single line
subscribe_categories ["先端技術", "mysql"]

# multi line
subscribe_categories [
  "先端技術",
  "mysql"
]

Usage

After installed this plugin, executing fluentd with following configuration.

$ cat /etc/td-agent/td-agent.conf
<source>
  type             feedly
  access_token     YOUR_ACCESS_TOKEN
  state_file       /var/log/td-agent/feedly.state
  tag              input.feedly
  run_interval     30m
  fetch_time_range 1h
  fetch_time_range_on_startup 3h
</source>

<match input.feedly>
  type             file
  path             /tmp/feedly*.json
  symlink_path     /tmp/feedly.json
  format           json
  append           true
</match>

You can see the behavior about this plugin with this command.

# to check stdout of this plugin
$ tail -f /var/log/td-agent/td-agent.log
2014-10-16 14:47:01 +0900 [info]: Feedly: fetched articles. articles=416 request_option={:count=>1000, :continuation=>"148cfb7f516:9371a3c:726280cf", :newerThan=>1412228787000}
2014-10-16 15:02:02 +0900 [info]: Feedly: fetched articles. articles=492 request_option={:count=>1000, :continuation=>nil, :newerThan=>1413428521000}

# to check fetched articles
$ tail -f /tmp/feedly.json | jq "."

TODO

Pull requests are very welcome!!

Copyright

License

Apache License, Version 2.0

Contributing

Fork it ( https://github.com/[my-github-username]/fluent-plugin-feedly/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request