Robocop is a simple Rack middleware that inserts the
X-Robots-Tag into the headers of all your responses.
X-Robots-Tag can be used in place of a
robots.txt file or meta tags to tell crawlers what they're allowed to do with your content. See this article for more information.
Tested against Ruby 1.8.7, 1.9.2, 1.9.3, and Rubinius & JRuby in both 1.8 and 1.9 mode.
The simplest way to install Robocop is to use Bundler.
Add Robocop to your
To use Robocop in your Rails application, add the following line to your application config file (
config/application.rb for Rails 3,
config/environment.rb for Rails 2):
config.middleware.use :: do directives :all end
Other Rack Applications (Sinatra, Padrino, etc.)
Simple add the following to your
use :: do directives :all end
The following directives can be passed in to Robocop's configuration:
Directives (useragent agnostic)
If you just want to specify a list of directives for all useragents to follow, simply pass in a list of directives with the
config.middleware.use :: do directives :noindex, :nofollow end
If you want to give specific user agents unique sets of directives, you can do so by using the
config.middleware.use :: do useragent :googlebot do directive :all end directives :noindex, :nofollow end
It should be noted that if both the
directives options are passed in, the
useragents are output first in the header, followed by the generic directives.
- Re-factor & DRY up code
- Directive validation
- Add support for
- Sanity checks for directives that are passed in. e.g. passing all, noindex, nofollow doesn't make any sense and should not be allowed.
Note on Patches / Pull Requests
- Fork the project.
- Code your feature addition or bug fix.
- Add specs for it. This is important so we don't break it in a future version unintentionally.
- Commit, do not mess with Rakefile or version number. If you want to have your own version, that's fine but bump version in a commit by itself so we can ignore it when merging.
- Send a pull request. Bonus points for topic branches.
Copyright (c) 2012 PJ Kelly. See LICENSE for details.