Robots

Get robots.txt information from a site.

Installation

gem install Robots-txt

bundle add Robots-txt

If bundler is not being used to manage dependencies, install the gem by executing:

gem install Robots-txt

Usage

require './lib/robots'

u = Robots::Url.new('https://www.ebay.com')
u.allow
u.disallow
u.disallow_removed
u.allow_removed

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/Robots.