Module: Spider

Defined in:

lib/version.rb,
lib/queue.rb,
lib/recorder.rb,
lib/exclusion.rb,
lib/wait_time.rb

Overview

Class to encapsulate the crawl delay being used. Clamps the value to a maximum amount and implements an exponential backoff function for responding to rate limit requests

Defined Under Namespace

Classes: ExclusionParser, VisitQueue, VisitRecorder, WaitTime

Constant Summary

VERSION =

"0.2.1"