Class: Unbreakable::Scraper
- Inherits:
-
Object
- Object
- Unbreakable::Scraper
- Extended by:
- Forwardable
- Defined in:
- lib/unbreakable/scraper.rb
Overview
You may implement a scraper by subclassing this class:
require 'open-uri'
class MyScraper < Unbreakable::Scraper
# Stores the contents of +http://www.example.com/+ in +index.html+.
def retrieve(args)
store(:path => 'index.html'){ open('http://www.example.com/').read }
end
# Processes +index.html+.
def process(args)
fetch('index.html').process(:transform).apply
end
# Alternatively, you can just set the files to fetch, which will be
# processed using a +:transform+ processor which you must implement.
def processable
['index.html']
end
end
To configure:
scraper.configure do |c|
c.datastore = MyDataStore.new # default Unbreakable::DataStorage::FileDataStore.new(scraper)
c.log = Logger.new('/path/to/file') # default Logger.new(STDOUT)
c.datastore. = true # default false
end
The following instance methods must be implemented in sub-classes:
-
retrieve
-
process
orprocessable
Instance Method Summary collapse
-
#initialize ⇒ Scraper
constructor
Initializes a Dragonfly app for storage and processing.
-
#opts ⇒ OptionParser
Returns an option parser.
-
#parse(temp_object_or_uid, encoding = 'utf-8') ⇒ Object
Parses a JSON, HTML, XML, or YAML file.
-
#process(args) ⇒ Object
Processes cached files into machine-readable data.
-
#processable ⇒ Array<String>
Returns a list of record IDs to process.
-
#retrieve(args) ⇒ Object
Caches remote files to the datastore for later processing.
-
#run(args) ⇒ Object
Runs the command.
-
#store(opts = {}, &block) ⇒ Object
Stores a record in the datastore.
Constructor Details
#initialize ⇒ Scraper
Initializes a Dragonfly app for storage and processing.
48 49 50 51 52 53 54 55 56 57 58 |
# File 'lib/unbreakable/scraper.rb', line 48 def initialize @app = Dragonfly[SecureRandom.hex.to_sym] # defaults to Logger.new('/var/tmp/dragonfly.log') @app.log = Logger.new(STDOUT) # defaults to Dragonfly::DataStorage::FileDataStore.new @app.datastore = Unbreakable::DataStorage::FileDataStore.new(self) # defaults to '/var/tmp/dragonfly' @app.datastore.root_path = '/var/tmp/unbreakable' # defaults to true @app.datastore. = false end |
Instance Method Details
#opts ⇒ OptionParser
Returns an option parser.
62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
# File 'lib/unbreakable/scraper.rb', line 62 def opts if @opts.nil? @opts = OptionParser.new @opts. = "usage: \#{@opts.program_name} [options] <command> [<args>]\n\nThe most commonly used commands are:\nretrieve Cache remote files to the datastore for later processing\nprocess Process cached files into machine-readable data\nconfig Print the current configuration\n" @opts.separator '' @opts.separator 'Specific options:' extract_configuration @app @opts.separator '' @opts.separator 'General options:' @opts.on_tail('-h', '--help', 'Display this screen') do puts @opts exit end end @opts end |
#parse(temp_object_or_uid, encoding = 'utf-8') ⇒ Object
Parses a JSON, HTML, XML, or YAML file.
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
# File 'lib/unbreakable/scraper.rb', line 124 def parse(temp_object_or_uid, encoding = 'utf-8') temp_object = temp_object_or_uid.is_a?(Dragonfly::TempObject) ? temp_object_or_uid : fetch(temp_object_or_uid) string = temp_object.data case File.extname temp_object.path when '.json' begin require 'yajl' Yajl::Parser.parse string rescue LoadError require 'json' JSON.parse string end when '.html' require 'nokogiri' Nokogiri::HTML string, nil, encoding when '.xml' require 'nokogiri' Nokogiri::XML string, nil, encoding when '.yml', '.yaml' require 'yaml' YAML.load string else string end end |
#process(args) ⇒ Object
Processes cached files into machine-readable data.
158 159 160 161 162 |
# File 'lib/unbreakable/scraper.rb', line 158 def process(args) processable.each do |record| fetch(record).process(:transform, :args => args).apply end end |
#processable ⇒ Array<String>
Returns a list of record IDs to process.
166 167 168 |
# File 'lib/unbreakable/scraper.rb', line 166 def processable raise NotImplementedError end |
#retrieve(args) ⇒ Object
Caches remote files to the datastore for later processing.
152 153 154 |
# File 'lib/unbreakable/scraper.rb', line 152 def retrieve(args) raise NotImplementedError end |
#run(args) ⇒ Object
Only call this method once per scraper instance.
Runs the command. Most often run from a command-line script as:
scraper.run(ARGV)
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 |
# File 'lib/unbreakable/scraper.rb', line 94 def run(args) opts.parse!(args) command = args.shift case command when 'retrieve' retrieve(args) when 'process' process(args) when 'config' print_configuration @app when nil puts opts else opts.abort "'#{command}' is not a #{opts.program_name} command. See '#{opts.program_name} --help'." end end |
#store(opts = {}, &block) ⇒ Object
Stores a record in the datastore.
114 115 116 |
# File 'lib/unbreakable/scraper.rb', line 114 def store(opts = {}, &block) datastore.defer_store(opts, &block) end |