Class: Wgit::Indexer

Inherits:
Object
  • Object
show all
Defined in:
lib/wgit/indexer.rb

Overview

Class which crawls and saves the Documents to a database. Can be thought of as a combination of Wgit::Crawler and Wgit::Database.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(database = Wgit::Database.new, crawler = Wgit::Crawler.new) ⇒ Indexer

Initialize the Indexer.

Parameters:

  • database (Wgit::Database) (defaults to: Wgit::Database.new) —

    The database instance (already initialized and connected) used to index.

  • crawler (Wgit::Crawler) (defaults to: Wgit::Crawler.new) —

    The crawler instance used to index.



21
22
23
24
 
# File 'lib/wgit/indexer.rb', line 21

def initialize(database = Wgit::Database.new, crawler = Wgit::Crawler.new)
  @db      = database
  @crawler = crawler
end

Instance Attribute Details

#crawler ⇒ Object (readonly)

The crawler used to index the WWW.



11
12
13
 
# File 'lib/wgit/indexer.rb', line 11

def crawler
  @crawler
end

#db ⇒ Object (readonly) Also known as: database

The database instance used to store Urls and Documents in.



14
15
16
 
# File 'lib/wgit/indexer.rb', line 14

def db
  @db
end

Instance Method Details

#index_site(url, insert_externals: false, follow: :default, allow_paths: nil, disallow_paths: nil) {|doc| ... } ⇒ Integer Also known as: index_r

Crawls a single website's pages and stores them into the database. There is no max download limit so be careful which sites you index. Logs info on the crawl using Wgit.logger as it goes along. This method will honour the site's robots.txt and 'noindex' requests.

Parameters:

  • url (Wgit::Url) —

    The base Url of the website to crawl.

  • insert_externals (Boolean) (defaults to: false) —

    Whether or not to insert the website's external Url's into the database.

  • follow (String) (defaults to: :default) —

    The xpath extracting links to be followed during the crawl. This changes how a site is crawled. Only links pointing to the site domain are allowed. The :default is any <a> href returning HTML.

  • allow_paths (String, Array<String>) (defaults to: nil) —

    Filters the follow: links by selecting them if their path File.fnmatch? one of allow_paths.

  • disallow_paths (String, Array<String>) (defaults to: nil) —

    Filters the follow links by rejecting them if their path File.fnmatch? one of disallow_paths.

Yields:

  • (doc) —

    Given the Wgit::Document of each crawled web page before it's inserted into the database allowing for prior manipulation. Return nil or false from the block to prevent the document from being saved into the database.

Returns:

  • (Integer) —

    The total number of webpages/documents indexed.



127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
# File 'lib/wgit/indexer.rb', line 127

def index_site(
  url, insert_externals: false, follow: :default,
  allow_paths: nil, disallow_paths: nil
)
  parser = parse_robots_txt(url)
  if parser&.no_index?
    upsert_url_and_redirects(url)

    return 0
  end

  allow_paths, disallow_paths = merge_paths(parser, allow_paths, disallow_paths)
  crawl_opts = { follow:, allow_paths:, disallow_paths: }
  total_pages_indexed = 0

  ext_urls = @crawler.crawl_site(url, **crawl_opts) do |doc|
    next if no_index?(@crawler.last_response, doc)

    result = block_given? ? yield(doc) : true

    if result && !doc.empty?
      upsert_doc(doc)
      total_pages_indexed += 1
    end
  end

  upsert_url_and_redirects(url)
  upsert_external_urls(ext_urls) if insert_externals && ext_urls

  Wgit.logger.info("Crawled and indexed #{total_pages_indexed} documents \
for the site: #{url}")

  total_pages_indexed
end

#index_url(url, insert_externals: false) {|doc| ... } ⇒ Object

Crawls a single webpage and stores it into the database. There is no max download limit so be careful of large pages. Logs info on the crawl using Wgit.logger as it goes along. This method will honour the site's robots.txt and 'noindex' requests in relation to the given url.

Parameters:

  • url (Wgit::Url) —

    The webpage Url to crawl.

  • insert_externals (Boolean) (defaults to: false) —

    Whether or not to insert the webpages external Url's into the database.

Yields:

  • (doc) —

    Given the Wgit::Document of the crawled webpage, before it's inserted into the database allowing for prior manipulation. Return nil or false from the block to prevent the document from being saved into the database.



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'lib/wgit/indexer.rb', line 198

def index_url(url, insert_externals: false)
  parser = parse_robots_txt(url)
  if parser && (parser.no_index? || contains_path?(parser.disallow_paths, url))
    upsert_url_and_redirects(url)

    return
  end

  document = @crawler.crawl_url(url) do |doc|
    break if no_index?(@crawler.last_response, doc)

    result = block_given? ? yield(doc) : true
    upsert_doc(doc) if result && !doc.empty?
  end

  upsert_url_and_redirects(url)

  ext_urls = document&.external_links
  upsert_external_urls(ext_urls) if insert_externals && ext_urls

  nil
end

#index_urls(*urls, insert_externals: false) {|doc| ... } ⇒ Object Also known as: index

Crawls one or more webpages and stores them into the database. There is no max download limit so be careful of large pages. Logs info on the crawl using Wgit.logger as it goes along. This method will honour the site's robots.txt and 'noindex' requests in relation to the given urls.

Parameters:

  • urls (*Wgit::Url) —

    The webpage Url's to crawl.

  • insert_externals (Boolean) (defaults to: false) —

    Whether or not to insert the webpages external Url's into the database.

Yields:

  • (doc) —

    Given the Wgit::Document of the crawled webpage, before it's inserted into the database allowing for prior manipulation. Return nil or false from the block to prevent the document from being saved into the database.

Raises:

  • (StandardError) —

    if no urls are provided.



176
177
178
179
180
181
182
183
 
# File 'lib/wgit/indexer.rb', line 176

def index_urls(*urls, insert_externals: false, &block)
  raise 'You must provide at least one Url' if urls.empty?

  opts = { insert_externals: }
  Wgit::Utils.each(urls) { |url| index_url(url, **opts, &block) }

  nil
end

#index_www(max_sites: -1,, max_data: 1_048_576_000, max_urls_per_iteration: 10) ⇒ Object

Retrieves uncrawled url's from the database and recursively crawls each site storing their internal pages into the database and adding their external url's to be crawled later on. Logs info on the crawl using Wgit.logger as it goes along. This method will honour all site's robots.txt and 'noindex' requests.

Parameters:

  • max_sites (Integer) (defaults to: -1,) —

    The number of separate and whole websites to be crawled before the method exits. Defaults to -1 which means the crawl will occur until manually stopped (Ctrl+C), the max_data has been reached, or it runs out of external urls to index.

  • max_data (Integer) (defaults to: 1_048_576_000) —

    The maximum amount of bytes that will be scraped from the web (default is 1GB). Note, that this value is used to determine when to stop crawling; it's not a guarantee of the max data that will be obtained.

  • max_urls_per_iteration (Integer) (defaults to: 10) —

    The maximum number of uncrawled urls to index for each iteration, before checking max_sites and max_data, possibly ending the crawl.



43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/wgit/indexer.rb', line 43

def index_www(max_sites: -1, max_data: 1_048_576_000, max_urls_per_iteration: 10)
  if max_sites.negative?
    Wgit.logger.info("Indexing until the database has been filled or it \
runs out of urls to crawl (which might be never)")
  end
  site_count = 0

  while keep_crawling?(site_count, max_sites, max_data)
    Wgit.logger.info("Current database size: #{@db.size}")

    uncrawled_urls = @db.uncrawled_urls(limit: max_urls_per_iteration)

    if uncrawled_urls.empty?
      Wgit.logger.info('No urls to crawl, exiting')

      return
    end
    Wgit.logger.info("Starting indexing loop for: #{uncrawled_urls.map(&:to_s)}")

    docs_count = 0
    urls_count = 0

    uncrawled_urls.each do |url|
      unless keep_crawling?(site_count, max_sites, max_data)
        Wgit.logger.info("Reached max number of sites to crawl or \
database capacity, exiting")

        return
      end
      site_count += 1

      parser = parse_robots_txt(url)
      if parser&.no_index?
        upsert_url_and_redirects(url)

        next
      end

      site_docs_count = 0
      ext_links = @crawler.crawl_site(
        url, allow_paths: parser&.allow_paths, disallow_paths: parser&.disallow_paths
      ) do |doc|
        next if doc.empty? || no_index?(@crawler.last_response, doc)

        upsert_doc(doc)
        docs_count += 1
        site_docs_count += 1
      end

      upsert_url_and_redirects(url)

      urls_count += upsert_external_urls(ext_links)
    end

    Wgit.logger.info("Crawled and indexed documents for #{docs_count} \
url(s) during this iteration")
    Wgit.logger.info("Found and saved #{urls_count} external url(s) for \
future iterations")
  end

  nil
end

#keep_crawling?(site_count, max_sites, max_data) ⇒ Boolean (protected)

Returns whether or not to keep crawling based on the DB size and current loop iteration.

Parameters:

  • site_count (Integer) —

    The current number of crawled sites.

  • max_sites (Integer) —

    The maximum number of sites to crawl before stopping. Use -1 for an infinite number of sites.

  • max_data (Integer) —

    The maximum amount of data to crawl before stopping.

Returns:

  • (Boolean) —

    True if the crawl should continue, false otherwise.



232
233
234
235
236
237
 
# File 'lib/wgit/indexer.rb', line 232

def keep_crawling?(site_count, max_sites, max_data)
  return false if @db.size >= max_data
  return true  if max_sites.negative?

  site_count < max_sites
end

#upsert_doc(doc) ⇒ Object (protected)

Write the doc to the DB. Note that the unique url index on the documents collection deliberately prevents duplicate inserts. If the document already exists, then it will be updated in the DB.

Parameters:



244
245
246
247
248
249
250
 
# File 'lib/wgit/indexer.rb', line 244

def upsert_doc(doc)
  if @db.upsert(doc)
    Wgit.logger.info("Saved document for url: #{doc.url}")
  else
    Wgit.logger.info("Updated document for url: #{doc.url}")
  end
end

#upsert_external_urls(urls) ⇒ Integer (protected)

Write the external urls to the DB. For any external url, its origin will be inserted e.g. if the external url is http://example.com/contact then http://example.com will be inserted into the database. Note that the unique url index on the urls collection deliberately prevents duplicate inserts.

Parameters:

  • urls (Array<Wgit::Url>) —

    The external urls to write to the DB.

Returns:

  • (Integer) —

    The number of upserted urls.



271
272
273
274
275
276
277
278
279
280
281
282
 
# File 'lib/wgit/indexer.rb', line 271

def upsert_external_urls(urls)
  urls = urls
         .reject(&:invalid?)
         .map(&:to_origin)
         .uniq
  return 0 if urls.empty?

  count = @db.bulk_upsert(urls)
  Wgit.logger.info("Saved #{count} external urls")

  count
end

#upsert_url_and_redirects(url) ⇒ Integer (protected)

Upsert the url and its redirects, setting all to crawled = true.

Parameters:

  • url (Wgit::Url) —

    The url to write to the DB.

Returns:

  • (Integer) —

    The number of upserted urls (url + redirect urls).



256
257
258
259
260
261
 
# File 'lib/wgit/indexer.rb', line 256

def upsert_url_and_redirects(url)
  url.crawled = true unless url.crawled?

  # Upsert the url and any url redirects, setting them as crawled also.
  @db.bulk_upsert(url.redirects_journey)
end