Method: Wgit::DSL#crawl

Defined in:
lib/wgit/dsl.rb

#crawl(*urls, follow_redirects: true) {|doc| ... } ⇒ Wgit::Document Also known as: crawl_url

Crawls one or more individual urls using Wgit::Crawler#crawl_url underneath. If no urls are provided, then the start URL is used.

Yields:

  • (doc) —

    Given each crawled page (Wgit::Document); this is the only way to interact with them.

Raises:

  • (StandardError) —

    If no urls are provided and no start URL has been set.



99
100
101
102
103
104
105
 
# File 'lib/wgit/dsl.rb', line 99

def crawl(*urls, follow_redirects: true, &block)
  urls = (@dsl_start || []) if urls.empty?
  raise DSL_ERROR__NO_START_URL if urls.empty?

  urls.map! { |url| Wgit::Url.parse(url) }
  get_crawler.crawl_urls(*urls, follow_redirects:, &block)
end