Module: SiteMapper

Defined in:
lib/site_mapper.rb,
lib/site_mapper/robots.rb,
lib/site_mapper/logger.rb,
lib/site_mapper/crawler.rb,
lib/site_mapper/version.rb,
lib/site_mapper/request.rb,
lib/site_mapper/crawl_url.rb

Overview

Find all links on domain to domain

Defined Under Namespace

Classes: CrawlUrl, Crawler, Logger, Request, Robots

Constant Summary collapse

'https://rubygems.org/gems/site_mapper'
USER_AGENT =

SiteMapper User-Agent

"SiteMapper/#{SiteMapper::VERSION} (+#{INFO_LINK})"
VERSION =

Gem version

'0.0.11'

Class Method Summary collapse

Class Method Details

.map(link, options = {}) ⇒ Array

Map all links on a given site.

Examples:

Collect all URLs from example.com

SiteMapper.map('example.com')

Collect all URLs from example.com with custom User-agent

SiteMapper.map('example.com', user_agent: 'MyUserAgent')

Collect all URLs from example.com with custom logger class

class MyLogger
  def self.log(msg);     puts msg;end
  def self.err_log(msg); puts msg;end
end
SiteMapper.map('example.com', logger: MyLogger)

Parameters:

  • link (String)

    to domain

  • options (Hash) (defaults to: {})

    hash

Returns:

  • (Array)

    with links.



32
33
34
35
36
# File 'lib/site_mapper.rb', line 32

def self.map(link, options = {})
  set_logger(options.delete(:logger))
  options = { user_agent: USER_AGENT }.merge(options)
  Crawler.collect_urls(link, options) { |url| yield(url) if block_given? }
end

.set_logger(logger) ⇒ Object

Set logger.

Examples:

set system logger

SiteMapper.set_logger(:system)

set nil logger

SiteMapper.set_logger(:nil)

set your own logger

SiteMapper.set_logger(YourLogger)

Parameters:

  • logger (Object)


46
47
48
49
50
51
52
53
# File 'lib/site_mapper.rb', line 46

def self.set_logger(logger)
  return if logger.nil?
  if logger.is_a?(Symbol)
    Logger.use_logger_type(logger)
  else
    Logger.use_logger(logger)
  end
end