Class: RightScraper::Scrapers::Base

Inherits:
Object
  • Object
show all
Defined in:
lib/right_scraper/scrapers/base.rb

Overview

Base class for all scrapers. Subclasses should override #find_next which instantiates the resource from the file system.

Direct Known Subclasses

Cookbook, Workflow

Constant Summary collapse

@@types =

(Hash) Lookup table from textual description of scraper type (‘cookbook’ or ‘workflow’ currently) to the class that represents that scraper.

{}

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#resources ⇒ Object (readonly)

Scraped resources



32
33
34
 
# File 'lib/right_scraper/scrapers/base.rb', line 32

def resources
  @resources
end

Class Method Details

.scraper(options) ⇒ Object

Initialize scraper

Options

:kind

Scraper type, one of :cookbook or :workflow

:repo_dir

Required, path to directory containing files

to be scraped
:ignorable_paths

List of directory names that should

be ignored by scraper
:scanners

List of Scanner classes to use, optional

:builders

List of Builder classes to use, optional

Return

scraper(Scrapers::Base)

Corresponding scraper instance



47
48
49
50
51
52
 
# File 'lib/right_scraper/scrapers/base.rb', line 47

def self.scraper(options)
  scraper_kind = options.delete(:kind)
  scraper_class = @@types[scraper_kind]
  raise "Can't understand how to build scraper #{scraper_kind}" if scraper_class.nil?
  scraper = scraper_class.new(options)
end

Instance Method Details

#close ⇒ Object

Close any opened file descriptor

Return

true

Always return true



99
100
101
102
103
104
105
106
 
# File 'lib/right_scraper/scrapers/base.rb', line 99

def close
  @builder.finish
  if @stack && !@stack.empty?
    @stack.each {|s| s.close}
    @stack = []
  end
  true
end

#next_resource ⇒ Object

Return the next resource in the filesystem, or nil if none. As a part of building the resources, invokes the builders. A resource can be a cookbook, a workflow, a RightScript etc.

Returns

Object

next resource in filesystem, or nil if none.



82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/right_scraper/scrapers/base.rb', line 82

def next_resource
  @logger.operation(:next) do
    next nil if @next.nil?

    value = @next
    @next = search_dirs
    while @next.nil? && !@queue.empty?
      pop_queue
    end
    value
  end
end

#scrape ⇒ Object

Do the scrape! Extract all resources from directory Call this method or call ‘next_resource’ to retrieve resources one by one (you must then call ‘close’ yourself) Fill @resources

Return

resources<Array>

List of all scraped resources



62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/right_scraper/scrapers/base.rb', line 62

def scrape
  @resources = []
  begin
    resource = next_resource
    until resource.nil?
      @resources << resource
      resource = next_resource
    end
  ensure
    close
  end
  @resources
end