Class: RightScraper::Scrapers::Base

Inherits:
RegisteredBase show all
Defined in:
lib/right_scraper/scrapers/base.rb

Overview

Base class for all scrapers. Subclasses should override #find_next which instantiates the resource from the file system.

Direct Known Subclasses

Cookbook, Workflow

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from RegisteredBase

query_registered_type, register_class, register_self, registered_types

Instance Attribute Details

#resources ⇒ Object (readonly)

Scraped resources



36
37
38
 
# File 'lib/right_scraper/scrapers/base.rb', line 36

def resources
  @resources
end

Class Method Details

.registration_module ⇒ Module

Returns module for registered repository types.

Returns:

  • (Module) —

    module for registered repository types



39
40
41
 
# File 'lib/right_scraper/scrapers/base.rb', line 39

def self.registration_module
  ::RightScraper::Scrapers
end

.scraper(options) ⇒ Object

Initialize scraper

Options

:kind

Scraper type, one of :cookbook or :workflow

:repo_dir

Required, path to directory containing files

to be scraped
:ignorable_paths

List of directory names that should

be ignored by scraper
:scanners

List of Scanner classes to use, optional

:builders

List of Builder classes to use, optional

Return

scraper(Scrapers::Base)

Corresponding scraper instance



56
57
58
59
60
 
# File 'lib/right_scraper/scrapers/base.rb', line 56

def self.scraper(options)
  scraper_kind = options.delete(:kind)
  scraper_class = query_registered_type(scraper_kind)
  scraper_class.new(options)
end

Instance Method Details

#close ⇒ Object

Close any opened file descriptor

Return

true

Always return true



107
108
109
110
111
112
113
114
 
# File 'lib/right_scraper/scrapers/base.rb', line 107

def close
  @builder.finish
  if @stack && !@stack.empty?
    @stack.each {|s| s.close}
    @stack = []
  end
  true
end

#next_resource ⇒ Object

Return the next resource in the filesystem, or nil if none. As a part of building the resources, invokes the builders. A resource can be a cookbook, a workflow, a RightScript etc.

Returns

Object

next resource in filesystem, or nil if none.



90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/right_scraper/scrapers/base.rb', line 90

def next_resource
  @logger.operation(:next) do
    next nil if @next.nil?

    value = @next
    @next = search_dirs
    while @next.nil? && !@queue.empty?
      pop_queue
    end
    value
  end
end

#scrape ⇒ Object

Do the scrape! Extract all resources from directory Call this method or call ‘next_resource’ to retrieve resources one by one (you must then call ‘close’ yourself) Fill @resources

Return

resources<Array>

List of all scraped resources



70
71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/right_scraper/scrapers/base.rb', line 70

def scrape
  @resources = []
  begin
    resource = next_resource
    until resource.nil?
      @resources << resource
      resource = next_resource
    end
  ensure
    close
  end
  @resources
end