Class: RightScraper::Retrievers::CheckoutBasedRetriever

Inherits:
Base
  • Object
show all
Defined in:
lib/right_scraper/retrievers/checkout.rb

Overview

Base class for retrievers that want to do version control operations (CVS, SVN, etc.). Subclasses can get away with implementing only Retrievers::Base#available? and #do_checkout but to support incremental operation need to implement #exists? and #do_update, in addition to Retrievers::Base#ignorable_paths.

Direct Known Subclasses

Git, Svn

Instance Attribute Summary

Attributes inherited from Base

#max_bytes, #max_seconds, #repo_dir, #repository

Instance Method Summary collapse

Methods inherited from Base

#available?, #ignorable_paths, #initialize, repo_dir

Constructor Details

This class inherits a constructor from RightScraper::Retrievers::Base

Instance Method Details

#do_checkout ⇒ Object

Perform a de novo full checkout of the repository. Subclasses must override this to do anything useful.



73
74
75
 
# File 'lib/right_scraper/retrievers/checkout.rb', line 73

def do_checkout
  FileUtils.mkdir_p(@repo_dir)
end

#do_update ⇒ Object

Perform an incremental update of the checkout. Subclasses that want to handle incremental updating need to override this.



67
68
69
 
# File 'lib/right_scraper/retrievers/checkout.rb', line 67

def do_update
  do_checkout
end

#exists? ⇒ Boolean

Return true if a checkout exists.

Returns

Boolean

true if the checkout already exists (and thus incremental updating can occur).

Returns:

  • (Boolean) 


61
62
63
 
# File 'lib/right_scraper/retrievers/checkout.rb', line 61

def exists?
  false
end

#retrieve ⇒ Object

Check out repository into the directory. Occurs between variable initialization and beginning scraping.

Raises:



36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/right_scraper/retrievers/checkout.rb', line 36

def retrieve
  raise RetrieverError.new("retriever is unavailable") unless available?
  if exists?
    begin
      @logger.operation(:updating) do
        do_update
      end
    rescue Exception => e
      FileUtils.remove_entry_secure @basedir
      @logger.operation(:checkout, "switching to using checkout") do
        do_checkout
      end
    end
  else
    @logger.operation(:checkout) do
      do_checkout
    end
  end
end