Module: OpencBot

Includes:
Helpers::Text, ScraperWiki
Included in:
MyModule, CompanyFetcherBot, SimpleOpencBot
Defined in:
lib/openc_bot/helpers/_csv.rb,
lib/openc_bot.rb,
lib/openc_bot/version.rb,
lib/openc_bot/exceptions.rb,
lib/openc_bot/helpers/html.rb,
lib/openc_bot/helpers/text.rb,
lib/openc_bot/helpers/dates.rb,
lib/openc_bot/incrementers/base.rb,
lib/openc_bot/bot_data_validator.rb,
lib/openc_bot/company_fetcher_bot.rb,
lib/openc_bot/incrementers/common.rb,
lib/openc_bot/helpers/alpha_search.rb,
lib/openc_bot/helpers/register_methods.rb,
lib/openc_bot/helpers/incremental_search.rb

Overview

This is in _csr.rb to avoid requiring it when we mean to require the system csv library.

Defined Under Namespace

Modules: BotDataValidator, CompanyFetcherBot, Helpers Classes: AsciiIncrementer, BaseIncrementer, DatabaseError, InvalidDataError, ManualIncrementer, NotFoundError, NumericIncrementer, OpencBotError, RecordInvalid

Constant Summary collapse

VERSION = 
"0.0.46"

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Helpers::Text

#normalise_utf8_spaces, #strip_all_spaces

Class Method Details

.extended(obj) ⇒ Object

When deciding on the location of the SQLite databases we need to set the directory relative to the directory of the file/app that includes the gem, not the gem itself. Doing it this way, and setting a class variable feels ugly, but this appears to be difficult in Ruby, esp as the file may ultimately be called by another process, e.g. the main OpenCorporates app or the console, whose main directory is unrelated to where the databases are stored (which means we can’t use Dir.pwd etc). The only time we know about the directory is when the module is called to extend the file, and we capture that in the @app_directory class variable



69
70
71
72
73
 
# File 'lib/openc_bot.rb', line 69

def self.extended(obj)
  path, = caller[0].partition(":")
  path = File.expand_path(File.join(File.dirname(path),'..'))
  @@app_directory = path
end

Instance Method Details

#db_location ⇒ Object 



83
84
85
 
# File 'lib/openc_bot.rb', line 83

def db_location
  File.expand_path(File.join(@@app_directory, 'db', db_name))
end

#db_name ⇒ Object 



75
76
77
78
79
80
81
 
# File 'lib/openc_bot.rb', line 75

def db_name
  if is_a?(Module)
    "#{self.name.downcase}.db"
  else
    "#{self.class.name.downcase}.db"
  end
end

#export(opts = {}) ⇒ Object 



48
49
50
51
52
53
 
# File 'lib/openc_bot.rb', line 48

def export(opts={})
  export_data(opts).each do |record|
    $stdout.puts record.to_json
    $stdout.flush
  end
end

#insert_or_update(uniq_keys, values_hash, tbl_name = 'ocdata') ⇒ Object 



20
21
22
 
# File 'lib/openc_bot.rb', line 20

def insert_or_update(uniq_keys, values_hash, tbl_name='ocdata')
  sqlite_magic_connection.insert_or_update(uniq_keys, values_hash, tbl_name)
end

#root_directory ⇒ Object

Returns the root directory of the bot (not this gem). Assumes the bot file that extends its functionality using this bot is in a directory (lib) inside the root directory



35
36
37
 
# File 'lib/openc_bot.rb', line 35

def root_directory
  @@app_directory
end

#save_data(uniq_keys, values_array, tbl_name = 'ocdata') ⇒ Object 



24
25
26
 
# File 'lib/openc_bot.rb', line 24

def save_data(uniq_keys, values_array, tbl_name='ocdata')
  save_sqlite(uniq_keys, values_array, tbl_name)
end

#save_run_report(report_hash) ⇒ Object 



28
29
30
31
 
# File 'lib/openc_bot.rb', line 28

def save_run_report(report_hash)
  json_report = report_hash.to_json
  save_data([:run_at], { :report => json_report, :run_at => Time.now.to_s }, :ocrunreports)
end

#spotcheck ⇒ Object 



55
56
57
 
# File 'lib/openc_bot.rb', line 55

def spotcheck
  $stdout.puts JSON.pretty_generate(spotcheck_data)
end

#sqlite_busy_timeout ⇒ Object 



94
95
96
 
# File 'lib/openc_bot.rb', line 94

def sqlite_busy_timeout
  self.const_defined?('SQLITE_BUSY_TIMEOUT') && self.const_get('SQLITE_BUSY_TIMEOUT')
end

#sqlite_magic_connection ⇒ Object

Override default in ScraperWiki gem



88
89
90
91
92
 
# File 'lib/openc_bot.rb', line 88

def sqlite_magic_connection
  db = @config ? @config[:db] : File.expand_path(File.join(@@app_directory, 'db', db_name))
  options = sqlite_busy_timeout ? {:busy_timeout => sqlite_busy_timeout} : {:busy_timeout => 10000}
  @sqlite_magic_connection ||= SqliteMagic::Connection.new(db, options)
end

#table_summary ⇒ Object 



98
99
100
101
102
 
# File 'lib/openc_bot.rb', line 98

def table_summary
  field_names = sqlite_magic_connection.execute('PRAGMA table_info(ocdata)').collect{|c| c['name']}
  select_sql = "COUNT(1) Total, " + field_names.collect{ |fn| "COUNT(#{fn}) #{fn}_not_null" }.join(', ') + " FROM ocdata"
  select(select_sql).first
end

#unlock_database ⇒ Object 



39
40
41
 
# File 'lib/openc_bot.rb', line 39

def unlock_database
  sqlite_magic_connection.execute("BEGIN TRANSACTION; END;")
end

#verbose? ⇒ Boolean

Convenience method that returns true if VERBOSE environmental variable set (at the moment whatever it is set to)

Returns:

  • (Boolean) 


44
45
46
 
# File 'lib/openc_bot.rb', line 44

def verbose?
  ENV['VERBOSE']
end