Class: Replicate::Loader

Inherits:
Emitter show all
Defined in:
lib/replicate/loader.rb

Overview

Load replicants in a streaming fashion.

The Loader reads [type, id, attributes] replicant tuples and creates objects in the current environment.

Objects are expected to arrive in order such that a record referenced via foreign key always precedes the referencing record. The Loader maintains a mapping of primary keys from the dump system to the current environment. This mapping is used to properly establish new foreign key values on all records inserted.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Emitter

#complete, #emit, #listen, #use

Constructor Details

#initialize ⇒ Loader

Returns a new instance of Loader.



17
18
19
20
21
 
# File 'lib/replicate/loader.rb', line 17

def initialize
  @keymap = Hash.new { |hash,k| hash[k] = {} }
  @stats  = Hash.new { |hash,k| hash[k] = 0 }
  super
end

Instance Attribute Details

#stats ⇒ Object (readonly)

Stats hash.



15
16
17
 
# File 'lib/replicate/loader.rb', line 15

def stats
  @stats
end

Instance Method Details

#constantize(string) ⇒ Object

1.9 implement has the inherit argument to const_defined?. Use it!



135
136
137
138
139
140
141
142
143
 
# File 'lib/replicate/loader.rb', line 135

def constantize(string)
  string.split('::').inject ::Object do |namespace, name|
    if namespace.const_defined?(name)
      namespace.const_get(name)
    else
      namespace.const_missing(name)
    end
  end
end

#feed(type, id, attrs) ⇒ Object

Feed a single replicant tuple into the loader.

type - The class to create. Must respond to load_replicant. id - The remote system’s id for this object. attrs - Hash of primitively typed objects.

Returns the need object resulting from the load operation.



45
46
47
48
49
50
 
# File 'lib/replicate/loader.rb', line 45

def feed(type, id, attrs)
  type = type.to_s
  object = load(type, id, attrs)
  @stats[type] += 1
  emit type, id, attrs, object
end

#load(type, id, attributes) ⇒ Object

Load an individual replicant into the underlying datastore.

type - Model class name as a String. id - Primary key id of the record on the dump system. This must be

translated to the local system and stored in the keymap.

attrs - Hash of attributes to set on the new record.

Returns the new object instance.



77
78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/replicate/loader.rb', line 77

def load(type, id, attributes)
  model_class = constantize(type)
  translate_ids type, id, attributes
  begin
    new_id, instance = model_class.load_replicant(type, id, attributes)
  rescue => boom
    warn "error: loading #{type} #{id} #{boom.class} #{boom}"
    raise
  end
  register_id instance, type, id, new_id
  instance
end

#log_to(out = $stderr, verbose = false, quiet = false) ⇒ Object

Register a filter to write status information to the given stream. By default, a single line is used to report object counts while the dump is in progress; dump counts for each class are written when complete. The verbose and quiet options can be used to increase or decrease verbository.

out - An IO object to write to, like stderr. verbose - Whether verbose output should be enabled. quiet - Whether quiet output should be enabled.

Returns the Replicate::Status object.



34
35
36
 
# File 'lib/replicate/loader.rb', line 34

def log_to(out=$stderr, verbose=false, quiet=false)
  use Replicate::Status, 'load', out, verbose, quiet
end

#read(io) ⇒ Object

Read multiple [type, id, attrs] replicant tuples from io and call the feed method to load and filter the object.



54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/replicate/loader.rb', line 54

def read(io)
  ActiveRecord::Base.transaction do
    begin
      if ActiveRecord::Base.connection.instance_values["config"][:adapter] == "postgresql"
        ActiveRecord::Base.connection.execute("set constraints all deferred")
      end
      while object = Marshal.load(io)
        type, id, attrs = object
        feed type, id, attrs
      end
    rescue EOFError
    end
  end
end

#register_id(object, type, remote_id, local_id) ⇒ Object

Register an id in the keymap. Every object loaded must be stored here so that key references can be resolved.



122
123
124
125
126
127
128
129
 
# File 'lib/replicate/loader.rb', line 122

def register_id(object, type, remote_id, local_id)
  @keymap[type.to_s][remote_id] = local_id
  c = object.class
  while !['Object', 'ActiveRecord::Base'].include?(c.name)
    @keymap[c.name][remote_id] = local_id
    c = c.superclass
  end
end

#translate_ids(type, id, attributes) ⇒ Object

Translate remote system id references in the attributes hash to their local system id values. The attributes hash may include special id values like this:

{ 'title'         => 'hello there',
  'repository_id' => [:id, 'Repository', 1234],
  'label_ids'     => [:id, 'Label', [333, 444, 555, 666, ...]]
  ... }

These values are translated to local system ids. All object references must be loaded prior to the referencing object.



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'lib/replicate/loader.rb', line 99

def translate_ids(type, id, attributes)
  attributes.each do |key, value|
    next unless value.is_a?(Array) && value[0] == :id
    referenced_type, value = value[1].to_s, value[2]
    local_ids =
      Array(value).map do |remote_id|
        if local_id = @keymap[referenced_type][remote_id]
          local_id
        else
          warn "warn: #{referenced_type}(#{remote_id}) not in keymap, " +
               "referenced by #{type}(#{id})##{key}"
        end
      end
    if value.is_a?(Array)
      attributes[key] = local_ids
    else
      attributes[key] = local_ids[0]
    end
  end
end