Class: MongoMapperParallel

Inherits:
Object
  • Object
show all
Defined in:
lib/mongo_mapper_parallel.rb

Defined Under Namespace

Classes: Key, ProgressError

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(opts = {}) ⇒ MongoMapperParallel

Instantiates the parallel operation object with the right class, javascript function, and field

Options Hash (opts):

  • :class (Class) —

    the Mongo collection’s Ruby Class to execute operations on.

  • :javascript (String) —

    the Javascript function in String format

  • :args (Array, Hash) —

    the arguments to pass to the Javascript function

  • :split (String, Symbol) —

    the field to split the computation on – typically an indexed unique property of the resources in the collection.

  • :maxChunkSizeBytes (Fixnum) —

    the size of the chunks to parallelize. Defaults to ‘32*1024*1024 = 33554432`.



88
89
90
91
92
93
94
95
96
 
# File 'lib/mongo_mapper_parallel.rb', line 88

def initialize(opts={})
  @command_class = opts[:class]
  @javascript    = opts[:javascript]
  @args          = opts[:args]
  @split         = opts[:split] # name, title, etc...
  @splitSize     = opts[:maxChunkSizeBytes] || 32*1024*1024
  get_split_keys()
  self
end

Instance Attribute Details

#args ⇒ Object

Returns the value of attribute args.



13
14
15
 
# File 'lib/mongo_mapper_parallel.rb', line 13

def args
  @args
end

#command_class ⇒ Object

Returns the value of attribute command_class.



11
12
13
 
# File 'lib/mongo_mapper_parallel.rb', line 11

def command_class
  @command_class
end

#javascript ⇒ Object

Returns the value of attribute javascript.



12
13
14
 
# File 'lib/mongo_mapper_parallel.rb', line 12

def javascript
  @javascript
end

#split_keys ⇒ Object (readonly)

runs distributed computation over a Mongo collection



10
11
12
 
# File 'lib/mongo_mapper_parallel.rb', line 10

def split_keys
  @split_keys
end

Instance Method Details

#advance(percentage) ⇒ MongoMapperParallel

In case of stalled progress you can skip ahead by a percentage and mark the keys as completed.



116
117
118
119
120
121
122
123
124
 
# File 'lib/mongo_mapper_parallel.rb', line 116

def advance percentage
  if percentage.class != Float
    raise TypeError.new "Can only advance by a Float value."
  elsif percentage > 1.0 or percentage < 0.0
    raise ProgressError.new "Can only advance by a Float between 0.0 and 1.0."
  end
  @split_keys[0..(@split_keys.length*percentage).to_i].each {|i| i.completed = true}
  self
end

#get_split_keys ⇒ Array<MongoMapperParallel::Key>

Obtains the splitVectors keys to find chunks to parallelize via the MongoDB splitVector command.



71
72
73
74
75
76
 
# File 'lib/mongo_mapper_parallel.rb', line 71

def get_split_keys
  @split_keys, splits = [], @command_class.database.command({splitVector: "#{@command_class.database.name}.#{@command_class.collection.name}", keyPattern: {@split.to_sym => 1}, maxChunkSizeBytes: @splitSize })["splitKeys"]
  splits.each_with_index do |split_key,k|
    @split_keys << MongoMapperParallel::Key.new(:compiler => self, :key => split_key[@split.to_s], :future_key => (splits[k+1] ? splits[k+1][@split.to_s] : nil))
  end
end

#run ⇒ Object

Starts the parallel processing using Parallel.



100
101
102
103
104
105
106
107
 
# File 'lib/mongo_mapper_parallel.rb', line 100

def run
  total = @split_keys.length
  Parallel.each_with_index(@split_keys) do |section,k|
    if !section.completed then section.compile end
    JRProgressBar.show(k,total)
  end
  puts "Success".green
end