Class: MapReduce

Inherits:
Object
  • Object
show all
Defined in:
lib/learn_mapreduce.rb

Overview

Small Framwork to learn MapReduce

Instance Method Summary collapse

Constructor Details

#initialize ⇒ MapReduce

Returns a new instance of MapReduce.



3
4
5
6
 
# File 'lib/learn_mapreduce.rb', line 3

def initialize
   @mapper = nil
   @reducer = nil
end

Instance Method Details

#map(&mapper) ⇒ Object

Set the mapper function

Example:

mr = MapReduce.new
mr.map do |record|
  ...
end

Arguments:

mapper: (block)



18
19
20
 
# File 'lib/learn_mapreduce.rb', line 18

def map &mapper
   @mapper = mapper
end

#reduce(&reducer) ⇒ Object

Set the reducer function

Example:

mr = MapReduce.new
mr.reduce do |key, list_of_values|
  ...
end

Arguments:

reducer: (block)



32
33
34
 
# File 'lib/learn_mapreduce.rb', line 32

def reduce &reducer
   @reducer = reducer
end

#run(data) ⇒ Object

Run a MapReduce algorithm on the data provided

Example:

mr = MapReduce.new
mr.run(data)

Arguments:

data: (array)

Returns:

[emitted, shuffled, output]



47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/learn_mapreduce.rb', line 47

def run data
   throw "Mapper missing!" unless @mapper
   throw "Reducer missing!" unless @reducer
   
   map_context = MapContext.new
   data.each { |record|
      map_context.instance_exec(record, &@mapper)
   }

   reduce_context = ReduceContext.new
   map_context.shuffled.each { |key, list_of_values|
      reduce_context.instance_exec(key, list_of_values, &@reducer)
   }

   return [
      map_context.emitted,
      map_context.shuffled,
      reduce_context.output
   ]
end