Class: MapReduce

Inherits:
Object
  • Object
show all
Defined in:
lib/learn_mapreduce.rb

Overview

Small Framwork to learn MapReduce

Instance Method Summary collapse

Constructor Details

#initialize ⇒ MapReduce

Returns a new instance of MapReduce.



3
4
5
6
 
# File 'lib/learn_mapreduce.rb', line 3

def initialize
   @mapper = nil
   @reducer = nil
end

Instance Method Details

#map(&mapper) ⇒ Object

Call this function with a block to set the mapper function The mapper takes a record as argument



10
11
12
 
# File 'lib/learn_mapreduce.rb', line 10

def map &mapper
   @mapper = mapper
end

#reduce(&reducer) ⇒ Object

Call this function with a block to set the reducer function The reducer takes a key and a list of values as arguments



16
17
18
 
# File 'lib/learn_mapreduce.rb', line 16

def reduce &reducer
   @reducer = reducer
end

#run(data) ⇒ Object

Run a MapReduce algorithm on the data provided Returns a hash containing the emitted tuples, the shuffled tuples and the output



22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/learn_mapreduce.rb', line 22

def run data
   throw "Mapper missing!" unless @mapper
   throw "Reducer missing!" unless @reducer
   
   map_context = MapContext.new
   data.each { |record|
      map_context.instance_exec(record, &@mapper)
   }

   reduce_context = ReduceContext.new
   map_context.shuffled.each { |key, list_of_values|
      reduce_context.instance_exec(key, list_of_values, &@reducer)
   }

   return [
      map_context.emitted,
      map_context.shuffled,
      reduce_context.output
   ]
end