Class: Skynet::Partitioners::ArrayDataSplitByFirstEntry

Inherits:
Skynet::Partitioners show all
Defined in:
lib/skynet/skynet_partitioners.rb

Class Method Summary collapse

Methods included from SkynetDebugger

#args_pp, #debug, #debug_header, #error, #fatal, included, #info, #log, #printlog, #stderr, #stdout, #warn

Class Method Details

.reduce_partition(post_map_data, new_partitions) ⇒ Object

Smarter partitioner for array data, generates simple sum of array and ensures that all arrays sharing that key go into the same partition.



73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/skynet/skynet_partitioners.rb', line 73

def self.reduce_partition(post_map_data, new_partitions)
  partitions = []
  (0..new_partitions - 1).each { |i| partitions[i] = Array.new }
  cnt = 0
  post_map_data.each do |partition|
    partition.each do |array|            
      next unless array.is_a?(Array) and array.size >= 2
      if array[0].kind_of?(Fixnum)
        key = array[0] % new_partitions
      elsif array[0].kind_of?(String)
        key = array[0].sum % new_partitions
      else                 
        cnt += 1
        key = cnt % new_partitions
      end
      partitions[key] << array
    end
  end
  partitions
end