Module: Wukong::Hadoop::LocalInvocation

Included in:
HadoopRunner
Defined in:
lib/wukong-hadoop/runner/local_invocation.rb

Overview

Provides methods for executing a map/reduce job locally on the command-line.

Instance Method Summary collapse

Instance Method Details

#cat_input ⇒ Object

:nodoc:



34
35
36
37
38
 
# File 'lib/wukong-hadoop/runner/local_invocation.rb', line 34

def cat_input
  return unless input_paths && (!input_paths.empty?)
  paths = Shellwords.join(input_paths.split(','))
  "cat #{paths}"
end

#cat_output ⇒ Object

:nodoc:



41
42
43
44
 
# File 'lib/wukong-hadoop/runner/local_invocation.rb', line 41

def cat_output
  return unless output_path
  "> #{output_path}"
end

#local_commandline ⇒ String

Returns the full local command used by Wukong-Hadoop when simulating a map/reduce job on the command-line.

You should be able to run this commmand directly to simulate the job yourself.

Returns:

  • (String) 


16
17
18
19
20
21
22
23
 
# File 'lib/wukong-hadoop/runner/local_invocation.rb', line 16

def local_commandline
  [
   [cat_input, mapper_commandline].tap do |pipeline|
     pipeline.concat([sort_commandline, reducer_commandline]) if reduce?
   end.flatten.compact.join(' | '),
   cat_output
  ].flatten.compact.join(' ')
end

#sort_commandline ⇒ String

Returns the sort command used by Wukong-Hadoop when simulating a map/reduce job on the command-line.

Returns:

  • (String) 


29
30
31
 
# File 'lib/wukong-hadoop/runner/local_invocation.rb', line 29

def sort_commandline
  settings[:sort_command]
end