Class: RemoteHadoopGem

Inherits:
Object
  • Object
show all
Defined in:
lib/remoteHadoopGem.rb

Class Method Summary collapse

Class Method Details

.cleanTempFiles(username, host, keyPathFile) ⇒ Object

****** cleanTempFiles ******

function to clean all the temporary files created by commandToHadoopJobID require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine)



114
115
116
 
# File 'lib/remoteHadoopGem.rb', line 114

def self.cleanTempFiles(username, host, keyPathFile)
  `ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; rm ./tempJobID_* "`
end

.commandToHadoop(username, host, keyPathFile, command) ⇒ Object

****** commandToHadoop ******

function to run a command in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) command to execute



15
16
17
18
 
# File 'lib/remoteHadoopGem.rb', line 15

def self.commandToHadoop(username, host, keyPathFile, command)
  res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; #{command}"`
  return "#{res}"
end

.commandToHadoopJobID(username, host, keyPathFile, command) ⇒ Object

****** commandToHadoopJobID ******

function to run a command in a remote hadoop machine; returns the <job-id> require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) command to execute



30
31
32
33
34
35
36
37
38
39
40
 
# File 'lib/remoteHadoopGem.rb', line 30

def self.commandToHadoopJobID(username, host, keyPathFile, command)
  # create a file with a random name to obtained the jobID
  filename="tempJobID_"+UUIDTools::UUID.random_create.to_s
  #`ssh -n -f -i "#{keyPathFile}" "#{username}"@"#{host}" 'sh -c "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; nohup #{command} >/dev/null 2>#{filename}.txt & " '`
  `ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" 'sh -c "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; nohup #{command} >/dev/null 2>#{filename}.txt & " '`
  res=`sleep 10; ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "cat /hadoop/#{filename}.txt | grep 'Running job' " `
  `ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "rm /hadoop/#{filename}.txt"`
  jobid=res.split[6]
  
  return "#{jobid}"
end

.copyFileTo(username, host, keyPathFile, filePath, destFilePath) ⇒ Object

****** copyFileTo ******

function to copy a file in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) filePath (path of the local file) 5) destFilePath (path on the remote hadoop node)



53
54
55
 
# File 'lib/remoteHadoopGem.rb', line 53

def self.copyFileTo(username, host, keyPathFile, filePath, destFilePath)
  res=`scp -r -i "#{keyPathFile}" -o StrictHostKeyChecking=no #{filePath}  "#{username}"@"#{host}":#{destFilePath}`
end

.jobsList(username, host, keyPathFile) ⇒ Object

****** jobsList ******

function to list the running jobs in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine)



83
84
85
86
 
# File 'lib/remoteHadoopGem.rb', line 83

def self.jobsList(username, host, keyPathFile)
  res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; hadoop job -list "`
  return "#{res}"
end

.jobStatus(username, host, keyPathFile, job_id) ⇒ Object

****** jobStatus ******

function to read the status of a running job in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) job_id (id of the job)



99
100
101
102
 
# File 'lib/remoteHadoopGem.rb', line 99

def self.jobStatus(username, host, keyPathFile, job_id)
  res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; hadoop job -status #{job_id} "`
  return "#{res}"
end

.readFileFrom(username, host, keyPathFile, destFilePath) ⇒ Object

****** readFileFrom ******

function to read a file from a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) destFilePath (path of the remote file to read)



68
69
70
71
 
# File 'lib/remoteHadoopGem.rb', line 68

def self.readFileFrom(username, host, keyPathFile, destFilePath)
  res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "cat #{destFilePath}"`
  return "#{res}"
end