Class: RemoteHadoopGem
- Inherits:
-
Object
- Object
- RemoteHadoopGem
- Defined in:
- lib/remoteHadoopGem.rb
Class Method Summary collapse
- .cleanTempFiles(username, host, keyPathFile) ⇒ Object
-
.commandToHadoop(username, host, keyPathFile, command) ⇒ Object
****** commandToHadoop ****** .
- .commandToHadoopJobID(username, host, keyPathFile, command) ⇒ Object
-
.copyFileTo(username, host, keyPathFile, filePath, destFilePath) ⇒ Object
****** copyFileTo ****** .
-
.jobsList(username, host, keyPathFile) ⇒ Object
****** jobsList ****** .
-
.jobStatus(username, host, keyPathFile, job_id) ⇒ Object
****** jobStatus ****** .
-
.readFileFrom(username, host, keyPathFile, destFilePath) ⇒ Object
****** readFileFrom ****** .
-
.tryOne ⇒ Object
****** commandToHadoopJobID ****** .
Class Method Details
.cleanTempFiles(username, host, keyPathFile) ⇒ Object
107 108 109 |
# File 'lib/remoteHadoopGem.rb', line 107 def self.cleanTempFiles(username, host, keyPathFile) `ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; rm ./tempJobID_* "` end |
.commandToHadoop(username, host, keyPathFile, command) ⇒ Object
****** commandToHadoop ******
function to run a command in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) command to execute
15 16 17 18 |
# File 'lib/remoteHadoopGem.rb', line 15 def self.commandToHadoop(username, host, keyPathFile, command) res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; #{command}"` return "#{res}" end |
.commandToHadoopJobID(username, host, keyPathFile, command) ⇒ Object
32 33 34 35 36 37 38 39 40 41 42 |
# File 'lib/remoteHadoopGem.rb', line 32 def self.commandToHadoopJobID(username, host, keyPathFile, command) # create a file with a random name to obtained the jobID filename="tempJobID_"+UUIDTools::UUID.random_create.to_s #`ssh -n -f -i "#{keyPathFile}" "#{username}"@"#{host}" 'sh -c "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; nohup #{command} >/dev/null 2>#{filename}.txt & " '` `ssh -i "#{keyPathFile}" "#{username}"@"#{host}" 'sh -c "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; nohup #{command} >/dev/null 2>#{filename}.txt & " '` res=`sleep 10; ssh -i "#{keyPathFile}" "#{username}"@"#{host}" "cat /hadoop/#{filename}.txt | grep 'Running job' " ` `ssh -i "#{keyPathFile}" "#{username}"@"#{host}" "rm /hadoop/#{filename}.txt"` jobid=res.split[6] return "#{jobid}" end |
.copyFileTo(username, host, keyPathFile, filePath, destFilePath) ⇒ Object
****** copyFileTo ******
function to copy a file in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) filePath (path of the local file) 5) destFilePath (path on the remote hadoop node)
55 56 57 |
# File 'lib/remoteHadoopGem.rb', line 55 def self.copyFileTo(username, host, keyPathFile, filePath, destFilePath) res=`scp -r -i "#{keyPathFile}" -o StrictHostKeyChecking=no #{filePath} "#{username}"@"#{host}":#{destFilePath}` end |
.jobsList(username, host, keyPathFile) ⇒ Object
****** jobsList ******
function to list the running jobs in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine)
85 86 87 88 |
# File 'lib/remoteHadoopGem.rb', line 85 def self.jobsList(username, host, keyPathFile) res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; hadoop job -list "` return "#{res}" end |
.jobStatus(username, host, keyPathFile, job_id) ⇒ Object
****** jobStatus ******
function to read the status of a running job in a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) job_id (id of the job)
101 102 103 104 |
# File 'lib/remoteHadoopGem.rb', line 101 def self.jobStatus(username, host, keyPathFile, job_id) res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "export PATH=/hadoop/bin/:$PATH; cd /hadoop/; hadoop job -status #{job_id} "` return "#{res}" end |
.readFileFrom(username, host, keyPathFile, destFilePath) ⇒ Object
****** readFileFrom ******
function to read a file from a remote hadoop machine require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) destFilePath (path of the remote file to read)
70 71 72 73 |
# File 'lib/remoteHadoopGem.rb', line 70 def self.readFileFrom(username, host, keyPathFile, destFilePath) res=`ssh -i "#{keyPathFile}" -o StrictHostKeyChecking=no "#{username}"@"#{host}" "cat #{destFilePath}"` return "#{res}" end |
.tryOne ⇒ Object
****** commandToHadoopJobID ******
function to run a command in a remote hadoop machine; returns the <job-id> require: 1) username (to access the machine via ssh) 2) host (IP of the hadoop machine) 3) keyPathFile (path of the private key to access the machine) 4) command to execute
29 30 31 |
# File 'lib/remoteHadoopGem.rb', line 29 def self.tryOne() return "yO!" end |