Module: WebHDFS::FileUtils
- Defined in:
- lib/webhdfs/fileutils.rb
Class Method Summary collapse
-
.append(path, body, options = {}) ⇒ Object
Public: Append to HDFS file.
-
.chmod(mode, list, options = {}) ⇒ Object
Public: Change permission of one or more directories/files.
-
.chown(user, group, list, options = {}) ⇒ Object
Public: Change an ownership of one or more directories/files.
-
.copy_from_local(file, path, options = {}) ⇒ Object
Public: Copy local file into HDFS.
-
.copy_to_local(path, file, options = {}) ⇒ Object
Public: Copy remote HDFS file into local.
-
.mkdir(list, options = {}) ⇒ Object
Public: Create one or more directories.
-
.mkdir_p ⇒ Object
Public: Create one or more directories.
-
.private_module_function(name) ⇒ Object
Internal: make functin private.
-
.rename(src, dst, options = {}) ⇒ Object
Public: Rename a file or directory.
-
.rm(list, options = {}) ⇒ Object
Public: Remove one or more directories or files.
-
.rmr(list, options = {}) ⇒ Object
Public: Remove one or more directories/files recursively.
-
.set_atime(list, time, options = {}) ⇒ Object
Public: Set an access time of files.
-
.set_httpfs_mode(mode = true) ⇒ Object
Public: Set httpfs mode enable/disable.
-
.set_mtime(list, time, options = {}) ⇒ Object
Public: Set a modification time of files.
-
.set_repl_factor(list, num, options = {}) ⇒ Object
Public: Set a replication factor of files.
-
.set_server(host, port, user = nil, doas = nil) ⇒ Object
Public: Set hostname and port number of WebHDFS.
Instance Method Summary collapse
-
#client ⇒ Object
Internal.
-
#fu_log(msg) ⇒ Object
Internal: Logging.
Class Method Details
.append(path, body, options = {}) ⇒ Object
Public: Append to HDFS file
path - HDFS file path body - contents options - :buffersize, :verbose
Examples
FileUtils.append 'remote_path', 'contents'
102 103 104 105 106 |
# File 'lib/webhdfs/fileutils.rb', line 102 def append(path, body, ={}) opts = .dup fu_log "append #{body.bytesize} bytes to #{path}" if opts.delete(:verbose) client.append(path, body, opts) end |
.chmod(mode, list, options = {}) ⇒ Object
216 217 218 219 220 221 222 223 224 225 |
# File 'lib/webhdfs/fileutils.rb', line 216 def chmod(mode, list, ={}) opts = .dup list = [list].flatten fu_log sprintf('chmod %o %s', mode, list.join(' ')) if opts.delete(:verbose) mode = ('%03o' % mode) if mode.is_a? Integer c = client list.each { |entry| c.chmod(entry, mode, opts) } end |
.chown(user, group, list, options = {}) ⇒ Object
240 241 242 243 244 245 246 247 248 249 250 |
# File 'lib/webhdfs/fileutils.rb', line 240 def chown(user, group, list, ={}) opts = .dup list = [list].flatten fu_log sprintf('chown %s%s', [user,group].compact.join(':') + ' ', list.join(' ')) if opts.delete(:verbose) c = client list.each { |entry| c.chown(entry, {:owner => user, :group => group}) } end |
.copy_from_local(file, path, options = {}) ⇒ Object
Public: Copy local file into HDFS
file - local file path path - HDFS file path options - :overwrite, :blocksize, :replication, :mode, :buffersize, :verbose
Examples
FileUtils.copy_from_local 'local_file', 'remote_file'
57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
# File 'lib/webhdfs/fileutils.rb', line 57 def copy_from_local(file, path, ={}) opts = .dup fu_log "copy_from_local local=#{file} hdfs=#{path}" if opts.delete(:verbose) if mode = opts.delete(:mode) mode = ('%03o' % mode) if mode.is_a? Integer else mode = '644' end opts[:permission] = mode opts[:overwrite] ||= true client.create(path, File.new(file, 'rb').read(File.size(file)), opts) end |
.copy_to_local(path, file, options = {}) ⇒ Object
Public: Copy remote HDFS file into local
path - HDFS file path file - local file path options - :offset, :length, :buffersize, :verbose
Examples
FileUtils.copy_to_local 'remote_file', 'local_file'
83 84 85 86 87 88 89 |
# File 'lib/webhdfs/fileutils.rb', line 83 def copy_to_local(path, file, ={}) opts = .dup fu_log "copy_to_local hdfs=#{path} local=#{file}" if opts.delete(:verbose) File.open(file, "wb") do |f| f.write client.read(path, opts) end end |
.mkdir(list, options = {}) ⇒ Object
120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
# File 'lib/webhdfs/fileutils.rb', line 120 def mkdir(list, ={}) opts = .dup list = [list].flatten fu_log "mkdir #{[:mode] ? ('-m %03o ' % [:mode]) : ''}#{list.join ' '}" if opts.delete(:verbose) if mode = opts[:mode] mode = ('0%03o' % mode) if mode.is_a? Integer else mode = '0755' end c = client list.each { |dir| c.mkdir(dir, {:permission => mode}) } end |
.mkdir_p ⇒ Object
Public: Create one or more directories.
list - directory name, or list of them options - :mode, :verbose
Examples
FileUtils.mkdir 'test'
FileUtils.mkdir %w( tmp data )
FileUtils.mkdir 'tmp', :mode => 0700
Public: Create one or more directories recursively.
list - directory name, or list of them options - :mode, :verbose
Examples
FileUtils.mkdir_p 'dir/subdir'
FileUtils.mkdir_p %w( tmp data )
FileUtils.mkdir_p 'dir/subdir', :mode => 0700
147 148 149 150 151 152 153 154 155 156 157 158 159 160 |
# File 'lib/webhdfs/fileutils.rb', line 147 def mkdir(list, ={}) opts = .dup list = [list].flatten fu_log "mkdir #{[:mode] ? ('-m %03o ' % [:mode]) : ''}#{list.join ' '}" if opts.delete(:verbose) if mode = opts[:mode] mode = ('0%03o' % mode) if mode.is_a? Integer else mode = '0755' end c = client list.each { |dir| c.mkdir(dir, {:permission => mode}) } end |
.private_module_function(name) ⇒ Object
Internal: make functin private
320 321 322 323 |
# File 'lib/webhdfs/fileutils.rb', line 320 def self.private_module_function(name) module_function name private_class_method name end |
.rename(src, dst, options = {}) ⇒ Object
Public: Rename a file or directory.
src - from dst - to options - :verbose
Examples
FileUtils.rename 'from', 'to'
198 199 200 201 202 |
# File 'lib/webhdfs/fileutils.rb', line 198 def rename(src, dst, ={}) opts = .dup fu_log "rename #{src} #{dst}" if opts.delete(:verbose) client.rename(src, dst, opts) end |
.rm(list, options = {}) ⇒ Object
161 162 163 164 165 166 167 168 169 |
# File 'lib/webhdfs/fileutils.rb', line 161 def rm(list, ={}) opts = .dup list = [list].flatten fu_log "rm #{list.join ' '}" if opts.delete(:verbose) c = client list.each { |dir| c.delete(dir, {:recursive => opts[:recursive] || false}) } end |
.rmr(list, options = {}) ⇒ Object
183 184 185 |
# File 'lib/webhdfs/fileutils.rb', line 183 def rmr(list, ={}) self.rm(list, .merge({:recursive => true})) end |
.set_atime(list, time, options = {}) ⇒ Object
Public: Set an access time of files
list - file/directory name or list of them time - new access time options - :verbose
Examples
FileUtils.set_atime 'file', Time.now
285 286 287 288 289 290 291 292 293 294 |
# File 'lib/webhdfs/fileutils.rb', line 285 def set_atime(list, time, ={}) opts = .dup list = [list].flatten time = time.to_i fu_log sprintf('set_atime %s %d', list.join(' '), time) if opts.delete(:verbose) c = client list.each { |entry| c.touch(entry, {:accesstime => time}) } end |
.set_httpfs_mode(mode = true) ⇒ Object
Public: Set httpfs mode enable/disable
mode - boolean (default true)
Examples
FileUtils.set_httpfs_mode
42 43 44 |
# File 'lib/webhdfs/fileutils.rb', line 42 def set_httpfs_mode(mode=true) @fu_httpfs_mode = mode end |
.set_mtime(list, time, options = {}) ⇒ Object
Public: Set a modification time of files
list - file/directory name or list of them time - new modification time options - :verbose
Examples
FileUtils.set_mtime 'file', Time.now
307 308 309 310 311 312 313 314 315 316 |
# File 'lib/webhdfs/fileutils.rb', line 307 def set_mtime(list, time, ={}) opts = .dup list = [list].flatten time = time.to_i fu_log sprintf('set_mtime %s %d', list.join(' '), time) if opts.delete(:verbose) c = client list.each { |entry| c.touch(entry, {:modificationtime => time}) } end |
.set_repl_factor(list, num, options = {}) ⇒ Object
Public: Set a replication factor of files
list - file/directory name or list of them num - replication factor options - :verbose
Examples
FileUtils.set_repl_factor 'file', 3
263 264 265 266 267 268 269 270 271 272 |
# File 'lib/webhdfs/fileutils.rb', line 263 def set_repl_factor(list, num, ={}) opts = .dup list = [list].flatten fu_log sprintf('set_repl_factor %s %d', list.join(' '), num) if opts.delete(:verbose) c = client list.each { |entry| c.replication(entry, num, opts) } end |
.set_server(host, port, user = nil, doas = nil) ⇒ Object
Public: Set hostname and port number of WebHDFS
host - hostname port - port user - username doas - proxy user name
Examples
FileUtils.set_server 'localhost', 50070
26 27 28 29 30 31 |
# File 'lib/webhdfs/fileutils.rb', line 26 def set_server(host, port, user=nil, doas=nil) @fu_host = host @fu_port = port @fu_user = user @fu_doas = doas end |
Instance Method Details
#client ⇒ Object
Internal
336 337 338 339 340 341 342 343 344 345 346 347 |
# File 'lib/webhdfs/fileutils.rb', line 336 def client client = WebHDFS::Client.new(@fu_host, @fu_port, @fu_user, @fu_doas) if @fu_httpfs_mode client.httpfs_mode = true end if @fu_auth_type == :kerberos client.auth_type = @fu_auth_type client.keytab = @fu_keytab client.pass = @fu_pass end client end |
#fu_log(msg) ⇒ Object
Internal: Logging
328 329 330 331 332 |
# File 'lib/webhdfs/fileutils.rb', line 328 def fu_log(msg) @fileutils_output ||= $stderr @fileutils_label ||= '' @fileutils_output.puts @fileutils_label + msg end |