Module: IMW::Schemes::HDFSDirectory

Defined in:
lib/imw/schemes/hdfs.rb

Overview

Defines methods for listing contents of HDFS directories.

Instance Method Summary collapse

Instance Method Details

#contents ⇒ Array<String>

Return the paths of all files and directories directly below this directory on the HDFS.

Returns:



221
222
223
224
225
226
227
228
 
# File 'lib/imw/schemes/hdfs.rb', line 221

def contents
  returning([]) do |paths|
    HDFS.fs(:ls, path) do |line|
      next if line =~ /^Found.*items$/
      paths << line.split.last
    end
  end
end

#join(*paths) ⇒ IMW::Resource

Return the resource at the base path of this resource joined to path.

IMW.open('hdfs:///path/to/dir').join('subdir')
#=> IMW::Resource at 'hdfs:///path/to/dir/subdir'

Parameters:

Returns:



246
247
248
 
# File 'lib/imw/schemes/hdfs.rb', line 246

def join *paths
  IMW.open(File.join(stripped_uri.to_s, *paths))
end

#resources ⇒ Array<IMW::Resource>

Return the resources directly below this directory on the HDFS.

Returns:



234
235
236
 
# File 'lib/imw/schemes/hdfs.rb', line 234

def resources
  contents.map { |path| IMW.open(path) }
end