Module: Smash::CloudPowers::Storage

Includes:
AwsResources, Helpers
Included in:
Smash::CloudPowers, Bucket, Local, Smash::CloudPowers::Synapse::Queue::Board
Defined in:
lib/cloud_powers/storage.rb,
lib/cloud_powers/storage/local.rb,
lib/cloud_powers/storage/bucket.rb

Defined Under Namespace

Classes: Bucket, Local

Instance Method Summary collapse

Methods included from Helpers

#create_logger, #log_file, #logger

Methods included from PathHelp

#common_delimiter, #expand_path, #file_exists?, #file_search, #filename?, #job_exist?, #job_path, #job_require_path, #path_search, #paths_gcd, #paths_lcd, #to_path, #to_pathname, #to_realpath, #touch, #zlib_path

Methods included from LogicHelp

#attr_map, #called_from, #i_var_hash, #instance_attr_accessor, #smart_retry, #update_message_body

Methods included from LangHelp

#deep_modify_keys_with, #extract!, #find_and_remove, #format_error_message, #from_json, #modify_keys_with, #to_basic_hash, #to_camel, #to_hyph, #to_i_var, #to_pascal, #to_ruby_file_name, #to_snake, #valid_json?, #valid_url?

Methods included from AwsResources

#ec2, #image, #kinesis, #queue_poller, #region, #s3, #sns, #sqs

Methods included from Zenv

#env_vars, #i_vars, #lsof_cwd, #pid, #proc_cwd, #process_search, #project_root, #project_root=, #ps_cwd, #system_vars, #zfind, #zselect

Methods included from Auth

creds, region

Instance Method Details

#all_storage(queries = []) ⇒ Object

Get all existing Storage. If there isn’t any existing Storage objects, get some delegates for the available types. If parameters are passed, narrow down the result set with the params.

Parameters

  • queries String|Symbol|Storage - a

    • String will represent a name and gets searched for using delegates. If a location is found with no Storage object that matches, it builds it/them.

    • Symbols are used to narrow down the types used in the search.

    • Storage This method also allows you to pass in Storage objects and have them passed into the list of returned Storage objects. This is useful for allowing this method to be used in other methods

Returns

  • Array of Smash::CloudPowers::Storage[::?]



37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/cloud_powers/storage.rb', line 37

def all_storage(queries = [])
  types, passed, names = sort_all_storage_params(queries)
  # combine the search efforts from above
  found = existing_storage(types) + passed
  found_names = found.map(&:name)

  # make sure all the names that were not found are represented as a new
  # Storage object, if it isn't already
  unmatched_names = names - found_names

  # if all the names are represented, return the correct results
  if unmatched_names.empty?
    (found + storage_delegates(types)).uniq
  else
    # or build ones that should be and add it to the results, then return
    # them all.
    storage_delegates(types).map do |storage|
      unlinked_storages = storage.select(*unmatched_names).reject do |n|
        filename? n
      end

      unlinked_storages.map do |unlinked_storage|
        # TODO create a method in path_help to separate.  it can be used
        # here and instead of doing it in to_snake, to_snake can use it
        # too
        clues = unlinked_storage.to_s.split(%r"#{common_delimiter}")

        config = {
          name:         clues.pop,
          type:         storage.type,
          origin:       clues.join(common_delimiter)
        }

        config = storage_config(storage.type).merge(config)
        build_storage(**config)
      end
    end.flatten
  end
end

#build_storage(name:, type: nil, **config) {|storage_resource| ... } ⇒ Object

This method builds a CloudPowers::Storage:: object for you to use but doesn’t invoke the #create!() method, so no API call is made, no directories are created, etc. This can be used even if the storage already exists.

Parameters

  • name String - name of the storage you want to interact with

Returns Storage::

Example

storage_object = build_queue('exampleQueue')
storage_object.select('job')
=> job.rb

Yields:

  • (storage_resource) 


176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
 
# File 'lib/cloud_powers/storage.rb', line 176

def build_storage(name:, type: nil, **config)
  type = to_pascal(type || :local)

  storage_resource = Smash::CloudPowers::Storage.const_get(type).
    build(name: to_camel(name), type: (type || :storage), **config)

  yield storage_resource if block_given?

  attr_map(storage_resource.call_name => storage_resource) do |attribute, resource|
    instance_attr_accessor attribute
    resource
  end

  storage_resource
end

#create_storage(name:, type: nil, **config) ⇒ Object

Create a storage without explicitly creating an object, of whatever type we’re working with, here. The types can be of any CloudPowers::Storage or any other class, as long as it follows the CloudPowers::Createable Interface

Parameters

  • name String - The name of the Queue to be created

Returns CloudPowers::Storage::

Example

create_queue('exampleQueue')
get_queue_message_count



207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
 
# File 'lib/cloud_powers/storage.rb', line 207

def create_storage(name:, type: nil, **config)
  # default is :local
  type, config[:type] = type.nil? ? [:local, :storage] : [type, type]

  storage_resource =
    Smash::CloudPowers::Storage.const_get(to_pascal(type)).create!(
      name: to_camel(name), **config
    )

  attr_map(storage_resource.call_name => storage_resource) do |attribute, resource|
    instance_attr_accessor attribute
    resource
  end

  storage_resource
end

#existing_storage(types = all_storage_types) ⇒ Object 



13
14
15
16
17
18
 
# File 'lib/cloud_powers/storage.rb', line 13

def existing_storage(types = all_storage_types)
  normalized_types = types.map { |type| to_pascal(type) }
  i_vars.select do |k,v|
    normalized_types.include?(to_pascal(v.type)) rescue nil
  end.values
end

#local_job_file_exists?(file) ⇒ Boolean

Returns:

  • (Boolean) 


77
78
79
80
 
# File 'lib/cloud_powers/storage.rb', line 77

def local_job_file_exists?(file)
  path = job_path(to_ruby_file_name(file))
  storage_select(file, location: [:local, path]).count > 0
end

#search(bucket, pattern) ⇒ Object

Search through a bucket to find a file, based on a regex

Parameters

  • bucket String - the bucket to search through in AWS

  • pattern Regexp|nil - the Regex pattern you want to use for the search. nil doesn’t cause an exception but probably returns []

Example

matches = search('neuronJobs', /[Dd]emo\w*/)
# => ['Aws::S3::Type::ListObjectOutPut','Aws::S3::Type::ListObjectOutPut',...] # anything that matched that regex
matches.first.contents.size
# => 238934 # integer representation of the file size



145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/cloud_powers/storage.rb', line 145

def search(bucket, pattern)
  # this guard allows methods that return nil to be used as the 'bucket',
  # like #zfind()
  return [] if bucket.nil?

  begin
    pattern = /#{pattern}/ unless pattern.kind_of? Regexp
    s3.list_objects(bucket: bucket).contents.select do |o|
      o.key =~ pattern || /#{o.key}/ =~ pattern.to_s
    end
  rescue Aws::S3::Errors::NoSuchBucket => e
    logger.info format_error_message e
    return # log that the bucket doesn't exist but don't explode
  end
end

#send_logs_to_s3 ⇒ Object

Send the log files to the S3 log file bucket

Returns Aws::S3::Type::PutObjectOutput



228
229
230
231
232
233
234
235
236
 
# File 'lib/cloud_powers/storage.rb', line 228

def send_logs_to_s3
  File.open(log_file) do |file|
    s3.put_object(
      bucket: log_bucket,
      key: instance_id,
      body: file
    )
  end
end

#source_job(file) ⇒ Object

Searches a local jobs storage location for the given file name if it exists - exit the method if it does not exist - get the file from s3 and place it in the directory that was just searched bucket using #zfind()

Parameters

  • file String - the name of the file we’re searching for

Returns nothing

Example

  • file tree

    project_root- |
              |_sub_directory
              |              |_current_file.rb
              |_job_storage
              |             |_demorific.rb
              |             |_foobar.rb

  • code:

    source_job('demorific')
# file tree doesn't change because this file exists
source_job('custom_greetings')
# file tree now looks like this

  • new file tree

    project_root- |
              |_sub_directory
              |              |_current_file.rb
              |_job_storage
              |             |_demorific.rb
              |             |_foobar.rb
              |             |_custom_greetings.js # could be an after effects JS script



114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/cloud_powers/storage.rb', line 114

def source_job(file)
  storage_delegates(:bucket).map do |delegate|
    delegate.find(file, location: zfind('jobs storage')) do |f, data|
      f.write data
    end
  end.flatten
  # TODO: better path management
  # bucket = zfind('jobs storage')
  # if job_path(to_ruby_file_name(file)).nil?
  #   objects = s3.list_objects(bucket: bucket).contents.select do |f|
  #     /#{Regexp.escape file}/i =~ f.key
  #   end

  #   objects.each do |obj|
  #     s3.get_object(bucket: bucket, key: obj.key, response_target: job_path(file))
  #   end
  # end
end

#storage_select(*names, **opts) ⇒ Object

Search through all Storage or just the ones that fit your search parameters. The searching algorithm takes you through existing Storage objects and the locations they represent, like local the file system and AWS S3 Buckets

Parameters

  • list of String - the name(s) of the file, direcotry, bucket, etc you are looking for

  • list of KeyValue pairs - search options that are used for narrowing down your objects

    • :location :local|:bucket|String

Returns

  • Array Objects - can be the return value of whichever Storage object

found something that matched the param(s)

Notes:

  • Narrowing down the search results speeds up the search

  • If a block is passed to this method, it will be run in the #select method of the current Storage type

  • See the #select methods for all the Storage classes you are using



260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
 
# File 'lib/cloud_powers/storage.rb', line 260

def storage_select(*names, **opts)
  # make sure to process all possible types of names that can come in.
  # symbols are reserved for types, not names.
  locations = [opts.delete(:location) || opts.delete(:locations)].flatten
  locations = (locations + extract!(names) { |n| n.kind_of? Symbol }).uniq

  all_storage(locations).map do |storage|
    if block_given?
      storage.select(*names, **opts) do |object|
        yield object
      end
    else
      storage.select(*names, **opts)
    end
  end.flatten
end