Module: Smash::CloudPowers::Storage

Includes:
AwsResources
Included in:
Smash::CloudPowers
Defined in:
lib/cloud_powers/storage.rb

Instance Method Summary collapse

Methods included from AwsResources

#ec2, #image, #kinesis, #queue_poller, #region, #s3, #sns, #sqs

Methods included from Zenv

#env_vars, #file_tree_search, #i_vars, #project_root, #project_root=, #system_vars, #zfind

Methods included from Helpers

#create_logger, #log_file, #logger

Methods included from PathHelp

#job_exist?, #job_home, #job_path, #job_require_path

Methods included from LogicHelp

#attr_map, #called_from, #instance_attr_accessor, #smart_retry, #update_message_body

Methods included from LangHelp

#deep_modify_keys_with, #find_and_remove, #format_error_message, #from_json, #modify_keys_with, #to_basic_hash, #to_camel, #to_hyph, #to_i_var, #to_pascal, #to_ruby_file_name, #to_snake, #valid_json?, #valid_url?

Methods included from Auth

creds, region

Instance Method Details

#local_job_file_exists?(file) ⇒ Boolean

Returns:

  • (Boolean) 


9
10
11
 
# File 'lib/cloud_powers/storage.rb', line 9

def local_job_file_exists?(file)
  File.exists?(job_path(to_ruby_file_name(file)))
end

#search(bucket, pattern) ⇒ Object

Search through a bucket to find a file, based on a regex

Parameters

  • bucket String - the bucket to search through in AWS

  • pattern Regexp|nil - the Regex pattern you want to use for the search. nil doesn’t cause an exception but probably returns []

Example

matches = search('neuronJobs', /[Dd]emo\w*/)
# => ['Aws::S3::Type::ListObjectOutPut','Aws::S3::Type::ListObjectOutPut',...] # anything that matched that regex
matches.first.contents.size
# => 238934 # integer representation of the file size



71
72
73
74
75
76
77
78
79
80
81
82
83
 
# File 'lib/cloud_powers/storage.rb', line 71

def search(bucket, pattern)
  return [] if bucket.nil?

  begin
    pattern = /#{pattern}/ unless pattern.kind_of? Regexp
    s3.list_objects(bucket: bucket).contents.select do |o|
      o.key =~ pattern || /#{o.key}/ =~ pattern.to_s
    end
  rescue Aws::S3::Errors::NoSuchBucket => e
    logger.info format_error_message e
    return # log that the bucket doesn't exist but don't explode
  end
end

#send_logs_to_s3 ⇒ Object

Send the log files to the S3 log file bucket

Returns Aws::S3::Type::PutObjectOutput



89
90
91
92
93
94
95
96
97
 
# File 'lib/cloud_powers/storage.rb', line 89

def send_logs_to_s3
  File.open(log_file) do |file|
    s3.put_object(
      bucket: log_bucket,
      key: instance_id,
      body: file
    )
  end
end

#source_job(file) ⇒ Object

Searches a local job storage location for the given file name if it exists - exit the method if it does not exist - get the file from s3 and place it in the directory that was just searched bucket using #zfind()

Parameters

  • file String - the name of the file we’re searching for

Returns nothing

Example

  • file tree

    project_root- |
              |_sub_directory
              |              |_current_file.rb
              |_job_storage
              |             |_demorific.rb
              |             |_foobar.rb

  • code:

    source_job('demorific')
# file tree doesn't change because this file exists
source_job('custom_greetings')
# file tree now looks like this

  • new file tree

    project_root- |
              |_sub_directory
              |              |_current_file.rb
              |_job_storage
              |             |_demorific.rb
              |             |_foobar.rb
              |             |_custom_greetings.js # could be an after effects JS script



45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/cloud_powers/storage.rb', line 45

def source_job(file)
  # TODO: better path management
  bucket = zfind('job storage')
  if job_path(to_ruby_file_name(file)).nil?
    objects = s3.list_objects(bucket: bucket).contents.select do |f|
      /#{Regexp.escape file}/i =~ f.key
    end

    objects.each do |obj|
      s3.get_object(bucket: bucket, key: obj.key, response_target: job_path(file))
    end
  end
end