Class: AWS::Flow::S3DataConverter
- Inherits:
-
Object
- Object
- AWS::Flow::S3DataConverter
- Defined in:
- lib/aws/decider/data_converter.rb
Overview
S3DataConverter uses YAMLDataConverter internally to serialize and deserialize ruby objects. Additionally it stores objects larger than 32k characeters in AWS S3 and returns a serialized s3 link to be deserialized remotely. It caches objects locally to minimize calls to S3.
AWS Flow Framework for Ruby doesn’t delete files from S3 to prevent loss of data. It is recommended that users use Object Lifecycle Management in AWS S3 to auto delete files.
More information about object expiration can be found at: docs.aws.amazon.com/AmazonS3/latest/dev/ObjectExpiration.html
Defined Under Namespace
Classes: S3Cache
Class Attribute Summary collapse
-
.conv ⇒ Object
Returns the value of attribute conv.
Instance Attribute Summary collapse
-
#bucket ⇒ Object
readonly
Returns the value of attribute bucket.
-
#cache ⇒ Object
readonly
Returns the value of attribute cache.
-
#converter ⇒ Object
readonly
Returns the value of attribute converter.
Class Method Summary collapse
-
.converter ⇒ Object
Ensures singleton.
Instance Method Summary collapse
-
#delete_from_s3(s3_filename) ⇒ Object
private
Helper method to delete an s3 file.
-
#dump(object) ⇒ Object
Serializes a ruby object into a string.
-
#get_from_s3(s3_filename) ⇒ Object
private
Helper method to read an s3 file.
-
#initialize(bucket) ⇒ S3DataConverter
constructor
A new instance of S3DataConverter.
-
#load(source) ⇒ Object
Deserializes a string into a ruby object.
-
#put_to_s3(string) ⇒ Object
private
Helper method to write a string to an s3 file.
Constructor Details
#initialize(bucket) ⇒ S3DataConverter
Returns a new instance of S3DataConverter.
111 112 113 114 115 116 117 |
# File 'lib/aws/decider/data_converter.rb', line 111 def initialize(bucket) @bucket = bucket @cache = S3Cache.new s3 = AWS::S3.new s3.buckets.create(bucket) unless s3.buckets[bucket].exists? @converter = FlowConstants.default_data_converter end |
Class Attribute Details
.conv ⇒ Object
Returns the value of attribute conv.
94 95 96 |
# File 'lib/aws/decider/data_converter.rb', line 94 def conv @conv end |
Instance Attribute Details
#bucket ⇒ Object (readonly)
Returns the value of attribute bucket.
91 92 93 |
# File 'lib/aws/decider/data_converter.rb', line 91 def bucket @bucket end |
#cache ⇒ Object (readonly)
Returns the value of attribute cache.
91 92 93 |
# File 'lib/aws/decider/data_converter.rb', line 91 def cache @cache end |
#converter ⇒ Object (readonly)
Returns the value of attribute converter.
91 92 93 |
# File 'lib/aws/decider/data_converter.rb', line 91 def converter @converter end |
Class Method Details
.converter ⇒ Object
Ensures singleton
97 98 99 100 101 102 103 104 105 106 107 |
# File 'lib/aws/decider/data_converter.rb', line 97 def converter return self.conv if self.conv name = ENV['AWS_SWF_BUCKET_NAME'] if name.nil? raise "Need a valid S3 bucket name to initialize S3DataConverter."\ " Please set the AWS_SWF_BUCKET_NAME environment variable with the"\ " bucket name." end self.conv ||= self.new(name) return self.conv end |
Instance Method Details
#delete_from_s3(s3_filename) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Helper method to delete an s3 file
197 198 199 200 |
# File 'lib/aws/decider/data_converter.rb', line 197 def delete_from_s3(s3_filename) s3 = AWS::S3.new s3.buckets[@bucket].objects.delete(s3_filename) end |
#dump(object) ⇒ Object
Serializes a ruby object into a string. If the size of the converted string is greater than 32k characters, the string is uploaded to an AWS S3 file and a serialized hash containing the filename is returned instead. The filename is generated at random in the following format - rubyflow_data_<UUID>.
The format of the returned serialized hash is - { s3_filename: <filename> }
131 132 133 134 135 136 137 138 139 |
# File 'lib/aws/decider/data_converter.rb', line 131 def dump(object) string = @converter.dump(object) ret = string if string.size > 32768 filename = put_to_s3(string) ret = @converter.dump({ s3_filename: filename }) end ret end |
#get_from_s3(s3_filename) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Helper method to read an s3 file
179 180 181 182 183 184 185 186 187 188 189 190 |
# File 'lib/aws/decider/data_converter.rb', line 179 def get_from_s3(s3_filename) return @cache[s3_filename] if @cache[s3_filename] s3 = AWS::S3.new s3_object = s3.buckets[@bucket].objects[s3_filename] begin ret = s3_object.read @cache[s3_filename] = ret rescue AWS::S3::Errors::NoSuchKey => e raise "Could not find key #{s3_filename} in bucket #{@bucket} on S3. #{e}" end return ret end |
#load(source) ⇒ Object
Deserializes a string into a ruby object. If the deserialized string is a ruby hash of the format { s3_filename: <filename> }, then it will first look for the file in a local cache. In case of a cache miss, it will try to download the file from AWS S3, deserialize the contents of the file and return the new object.
150 151 152 153 154 155 156 157 |
# File 'lib/aws/decider/data_converter.rb', line 150 def load(source) object = @converter.load(source) ret = object if object.is_a?(Hash) && object[:s3_filename] ret = @converter.load(get_from_s3(object[:s3_filename])) end ret end |
#put_to_s3(string) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Helper method to write a string to an s3 file. A random filename is generated of the format - rubyflow_data_<UUID>
166 167 168 169 170 171 172 |
# File 'lib/aws/decider/data_converter.rb', line 166 def put_to_s3(string) filename = "rubyflow_data_#{SecureRandom.uuid}" s3 = AWS::S3.new s3.buckets[@bucket].objects.create(filename, string) @cache[filename] = string return filename end |