Class: ActiveStorageEncryption::ResumableGCSUpload

Inherits:
Object
  • Object
show all
Defined in:
lib/active_storage_encryption/resumable_gcs_upload.rb

Overview

Unlike the AWS SDKs, the Ruby GCP SDKs do not have a built-in resumable upload feature, while that feature is well-supported by GCP (and has been supported for a long while). This module provides resumable uploads in an IO-like package, giving you an object you can write to.

file = @bucket.file("upload.bin", skip_lookup: true)
upload = ActiveStorageEncryption::ResumableGCSUpload.new(file)
upload.stream do |io|
   io.write("Hello resumable")
   20.times { io.write(Random.bytes(1.megabyte)) }
end

Note that to perform the resumable upload your IAM identity or machine identity must have either a correct key for accessing Cloud Storage, or - alternatively - run under a service account that is permitted to sign blobs. This maps to the “iam.serviceAccountTokenCreator” role - see github.com/googleapis/google-cloud-ruby/issues/13307 and cloud.google.com/iam/docs/service-account-permissions

Defined Under Namespace

Classes: ByteChunker, RangedPutIO

Constant Summary collapse

CHUNK_SIZE_FOR_UPLOADS =

AWS recommend 5MB as the default part size for multipart uploads. GCP recommend doing “less requests” in general, and they mandate that all parts except last are a multile of 256*1024. Knowing that we will need to hold a buffer of that size, let’s just assume that the 5MB that AWS uses is a good number for part size.

5 * 1024 * 1024

Instance Method Summary collapse

Constructor Details

#initialize(file, content_type: "binary/octet-stream", **signed_url_options) ⇒ ResumableGCSUpload 



139
140
141
142
143
 
# File 'lib/active_storage_encryption/resumable_gcs_upload.rb', line 139

def initialize(file, content_type: "binary/octet-stream", **signed_url_options)
  @file = file
  @content_type = content_type
  @signed_url_options = url_issuer_and_signer.merge(signed_url_options)
end

Instance Method Details

#stream {|writable| ... } ⇒ Object

Yields:

  • (writable) 


146
147
148
149
150
151
152
153
154
155
 
# File 'lib/active_storage_encryption/resumable_gcs_upload.rb', line 146

def stream(&blk)
  session_start_url = @file.signed_url(method: "POST", content_type: @content_type, headers: {"x-goog-resumable": "start"}, **@signed_url_options)
  response = Net::HTTP.post(URI(session_start_url), "", {"content-type" => @content_type, "x-goog-resumable" => "start"})
  raise "Expected HTTP status code to be 201, got #{response.code}" unless response.code.to_i == 201

  resumable_upload_session_put_url = response["location"]
  writable = RangedPutIO.new(resumable_upload_session_put_url, content_type: @content_type, chunk_size: CHUNK_SIZE_FOR_UPLOADS)
  yield(writable)
  writable.finish
end