Class: S3Publisher

Inherits:
Object
  • Object
show all
Defined in:
lib/s3-publisher.rb

Overview

You can either use the block syntax, or:

  • instantiate a class

  • queue data to be published with push

  • call run to actually upload the data to S3

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(bucket_name, opts = {}) ⇒ S3Publisher

Additional keys will be passed through to the Aws::S3::Client init, including: See docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#initialize-instance_method for full details.

Parameters:

  • bucket_name (String)
  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :base_path (String) —

    Path prepended to supplied file_name on upload

  • :workers (Integer) —

    Number of threads to use when pushing to S3. Defaults to 3.

  • :logger (Object) —

    A logger object to recieve ‘uploaded’ messages. Defaults to STDOUT.

  • :region (String) —

    AWS region to use, if different than global Aws config

  • :credentials (Object) —

    • :access_key_id, :secret_access_key, and :session_token options, if different than global Aws config



36
37
38
39
40
41
42
43
 
# File 'lib/s3-publisher.rb', line 36

def initialize bucket_name, opts={}
  @publish_queue = Queue.new
  @workers_to_use = opts.delete(:workers) || 3
  @logger         = opts.delete(:logger)  || $stdout
  @bucket_name, @base_path = bucket_name, opts.delete(:base_path)

  @s3 = Aws::S3::Client.new(opts)
end

Instance Attribute Details

#base_path ⇒ Object (readonly)

Returns the value of attribute base_path.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def base_path
  @base_path
end

#bucket_name ⇒ Object (readonly)

Returns the value of attribute bucket_name.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def bucket_name
  @bucket_name
end

#logger ⇒ Object (readonly)

Returns the value of attribute logger.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def logger
  @logger
end

#workers_to_use ⇒ Object (readonly)

Returns the value of attribute workers_to_use.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def workers_to_use
  @workers_to_use
end

Class Method Details

.publish(bucket_name, opts = {}) {|p| ... } ⇒ Object

Block style. run is called for you on block close.

S3Publisher.publish('my-bucket') do |p|
  p.push('test.txt', '123abc')
end

Yields:

  • (p) 


20
21
22
23
24
 
# File 'lib/s3-publisher.rb', line 20

def self.publish bucket_name, opts={}, &block
  p = self.new(bucket_name, opts)
  yield(p)
  p.run
end

Instance Method Details

#inspect ⇒ Object 



101
102
103
 
# File 'lib/s3-publisher.rb', line 101

def inspect
  "#<S3Publisher:#{bucket_name}>"
end

#push(key_name, opts = {}) ⇒ Object

Queues a file to be published. You can provide :data as a string, or a path to a file with :file. :file references won’t be evaluated until publish-time, reducing memory overhead.

Parameters:

  • key_name (String) —

    The name of the file on S3. base_path will be prepended if supplied.

  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :data (String) —

    a string to be published

  • :file (String) —

    path to a file to publish

  • :gzip (Boolean) —

    gzip file contents? defaults to true.

  • :ttl (Integer) —

    TTL in seconds for cache-control header. defaults to 5.

  • :cache_control (String) —

    specify Cache-Control header directly if you don’t like the default

  • :content_type (String) —

    no need to specify if default based on extension is okay. But if you need to force, you can provide :xml, :html, :text, or your own custom string.



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/s3-publisher.rb', line 57

def push key_name, opts={}
  write_opts = { acl: 'public-read' }
  
  key_name = "#{base_path}/#{key_name}" unless base_path.nil?
  
  # Setup data.
  if opts[:data]
    contents = opts[:data]
  elsif opts[:file]
    contents = Pathname.new(opts[:file])
    raise ArgumentError, "'#{opts[:file]}' does not exist!" if !contents.exist?
  else
    raise ArgumentError, "A :file or :data attr must be provided to publish to S3!"
  end

  # Then Content-Type
  if opts[:content_type]
    write_opts[:content_type] = opts[:content_type]
  else
    matching_mimes = MIME::Types.type_for(key_name)
    raise  ArgumentError, "Can't infer the content-type for '#{key_name}'! Please specify with the :content_type opt." if matching_mimes.empty?
    write_opts[:content_type] = matching_mimes.first.to_s
  end

  # And Cache-Control
  if opts.has_key?(:cache_control)
    write_opts[:cache_control] = opts[:cache_control]
  else
    write_opts[:cache_control] = "max-age=#{opts[:ttl] || 5}"
  end

  opts[:gzip] = true unless opts.has_key?(:gzip)

  @publish_queue.push({ key_name: key_name, contents: contents, write_opts: write_opts, gzip: opts[:gzip] })
end

#run ⇒ Object

Process queued uploads and push to S3



94
95
96
97
98
99
 
# File 'lib/s3-publisher.rb', line 94

def run
  threads = []
  workers_to_use.times { threads << Thread.new { publish_from_queue } }
  threads.each { |t| t.join }
  true
end