Class: S3Publisher

Inherits:
Object
  • Object
show all
Defined in:
lib/s3-publisher.rb

Overview

You can either use the block syntax, or:

  • instantiate a class

  • queue data to be published with push

  • call run to actually upload the data to S3

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(bucket_name, opts = {}) ⇒ S3Publisher

Returns a new instance of S3Publisher.

Parameters:

  • bucket_name (String)
  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :base_path (String) —

    Path prepended to supplied file_name on upload

  • :workers (Integer) —

    Number of threads to use when pushing to S3. Defaults to 3.

  • :logger (Object) —

    A logger object to recieve ‘uploaded’ messages. Defaults to STDOUT.

  • :access_key_id (String) —

    AWS access key to use, if different than global AWS config

  • :secret_access_key (String) —

    AWS secret access key to use, if different than global AWS config

  • :region (String) —

    AWS region to use, if different than global AWS config

Raises:

  • (ArgumentError) 


34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
# File 'lib/s3-publisher.rb', line 34

def initialize bucket_name, opts={}
  @publish_queue = Queue.new
  @workers_to_use = opts[:workers] || 3
  @logger = opts[:logger] || $stdout

  s3_opts = {}
  s3_opts[:access_key_id]     = opts[:access_key_id]     if opts.key?(:access_key_id)
  s3_opts[:secret_access_key] = opts[:secret_access_key] if opts.key?(:secret_access_key)
  s3_opts[:region]            = opts[:region]            if opts.key?(:region)

  @s3 = AWS::S3.new(s3_opts)

  @bucket_name, @base_path = bucket_name, opts[:base_path]
  raise ArgumentError, "#{bucket_name} doesn't seem to be a valid bucket on your account" if @s3.buckets[bucket_name].nil?
end

Instance Attribute Details

#base_path ⇒ Object (readonly)

Returns the value of attribute base_path.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def base_path
  @base_path
end

#bucket_name ⇒ Object (readonly)

Returns the value of attribute bucket_name.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def bucket_name
  @bucket_name
end

#logger ⇒ Object (readonly)

Returns the value of attribute logger.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def logger
  @logger
end

#workers_to_use ⇒ Object (readonly)

Returns the value of attribute workers_to_use.



14
15
16
 
# File 'lib/s3-publisher.rb', line 14

def workers_to_use
  @workers_to_use
end

Class Method Details

.publish(bucket_name, opts = {}) {|p| ... } ⇒ Object

Block style. run is called for you on block close.

S3Publisher.publish('my-bucket') do |p|
  p.push('test.txt', '123abc')
end

Yields:

  • (p) 


20
21
22
23
24
 
# File 'lib/s3-publisher.rb', line 20

def self.publish bucket_name, opts={}, &block
  p = self.new(bucket_name, opts)
  yield(p)
  p.run
end

Instance Method Details

#inspect ⇒ Object 



106
107
108
 
# File 'lib/s3-publisher.rb', line 106

def inspect
  "#<S3Publisher:#{bucket_name}>"
end

#push(key_name, opts = {}) ⇒ Object

Queues a file to be published. You can provide :data as a string, or a path to a file with :file. :file references won’t be evaluated until publish-time, reducing memory overhead.

Parameters:

  • key_name (String) —

    The name of the file on S3. base_path will be prepended if supplied.

  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :data (String) —

    a string to be published

  • :file (String) —

    path to a file to publish

  • :gzip (Boolean) —

    gzip file contents? defaults to true.

  • :ttl (Integer) —

    TTL in seconds for cache-control header. defaults to 5.

  • :cache_control (String) —

    specify Cache-Control header directly if you don’t like the default

  • :content_type (String) —

    no need to specify if default based on extension is okay. But if you need to force, you can provide :xml, :html, :text, or your own custom string.



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
 
# File 'lib/s3-publisher.rb', line 62

def push key_name, opts={}
  write_opts = { acl: 'public-read' }
  
  key_name = "#{base_path}/#{key_name}" unless base_path.nil?
  
  # Setup data.
  if opts[:data]
    contents = opts[:data]
  elsif opts[:file]
    contents = Pathname.new(opts[:file])
    raise ArgumentError, "'#{opts[:file]}' does not exist!" if !contents.exist?
  else
    raise ArgumentError, "A :file or :data attr must be provided to publish to S3!"
  end

  # Then Content-Type
  if opts[:content_type]
    write_opts[:content_type] = opts[:content_type]
  else
    matching_mimes = MIME::Types.type_for(key_name)
    raise  ArgumentError, "Can't infer the content-type for '#{key_name}'! Please specify with the :content_type opt." if matching_mimes.empty?
    write_opts[:content_type] = matching_mimes.first.to_s
  end

  # And Cache-Control
  if opts.has_key?(:cache_control)
    write_opts[:cache_control] = opts[:cache_control]
  else
    write_opts[:cache_control] = "max-age=#{opts[:ttl] || 5}"
  end

  opts[:gzip] = true unless opts.has_key?(:gzip)

  @publish_queue.push({ key_name: key_name, contents: contents, write_opts: write_opts, gzip: opts[:gzip] })
end

#run ⇒ Object

Process queued uploads and push to S3



99
100
101
102
103
104
 
# File 'lib/s3-publisher.rb', line 99

def run
  threads = []
  workers_to_use.times { threads << Thread.new { publish_from_queue } }
  threads.each { |t| t.join }
  true
end