ProcessorPool
by Ari Lerner and Ron Evans
CitrusByte
http://blog.citrusbyte.com
== DESCRIPTION:
Provides a simple load-balancing solution for Amazon's EC2 - S3 backend.
Usage would include an upload server, a image and video processing and transcoding server, all non-blocking separate from an application server.
== FEATURES/PROBLEMS:
* Need to find a keep-alive solution
== SYNOPSIS:
require "rubygems"
require "lib/processor.rb"
# Set the access_key_id and the secret_access_key provided by amazon
access_key_id, secret_access_key = '11H31W1044QZ9QQF48G2', 'a8Xb46qggIeDfm1y/RUL48Ullq87MxgOKCzCmw1c'
# And start the processors!
ProcessorPool.start(access_key_id, secret_access_key)
# ProcessorPool is built on top of Sinatra (http://sinatra.rubyforge.org/) and uses the same idioms.
# For more information, check out Sinatra (at http://sinatra.rubyforge.org/):
get '/' do
"Processing data"
end
== IMPLEMENTATION
# Uploading and processing
# To use it without inline uploading
# Upload like you normally would do, but in the controller, just pass the variables to the offsite storage and continue on as you would. This won't block the rails mongrel from continuing on. Of course, you also need to check to make sure the file does get uploaded and processed at this point.
# Inline uploading
# In Rails, you can redirect the the upload to the processor pool in the form. For instance:
<% form_for @media, :id => "fileform", :url => upload_url, :multipart => true do %>
# To do the inline uploading, you should call a javascript method to interact with the user to let them know the uploading has begun. For instance:
<button class="button" id="buttonUpload" onclick="return beginFileUpload();">Upload</button>
<script type="text/javascript">
function beginFileUpload()
// todo: some kind of feedback that upload has started...
$("#uploading").show();
...
# For ease of use, we recommend embedding this helper function somewhere:
def url_for_upload_server
if RAILS_ENV == 'production'
# determine the upload server to try to use
p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
"http://#{p.hostname:4567/new"
else
p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
"http://localhost:4567/new"
end
end
# The reset is an exercise for you to complete.
# Other ideas, using similar methodology would be to upload videos and do the processing there, to run awy sort of processor intensive computations off-site.
== CONVENIENCE METHODS
# There are a number of convenience methods included with processor_pool. It is intended to work with Amazon's S3 service.
AWS::S3::S3Object.store('/folder/to/greeting.txt', 'hello world!', 'ron', :use_virtual_directories => true)
# Alternatively, you can use the included RemoteDirectory class
dir = RemoteDirectory.new("/folder/name")
# Save one file to the S3 directory
dir.save_to_s3("movie.mpg", "movies", "encoded")
# Copy all the files in the directory to the s3 bucket
dir.copy_to_s3("movies", "encoded")
dir.files # Returns a list of all the files in the bucket
== REQUIREMENTS:
* aws/s3
* net/ping
* open-uri
* sinatra
== INSTALL:
* sudo gem install processor_pool
== LICENSE:
(The MIT License)
Copyright (c) 2008 Ari Lerner, CitrusByte and Ron Evans
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
by Ari Lerner and Ron Evans
CitrusByte
http://blog.citrusbyte.com
== DESCRIPTION:
Provides a simple load-balancing solution for Amazon's EC2 - S3 backend.
Usage would include an upload server, a image and video processing and transcoding server, all non-blocking separate from an application server.
== FEATURES/PROBLEMS:
* Need to find a keep-alive solution
== SYNOPSIS:
require "rubygems"
require "lib/processor.rb"
# Set the access_key_id and the secret_access_key provided by amazon
access_key_id, secret_access_key = '11H31W1044QZ9QQF48G2', 'a8Xb46qggIeDfm1y/RUL48Ullq87MxgOKCzCmw1c'
# And start the processors!
ProcessorPool.start(access_key_id, secret_access_key)
# ProcessorPool is built on top of Sinatra (http://sinatra.rubyforge.org/) and uses the same idioms.
# For more information, check out Sinatra (at http://sinatra.rubyforge.org/):
get '/' do
"Processing data"
end
== IMPLEMENTATION
# Uploading and processing
# To use it without inline uploading
# Upload like you normally would do, but in the controller, just pass the variables to the offsite storage and continue on as you would. This won't block the rails mongrel from continuing on. Of course, you also need to check to make sure the file does get uploaded and processed at this point.
# Inline uploading
# In Rails, you can redirect the the upload to the processor pool in the form. For instance:
<% form_for @media, :id => "fileform", :url => upload_url, :multipart => true do %>
# To do the inline uploading, you should call a javascript method to interact with the user to let them know the uploading has begun. For instance:
<button class="button" id="buttonUpload" onclick="return beginFileUpload();">Upload</button>
<script type="text/javascript">
function beginFileUpload()
// todo: some kind of feedback that upload has started...
$("#uploading").show();
...
# For ease of use, we recommend embedding this helper function somewhere:
def url_for_upload_server
if RAILS_ENV == 'production'
# determine the upload server to try to use
p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
"http://#{p.hostname:4567/new"
else
p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
"http://localhost:4567/new"
end
end
# The reset is an exercise for you to complete.
# Other ideas, using similar methodology would be to upload videos and do the processing there, to run awy sort of processor intensive computations off-site.
== CONVENIENCE METHODS
# There are a number of convenience methods included with processor_pool. It is intended to work with Amazon's S3 service.
AWS::S3::S3Object.store('/folder/to/greeting.txt', 'hello world!', 'ron', :use_virtual_directories => true)
# Alternatively, you can use the included RemoteDirectory class
dir = RemoteDirectory.new("/folder/name")
# Save one file to the S3 directory
dir.save_to_s3("movie.mpg", "movies", "encoded")
# Copy all the files in the directory to the s3 bucket
dir.copy_to_s3("movies", "encoded")
dir.files # Returns a list of all the files in the bucket
== REQUIREMENTS:
* aws/s3
* net/ping
* open-uri
* sinatra
== INSTALL:
* sudo gem install processor_pool
== LICENSE:
(The MIT License)
Copyright (c) 2008 Ari Lerner, CitrusByte and Ron Evans
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.