MultiSync

:heavy_exclamation_mark: currently a functioning WIP thats not quite finished yet but its close! :heavy_exclamation_mark:

Flexible synchronisation for your assets.

MultiSync stands on the shoulders of giants. On one side is Celluloid allowing for the synchronisation of assets to be highly parallel. On the other is Fog::Storage allowing MulitSync to support various well known storage services.

What that means is when your configuring MultiSync your creating various pools of workers which then distrubute the work behind synchronising your assets. Meaning that when your site has thousands of files, you get alot more :boom: for your :dollar: in less :alarm_clock:.

MultiSync tries to expose its asset synchronisation in a flexible way which should allow you to define how and where your assets live. Where possible though, MultiSync will try to provide support for various well known libraries.

Listed below are examples of how to get setup and started.

Installation

gem 'multi_sync', '~> 0.0.4'

require 'multi_sync'

MultiSync.configure do |config|
  # config.verbose = false  # turn on verbose logging (defaults to false)
  # config.force = false  # force syncing of outdated_files (defaults to false)
  # config.run_on_build = true # when within a framework which `builds` assets, whether to sync afterwards (defaults to true)
  # config.sync_outdated_files = true # when an outdated file is found whether to replace it (defaults to true)
  # config.delete_abandoned_files = true # when an abondoned file is found whether to remove it (defaults to true)
  # config.upload_missing_files = true # when a missing file is found whether to upload it (defaults to true)
  # config.target_pool_size = 8 # how many threads you would like to open for each target (defaults to the amount of CPU core's your machine has)
  # config.max_sync_attempts = 3 # how many times a file should be retried if there was an error during sync (defaults to 3)
end

Fog credentials support

MultiSync supports utilising Fog Credentials. Simply specify either a FOG_RC or .fog and we'll use it as the base for any :credentials used in a target.

Features / Usage Examples

MultiSync in its simplist form consists of three objects. sources, resources and targets. A source defines how and where a list of files (or resources) can be found. A resource represents a file from a source with additional properties (such as how to compare them). A target is destination which resources can be synchronised against.

Source

All sources takes one argument which is a Hash of configuration detailed below. There are currently two type's of sources which are

Source Types

  • local_source - Uses all files within the source_dir
  • manifest_source - Tries to find a Sprockets manifest.{yml,json} file within the source_dir
Key Type Default Description
source_dir Pathname, String nil The location this source should use
resource_options Hash {} A hash of options for this source's resources
targets Symbol, Array All targets The target(s) this source should sync against
include String (shell glob) **/* A shell globe to use for inclusion
exclude String (shell glob) nil A shell globe to use for exclusion 
# A `local` `source` which will use all files within '../build'
local_source({
  source_dir: '../build'
})

# A `manifest` `source` which will use a Sprockets ':manifest' within '../public/assets'
manifest_source({
  source_dir: '../public/assets'
})

# A `local` `source` which will use all files
# within '../build' including only 'mp4, mpg, mov' files
local_source({
  source_dir: '../build',
  include: '*.{mp4,mpg,mov}'
})

# A `local` `source` which will use all files
# within '../build' excluding any 'jpg, gif, png' files
local_source({
  source_dir: '../build',
  exclude: '*.{jpg,gif,png}'
})

# A `manifest` `source` which will use use a Sprockets `manifest`
# within '../public/assets' including only 'jpg, gif, png' files
# which sets `cache_control` and `expires` headers
manifest_source({
  source_dir: '../public/assets',
  include: '*.{jpg,gif,png}',
  resource_options: {
    cache_control: 'public, max-age=31557600',
    expires: CGI.rfc1123_date(Time.now + 31557600)
  }
})

Target

All targets takes one argument which is a Hash of configuration detailed below. There is currently only one target type which is:

Target Types

  • aws_target - Synchronises to aws (S3)
Key Type Default Description
target_dir Pathname, String nil the name of the target's directory (eg s3 bucket name)
destination_dir Pathname, String nil the name of the target destination's directory (eg folder within target)
credentials Hash inherits Fog Credentials credentionals needed by Fog 
# An `aws` `target` which will sync to the root of a bucket named 's3-bucket-name'
# in region 'us-east-1', with access_key_id 'xxx', and secret_access_key 'xxx'
aws_target({
  target_dir: 's3-bucket-name'
  credentials: {
    region: 'us-east-1',
    aws_access_key_id: 'xxx',
    aws_secret_access_key: 'xxx'
  }
})

# An `aws` `target` which will sync to a bucket named 's3-bucket-name'
# using credentials sourced from Fog's credentials (.fog or FOG_RC)
aws_target({
  target_dir: 's3-bucket-name'
})

# An `aws` `target` which will sync to a bucket named 's3-bucket-name'
# within a directory named 'directory-within-s3'
# in region 'us-east-1', with access_key_id 'xxx', and secret_access_key 'xxx'
aws_target({
  target_dir: 's3-bucket-name'
  destination_dir: 'directory-within-s3'
  credentials: {
    region: 'us-east-1',
    aws_access_key_id: 'xxx',
    aws_secret_access_key: 'xxx'
  }
})

Supported Libraries

Badges

Gem Version Build Status Code Quality Gittip

Supported Storage Services

Behind the scenes we're using Fog::Storage which allows us to support the most popular storage providers

Supported Ruby Versions

This library aims to support and is tested against the following Ruby implementations: