tus-ruby-server

A Ruby server for the tus resumable upload protocol. It implements the core 1.0 protocol, along with the following extensions:

Installation

gem "tus-server"

Usage

Tus-ruby-server provides a Tus::Server Roda app, which you can run in your config.ru. That way you can run Tus::Server both as a standalone app or as part of your main app.

# config.ru
require "tus/server"

map "/files" do
  run Tus::Server
end

run YourApp

While this is the most flexible option, it's not optimal in terms of performance; see the Goliath section for an alternative approach.

Now you can tell your tus client library (e.g. tus-js-client) to use this endpoint:

// using tus-js-client
new tus.Upload(file, {
  endpoint: "http://localhost:9292/files",
  chunkSize: 5*1024*1024, // required unless using Goliath or Unicorn
  // ...
})

After the upload is complete, you'll probably want to attach the uploaded file to a database record. Shrine is one file attachment library that integrates nicely with tus-ruby-server, see shrine-tus-demo for an example integration.

Goliath

Among all the existing Ruby web servers, Goliath is probably the ideal one to run tus-ruby-server on. It's built on top of EventMachine, making it asynchronous both in reading the request body and writing to the response body. Goliath also allows tus-ruby-server to handle interrupted requests, by saving data that has been uploaded until the interruption. This means that with Goliath it's not mandatory for client to chunk the upload into multiple requests in order to achieve resumable upload (which would be the case for most other web servers).

It's recommended that you use goliath-rack_proxy for running your tus server app:

# Gemfile
gem "tus-server", "~> 1.0"
gem "goliath-rack_proxy", "~> 1.0"

# tus.rb
require "tus/server"
require "goliath/rack_proxy"

# any additional Tus::Server configuration you want to put in here

class GoliathTusServer < Goliath::RackProxy
  rack_app Tus::Server
  rewindable_input false # set to true if you're using checksums
end

$ ruby tus.rb --stdout # enable logging

This will run the tus server app on the root URL; if you want to run it on some path you can use Rack::Builder:

class GoliathTusServer < Goliath::RackProxy
  rack_app Rack::Builder.new {
    map "/files" do
      run Tus::Server
    end
  }
  rewindable_input false # set to true if you're using checksums
end

Unicorn

Like Goliath, Unicorn also support streaming uploads, and tus-ruby-server knows how to automatically recover from Unicorn::ClientShutdown exceptions during upload, storing data that it has received up until that point. Just note that in order to achieve streaming uploads, Nginx should be configured not to buffer incoming requests, and to disable worker timeout to enable long running upload requests:

timeout 60*60*24*30 # set worker timeout to 30 days

But it's also fine to have Nginx buffer requests, just note that in this case Nginx won't forward incomplete upload requests to tus-ruby-server, so in order for resumable upload to be possible the client needs to send data in multiple upload requests (which can then be retried individually).

Unless you're using the "checksum" tus feature, you might want to consider disabling rewindability of request body, to prevent Unicorn from additionally caching received data onto the disk (since that's not necessary unless request body needs to be rewinded).

# config/unicorn.rb
# ...

rewindable_input false

Other web servers

It's perfectly feasible to run tus-ruby-server on web servers other than Goliath or Unicorn (even necessary if you want to run it inside another app). Just keep in mind that most other web servers don't support request streaming, which means that tus-ruby-server will be able to start processing upload requess only once the whole request body has been received. Additionally, incomplete upload requests won't be forwarded to tus-ruby-server, so in order for resumable upload to be possible the client needs to send data in multiple upload requests (which can then be retried individually).

Storage

Filesystem

By default Tus::Server stores uploaded files to disk, in the data/ directory. You can configure a different directory:

require "tus/storage/filesystem"

Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("public/cache")

If the configured directory doesn't exist, it will automatically be created. By default the UNIX permissions applied will be 0644 for files and 0755 for directories, but you can set different permissions:

Tus::Storage::Filesystem.new("data", permissions: 0600, directory_permissions: 0777)

One downside of filesystem storage is that it doesn't work by default if you want to run tus-ruby-server on multiple servers, you'd have to set up a shared filesystem between the servers. Another downside is that you have to make sure your servers have enough disk space. Also, if you're using Heroku, you cannot store files on the filesystem as they won't persist.

All these are reasons why you might store uploaded data on a different storage, and luckily tus-ruby-server ships with two more storages.

MongoDB GridFS

MongoDB has a specification for storing and retrieving large files, called "GridFS". Tus-ruby-server ships with Tus::Storage::Gridfs that you can use, which uses the Mongo gem.

gem "mongo", ">= 2.2.2", "< 3"

require "tus/storage/gridfs"

client = Mongo::Client.new("mongodb://127.0.0.1:27017/mydb")
Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: client)

You can change the database prefix (defaults to fs):

Tus::Storage::Gridfs.new(client: client, prefix: "fs_temp")

By default MongoDB Gridfs stores files in chunks of 256KB, but you can change that with the :chunk_size option:

Tus::Storage::Gridfs.new(client: client, chunk_size: 1*1024*1024) # 1 MB

Note that if you're using the concatenation tus feature with Gridfs, all partial uploads except the last one are required to fill in their Gridfs chunks, meaning the length of each partial upload needs to be a multiple of the :chunk_size number.

Amazon S3

Amazon S3 is probably one of the most popular services for storing files, and tus-ruby-server ships with Tus::Storage::S3 which utilizes S3's multipart API to upload files, and depends on the aws-sdk-s3 gem.

gem "aws-sdk-s3", "~> 1.2"

require "tus/storage/s3"

Tus::Server.opts[:storage] = Tus::Storage::S3.new(
  access_key_id:     "abc",
  secret_access_key: "xyz",
  region:            "eu-west-1",
  bucket:            "my-app",
)

One thing to note is that S3's multipart API requires each chunk except the last to be 5MB or larger, so that is the minimum chunk size that you can specify on your tus client if you want to use the S3 storage.

If you want to files to be stored in a certain subdirectory, you can specify a :prefix in the storage configuration.

Tus::Storage::S3.new(prefix: "tus", **options)

You can also specify additional options that will be fowarded to Aws::S3::Client#create_multipart_upload using :upload_options.

Tus::Storage::S3.new(upload_options: {content_disposition: "attachment"}, **options)

All other options will be forwarded to Aws::S3::Client#initialize, so you can for example change the :endpoint to use S3's accelerate host:

Tus::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **options)

Other storages

If none of these storages suit you, you can write your own, you just need to implement the same public interface:

def create_file(uid, info = {})            ... end
def concatenate(uid, part_uids, info = {}) ... end
def patch_file(uid, io, info = {})         ... end
def update_info(uid, info)                 ... end
def read_info(uid)                         ... end
def get_file(uid, info = {}, range: nil)   ... end
def delete_file(uid, info = {})            ... end
def expire_files(expiration_date)          ... end

Maximum size

By default the size of files the tus server will accept is unlimited, but you can configure the maximum file size:

Tus::Server.opts[:max_size] = 5 * 1024*1024*1024 # 5GB

Expiration

Tus-ruby-server automatically adds expiration dates to each uploaded file, and refreshes this date on each PATCH request. By default files expire 7 days after they were last updated, but you can modify :expiration_time:

Tus::Server.opts[:expiration_time] = 2*24*60*60 # 2 days

Tus-ruby-server won't automatically delete expired files, but each storage knows how to expire old files, so you just have to set up a recurring task that will call #expire_files.

expiration_time = Tus::Server.opts[:expiration_time]
tus_storage     = Tus::Server.opts[:storage]
expiration_date = Time.now.utc - expiration_time

tus_storage.expire_files(expiration_time)

Download

In addition to implementing the tus protocol, tus-ruby-server also comes with a GET endpoint for downloading the uploaded file, which streams the file from the storage into the response body.

The endpoint will automatically use the following Upload-Metadata values if they're available:

  • content_type -- used in the Content-Type response header
  • filename -- used in the Content-Disposition response header

The Content-Disposition header will be set to "inline" by default, but you can change it to "attachment" if you want the browser to always force download:

Tus::Server.opts[:disposition] = "attachment"

The download endpoint supports Range requests, so you can use the tus file URL as src in <video> and <audio> HTML tags.

Checksum

The following checksum algorithms are supported for the checksum extension:

  • SHA1
  • SHA256
  • SHA384
  • SHA512
  • MD5
  • CRC32

Tests

Run tests with

$ bundle exec rake test # unit tests
$ bundle exec cucumber  # acceptance tests

Inspiration

The tus-ruby-server was inspired by rubytus.

License

MIT