tus-ruby-server

A Ruby server for the tus resumable upload protocol. It implements the core 1.0 protocol, along with the following extensions:

Installation

gem "tus-server"

Usage

Tus-ruby-server provides a Tus::Server Roda app, which you can run in your config.ru. That way you can run Tus::Server both as a standalone app or as part of your main app (though it's recommended to run it as a standalone app, as explained in the "Performance considerations" section of this README).

# config.ru
require "tus/server"

map "/files" do
  run Tus::Server
end

run YourApp

Now you can tell your tus client library (e.g. tus-js-client) to use this endpoint:

// using tus-js-client
new tus.Upload(file, {
  endpoint: "http://localhost:9292/files",
  chunkSize: 5*1024*1024, // 5MB
  // ...
})

After the upload is complete, you'll probably want to attach the uploaded file to a database record. Shrine is one file attachment library that integrates nicely with tus-ruby-server, see shrine-tus-demo for an example integration.

Storage

Filesystem

By default Tus::Server stores uploaded files to disk, in the data/ directory. You can configure a different directory:

require "tus/storage/filesystem"

Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("public/cache")

If the configured directory doesn't exist, it will automatically be created. By default the UNIX permissions applied will be 0644 for files and 0755 for directories, but you can set different permissions:

Tus::Storage::Filesystem.new("data", permissions: 0600, directory_permissions: 0777)

One downside of filesystem storage is that it doesn't work by default if you want to run tus-ruby-server on multiple servers, you'd have to set up a shared filesystem between the servers. Another downside is that you have to make sure your servers have enough disk space. Also, if you're using Heroku, you cannot store files on the filesystem as they won't persist.

All these are reasons why you might store uploaded data on a different storage, and luckily tus-ruby-server ships with two more storages.

MongoDB GridFS

MongoDB has a specification for storing and retrieving large files, called "GridFS". Tus-ruby-server ships with Tus::Storage::Gridfs that you can use, which uses the Mongo gem.

gem "mongo", ">= 2.2.2", "< 3"

require "tus/storage/gridfs"

client = Mongo::Client.new("mongodb://127.0.0.1:27017/mydb")
Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: client)

You can change the database prefix (defaults to fs):

Tus::Storage::Gridfs.new(client: client, prefix: "fs_temp")

By default MongoDB Gridfs stores files in chunks of 256KB, but you can change that with the :chunk_size option:

Tus::Storage::Gridfs.new(client: client, chunk_size: 1*1024*1024) # 1 MB

Note that if you're using the concatenation tus feature with Gridfs, all partial uploads except the last one are required to fill in their Gridfs chunks, meaning the length of each partial upload needs to be a multiple of the :chunk_size number.

Amazon S3

Amazon S3 is probably one of the most popular services for storing files, and tus-ruby-server ships with Tus::Storage::S3 which utilizes S3's multipart API to upload files, and depends on the aws-sdk gem.

gem "aws-sdk", "~> 2.1"

require "tus/storage/s3"

Tus::Server.opts[:storage] = Tus::Storage::S3.new(
  access_key_id:     "abc",
  secret_access_key: "xyz",
  region:            "eu-west-1",
  bucket:            "my-app",
)

One thing to note is that S3's multipart API requires each chunk except the last to be 5MB or larger, so that is the minimum chunk size that you can specify on your tus client if you want to use the S3 storage.

If you want to files to be stored in a certain subdirectory, you can specify a :prefix in the storage configuration.

Tus::Storage::S3.new(prefix: "tus", **options)

You can also specify additional options that will be fowarded to Aws::S3::Client#create_multipart_upload using :upload_options.

Tus::Storage::S3.new(upload_options: {content_disposition: "attachment"}, **options)

All other options will be forwarded to Aws::S3::Client#initialize, so you can for example change the :endpoint to use S3's accelerate host:

Tus::Storage::S3.new(endpoint: "https://s3-accelerate.amazonaws.com", **options)

Other storages

If none of these storages suit you, you can write your own, you just need to implement the same public interface:

def create_file(uid, info = {})            ... end
def concatenate(uid, part_uids, info = {}) ... end
def patch_file(uid, io, info = {})         ... end
def update_info(uid, info)                 ... end
def read_info(uid)                         ... end
def get_file(uid, info = {}, range: nil)   ... end
def delete_file(uid, info = {})            ... end
def expire_files(expiration_date)          ... end

Maximum size

By default the size of files the tus server will accept is unlimited, but you can configure the maximum file size:

Tus::Server.opts[:max_size] = 5 * 1024*1024*1024 # 5GB

Expiration

Tus-ruby-server automatically adds expiration dates to each uploaded file, and refreshes this date on each PATCH request. By default files expire 7 days after they were last updated, but you can modify :expiration_time:

Tus::Server.opts[:expiration_time] = 2*24*60*60 # 2 days

Tus-ruby-server won't automatically delete expired files, but each storage knows how to expire old files, so you just have to set up a recurring task that will call #expire_files.

expiration_date = Time.now.utc - Tus::Server.opts[:expiration_time]
Tus::Server.opts[:storage].expire_files(expiration_date)

Download

In addition to implementing the tus protocol, tus-ruby-server also comes with a GET endpoint for downloading the uploaded file, which streams the file directly from the storage.

The endpoint will automatically use the following Upload-Metadata values if they're available:

  • content_type -- used in the Content-Type response header
  • filename -- used in the Content-Disposition response header

The Content-Disposition header will be set to "inline" by default, but you can change it to "attachment" if you want the browser to always force download:

Tus::Server.opts[:disposition] = "attachment"

The download endpoint supports Range requests, so you can use the tus file URL as src in <video> and <audio> HTML tags.

Checksum

The following checksum algorithms are supported for the checksum extension:

  • SHA1
  • SHA256
  • SHA384
  • SHA512
  • MD5
  • CRC32

Performance considerations

Buffering

When handling file uploads it's important not be be vulnerable to slow-write clients. That means you need to make sure that your web/application server buffers the request body locally before handing the request to the request worker.

If the request body is not buffered and is read directly from the socket when it has already reached your Rack application, your application throughput will be severly impacted, because the workers will spend majority of their time waiting for request body to be read from the socket, and in that time they won't be able to serve new requests.

Puma will automatically buffer the whole request body in a Tempfile, before fowarding the request to your Rack app. Unicorn and Passenger will not do that, so it's highly recommended to put a frontend server like Nginx in front of those web servers, and configure it to buffer the request body.

Chunking

The tus protocol specifies

The Server SHOULD always attempt to store as much of the received data as possible.

The tus-ruby-server Rack application supports saving partial data for if the PATCH request gets interrupted before all data has been sent, but I'm not aware of any Rack-compliant web server that will forward interrupted requests to the Rack app.

This means that for resumable upload to be possible with tus-ruby-server in general, the file must be uploaded in multiple chunks; the client shouldn't rely that server will store any data if the PATCH request was interrupted.

// using tus-js-client
new tus.Upload(file, {
  endpoint: "http://localhost:9292/files",
  chunkSize: 5*1024*1024, // required option
  // ...
})

Downloading

Tus-ruby-server has a download endpoint which streams the uploaded file to the client. Unfortunately, with most classic web servers this endpoint will be vulnerable to slow-read clients, because the worker is only done once the whole response body has been received by the client. Web servers that are not vulnerable to slow-read clients include Goliath/Thin and Reel.

So, depending on your requirements, you might want to avoid displaying the uploaded file in the browser (making the user download the file directly from the tus server), until it has been moved to a permanent storage. You might also want to consider copying finished uploads to permanent storage directly from the underlying tus storage, instead of downloading them through the app.

Tests

Run tests with

$ rake test

The S3 tests are excluded by default, but you can include them by setting the $S3 environment variable.

$ S3=1 rake test

For running S3 tests you need to create an .env with the S3 credentials:

# .env
S3_BUCKET="..."
S3_REGION="..."
S3_ACCESS_KEY_ID="..."
S3_SECRET_ACCESS_KEY="..."

Inspiration

The tus-ruby-server was inspired by rubytus.

License

MIT