Thingfish Filesystem Datastore

Description

This is a data storage plugin for the Thingfish digital asset manager. It provides persistent storage for uploaded data to a simple filesystem path.

Authors

Installation

$ gem install thingfish-datastore-filesystem

Basic Usage

By default, data will be stored in a ‘thingfish’ directory, underneath your system’s tmpdir. (For most systems, this will be /tmp/thingfish.) This directory will need to exist before starting the Thingfish handler.

As with Thingfish itself, this plugin uses Configurability to modify default behaviors.

Here’s an example configuration file that enables this plugin and modifies the default storage path:

---
thingfish:
  datastore: filesystem
  filesystem_datastore:
    root_path: /net/storage/thingfish-data

Advanced Usage

This plugin can utilize various advanced features of the Mongrel2 daemon that can offload large data transfers to the Mongrel2 daemon directly. This frees the Thingfish handler(s) up for servicing new requests quickly, and dramatically improves performance for concurrent data transfers and large files.

Assumptions

Because of Mongrel2’s flexible architecture, you could equally run everything under a single machine, or scale Thingfish out as a farm, spanning hundreds of handlers across individual nodes, all with the same configuration. In any case, Your Thingfish handlers need access to the same physical storage device(s) that the Mongrel2 daemon does.

How you accomplish this is entirely up to you, though NFS is a likely choice. There are two directories that the Mongrel2 server and the Thingfish handlers will need access to – one for downloads (this plugin’s ‘root_path’ directory), and one for uploads (the Mongrel2 async spool directory.)

Downloads

To enable this functionality for downloads, you’ll need to add the ‘sendfile’ filter in your Mongrel2 server configuration. The path to this library will vary, depending on where Mongrel2 was installed on your system – but it should look like something that resembles:

xrequest ‘/usr/local/lib/mongrel2/filters/sendfile.so’

That filter is automatically built alongside Mongrel2, it just needs to be switched on. Ensure the ‘root_path’ is accessible from the Thingfish handler(s) and the Mongrel2 daemon.

Uploads

Uploads also require altering Mongrel2 server settings:

setting ‘limits.content_length’, 65536

This switches on async uploads, for any upload larger than 64k. Anything under 64k is kept in RAM, and handled internally by the Thingfish handler. Adjust to taste.

setting ‘upload.temp_store’, ‘/network/storage/mongrel2.upload.XXXXXXX’

For files larger than ‘limits.content_length’ size, asynchronously spool the upload to this location. When the file upload has completed, Mongrel2 notifies Thingfish, and the spooled upload is properly stored. This path must be accessible from the Thingfish handler(s) and the Mongrel2 daemon.

setting ‘upload.temp_store_mode’, ‘0600’

While this is not required, it is recommended to spool files so they are readable only to the Mongrel2 process. It is your responsibility to ensure the user account running the Thingfish handler also has read/write access to these files.

License

Copyright © 2014-2017, Michael Granger and Mahlon E. Smith All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  • Neither the name of the author/s, nor the names of the project’s contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.