rest-ftp-daemon
This is a pretty simple FTP client daemon, controlled through a RESTfull API.
API documentation is maintained on Apiary
Features
As of today, its main features are :
- Allow environment-specific configuration in a YAML file
- Delegate a transfer job by
POST
`'ing a simple JSON structure - Spawn a dedicated thread to handle this job in its own context
- Report transfer status, progress and errors for each job in realtime
- Expose JSON status of workers on
GET /jobs/
for automated monitoring - Parralelize jobs as soon as they arrive
- Handle job queues and priority as an attribute of the job
- Allow dynamic evaluation of priorities, and change of any attribute until the job is picked
- Provide RESTful notifications to the requesting client
- Allow authentication in FTP target in a standard URI-format
- Allow configuration-based path templates to abstract local mounts or remote FTPs (endpoint tokens)
- Remote supported protocols: FTP and FTPs
- Allow main file transfer protocols: sFTP, FTPs / FTPes
- Automatically clean-up jobs after a configurable amount of time (failed, finished)
- Current bitrate on the last blocks chunk updated in the job attributes
- Global bitrate on the whole file transfer is re-computed after the transfer finishes
- Daemon process is tagged with its name and environement in process lists
- Allow basic patterns in source filename to match multiple files (/dir/file*.jpg)
Expected features in a short-time range :
- Allow change of priorities or other attributes after a job has been started
- Offer a basic dashboard directly within the daemon HTTP interface
- Periodically send an update-notification with transfer status and progress
- Allow fallback file source when first file path is unavailable (failover)
- Provide swagger-style API documentation
- Authenticate API clients
- Allow to specify random remote/local source/target
- Allow more transfer protocols (sFTP, HTTP POST etc)
Known bugs :
- As this project is based on SettingsLogic, which in turns uses Syck YAML parser, configuration merge from "defaults" section and environment-specific section is broken. A sub-tree defined for a specific environment, will overwrite the corresponding subtree from "defaults".
Installation
This project is available as a rubygem, requires Ruby 2.1 and rubygems installed.
You may use rbenv
and ruby-build
to get the right Ruby version. If this is your case, ensure that ruby-build definitions are up-to-date and include ruby-2.1.0
# apt-get install ruby-build rbenv
# ruby-build --definitions | grep '2.1'
Otherwise, you way have to update ruby-build to include Ruby 2.1.0 definitions. On Debian, 2.1.0 is not included in Wheezy and appears in Jessie's version of the package.
Use a dedicated user for the daemon, switch to this user and enable rbenv
# adduser --disabled-password --gecos "" rftpd
# su rftpd -l
# echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
# echo 'eval "$(rbenv init -)"' >> ~/.bashrc
Install the right ruby version and activate it
# rbenv install 2.1.0
# rbenv local 2.1.0
# rbenv rehash
Update RubyGems and install the gem from rubygems.org
# gem update --system
# gem install rest-ftp-daemon --no-ri --no-rdoc
# rbenv rehash
# rest-ftp-daemon start
Finally start the daemon on the standart port, or on a specific port using -p
# rest-ftp-daemon -p 4000 start
Check that the daemon is running and providing its status info. If the daemon seems to exit as soon as it's launched, this may be due to logfiles that cannot be written on (check permissions or owner).
http://localhost:3200/
Configuration
Most of the configuration options live in a YAML configuration file, containing two main sections:
- the
defaults
section should be left as-is and will be used is no other environment-specific value is provided. - the
production
section can receive personnalized settings according to your environment-specific setup and paths.
Configuration priority is defined as follows (from most important to last resort):
- command-line parameters
- config file defaults section
- config file environment section
- application internal defaults
As a starting point, rest-ftp-daemon.yml.sample
is an exemple config file that can be copied into the expected location /etc/rest-ftp-daemon.yml
.
Default administrator credentials are admin/admin. Please change the password in this configuration file before starting any kind of production.
Logging
The application will not log to any file by default, if not specified in its configuration.
Otherwise separate logging paths can be provided for the Thin webserver, API related messages, and workers related messages. Providing and empty value will simply activate logging to STDOUT
.
Job cleanup
Job can be cleanup up after a certain amount of time, when they reach on of these status:
- failed, after conchita.clean_failed seconds
- finished, after conchita.clean_finished seconds
Cleanup is done on a regular basis, every few seconds (conchita.timer)
Usage examples
- Start a job to transfer a file named "file.iso" to a local FTP server
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
'{"source":"~/file.iso","target":"ftp://anonymous@localhost/incoming/dest2.iso"}' "http://localhost:3000/jobs"
- Start a job using endpoint tokens
First define nas
ans ftp1
in the configuration file :
defaults: &defaults
development:
<<: *defaults
endpoints:
nas: "~/"
ftp1: "ftp://anonymous@localhost/incoming/"
Thos tokens will be expanded when the job is ran :
curl -H "Content-Type: application/json" -X POST -D /dev/stdout -d \
'{"source":"~/file.dmg","priority":"3", target":"ftp://anonymous@localhost/incoming/dest4.dmg","notify":"http://requestb.in/1321axg1"}' "http://localhost:3000/jobs"
- Get info about a job with ID="q89j.1"
Both parameters q89j.1
and 1
will be accepted as ID in the API. Requests below are equivalent:
GET http://localhost:3100/jobs/q89j.1
GET http://localhost:3100/jobs/1
Doc TODO
Document /status
Document /routes
Document multiple-files upload
Document mkdir and overwrite options
Document counters
About
Bruno MEDICI Consultant