Asperalm - A Ruby library for Aspera transfers and "Amelia", the Multi Layer IBM Aspera Command Line Tool

Version : 0.9.20

Laurent/2016-2018

This gem provides a ruby API to Aspera transfers and a command line interface to Aspera Applications. Location: https://rubygems.org/gems/asperalm

Disclaimers:

  • Aspera, FASP are owned by IBM
  • This GEM is not endorsed/supported by IBM/Aspera
  • Use at your risk (not in production environments)
  • This gem is provided as-is, and is not intended to be a complete CLI, or industry-grade product.
  • some features may not be fully validated
  • IBM provides an officially supported Aspera CLI: http://downloads.asperasoft.com/en/downloads/62 .

That being said, mlia is very powerful and gets things done, it's also a great tool to learn Aspera APIs.

This manual addresses three parts:

  • mlia : ("Amelia") The Multi Layer IBM Aspera tool
  • asession : starting a FASP Session with JSON parameters
  • Asperalm : includes a Ruby "FASPManager"

In examples, command line operations (starting with $) are shown using a standard shell: bash.

Quick Start

This section guides you from installation, first use and advanced used.

First, follow the section: Installation (Ruby, Gem, FASP) to start using mlia.

Once the gem is installed, mlia shall be accessible:

$ mlia --version
0.9.20

First use

Once installation is completed, you can proceed to the first use with a demo server:

One liner:

$ mlia server browse / --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
:............:...........:......:........:...........................:.......................:
:   zmode    :   zuid    : zgid :  size  :           mtime           :         name          :
:............:...........:......:........:...........................:.......................:
: dr-xr-xr-x : asperaweb : fasp : 4096   : 2014-04-10 19:44:05 +0200 : aspera-test-dir-tiny  :
: drwxr-xr-x : asperaweb : fasp : 176128 : 2018-03-15 12:20:10 +0100 : Upload                :
: dr-xr-xr-x : asperaweb : fasp : 4096   : 2015-04-01 00:37:22 +0200 : aspera-test-dir-small :
: dr-xr-xr-x : asperaweb : fasp : 4096   : 2018-05-04 14:26:55 +0200 : aspera-test-dir-large :
:............:...........:......:........:...........................:.......................:

In order to make further calls more convenient, it is advised to define a option preset for the servers identification options. The following example will:

  • create a option preset
  • define it as default for "server" plugin
  • list files in a folder
  • download a file
$ mlia config id demoserver update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
updated: demoserver
$ mlia config id default set server demoserver
updated: default->server to demoserver
$ mlia server browse /aspera-test-dir-large
:............:...........:......:..............:...........................:............................:
:   zmode    :   zuid    : zgid :     size     :           mtime           :            name            :
:............:...........:......:..............:...........................:............................:
: -rw-rw-rw- : asperaweb : fasp : 10133504     : 2018-05-04 14:16:24 +0200 : ctl_female_2.fastq.partial :
: -rw-r--r-- : asperaweb : fasp : 209715200    : 2014-04-10 19:49:27 +0200 : 200MB                      :
: -rw-r--r-- : asperaweb : fasp : 524288000    : 2014-04-10 19:44:15 +0200 : 500MB                      :
: -rw-r--r-- : asperaweb : fasp : 5368709120   : 2014-04-10 19:45:52 +0200 : 5GB                        :
: -rw-r--r-- : asperaweb : fasp : 500000000000 : 2017-06-14 20:09:57 +0200 : 500GB                      :
: -rw-rw-rw- : asperaweb : fasp : 13606912     : 2018-05-04 14:20:21 +0200 : ctl_male_2.fastq.partial   :
: -rw-rw-rw- : asperaweb : fasp : 76           : 2018-05-04 14:13:18 +0200 : ctl_female_2.fastq.haspx   :
: -rw-rw-rw- : asperaweb : fasp : 647348       : 2018-05-04 14:26:39 +0200 : ctl_female_2.gz            :
: -rw-rw-rw- : asperaweb : fasp : 74           : 2018-05-04 14:16:00 +0200 : ctl_male_2.fastq.haspx     :
: -rw-r--r-- : asperaweb : fasp : 1048576000   : 2014-04-10 19:49:23 +0200 : 1GB                        :
: -rw-r--r-- : asperaweb : fasp : 104857600    : 2014-04-10 19:49:29 +0200 : 100MB                      :
: -rw-r--r-- : asperaweb : fasp : 10737418240  : 2014-04-10 19:49:04 +0200 : 10GB                       :
:............:...........:......:..............:...........................:............................:
$ mlia server download /aspera-test-dir-large/200MB
Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
complete

Going further

Get familiar with configuration, options, commands : Command Line Interface.

Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : Application Plugins

Installation

In order to use the tool or the gem, it is necessary to install those components:

  • ruby
  • asperalm
  • fasp

The following sections provide information on the installation.

Ruby

A ruby interpreter is required to run the tool or to use the gem. It is also required to have privilege to install gems.

Ruby 2.4+ is prefered, but it should also work with 2.0+.

Refer to the following sections for specific operating systems.

macOS

Ruby comes pre-installed on macOS. Starting with Macos Sierra, the version of Ruby is high enough. Nevertheless, installation of the gem requires: sudo gem install asperalm.

Alternatively, install "homebrew", from here: https://brew.sh/, and then install Ruby:

$ brew install ruby

Windows

On windows download Ruby from https://rubyinstaller.org/.

Linux

$ yum install ruby rubygems

Note that Ruby 2+ is required, if you have an older Linux (e.g. CentOS 6), you should install "rvm" https://rvm.io/ and install and use a newer Ruby.

asperalm gem

Once you have Ruby and rights to install gems: Install the gem and its dependencies:

$ gem install asperalm

To upgrade to the latest version:

$ gem update asperalm

FASP Protocol

Most file transfers will be done using the FASP protocol. This requires one of IBM Asprea transfer server or client with its license file (some are free):

  • IBM Aspera Connect Client (Free)
  • IBM Aspera Desktop Client (Free)
  • IBM Aspera CLI (Free)
  • IBM Aspera High Speed Transfer Server (Licensed)
  • IBM Aspera High Speed Transfer EndPoint (Licensed)

For instance, Aspera Connect Client can be installed by visiting the page: http://downloads.asperasoft.com/connect2/.

mlia will detect most of Aspera transfer products in standard locations and use the first one found. Refer to section Client for details.

Several methods are provided on how to effectively start a transfer, refer to section: Transfer Agents

Command Line Interface: mlia

The asperalm Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):

  • IBM Aspera High Speed Transfer Server (FASP and Node)
  • IBM Aspera on Cloud (including ATS)
  • IBM Aspera Faspex
  • IBM Aspera Shares
  • IBM Aspera Console
  • IBM Aspera Orchestrator
  • and more...

mlia provides the following features:

  • Supports most Aspera server products (on-premise and SaaS)
  • Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
  • Supports Commands, Option values and Parameters shortcuts
  • FASP Transfer Agents can be: FaspManager (local ascp), or Connect Client, or any transfer node
  • Transfer parameters can be altered by modification of transfer-spec, this includes requiring multi-session
  • Allows transfers from products to products, essentially at node level (using the node transfer agent)
  • Supports FaspStream creation (using Node API)
  • Supports Watchfolder creation (using Node API)
  • Additional command plugins can be written by the user
  • Supports download of faspex and Aspera on Cloud "external" links
  • Supports "legacy" ssh based FASP transfers and remote commands (ascmd)

Basic usage is displayed by executing:

$ mlia -h

Refer to sections: Usage and Sample Commands.

Not all mlia features are fully documented here, the user may explore commands on the command line.

Arguments : Commands and options

Arguments are the units of command line, as parsed by the shell, typically separated by spaces (and called "argv").

There are two types of arguments: Commands and Options. Example :

$ mlia command --option-name=VAL1 VAL2
  • executes command: command
  • with one option: option_name
  • this option has a value of: VAL1
  • the command has one additional argument: VAL2

When the value of a command, option or argument is constrained by a fixed list of values, it is possible to use the first letters of the value only, provided that it uniquely identifies a value. For example mlia conf ov is the same as mlia config overview.

The value of options and arguments is evaluated with the Extended Value Syntax.

Options

All options, e.g. --log-level=debug, are command line arguments that:

  • start with --
  • have a name, in lowercase, using - as word separator in name (e.g. --log-level=debug)
  • have a value, separated from name with a =
  • can be used by prefix, provided that it is unique. E.g. --log-l=debug is the same as --log-level=debug

Exceptions:

  • some options accept a short form, e.g. -Ptoto is equivalent to --preset=toto, refer to the manual or -h.
  • some options (flags) don't take a value, e.g. -r
  • the special option -- stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a -. Example:
$ mlia config echo -- --sample
"--sample"

Note that --sample is taken as an argument, and not option.

Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on comand line and evaluated in order.

The value for any options can come from the following locations (in this order, last value evaluated overrides previous value):

Environment variable starting with prefix: MLIA_ are taken as option values, e.g. MLIA_OPTION_NAME is for --option-name.

Options values can be displayed for a given command by providing the --show-config option: mlia node --show-config

Commands and Arguments

Command line arguments that are not options are either commands or arguments. If an argument must begin with -, then either use the @val: syntax (see Extended Values), or use the -- separator (see above).

Interactive Input

Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.

The behaviour can be controlled with:

  • --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
    • yes : missing mandatory parameters/options are asked to the user
    • no : missing mandatory parameters/options raise an error message
  • --ask-options=<yes|no> (default=no)
    • optional parameters/options are asked to user

Output Format

Command execution will result in output (terminal, stdout/stderr). The information displayed depends on the action. Types of result include:

  • single_object : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is atteribute value. Nested hashes are collapsed.
  • object_list : displayed as a 2 dimensional table: one line per item, one colum per attribute.
  • value_list : a table with one column.
  • empty : nothing
  • status : a message
  • other_struct : a complex structure that cannot be displayed as an array

The table style is :.: by default and can be customized with parameter: table_style (horizontal, vertical and intersection characters).

The style of output can be set using the format parameter, supporting:

  • table : Text table
  • ruby : Ruby code
  • json : JSON code
  • jsonpp : JSON pretty printed
  • yaml : YAML
  • csv : Comma Separated Values

Table output can be filtered using the select parameter. Example:

$ mlia aspera admin res user list --fields=name,email,ats_admin [email protected]:'{"per_page":1000}' [email protected]:'{"ats_admin":true}'
:...............................:..................................:...........:
:             name              :              email               : ats_admin :
:...............................:..................................:...........:
: John Custis                   : [email protected]                 : true      :
: Laurent Martin                : [email protected]              : true      :
:...............................:..................................:...........:

Note that select filters selected elements from the result of API calls, while the query parameters gives filtering parameters to the API when listing elements.

In a table format, when displaying "objects" (single, or list), by default, sub object are flatten (option flat_hash). So, object "user":{"id":1,"name":"toto"} will have attributes: user.id and user.name. Setting flat_hash to "false" will only display one field: "user" and value is the sub hash table. When in flatten mode, it is possible to filter fields by "dotted" field name.

Another option is display, which accepts values: info, data, error. Level info displays all messages (in table mode only). data do not display info messages, error display only error messages.

Extended Value Syntax

Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).

The value of options and arguments can optionally be retrieved using one of the following "readers":

  • @val:VALUE , prevent further special prefix processing, e.g. [email protected]:laurent sets the option username to value laurent.
  • @file:PATH , read value from a file (prefix "~/" is replaced with the users home folder), e.g. [email protected]:~/.ssh/mykey
  • @path:PATH , performs path expansion (prefix "~/" is replaced with the users home folder), e.g. [email protected]:~/sample_config.yml
  • @env:ENVVAR , read from a named env var, [email protected]:MYPASSVAR
  • @stdin: , read from stdin
  • @preset:NAME , get whole option preset value by name

In addition it is possible to decode a value, using one or multiple decoders :

  • @base64: decode a base64 encoded string
  • @json: decode JSON values (convenient to provide complex structures)
  • @zlib: uncompress data
  • @ruby: execute ruby code
  • @csvt: decode a titled CSV value

To display the result of an extended value, use the config echo command.

Example: read the content of the specified file, then, base64 decode, then unzip:

$ mlia config echo @zlib:@base64:@file:myfile.dat

Example: create a value as a hash, with one key and the value is read from a file:

$ mlia config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}' 

Example: read a csv file and create a list of hash for bulk provisioning:

$ cat test.csv 
name,email
lolo,[email protected]
toto,[email protected]
$ mlia config echo @csvt:@file:test.csv
:......:.....................:
: name :        email        :
:......:.....................:
: lolo : [email protected] :
: toto : [email protected]      :
:......:.....................:

Structured Value

Some options and parameters expect a Structured Value, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.

For instance, a transfer-spec is expected to be a Structured Value.

Structured values shall be described using the Extended Value Syntax. A convenient way to specify a Structured Value is to use the @json: decoder, and describe the value in JSON format. The @ruby: decoder can also be used. For an array of hash tables, the @csvt: decoder can be used.

It is also possible to provide a Structured Value in a file using @json:@file:<path>

Configuration file

mlia configuration and other runtime files (token cache, file lists) re stored in folder $HOME/.aspera/mlia.

An empty configuration file is created on the first execution of mlia. Nevertheless, there is no mandatory information required in this file, the use of it is optional.

Although the file is a standard YAML file, mlia provides commands to read and nmodify it using the config command.

All options for mlia commands can be set on command line, or by env vars, or using option presets in the configuratin file.

A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.

The default configuration file is: $HOME/.aspera/mlia/config.yaml (this can be overriden with option --config-file=path or equivalent env var).

So, finally, the configuration file is simply a catalog of pre-defined lists of options, called: option presets. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a option preset (e.g. mypreset) using the option: -Pmypreset or --preset=mypreset.

Option preset

A option preset is simply a collection of parameters and their associated values.

A named option preset can be modified directly using mlia, which will update the configuration file :

$ mlia config id &lt;option preset&gt; set|delete|show|initialize|update

The command update allows the easy creation of option preset by simply providing the options in their command line format, e.g. :

$ mlia config id demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera [email protected]:'{"precalculate_job_size":true}'
  • This creates a option preset demo_server with all provided options.

The command set allows setting individual options in a option preset.

$ mlia config id demo_server set password demoaspera

The command initialize, like update allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a Structured Value.

$ mlia config id demo_server initialize @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"demoaspera","ts":{"precalculate_job_size":true}}'

A good practice is to not manually edit the configurqtion file and use modification commands instead. If necessary, the configuration file can be edited (or simply consulted) with:

$ mlia config open

A full terminal based overview of the configuration can be displayed using:

$ mlia config over

A list of option preset can be displayed using:

$ mlia config list

Format

The configuration file is a hash in a YAML file. Example:

config:
  version: 0.3.7
default:
  server: demo_server
demo_server:
  url: ssh://demo.asperasoft.com:33001
  username: asperaweb
  password: demoaspera

We can see here:

  • The configuration was created with CLI version 0.3.7
  • the default option preset to load for plugin "server" is : demo_server
  • the option preset demo_server defines some parameters: the URL and credentials

Two option presets are reserved:

  • config contains a single value: version showing the CLI version used to create the configuration file. It is used to check compatibility.
  • default is reserved to define the default option preset name used for known plugins.

The user may create as many option presets as needed. For instance, a particular option preset can be created for a particular application instance and contain URL and credentials.

Values in the configuration also follow the Extended Value Syntax.

Note: if the user wants to use the Extended Value Syntax inside the configuration file, using the config id update command, the user shall use the @val: prefix. Example:

$ mlia config id my_aoc_org set private_key @val:@file:"$HOME/.aspera/mlia/aocapikey"

This creates the option preset:

...
my_aoc_org:
  private_key: @file:"/Users/laurent/.aspera/mlia/aocapikey"
...

So, the key file will be read only at execution time, but not be embedded in the configuration file.

Options are loaded using this algorithm:

  • if option '--preset=xxxx' is specified (or -Pxxxx), this reads the option preset specified from the configuration file.
    • else if option --no-default is specified, then dont load default
    • else it looks for the name of the default option preset in section "default" and loads it
  • environment variables are evaluated
  • command line options are evaluated

Parameters are evaluated in the order of command line.

To avoid loading the default option preset for a plugin, just specify a non existing configuration: -Pnone

On command line, words in parameter names are separated by a dash, in configuration file, separator is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.

Note: before version 0.4.5, some keys could be ruby symbols, from 0.4.5 all keys are strings. To convert olver versions, remove the leading ":" in front of keys.

The main plugin name is config, so it is possible to define a default option preset for the main plugin with:

$ mlia config id cli_default set interactive no
$ mlia config id default set config cli_default

A option preset value can be removed with unset:

$ mlia config id cli_default unset interactive

Examples

For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console, only username/password and url are required (either on command line, or from config file). Those can usually be provided on the command line:

$ mlia shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra 

This can also be provisioned in a config file:

1$ mlia config id shares06 set url https://10.25.0.6
2$ mlia config id shares06 set username john
3$ mlia config id shares06 set password 4sp3ra
4$ mlia config id default set shares shares06 
5$ mlia config overview
6$ mlia shares repo browse /

The three first commands build a option preset. Note that this can also be done with one single command:

$ mlia config id shares06 init @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'

The fourth command defines this option preset as the default option preset for the specified application ("shares"). The 5th command displays the content of configuration file in table format. Alternative option presets can be used with option "-P<option preset>" (or --preset=<option preset>)

Eventually, the last command shows a call to the shares application using default parameters.

Plugins

The CLI tool uses a plugin mechanism. The first level command (just after mlia on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application. For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".

Create your own plugin

$ mkdir -p ~/.aspera/mlia/plugins
$ cat<<EOF>~/.aspera/mlia/plugins/test.rb
require 'asperalm/cli/plugin'
module Asperalm
  module Cli
    module Plugins
      class Test < Plugin
        def declare_options; end
        def action_list; [];end
        def execute_action; puts "Hello World!"; end
      end # Test
    end # Plugins
  end # Cli
end # Asperalm
EOF

Debugging

The gem is equipped with traces. By default logging level is "warn". To increase debug level, use parameter log_level, so either command line --log-level=xx or env var MLIA_LOG_LEVEL.

Learning Aspera Product APIs (REST)

This CLI uses REST APIs. To display HTTP calls, use argument -r or --rest-debug, this is useful to display exact content or HTTP requests and responses.

In order to get traces of execution, use argument : --log-level=debug

Graphical Interactions: Browser and Text Editor

Some actions may require the use of a graphical tool:

  • a browser for Aspera on Cloud authentication (web auth method)
  • a text editor for configuration file edition

By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable. It is also possible to force the graphical mode with option --ui :

  • --ui=graphical forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
  • --ui=text forces a text environment, the URL or file path to open is displayed on terminal.

Transfer Agents

Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (ascp).

When a transfer needs to be started, a transfer-spec has been internally prepared. This transfer-spec will be executed by a transfer client, here called "Transfer Agent".

There are currently 3 agents:

  • direct : a local execution of ascp
  • connect : use of a local Connect Client
  • node : use of a potentially remote Aspera Transfer Node.

Note that all transfer operation are seen from the point of view of the agent. For instance, a node agent making an "upload", or "package send" operation, will effectively push files to the related server from the agent node.

mlia standadizes on the use of a transfer-spec instead of raw ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.

Direct (local ascp using FASPManager API)

By default the CLI will use a local FASP protocol. mlia will detect locally installed Aspera products. Refer to section Client.

IBM Aspera Connect Client GUI

By specifying option: --transfer=connect, mlia will start transfers using the locally installed Aspera Connect Client.

Aspera Node API : Node to node transfers

By specifying option: --transfer=node, the CLI will start transfers in an Aspera Transfer Server using the Node API, either on a local or remote node.

If a default node has been configured in the configuration file, then this node is used by default else the parameter --transfer-info is required. The node specification shall be a hash table with three keys: url, username and password, corresponding to the URL of the node API and associated credentials (node user or access key).

The --transfer-info parameter can directly specify a pre-configured option preset : [email protected]:<psetname> or specified using the option syntax : [email protected]:'{"url":"https://...","username":"theuser","password":"thepass"}'

Transfer Specification

Some commands lead to file transfer (upload/download), all parameters necessary for this transfer is described in a transfer-spec (Transfer Specification), such as:

  • server address
  • transfer user name
  • credentials
  • file list
  • etc...

mlia builds a default transfer-spec internally, so it is not necessary to provide additional parameters on the command line for this transfer.

If needed, it is possible to modify or add any of the supported transfer-spec parameter using the ts option. The ts option accepts a Structured Value containing one or several transfer-spec parameters.

It is possible to specify ascp options when the transfer option is set to direct using the special transfer-spec parameter: EX_ascp_args. Example: [email protected]:'{"EX_ascp_args":["-l","100m"]}'.

The use of a transfer-spec instead of ascp parameters has the advantage of:

  • common to all Transfer Agent
  • not dependent on command line limitations (special characters...)

A transfer-spec is a Hash table, so it is described on the command line with the Extended Value Syntax.

Transfer Parameters

All standard transfer-spec parameters can be overloaded. To display parameters, run in debug mode (--log-level=debug). transfer-spec can also be saved/overridden in the config file.

Columns:

  • F=Fasp Manager(local FASP execution)
  • N=remote node(node API)
  • C=Connect Client(web plugin)

Req/Def : Required or default value (- means emty)

Fields with EX_ prefix are specific extensions to local mode.

arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)

(UNDER CONSTRUCTION ref)

FieldReq/DefTypeFNCargDescription
directionRequiredstringYYY--modeDirection: "send" or "receive"
remote_hostRequiredstringYYY--hostIP or fully qualified domain name of the remote server
remote_userRequiredstringYYY--userRemote user. Default value is "xfer" on node or connect.
destination_rootRequiredstringYYYlast argDestination root directory.
title-stringNYY-Title of the transfer
tags-hashYYY--tags
--tags64
Metadata for transfer
token-stringYYYTOKEN
-W
Authorization token: Bearer, Basic or ATM
cookie-stringYYYCOOKIEMetadata for transfer (older,string)
remote_access_keyTODOstring?Node only?
source_root-string--source-prefix
--source-prefix64
Source root directory.(TODO: verify option)
fasp_port33001integerYYY-OSpecifies fasp (UDP) port.
ssh_port22 or 33001integerYYY-PSpecifies ssh (TCP) port.
rate_policyserver configstringYYY--policyValid literals include "low","fair","high" and "fixed".
symlink_policyfollowstringYYY--symbolic-linkscopy, follow, copy+force, skip. Default is follow. Handle source side symbolic links by following the link (follow), copying the link itself (copy), skipping (skip), or forcibly copying the link itself (copy+force).
target_rate_kbps-integerYYY-lSpecifies desired speed for the transfer.
min_rate_kbps0integerYYY-mSet the minimum transfer rate in kilobits per second.
ciphernonestringYYY-cin transit encryption type.
none, aes-128, aes-256
content_protectionencrypt
decrypt
string--file-crypt=encryption at rest
content_protection_password-stringPASSSpecifies a string password.
overwritediffstringYYY--overwriteOverwrite destination files with the source files of the same name.
never, always, diff, older, or diff+older
retry_durationstringTODOSpecifies how long to wait before retrying transfer. (e.g. "5min")
http_fallbackbool (node), integer-y
TODO
When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.
create_dirbooleanYYY-dSpecifies whether to create new directories.
precalculate_job_sizesrv. def.booleanYYY--precalculate-job-sizeSpecifies whether to precalculate the job size.
delete_sourcebooleanY??
remove_after_transferbooleanY?Specifies whether to remove file after transfer.
remove_empty_directoriesbooleanY?Specifies whether to remove empty directories.
multi_session1integerNYN-CSpecifies how many parts the transfer is in.
multi_session_thresholdnullintegerNYN-in bytes
dgram_sizeintegerY-Zin bytes
compressionintegerascp4 only, 0 / 1?
read_threadsinteger-ascp4 only
write_threadsinteger-ascp4 only
use_ascp4falsebooleanY-specify version of protocol
pathssource files (dest)arraypositional
--file-list
--file-pair-list
Contains a path to the source (required) and a path to the destination.
http_fallback_portintegerY-tSpecifies http port.
https_fallback_portintegertodoSpecifies https port.
cipher_allowedstring-returned by node API. Valid literals include "aes-128" and "none".
target_rate_cap_kbpsN??-Returned by upload/download_setup node api.
rate_policy_allowed-returned by node API. Specifies most aggressive rate policy that is allowed. Valid literals include "low", "fair","high" and "fixed".
ssh_private_key-stringYNNKEYPrivate key used for SSH authentication
remote_password-stringYYYPASSSSH session password
resume_policyfaspmgr:
none
other:
sparse_csum
stringYYY-knone,attrs,sparse_csum,full_csum
authenticationfaspmgr:
none
other:
sparse_csum
stringYYN-token: Aspera web keys are provided to allow transparent web based session initiation. on connect: password is not asked. Else, password is asked, and keys are not provided.
EX_ssh_key_paths-arrayYNN-iUse public key authentication and specify the private key file
EX_at_rest_password-stringYNNFILEPASSPassphrase used for at rest encryption or decryption
EX_proxy_password-stringYNNPROXY_PASSTODO
EX_fasp_proxy_url-stringYNN--proxySpecify the address of the Aspera high-speed proxy server
EX_http_proxy_url-stringYNN-xSpecify the proxy server address used by HTTP Fallback
EX_ascp_args-arrayYNNsameAdd command line arguments to ascp
EX_http_transfer_jpeg0integerYNN-jHTTP transfers as JPEG file

Destination folder for transfers

The destination folder is set by mlia by default to:

  • . for downloads
  • / for uploads

It is specified by the transfer-spec parameter destination_root. As such, it can be modified with option: [email protected]:'{"destination_root":"<path>"}'. The option to_folder provides an equivalent and convenient way to change this parameter: --to-folder=<path> .

Source files for transfers

When uploading, downloading or sending files, the user must specify the list of files to transfer. This is done by using the option: sources. This is simply the list of paths of files.

Note the special case when the source files are located on "Aspera on Cloud":

  • All files must be in the same source folder.
  • If there is a single file : specify the full path
  • For multiple files, specify the source folder as first item in the list followed by the list of file names.

Source files are located on "Aspera on cloud", when :

  • the server is Aspera on Cloud, and making a download / recv
  • the agent is Aspera on Cloud, and making an upload / send

There are 3 ways to specify the list of files:

The list of file path is provided at the end of the command line. If the sources is not provided, this is the default. Example:

[email protected] file1.ext file2.ext

or more simply:

file1.ext file2.ext

The file list is part of the standard transfer spec. Example:

[email protected] [email protected]:'{"paths"=>[{"source"=>"file1.ext"},{"source"=>"file1.ext"}]}'

The value must be an Array of String, specified using the Extended Value Syntax. Example:

[email protected]:'[{"source"=>"file1.ext"},{"source"=>"file1.ext"}]'

Support of multi-session

Multi session, i.e. starting a transfer of a file set using multiple sessions is supported on "direct" and "node" agents, not yet on connect.

  • when agent=node :
[email protected]:'{"multi_session":10,"multi_session_threshold":1}'

Multi-session is directly supported by the node daemon.

  • when agent=direct :
[email protected]:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'

Note: resume policy of "attr" may cause problems. "none" or "sparse_csum" shall be preferred.

Multi-session spawn is done by mlia.

Examples

  • Change target rate
[email protected]:'{"target_rate_kbps":500000}'
  • Override the FASP SSH port to a specific TCP port:
[email protected]:'{"ssh_port":33002}'
  • Force http fallback mode:
[email protected]:'{"http_fallback":"force"}'
  • Activate progress when not activated by default on server
[email protected]:'{"precalculate_job_size":true}'

Scheduling an exclusive execution

It is possible to ensure that a given command is only run once at a time with parameter: --lock-port=nnnn. This is especially usefull when scheduling a command on a regular basis, for instance involving transfers, and a transfer may last longer than the execution period.

This opens a local TCP server port, and fails if this port is already used, providing a local lock.

This option is used when the tools is executed automatically, for instance with "preview" generation.

Usually the OS native scheduler shall already provide some sort of such protection (windows scheduler has it natively, linux cron can leverage flock).

Sample Commands

A non complete list of commands used in unit tests:

mlia
mlia --no-default node --url=my_url_here --username=my_username_here --password=my_password_here --insecure=yes delete /500M.dat
mlia --no-default node --url=my_url_here --username=my_username_here --password=my_password_here --insecure=yes upload --to-folder=/Upload [email protected] [email protected]:'{"paths":[{"source":"500M.dat"}],"remote_password":"demoaspera","precalculate_job_size":true}' --transfer=node [email protected]:'{"url":"my_url_here","username":"my_username_here","password":"my_password_here"}' 
mlia --version)
mlia --version` TOOLNAME=$(EXENAME) erb README.erb.md > README.md
mlia -N server --url=my_url_here --username=my_username_here --password=my_password_here --format=nagios nagios transfer --to-folder=/Upload
mlia -N server --url=my_url_here --username=my_username_here --ssh-keys=~/.ssh/id_rsa --format=nagios nagios app_services
mlia -N server --url=my_url_here --username=my_username_here --ssh-keys=~/.ssh/id_rsa ctl all:status
mlia -N server --url=my_url_here --username=my_username_here --ssh-keys=~/.ssh/id_rsa nodeadmin -- -l
mlia -Pnode_lmdk08 --url=my_url_here --username=my_username_here --password=my_password_here node acc create [email protected]:'{"id":"aoc_1","secret":"'$(NODE_PASS)'","storage":{"type":"local","path":"/"}}'
mlia -Pnode_lmdk08 --url=my_url_here --username=my_username_here --password=my_password_here node acc delete --id=aoc_1
mlia -h
mlia aspera admin ats access_key --id=testkey2 delete
mlia aspera admin ats access_key --id=testkey2 node browse /
mlia aspera admin ats access_key --id=testkey3 delete
mlia aspera admin ats access_key create --cloud=aws --region=eu-west-1 [email protected]:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"sedemo-ireland","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/laurent"}}'
mlia aspera admin ats access_key create --cloud=softlayer --region=ams [email protected]:'{"id":"testkey2","name":"laurent key","storage":{"type":"softlayer_swift","container":"laurent","credentials":{"api_key":"my_api_key_here","username":"my_username_here"},"path":"/"}}'
mlia aspera admin ats access_key list --fields=name,id,secret
mlia aspera admin ats cluster clouds
mlia aspera admin ats cluster list
mlia aspera admin ats cluster show --cloud=aws --region=eu-west-1 
mlia aspera admin ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
mlia aspera admin eve [email protected]:'{"page":1,"per_page":2,"q":"*","sort":"-date"}'
mlia aspera admin res node v3 events --secret=my_secret_here
mlia aspera admin resource node --name=eudemo --secret=my_secret_here v3 access_key create [email protected]:'{"id":"testsub1","storage":{"path":"/folder1"}}'
mlia aspera admin resource node --name=eudemo --secret=my_secret_here v3 access_key delete --id=testsub1
mlia aspera admin resource node --name=eudemo --secret=my_secret_here v3 events
mlia aspera admin resource node --name=eudemo --secret=my_secret_here v4 browse /
mlia aspera admin resource node --name=eudemo --secret=my_secret_here v4 delete /folder1
mlia aspera admin resource node --name=eudemo --secret=my_secret_here v4 mkdir /folder1
mlia aspera admin resource workspace list
mlia aspera admin resource workspace_membership list --fields=ALL [email protected]:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
mlia aspera admin set_client_key ERuzXGuPA @file:$(APIKEY)
mlia aspera apiinfo
mlia aspera bearer_token --display=data --scope=user:all
mlia aspera faspex
mlia aspera files browse /
mlia aspera files delete /newname
mlia aspera files download --to-folder=sample_dest_folder --transfer=connect [email protected] /200KB.1
mlia aspera files file 18891
mlia aspera files find / --value='\.partial$$'
mlia aspera files http_node_download --to-folder=sample_dest_folder [email protected] /200KB.1
mlia aspera files mkdir /testfolder
mlia aspera files rename /testfolder newname
mlia aspera files transfer --from-folder=/ --to-folder=xxx [email protected] 200KB.1
mlia aspera files upload --to-folder=/ sample_file.bin
mlia aspera files v3 info
mlia aspera organization
mlia aspera packages list
mlia aspera packages list --format=csv --fields=id|head -n 1)
mlia aspera packages recv --id=ALL --once-only=yes --lock-port=12345
mlia aspera packages send [email protected]:'{"name":"my title","note":"my note","recipients":["[email protected]"]}' [email protected] sample_file.bin
mlia aspera packages send [email protected]:'{"name":"my title","recipients":["[email protected]"]}' [email protected]:'{"package_contact":true}' [email protected] sample_file.bin
mlia aspera user info modify @json:'{"name":"dummy change"}'
mlia aspera user info show
mlia aspera workspace
mlia ats access_key --id=testkey2 cluster
mlia ats access_key --id=testkey2 delete
mlia ats access_key --id=testkey2 node browse /
mlia ats access_key --id=testkey3 delete
mlia ats access_key create --cloud=aws --region=eu-west-1 [email protected]:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"sedemo-ireland","credentials":{"access_key_id":"my_access_key_id_here","secret_access_key":"my_secret_access_key_here"},"path":"/laurent"}}'
mlia ats access_key create --cloud=softlayer --region=ams [email protected]:'{"id":"testkey2","name":"laurent key","storage":{"type":"softlayer_swift","container":"laurent","credentials":{"api_key":"my_api_key_here","username":"my_username_here"},"path":"/"}}'
mlia ats access_key list --fields=name,id,secret
mlia ats api_key create
mlia ats api_key instances
mlia ats api_key list
mlia ats cluster clouds
mlia ats cluster list
mlia ats cluster show --cloud=aws --region=eu-west-1 
mlia ats cluster show --id=1f412ae7-869a-445c-9c05-02ad16813be2
mlia client available
mlia client connect id 'Aspera Connect for Windows' info
mlia client connect id 'Aspera Connect for Windows' links id 'Windows Installer' download --to-folder=.
mlia client connect id 'Aspera Connect for Windows' links list
mlia client connect list
mlia client current
mlia conf wiz --url=my_url_here --config-file=$(SAMPLE_CONFIG_FILE) --client-id=$(HIDE_CLIENT_ID) --client-secret=$(HIDE_CLIENT_SECRET) --pkeypath=''
mlia config export
mlia config genkey sample_dest_folder/mykey
mlia config id conf_name delete
mlia config id conf_name initialize @json:'{"p1":"v1","p2":"v2"}'
mlia config id conf_name set param value
mlia config id conf_name show
mlia config id conf_name update --p1=v1 --p2=v2
mlia config id default set shares conf_name
mlia config list
mlia config open
mlia config overview
mlia config plugins
mlia console transfer current list 
mlia faspex nagios_check
mlia faspex package list
mlia faspex package list --fields=delivery_id --format=csv --box=sent|tail -n 1) --box=sent
mlia faspex package recv --to-folder=sample_dest_folder --id=ALL --once-only=yes
mlia faspex package send [email protected]:'{"title":"my title","recipients":["[email protected]"]}' [email protected] sample_file.bin 
mlia faspex recv_publink 'https://faspex.mycompany.com/aspera/faspex/external_deliveries/78780?passcode=a003aaf2f53e3869126b908525084db6bebc7031' --insecure=yes
mlia faspex source name "Server Files" node br /
mlia node async --id=1 counters 
mlia node async --id=1 summary 
mlia node async list
mlia node browse / -r
mlia node delete sample_dest_folder200KB.1
mlia node download --to-folder=sample_dest_folder [email protected] sample_dest_folder200KB.1
mlia node info
mlia node nagios_check
mlia node service --id=service1 delete
mlia node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
mlia node service list
mlia node transfer list
mlia node upload --to-folder=sample_dest_folder [email protected] sample_file.bin
mlia orchestrator info
mlia orchestrator plugins
mlia orchestrator processes
mlia orchestrator workflow --id=10 inputs
mlia orchestrator workflow --id=10 start [email protected]:'{"Param":"laurent"}'
mlia orchestrator workflow --id=10 start [email protected]:'{"Param":"laurent"}' --result=ResultStep:Complete_status_message
mlia orchestrator workflow --id=10 status
mlia orchestrator workflow list
mlia orchestrator workflow status
mlia preview events --once-only=yes --skip-types=office
mlia preview scan --skip-types=office --log-level=info
mlia preview test ~/Documents/Samples/anatomic-2k/TG18-CH/TG18-CH-2k-01.dcm --log-level=debug png --video=clips
mlia server browse /
mlia server browse /Upload/target_hot
mlia server cp /Upload/200KB.1 /Upload/200KB.2
mlia server delete /Upload/200KB.1
mlia server delete /Upload/target_hot
mlia server delete /Upload/to.delete
mlia server df
mlia server download [email protected] /Upload/200KB.1 --transfer=node
mlia server download --to-folder=sample_dest_folder [email protected] /Upload/200KB.1
mlia server du /
mlia server info
mlia server md5sum /Upload/200KB.1
mlia server mkdir /Upload/123 --logger=stdout
mlia server mkdir /Upload/target_hot
mlia server mv /Upload/200KB.2 /Upload/to.delete
mlia server rm /Upload/123
mlia server upload --to-folder=/Upload [email protected] sample_file.bin
mlia server upload --to-folder=/Upload/target_hot --lock-port=12345 [email protected]:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' [email protected] source_hot
mlia shares repository browse /
mlia shares repository delete /$(TEST_SHARE)/200KB.1
mlia shares repository download --to-folder=sample_dest_folder [email protected] /$(TEST_SHARE)/200KB.1
mlia shares repository upload --to-folder=/$(TEST_SHARE) --source[email protected] sample_file.bin
mlia shares2 appinfo
mlia shares2 organization list
mlia shares2 project list --organization=Sport
mlia shares2 repository browse /
mlia shares2 userinfo
mlia sync start [email protected]:'{"sessions":[{"name":"test","reset":true,"remote_dir":"/sync_test","local_dir":"contents","host":"10.25.0.8","user":"user1","private_key_path":"/Users/laurent/.ssh/id_rsa"}]}'

...and more

Usage

$ mlia -h
NAME
    mlia -- a command line tool for Aspera Applications (v0.9.20)

COMMAND: config
SUBCOMMANDS: gem_path, genkey, plugins, flush_tokens, list, overview, open, echo, id, documentation, wizard, export_to_cli, detect, coffee
OPTIONS:
        --value=VALUE                extended value for create, update, list filter
        --id=VALUE                   resource identifier (modify,delete,show)
        --override=ENUM              override existing value: yes, no
        --config-file=VALUE          read parameters from file in YAML format, current=/Users/laurent/.aspera/mlia/config.yaml
    -N, --no-default                 do not load default configuration for plugin
        --global-client-id=ENUM      wizard: AoC: use global or org specific jwt client id: yes, no
        --web-bootstrap=ENUM         wizard: AoC: first login with web: yes, no
        --pkeypath=VALUE             path to private key
SYNOPSIS
    mlia COMMANDS [OPTIONS] [ARGS]

DESCRIPTION
    Use Aspera application to perform operations on command line.
    Documentation and examples: https://rubygems.org/gems/asperalm
    execute: mlia conf doc

COMMANDS
    First level commands: 
    Note that commands can be written shortened (provided it is unique).

OPTIONS
    Options begin with a '-' (minus), and value is provided on command line.
    Special values are supported beginning with special prefix, like: @base64: @json: @zlib: @ruby: @csvt: @val: @file: @path: @env: @stdin: @preset:.
    Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'

ARGS
    Some commands require mandatory arguments, e.g. a path.

OPTIONS: global
        --table-style=VALUE          table display style, current=:.:
    -h, --help                       Show this message.
        --bash-comp                  generate bash completion for command
        --show-config                Display parameters used for the provided action.
    -r, --rest-debug                 more debug for HTTP calls
    -v, --version                    display version
        --display=ENUM               output only some information: info, data, error
        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
        --ui=ENUM                    method to start browser: text, graphical
        --log-level=ENUM             Log level: debug, info, warn, error, fatal, unknown
        --logger=ENUM                log method: stderr, stdout, syslog
        --format=ENUM                output format: table, ruby, json, jsonpp, yaml, csv, nagios
    -P, --presetVALUE                load the named option preset from current config file
        --fields=VALUE               comma separated list of fields, or ALL, or DEF
        --select=VALUE               select only some items in lists, extended value: hash (colum, value)
        --fasp-proxy=VALUE           URL of FASP proxy (dnat / dnats)
        --http-proxy=VALUE           URL of HTTP proxy (for http fallback)
        --lock-port=VALUE            prevent dual execution of a command, e.g. in cron
        --use-product=VALUE          which local product to use for ascp, current=FIRST
        --query=VALUE                additional filter for API calls (extended value) (some commands)
        --insecure=ENUM              do not validate HTTPS certificate: yes, no
        --flat-hash=ENUM             display hash values as additional keys: yes, no
        --once-only=ENUM             process only new items (some commands): yes, no
        --ts=VALUE                   override transfer spec values (Hash, use @json: prefix), current={}
        --to-folder=VALUE            destination folder for downloaded files
        --sources=VALUE              list of source files (see doc)
        --transfer=ENUM              type of transfer: direct, connect, node, aoc
        --transfer-info=VALUE        additional information for transfer client

COMMAND: shares
SUBCOMMANDS: repository, admin
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password

COMMAND: node
SUBCOMMANDS: nagios_check, events, space, info, mkdir, mklink, mkfile, rename, delete, browse, upload, download, postprocess, stream, transfer, cleanup, forward, access_key, watch_folder, service, async, central, asperabrowser
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --validator=VALUE            identifier of validator (optional for central)
        --asperabrowserurl=VALUE     URL for simple aspera web ui

COMMAND: orchestrator
SUBCOMMANDS: info, workflow, plugins, processes
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --params=VALUE               parameters hash table, use @json:{"param":"value"}
        --result=VALUE               specify result value as: 'work step:parameter'
        --synchronous=ENUM           work step:parameter expected as result: yes, no

COMMAND: aspera
SUBCOMMANDS: apiinfo, bearer_token, organization, user, workspace, packages, files, faspexgw, admin
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
COMMAND: ats
SUBCOMMANDS: cluster, access_key, api_key
OPTIONS:
        --ibm-api-key=VALUE          IBM API key, see https://console.bluemix.net/iam/#/apikeys
        --instance=VALUE             ATS instance in bluemix
        --ats-key=VALUE              ATS key identifier (ats_xxx)
        --ats-secret=VALUE           ATS key secret
        --params=VALUE               Parameters access key creation (@json:)
        --cloud=VALUE                Cloud provider
        --region=VALUE               Cloud region
        --auth=ENUM                  type of Oauth authentication: body_userpass, header_userpass, web, jwt, url_token
        --operation=ENUM             client operation for transfers: push, pull
        --client-id=VALUE            API client identifier in application
        --client-secret=VALUE        API client passcode
        --redirect-uri=VALUE         API client redirect URI
        --private-key=VALUE          RSA private key PEM value for JWT (prefix file path with @val:@file:)
        --workspace=VALUE            name of workspace
        --secret=VALUE               access key secret for node
        --eid=VALUE                  identifier
        --name=VALUE                 resource name
        --link=VALUE                 link to shared resource
        --public-token=VALUE         token value of public link
        --new-user-option=VALUE      new user creation option
        --from-folder=VALUE          share to share source folder
        --scope=VALUE                scope for AoC API calls
        --bulk=ENUM                  bulk operation: yes, no

COMMAND: xnode
SUBCOMMANDS: postprocess, cleanup, forward
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --filter-transfer=VALUE      Ruby expression for filter at transfer level (cleanup)
        --filter-file=VALUE          Ruby expression for filter at file level (cleanup)

COMMAND: ats
SUBCOMMANDS: cluster, access_key, api_key
OPTIONS:
        --secret=VALUE               Access key secret
        --ibm-api-key=VALUE          IBM API key, see https://console.bluemix.net/iam/#/apikeys
        --instance=VALUE             ATS instance in bluemix
        --ats-key=VALUE              ATS key identifier (ats_xxx)
        --ats-secret=VALUE           ATS key secret
        --params=VALUE               Parameters access key creation (@json:)
        --cloud=VALUE                Cloud provider
        --region=VALUE               Cloud region

COMMAND: client
SUBCOMMANDS: current, available, connect
OPTIONS:

COMMAND: faspex
SUBCOMMANDS: nagios_check, package, source, me, dropbox, recv_publink, v4, address_book, login_methods
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --delivery-info=VALUE        package delivery information (extended value)
        --source-name=VALUE          create package from remote source (by name)
        --storage=VALUE              Faspex local storage definition
        --box=ENUM                   package box: inbox, sent, archive

COMMAND: shares2
SUBCOMMANDS: repository, organization, project, team, share, appinfo, userinfo, admin
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --organization=VALUE         organization
        --project=VALUE              project
        --share=VALUE                share

COMMAND: preview
SUBCOMMANDS: scan, events, folder, check, test
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --file-access=ENUM           how to read and write files in repository: local, remote
        --skip-types=VALUE           skip types in comma separated list
        --skip-format=ENUM           skip this preview format (multiple possible): png, mp4
        --overwrite=ENUM             when to generate preview file: always, never, mtime
        --previews-folder=VALUE      preview folder in files
        --iteration-file=VALUE       path to iteration memory file
        --folder-reset-cache=ENUM    reset folder cache: no, header, read
        --temp-folder=VALUE          path to temp folder
        --skip-folders=VALUE         list of folder to skip
        --video=ENUM                 method to generate video: reencode, clips, preview
        --vid-offset-seconds=VALUE   generation parameter
        --vid-size=VALUE             generation parameter
        --vid-framecount=VALUE       generation parameter
        --vid-blendframes=VALUE      generation parameter
        --vid-framepause=VALUE       generation parameter
        --vid-fps=VALUE              generation parameter
        --vid-mp4-size-reencode=VALUE
                                     generation parameter
        --clips-offset-seconds=VALUE generation parameter
        --clips-size=VALUE           generation parameter
        --clips-length=VALUE         generation parameter
        --clips-count=VALUE          generation parameter
        --thumb-mp4-size=VALUE       generation parameter
        --thumb-img-size=VALUE       generation parameter
        --thumb-offset-fraction=VALUE
                                     generation parameter
        --validate-mime=ENUM         use magic number validation: yes, no
        --check-extension=ENUM       check extra file extensions: yes, no

COMMAND: sync
SUBCOMMANDS: start, admin
OPTIONS:
        --parameters=VALUE           extended value for session set definition
        --session-name=VALUE         name of session to use for admin commands, by default first one

COMMAND: server
SUBCOMMANDS: nagios, nodeadmin, userdata, configurator, ctl, download, upload, browse, delete, rename, ls, rm, mv, du, info, mkdir, cp, df, md5sum
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --ssh-keys=VALUE             one ssh key at a time

COMMAND: console
SUBCOMMANDS: transfer, nagios_check
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --filter-from=DATE           only after date
        --filter-to=DATE             only before date

Documentation : http://www.rubydoc.info/gems/asperalm


Note that actions and parameter values can be written in short form.

Application Plugins

mlia comes with several Aspera application plugins.

General: Application URL and Authentication

REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.

Those are using options:

  • url
  • username
  • password

Those can be provided using command line, parameter set, env var, see section above.

Aspera on Cloud relies on Oauth, refer to the Aspera on Cloud section.

Aspera on Cloud

Aspera on Cloud uses the more advanced Oauth mechanism for authentication (HTTP Basic authentication is not supported). This requires additional setup.

Configuration Wizard

mlia provides a configuration wizard, to invoke it do:

$ mlia config wizard --url=https://sedemo.ibmaspera.com

If the url parameter is not provided it will be asked on command line.

Follow instructions on terminal and interact in browser.

Configuration details

Several types of OAuth authentication are supported:

  • Web based authentication : authentication is made by user using a browser (simpler)
  • JSON Web Token (JWT) : authentication is secured by a private key (recommended)
  • URL Token : external users authentication with url tokens (external links)

The authentication method is controled by option auth.

For a quick start, follow the mandatory and sufficient section: API Client Registration (auth=web) as well as option preset for Aspera on Cloud.

For a more convenient, browser-less, experience follow the JWT section (auth=jwt) in addition to Client Registration.

In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.mlia saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.

API Client Registration

The first step is to declare mlia in Aspera on Cloud using the admin interface.

(official documentation: https://aspera.asperafiles.com/helpcenter/admin/organization/registering-an-api-client ).

Let's start by a registration with web based authentication (auth=web):

  • Open a web browser, log to your instance: e.g. https://myorg.ibmaspera.com/
  • Go to Apps→Admin→Organization→Integrations
  • Click "Create New"
    • Client Name: mlia
    • Redirect URIs: http://localhost:12345
    • Origins: localhost
    • uncheck "Prompt users to allow client to access"
    • leave the JWT part for now
  • Save

Note: for web based authentication, mlia listens on a local port (e.g. specified by the redirect_uri, in this example: 12345), and the browser will provide the OAuth code there. For `mlia, HTTP is required, and 12345 is the default port.

Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.

option preset for Aspera on Cloud

It is convenient to save several of those parameters in an option preset for mlia in its configuration file. Lets create an option preset called: my_aoc_org using ask interactive input (client info from previous step):

$ mlia config id my_aoc_org ask url client_id client_secret
option: url> https://myorg.ibmaspera.com/
option: client_id> BJLPObQiFw
option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
updated: my_aoc_org

(This can also be done in one line using the command config id my_aoc_org update --url=...)

Define this option preset as default configuration for the aspera plugin:

$ mlia config id default set aspera my_aoc_org

Note: Default auth method is web and default redirect_uri is http://localhost:12345. Leave those default values.

First Use

Once client has been registered and option preset created: mlia can be used:

$ mlia aspera files br /
Current Workspace: Default Workspace (default)
empty

Note that it requires a web based authentication. Refer to section Graphical Interactions to customize the way browser is started.

For direct browser-less authentication, follow the JWT section.

Activation of JSON Web Token (JWT) for direct authentication

In addition to basic API Client registration, the following steps are required for a Browser-less, Private Key-based authentication.

Key Pair Generation

In order to use JWT for Aspera on Cloud API client authentication, a private/public key pair must be generated (without passphrase) This can be done using any of the following method:

(TODO: add passphrase protection as option).

  • using the CLI:
$ mlia config genkey ~/.aspera/mlia/aocapikey
  • ssh-keygen:
$ ssh-keygen -t rsa -f ~/.aspera/mlia/aocapikey -N ''
  • openssl

(on some openssl implementation (mac) there is option: -nodes (no DES))

$ APIKEY=~/.aspera/mlia/aocapikey
$ openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
$ openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
$ openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
$ rm -f ${APIKEY}.protected

API Client JWT activation

JWT needs to be authorized in Aspera on Cloud. This can be done in two manners:

Graphically
  • Open a web browser, log to your instance: https://myorg.ibmaspera.com/
  • Go to Apps→Admin→Organization→Integrations
  • Click on the previously created application
  • select tab : "JSON Web Token Auth"
  • Modify options if necessary, for instance: activate both options in section "Settings"
  • Click "Save"
Using command line
$ mlia aspera admin res client list
:............:.........:
:     id     :  name   :
:............:.........:
: BJLPObQiFw : mlia :
:............:.........:
$ mlia aspera admin res client --id=BJLPObQiFw modify @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'
modified

User key registration

The public key must be assigned to your user. This can be done in two manners:

Graphically

open the previously generated public key located here: $HOME/.aspera/mlia/aocapikey.pub

  • Open a web browser, log to your instance: https://myorg.ibmaspera.com/
  • Click on the user's icon (top right)
  • Select "Account Settings"
  • Paste the Public Key in the "Public Key" section
  • Click on "Submit"
Using command line
$ mlia aspera admin res user list
:........:................:
:   id   :      name      :
:........:................:
: 109952 : Tech Support   :
: 109951 : LAURENT MARTIN :
:........:................:
$ mlia aspera user info modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/mlia/aocapikey.pub"))}'   
modified

Note: the aspera user info show command can be used to verify modifications.

option preset modification for JWT

To activate default use of JWT authentication for mlia using the option preset, do the folowing:

  • change auth method to JWT
  • provide location of private key
  • provide username to login as (OAuthg "subject")

Execute:

$ mlia config id my_aoc_org update --auth=jwt [email protected]:@file:~/.aspera/mlia/aocapikey [email protected]

Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.

After this last step, commands do not require web login anymore.

Administration

The admin command allows several administrative tasks (and require admin privilege).

It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the admin resource command.

Bulk operations are possible using option bulk (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the Extended Value Syntax.

To get more resources when doing request add:

[email protected]:'{"per_page":10000}'

other query parameters can be used:

[email protected]:'{"member_of_any_workspace":true,}'
[email protected]:'{"q":"laurent"}'

Refer to the AoC API for full list of query parameters.

Examples

  • Bulk creation
$ mlia aspera admin res user create --bulk=yes @json:'[{"email":"[email protected]"},{"email":"[email protected]"}]'
:.......:.........:
:  id   : status  :
:.......:.........:
: 98398 : created :
: 98399 : created :
:.......:.........:
  • Find with filter and delete
$ mlia aspera admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
:.......:........................:
:  id   :         email          :
:.......:........................:
: 98398 : [email protected] :
: 98399 : [email protected] :
:.......:........................:
$ thelist=$(echo $(mlia aspera admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email --field=id --format=csv)|tr ' ' ,)
$ echo $thelist
98398,98399
$ mlia aspera admin res user --bulk=yes [email protected]:[$thelist] delete
:.......:.........:
:  id   : status  :
:.......:.........:
: 98398 : deleted :
: 98399 : deleted :
:.......:.........:
  • Display current user's workspaces
$ mlia aspera user workspaces
:......:............................:
:  id  :            name            :
:......:............................:
: 16   : Engineering                :
: 17   : Marketing                  :
: 18   : Sales                      :
:......:............................:
  • Create a sub access key in a "node"

Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)

$ mlia aspera admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create [email protected]:'{"storage":{"path":"/folder1"}}'
  • Display transfer events (ops/transfer)
$ mlia aspera admin res node --secret=_secret_ v3 transfer list [email protected]:'[["q","*"],["count",5]]'
          # page=1&per_page=10&q=type:(file_upload+OR+file_delete+OR+file_download+OR+file_rename+OR+folder_create+OR+folder_delete+OR+folder_share+OR+folder_share_via_public_link)&sort=-date
          #[email protected]_files.read('events',{'q'=>'type:(file_upload OR file_download)'})[:data]
          # can add filters: tag=aspera.files.package_id%3DLA8OU3p8w
          #'tag'=>'aspera.files.package_id%3DJvbl0w-5A'
          # filter= 'id', 'short_summary', or 'summary'
          # count=nnn
          # tag=x.y.z%3Dvalue
          # iteration_token=nnn
          # after_time=2016-05-01T23:53:09Z
          # active_only=true|false
  • Display node events (events)
$ mlia aspera admin res node --secret=_secret_ v3 events
  • display members of a workspace
$ mlia aspera admin res workspace_membership list --fields=member_type,manager,member.email [email protected]:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
:.............:.........:..................................:
: member_type : manager :           member.email           :
:.............:.........:..................................:
: user        : true    : [email protected]            :
: user        : false   : [email protected] :
: user        : false   : [email protected]               :
: user        : false   : [email protected]         :
: group       : false   :                                  :
: user        : false   : [email protected]            :
:.............:.........:..................................:
  • get users who did not log since a date
$ mlia aspera admin res user list --fields=email [email protected]:'{"per_page":10000,"q":"last_login_at:<2018-05-28"}' 
:...............................:
:             email             :
:...............................:
: [email protected]          :
: [email protected]      :
:...............................:
  • list "Limited" users
$ mlia aspera admin res user list --fields=email [email protected]:'{"per_page":10000}' [email protected]:'{"member_of_any_workspace":false}'

Send a Package

Send a package:

$ mlia aspera packages send [email protected]:'{"name":"my title","note":"my note","recipients":["[email protected]","[email protected]"]}' [email protected] my_file.dat

Notes:

  • the value parameter can contain any supported package creation parameter. Refer to the API, or display an existing package.
  • to list recipients use fields: "recipients" and/or "bcc_recipients". mlia will resolve the list of email addresses to expected user ids. If a recipient is not already registered and the workspace allows external users, then the package is sent to an external user, and
    • if the option new_user_option is @json:{"package_contact":true} (default), then a public link is sent and the external user does not need to create an account.
    • if the option new_user_option is @json:{}, then external users are invited to join the workspace

Receive only new packages

Download Files

Download of files is straightforward with a specific syntax for the aspera files download action: Like other commands the source file list is provided as a list with the sources option. Nevertheless, consider this:

  • if only one source is provided, it is downloaded
  • if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).

Aspera Node (Transfer Server)

Simple Operations

It is possible to:

  • browse
  • transfer (upload / download)
  • ...

Central

The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transfered files.

Filtering can be applied:

$ mlia node central file list

by providing the validator option, offline transfer validation can be done.

FASP Stream

It is possible to start a FASPStream session using the node API:

Use the "node stream create" command, then arguments are provided as a transfer-spec.

$ mlia node stream create [email protected]:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --preset=stream

Watchfolder

Refer to Aspera documentation for watch folder creation.

mlia supports remote operations through the node API. Operations are:

  • Start watchd and watchfolderd services running as a system user having access to files
  • configure a watchfolder to define automated transfers
$ mlia node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
$ mlia node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
$ mlia node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'

Out of Transfer File Validation

Follow the Aspera Transfer Server configuration to activate this feature.

$ mlia node central file list --validator=mlia [email protected]:'{"file_transfer_filter":{"max_result":1}}'
:..............:..............:............:......................................:
: session_uuid :    file_id   :   status   :              path                    :
:..............:..............:............:......................................:
: 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
:..............:..............:............:......................................:
$ mlia node central file update --validator=mlia [email protected]:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
updated

Example: SHOD to ATS

Access to a "Shares on Demand" (SHOD) server on AWS is provided by a partner. And we need to transfer files from this third party SHOD instance into our Azure BLOB storage. Simply create an "Aspera Transfer Service" instance (https://ts.asperasoft.com), which provides access to the node API. Then create a configuration for the "SHOD" instance in the configuration file: in section "shares", a configuration named: awsshod. Create another configuration for the Azure ATS instance: in section "node", named azureats. Then execute the following command:

$ mlia node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node [email protected]:azureats

This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.

Create access key

$ mlia node access_key create [email protected]:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'

Client

The client plugin refers to the use of a local FASP client. It provides the following commands:

  • current : list current resources used for transfers
  • available : list Aspera transfer products available locally
  • connect : list,download connect client versions available on internet

List installed clients

Locally installed Aspera products can be listed with:

$ mlia client available
:..........................:................................................:
:           name           :                    app_root                    :
:..........................:................................................:
: Connect Client           : /Users/laurent/Applications/Aspera Connect.app :
: Aspera Enterprise Server : /Library/Aspera                                :
:..........................:................................................:

Selection of local client

By default, the special value FIRST is used and will select the first product in list. To select another product use option: use_product, either on command line: --use-product='Aspera Enterprise Server', or by setting as default:

$ mlia config id cli_default set use_product 'Aspera Enterprise Server'
updated: cli_default: use_product <- Aspera Enterprise Server
$ mlia config id default set config cli_default
updated: default: config <- cli_default
$ mlia client current
:........................:.......................................................................................:
:          name          :                                         path                                          :
:........................:.......................................................................................:
: bin_folder             : /Library/Aspera/bin                                                                   :
: ascp                   : /Library/Aspera/bin/ascp                                                              :
: ascp4                  : /Library/Aspera/bin/ascp4                                                             :
: ssh_bypass_key_dsa     : /Library/Aspera/var/aspera_tokenauth_id_dsa                                           :
: ssh_bypass_key_rsa     : /Library/Aspera/var/aspera_tokenauth_id_rsa                                           :
: fallback_cert          : /Library/Aspera/var/aspera_web_cert.pem                                               :
: fallback_key           : /Library/Aspera/var/aspera_web_key.pem                                                :
: plugin_https_port_file : /Users/laurent/Library/Application Support/Aspera/Enterprise Server/var/run/https.uri :
: log_folder             : /Users/laurent/Library/Logs/Aspera                                                    :
:........................:.......................................................................................:

List current resources used

$ mlia client current
:........................:............................................................................................:
:          name          :                                            path                                            :
:........................:............................................................................................:
: bin_folder             : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources                          :
: ssh_bypass_key_dsa     : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/asperaweb_id_dsa.openssh :
: ssh_bypass_key_rsa     : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/aspera_tokenauth_id_rsa  :
: fallback_cert          : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/aspera_web_cert.pem      :
: fallback_key           : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/aspera_web_key.pem       :
: localhost_cert         : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/localhost.crt            :
: localhost_key          : /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/localhost.key            :
: plugin_https_port_file : /Users/laurent/Library/Application Support/Aspera/Aspera Connect/var/run/https.uri         :
: log_folder             : /Users/laurent/Library/Logs/Aspera                                                         :
:........................:............................................................................................:

Installation of Connect Client on command line

$ mlia client connect list
:...............................................:......................................:..............:
:                      id                       :                title                 :   version    :
:...............................................:......................................:..............:
: urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E : Aspera Connect for Windows           : 3.7.0.138427 :
: urn:uuid:A3820D20-083E-11E2-892E-0800200C9A66 : Aspera Connect for Windows 64-bit    : 3.7.0.138427 :
: urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E : Aspera Connect for Windows XP        : 3.7.0.138427 :
: urn:uuid:55425020-083E-11E2-892E-0800200C9A66 : Aspera Connect for Windows XP 64-bit : 3.7.0.138427 :
: urn:uuid:D8629AD2-6898-4811-A46F-2AF386531BFF : Aspera Connect for Mac Intel 10.6    : 3.6.1.111259 :
: urn:uuid:D8629AD2-6898-4811-A46F-2AF386531BFF : Aspera Connect for Mac Intel         : 3.7.0.138427 :
: urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32          : 3.6.2.117442 :
: urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64          : 3.7.2.141527 :
:...............................................:......................................:..............:
$ mlia client connect id 'Aspera Connect for Mac Intel 10.6' links list
:.............................................:..........................:.......................................................................:..........:...............:
:                    title                    :           type           :                                 href                                  : hreflang :      rel      :
:.............................................:..........................:.......................................................................:..........:...............:
: Mac Intel Installer                         : application/octet-stream : bin/AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg                     : en       : enclosure     :
: Aspera Connect for Mac HTML Documentation   : text/html                :                                                                       : en       : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/ja-jp/pdf/Connect_User_3.7.0_OSX_ja-jp.pdf              : ja-jp    : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/en/pdf/Connect_User_3.7.0_OSX.pdf                       : en       : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/es-es/pdf/Connect_User_3.7.0_OSX_es-es.pdf              : es-es    : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/fr-fr/pdf/Connect_User_3.7.0_OSX_fr-fr.pdf              : fr-fr    : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf              : zh-cn    : documentation :
: Aspera Connect for Mac Release Notes        : text/html                : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en       : release-notes :
:.............................................:..........................:.......................................................................:..........:...............:
$ mlia client connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg

IBM Aspera High Speed Transfer Server

Works at FASP level (SSH/ascp/ascmd). (different from node api)

Example

One can test the "server" application using the well known demo server:

$ mlia config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
$ mlia config id default set server aspera_demo_server 
$ mlia server browse /aspera-test-dir-large
$ mlia server download /aspera-test-dir-large/200MB

This creates a option preset "aspera_demo_server" and set it as default for application "server"

IBM Aspera Faspex

Note that the command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.

Sending a Package

Provide delivery info in JSON, example:

[email protected]:'{"title":"my title","recipients":["[email protected]"]}'

a note can be added: "note":"Please ..."

metadata: "metadata":{"Meta1":"Val1","Meta2":"Val2"}

Note for full details, refer to: Reference on Developer Site

operation on dropboxes

Example:

$ mlia faspex v4 dropbox create [email protected]:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
$ mlia faspex v4 dropbox list
$ mlia faspex v4 dropbox delete --id=36

remote sources

Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this, the node API is used, for this it is required to add a section ":storage" that links a storage name to a node config and sub path.

Example:

my_faspex_conf:
  url: https://10.25.0.3/aspera/faspex
  username: admin
  password: MyPassword
  storage:
    testlaurent:
      node: "@preset:my_faspex_node"
      path: /myfiles
my_faspex_node:
  url: https://10.25.0.3:9092
  username: node_faspex
  password: MyPassword

In this example, a faspex storage named "testlaurent" exists in Faspex, and is located under the docroot in "/myfiles" (this must be the same as configured in Faspex). The node configuration name is "my_faspex_node" here.

Note: the v4 API provide an API for nodes and shares.

IBM Aspera Shares

Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and 2)

Aspera Transfer Service

ATS is usable either :

  • from an AoC subscription : mlia aspera admin ats

  • or from an IBM Cloud (bluemix) subscription : mlia ats

IBM Cloud ATS : creation of api key

First get your IBM Cloud APIkey, as described here: https://console.bluemix.net/docs/iam/userid_keys.html#userapikey

Execute:

$ mlia config id my_ibm_ats update --ibm-api-key=XXXX
$ mlia config id default set ats my_ibm_ats
$ mlia ats api_key instances
:......................................:
:               instance               :
:......................................:
: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee :
:......................................:
$ mlia config id my_ibm_ats update --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
$ mlia ats api_key create --ibm-api-key=XXXX --instance
:......................................:
:               instance               :
:......................................:
: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee :
:......................................:
$ mlia ats api_key create
:........:..............................................:
:  key   :                    value                     :
:........:..............................................:
: id     : ats_XXXXXXXXXXXXXXXXXXXXXXXX                 :
: secret : YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY :
:........:..............................................:
$ mlia config id my_ibm_ats update --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

Note: access key API is described here: https://ibm.ibmaspera.com/helpcenter/transfer-service

and here: https://developer.asperasoft.com/web/node/access-keys

Examples

Example: create access key on softlayer:

$ mlia ats access_key create --cloud=softlayer --region=ams [email protected]:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'

Example: create access key on AWS:

$ mlia ats access_key create --cloud=aws --region=eu-west-1 [email protected]:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'

Example: create access key on Azure SAS:

$ mlia ats access_key create --cloud=azure --region=eastus [email protected]:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'

(Note that the blob name is mandatory after server address and before parameters. and that parameter sr=c is mandatory.)

Example: create access key on Azure:

$ mlia ats access_key create --cloud=azure --region=eastus [email protected]:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'

delete all my access keys:

for k in $(mlia ats access_key list --field=id --format=csv);do mlia ats access_key id $k delete;done

IBM Aspera Sync

A basic plugin to start an "async" using mlia. The main advantage is the possibility to start from ma configuration file, using mlia standard options.

Preview

The preview plugin provides generation of previews for Aspera on Cloud.

The tool requires the following external tools:

  • ImageMagick : convert composite
  • OptiPNG : optipng
  • FFmpeg : ffmpeg ffprobe
  • Libreoffice : libreoffice

Preview Command

The preview plugin allows generation of preview files for Aspera on Cloud for on-premise nodes. (thumbnails and video previews)

The preview generator creates/updates a preview for files located on an access key main "storage root". Several candidate detection methods are supported.

This version requires ES 3.7.4+

This is related to:

https://aspera.asperafiles.com/helpcenter/admin/organization/installing-files-preview-maker

Configuration

Like any mlia commands, parameters can be passed on command line or using a configuration option preset. Example using a option preset:

$ mlia config id my_aoc_access_key update --url=https://localhost:9092 --username=my_access_key --password=my_secret
$ mlia config id default set preview my_aoc_access_key

Once can check if the access key is well configured using:

$ mlia -Pmy_aoc_access_key node browse /

Execution

The tool intentionally supports only a "one shot" mode. It needs to be run regularly to create or update preview files. It does not implement a loop. Running preview generation on a regular basis shall be done using the operating system scheduler (e.g. Cron on Linux, or Task Scheduler on Windows). Refer to section Scheduling.

Candidate detection

The tool will find candidates for preview generation using three commands:

  • events : only recently uploaded files will be tested
  • scan : deeply scan all files under the access key's "storage root"
  • folder : same as scan, but only on the specified folder's "file identifier"

Creation/Update

Once candidate are selected, once candidates are selected, a preview is always generated if it does not exist already, else if a preview already exist, it will be generated using one of three overwrite method:

  • always : preview is always generated, even if it already exists and is newer than original
  • never : preview is generated only if it does not exist already
  • mtime : preview is generated only if the original file is newer than the existing

Access to original files and preview creation

Standard open source tools are used to create thumnails and video previews. Those tools require that original files are accessible in the local file system and also write generated files on the local file system. The tool provides 2 ways to read and write files with the parameter: --file-access

If the preview generator is run on a system that has direct access to the file system, then the value local can be used.

If the preview generator does not have access to files on the file system (it is remote, no mount, or is an object storage), then the original file is first downloaded, then the result is uploaded, use method remote.

Options

To skip some folders, use the option : --skip-folders, note that it expects a list of path starting with slash, use the @json: notation, example:

$ mlia preview scan [email protected]:'["/not_here"]'

Examples

on command line:

$ mlia preview event --skip-types=office --file-access=remote --overwrite=always --iteration-file=/tmp/restart.txt --lock-port=12346

with crontab:

2-59 * * * * su -s /bin/bash - xfer -c 'timeout 10m mlia preview event --skip-types=office --lock-port=12346 --log-level=info --logger=syslog --iteration-file=/tmp/preview_restart.txt'
0 * * * *    su -s /bin/bash - xfer -c 'timeout 30m mlia preview scan  --skip-types=office --lock-port=12346 --log-level=info --logger=syslog'

External tools: Linux

Here shown on Redhat/CentOS

  • Imagemagick and optipng:
yum install -y ImageMagick optipng
  • FFmpeg
pushd /tmp
wget http://johnvansickle.com/ffmpeg/releases/ffmpeg-release-64bit-static.tar.xz
mkdir -p /opt/
cd /opt/
tar xJvf /tmp/ffmpeg-release-64bit-static.tar.xz
ln -s ffmpeg-* ffmpeg
ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin
popd
  • Libreoffice
yum install libreoffice
  • Xvfb

Although libreoffice is run headless, older versions may require an X server. If you get error running libreoffice headless, then install Xvfb:

yum install libreoffice Xvfb
cat<<EOF>/etc/init.d/xvfb 
# !/bin/bash
# chkconfig: 345 95 50
# description: Starts xvfb on display 42 for headless Libreoffice
if [ -z "\$1" ]; then
  echo "\`basename \$0\` {start|stop}"
  exit
fi
case "\$1" in
start) /usr/bin/Xvfb :42 -screen 0 1280x1024x8 -extension RANDR&;;
stop) killall Xvfb;;
esac
EOF
chkconfig xvfb on
service xvfb start

Tool: asession

This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: asession.

It aims at simplifying the startup of a FASP session from a programmatic stand point as formating a transfer-spec is:

  • common to Aspera Node API (HTTP POST /ops/transfer)
  • common to Aspera Connect API (browser javascript startTransfer)
  • easy to generate by using any third party language specific JSON library

This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.

The tool expect one single argument: a transfer-spec.

If not argument is provided, it assumes a value of: @json:@stdin:, i.e. a JSON formated transfer-spec on stdin.

Note that if JSON is the format, one has to specify @json: to tell the tool to decode the hash using JSON.

During execution, it generates all low level events, one per line, in JSON format on stdout.

Comparison of interfaces

feature/toolasessionascpFaspManager
language integrationanyanyC/C++
C#/.net
Go
Python
java
additional components to ascpRuby
Asperalm
-library
(headers)
startupJSON on stdin
(standard APIs:
JSON.generate
Process.spawn)
command line argumentsAPI
eventsJSON on stdoutnone by default
or need to open management port
and proprietary text syntax
callback

Simple session

MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'

echo "${MY_TSPEC}"|asession

Asynchronous commands and Persistent session

asession also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: LongParameter -> long_parameter

This is particularly useful for a persistent session ( with the transfer spec parameter: "keepalive":true )

$ asession
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"demoaspera","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
{"type":"DONE"}

(events from FASP are not shown in above example. They would appear after each command)

Example of language wrapper

Nodejs: https://www.npmjs.com/package/asperalm

Help

$ asession -h
USAGE
    asession <zero or one argument>

    If no argument is provided, default will be used: @json:@stdin, else:
       -h or --help to display this message
       a JSON value for transfer_spec, using the prefix: @json:
    The value can be either:
       the JSON description itself, e.g. @json:'{"xx":"yy",...}'
       @json:@stdin, if the JSON is provided from stdin
       @json:@file:<path>, if the JSON is provided from a file
    Asynchronous commands can be provided on STDIN, examples:
       {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
       {"type":"START","source":"xx","destination":"yy"}
       {"type":"DONE"}
Note: debug information can be placed on STDERR, using the "EX_loglevel" parameter in transfer spec (debug=0)
EXAMPLES
    asession @json:'{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}'
    echo '{"remote_host":...}'|asession @json:@stdin

Hot folder

Requirements

mlia maybe used as a simple hot folder engine. A hot folder being defined as a tool that:

  • locally (or remotely) detects new files in a top folder
  • send detected files to a remote (respectively, local) repository
  • only sends new files, do not re-send already sent files
  • optionally: sends only files that are not still "growing"
  • optionally: after transfer of files, deletes or moves to an archive

In addition: the detection should be made "continuously" or on specific time/date.

Setup procedure

The general idea is to rely on :

  • existing ascp features for detection and transfer
  • take advantage of mlia configuration capabilities and server side knowledge
  • the OS scheduler for reliability and continuous operation

ascp features

Interesting ascp features are found in its arguments: (see ascp manual):

  • ascp already takes care of sending only "new" files: option -k 1,2,3, or transfer_spec: resume_policy
  • ascp has some options to remove or move files after transfer: --remove-after-transfer, --move-after-transfer, --remove-empty-directories
  • ascp has an option to send only files not modified since the last X seconds: --exclude-newer-than (--exclude-older-than)
  • --src-base if top level folder name shall not be created on destination

Note that:

  • mlia takes transfer parameters exclusively as a transfer_spec, with --ts parameter.
  • not all native ascp arguments are available as standard transfer_spec parameters
  • native ascp arguments can be provided with the transfer spec parameter: EX_ascp_args (array), only for the "local" transfer agent (not connect or node)

server side and configuration

Virtually any transfer on a "repository" on a regular basis might emulate a hot folder. Note that file detection is not based on events (inotify, etc...), but on a stateless scan on source side.

Note: parameters may be saved in a option preset and used with -P.

Scheduling

Once mlia parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc... Refer to section Scheduling.

Example

$ mlia server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 [email protected]:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'

The local (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they dont grow anymore (based ona 8 second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).

Module: Asperalm

Main components:

  • Asperalm generic classes for REST and OAuth
  • Asperalm::Fasp: starting and monitoring transfers. It can be considered as a FASPManager class for Ruby.
  • Asperalm::Cli: mlia.

A working example can be found in the gem, example:

$ mlia config gem_path
$ cat $(mlia config gem_path)/../examples/transfer.rb

This sample code shows some example of use of the API as well as REST API. Note: although nice, it's probably a good idea to use RestClient for REST.

History

When I joined Aspera, there was only one CLI: ascp, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the ascp tool, I thought of an extended tool : eascp.pl which was accepting all ascp options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).

There were a few pitfalls:

  • The tool was written in the aging perl language while most Aspera application products (but the Transfer Server) are written in ruby.
  • The tool was only for transfers, but not able to call other products APIs

So, it evolved into mlia:

  • portable: works on platforms supporting ruby (and ascp)
  • easy to install with the gem utility
  • supports transfers with multiple Transfer Agents, that's why transfer parameters moved from ascp command line to transfer-spec (more reliable , more standard)
  • ruby is consistent with other Aspera products

BUGS

This is best effort code without official support, dont expect full capabilities. This code is not supported by IBM/Aspera. You can contact the author for bugs or features.

If you get message:

OpenSSH keys only supported if ED25519 is available

This means that you do not have ruby support for ED25519 SSH keys. You may either install the suggested Gems, or remove your ed25519 key from your .ssh folder to solve the issue.

Release Notes

  • version 0.9.20

    • improved wizard (prepare for AoC global client id)
    • preview generator: addedoption : --skip-format=
    • removed outdated pictures from this doc
  • version 0.9.19

    • added command aspera bearer --scope=xx
  • version 0.9.18

    • enhanced aspera admin events to support query
  • version 0.9.16

    • AoC transfers are now reported in activity app
    • new interface for Rest class authentication (keep backward compatibility)
  • version 0.9.15

    • new feature: "find" command in aspera files
    • sample code for transfer API
  • version 0.9.12

    • add nagios commands
    • support of ATS for IBM Cloud, removed old version based on aspera id
  • version 0.9.11

    • Breaking change: @stdin is now @stdin:
    • support of ATS for IBM Cloud, removed old version based on aspera id
  • version 0.9.10

    • Breaking change: parameter transfer-node becomes more generic: transfer-info
    • Display SaaS storage usage with command: aspera admin res node --id=nn info
    • cleaner way of specifying source file list for transfers
    • Breaking change: replaced download_mode option with http_download action
  • version 0.9.9

    • Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
    • Now supports "cargo" for Aspera on Cloud (automatic package download)
  • version 0.9.8

    • Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
    • AoC: share to share transfer with command "transfer"
  • version 0.9.7

    • homogeneous transfer spec for node and local
    • preview persistency goes to unique file by default
    • catch mxf extension in preview as video
    • Faspex: possibility to download all paclages by specifying id=ALL
    • Faspex: to come: cargo-like function to download only new packages with id=NEW
  • version 0.9.6

    • Breaking change: @param:is now @preset: and is generic
    • AoC: added command to display current workspace information
  • version 0.9.5

    • new parameter: new_user_option used to choose between public_link and invite of external users.
    • fixed bug in wizard, and wizard uses now product detection
  • version 0.9.4

    • Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
    • AoC Package send supports external users
    • new command to export AoC config to Aspera CLI config
  • version 0.9.3

    • REST error message show host and code
    • option for quiet display
    • modified transfer interface and allow token re-generation on error
    • async add admin command
    • async add db parameters
    • Breaking change: new option "sources" to specify files to transfer
  • version 0.9.2

    • Breaking change: changed AoC package creation to match API, see AoC section
  • version 0.9.1

    • Breaking change: changed faspex package creation to match API, see Faspex section
  • version 0.9

    • Renamed the CLI from aslmcli to mlia
    • Automatic rename and conversion of former config folder from aslmcli to mlia
  • version 0.7.6

    • add "sync" plugin
  • version 0.7

    • Breaking change: AoC package recv take option if for package instead of argument.
    • Breaking change: Rest class and Oauth class changed init parameters
    • AoC: receive package from public link
    • select by col value on output
    • added rename (AoC, node)
  • Version 0.6.19

Breaking change:

  • ats server list provisioned -> ats cluster list
  • ats server list clouds -> ats cluster clouds
  • ats server list instance --cloud=x --region=y -> ats cluster show --cloud=x --region=y
  • ats server id xxx -> ats cluster show --id=xxx
  • ats subscriptions -> ats credential subscriptions
  • ats api_key repository list -> ats credential cache list
  • ats api_key list -> ats credential list
  • ats access_key id xxx -> ats access_key --id=xxx

    • Version 0.6.18

some commands take now --id option instead of id command.

  • Version 0.6.15
  • Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in option preset "default".

TODO

  • remove rest and oauth classes and use ruby standard gems:

  • use Thor or any standard Ruby CLI manager

  • provide metadata in packages

  • deliveries to dropboxes

  • Going through proxy: use env var http_proxy and https_proxy, no_proxy

Contribution

Send comments !

Create your own plugin !