Asperalm - Laurent's Aspera Command Line Interface and Ruby Library

Version : 0.6.8

Laurent/2016-2018

This gem provides a ruby API to Aspera transfers and a command line interface to Aspera Applications.

Disclaimers:

  • This GEM is not endorsed/supported by IBM/Aspera
  • Use at your risk (not in production environments)
  • This gem is provided as-is, and is not intended to be a complete CLI, or industry-grade product. This is a sample.
  • IBM provides an officially supported Aspera CLI: http://downloads.asperasoft.com/en/downloads/62 .
  • Aspera, FASP are owned by IBM

That being said, the aslmcli tool is very powerful and gets things done, it's also a great tool to learn Aspera APIs.

This manual addresses three parts:

  • aslmcli tool
  • asession tool
  • Asperalm ruby module

In examples, command line operations (starting with $) are shown using a standard shell: bash.

Quick Start

To quick start using the aslmcli tool, follow the section: Installation (Ruby, Gem, FASP).

Once the gem is installed, the aslmcli tool can be used right away:

$ aslmcli --version
x.y.z

Then, follow the section relative to the product you want to inbteract with: Files, Faspex, ...

Detailed generic information on configuration can be found in section: Tool: aslmcli.

Installation

In order to use the tool or the gem, it is necessary to install those components:

  • ruby
  • asperalm
  • fasp

The following sections provide information on the installation.

Ruby

A ruby interpreter is required to run the tool or to use the gem. It is also required to have privilege to install gems.

Ruby 2.4+ is prefered, but it should also work with 2.0+.

Refer to the following sections for specific operating systems.

macOS

Ruby comes pre-installed on macOS. Nevertheless, installion of new gems on pre-installed ruby requires admin privilege (sudo) and the version is not the latest.

It is better to install "homebrew", from here: https://brew.sh/, and then install Ruby:

$ brew install ruby

Windows

On windows you can get it from here: https://rubyinstaller.org/.

Linux

$ yum install ruby rubygems

Note that Ruby 2+ is required, if you have an older Linux (e.g. CentOS 6), you should install "rvm" https://rvm.io/ and install and use a newer Ruby.

asperalm gem

Once you have ruby and rights to install gems: Install the gem and its dependencies:

$ gem install asperalm

FASP

For any FASP based file transfer, the FASP protocol and a valid license is required (e.g. if the server side is "connect" enabled, one can use the connect client license).

aslmcli will detect most of Aspera transfer products in standard locations and use the first one found.

Aspera Connect Client can be installed by visiting the page: http://downloads.asperasoft.com/connect2/

It is also possible to use the ascp binary from Aspera CLI, or any other transfer product (High Speed Transfer Server, etc...).

The connect client can be downloaded on command line using aslmcli, see section: Client.

Tool: aslmcli

The asperalm Gem provides a Ruby language based command line interface (CLI) which interacts with (mostly APIs using REST calls) Aspera Products:

  • Server
  • Node
  • Shares
  • Faspex
  • Console
  • Orchestrator
  • Files
  • ATS
  • and more...

The CLI tool that provides the following features:

  • Supports most Aspera server products (on-premise and SaaS)
  • Command options can be provided on command line, in configuration file, in env var, in files (products URL, credentials or any option)
  • Commands, Option values and Parameters can be provided in short format (must be unique)
  • FASP transfer agent can be: FaspManager (local ascp), or Connect Client, or a transfer node
  • Transfer parameters can be altered by modification of transferspec, this includes requiring multi-session transfer on nodes
  • Allows transfers from products to products, essentially at node level
  • Supports FaspStream creation (using Node API)
  • Additional command plugins can be written by the user
  • Supports download of faspex "external" links
  • Supports "legacy" ssh based FASP transfers and remote commands (ascmd)
  • REST and OAuth classes for use with Aspera products APIs

Basic usage is displayed by executing:

$ aslmcli -h

Refer to sections: Usage and Sample Commands.

Not all aslmcli features are fully documented here, the user may explore commands on the command line.

Commands, Arguments and Options

Commands, Options and Arguments are normally provided on command line, i.e. aslmcli command --option-name=VAL1 VAL2 takes "VAL1" as the value for option option_name, and "VAL2" as the value for first argument of command: command.

Options are command line arguments starting with a - (Note that some options are mandatory). Other command line arguments are considered as Commands and Arguments. Most options take a value, but a limited number of them are without values.

The special option "--" stops option processing, so following values are taken as arguments, including the ones starting with a -.

The value of options and arguments can be either a plain text or be provided using the Extended Value Syntax.

When the value of a command, option or argument is constrained by a fixed list of values, it is possible to use the first letters of the value only, provided that it uniquely identifies a value. For example aslmcli conf ov is the same as aslmcli config overview.

Interactive Input

Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.

The behaviour can be controlled with:

  • --interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
    • yes : missing parameters/options are asked to the user
    • no : missing parameters/options raise an error message
  • --ask-options=<yes|no> (default=no)
    • optional parameters/options are asked to user

Extended Value Syntax

Usually, values are specified by a simple string. But sometime it is required to read a value from a file, or decode it, or have a value more complex than a string (e.g. a structure (hash table)).

The value of options and arguments can optionally use the following special prefixes:

  • @val:VALUE , prevents further special prefix processing, e.g. [email protected]:laurent sets the option username to value laurent.
  • @file:PATH , read value from a file (prefix "~/" is replaced with $HOME), e.g. [email protected]:$HOME/.ssh/mykey
  • @env:ENVVAR , read from a named env var, [email protected]:MYPASSVAR

TODO: @csvfile : read a csv file and create an array of hash, whose keys are on first line, and values in following lines.

In addition it is possible to decode some values by prefixing :

  • @base64: to decode base64 encoded string
  • @json: to decode JSON values (convenient to provide complex structures)
  • @zlib: to uncompress data
  • @ruby: to execute ruby code, for instance read values from files.

Example: read the content of the specified file, then, base64 decode, then unzip:

@zlib:@base64:@file:myfile.dat

Example: create a value as a hash, with one key and the value is read from a file.

@ruby:'{"token_verification_key"=>File.read("pubkey.txt")}' 

Environment variable starting with prefix: ASLMCLI_ are taken as option values, i.e. ASLMCLI_OPTION_NAME is for --option-name.

Precedence: (TODO) : conf file, command line, env var

Some options have default values (defined in the tool). All options can be defined in a configuration file (see Configuration file). Options values can be displays for a given command by providing the --show-config option.

Structured Value

Some options and parameters expect a Structured Value, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.

A Structured Value requires the use of

For instance, a transfer-spec is expected to be a Structured Value.

A convenient way to specify a Structured Value is to use the @json: modifier, and describe the value in JSON format. Note that the @ruby: modifier can also be used.

It is also possible to provide a Structured Value in a file using @json:@file:<path>

Configuration file

aslmcli keeps configuration and cache files in folder $HOME/.aspera/aslmcli.

A default configuration file is created on first use, but there is no mandatory information required in this file. It is mainly used to define pre-sets of command options.

All options can be set on command line, or by env vars, or in the configuratin file. A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always have to specify those parameters on the command line.

The default configuration file is: $HOME/.aspera/aslmcli/config.yaml (this can be overriden with option --config-file=path).

It is composed of option presets containing pre-sets for options.

Creation and Modification of option presets

The configuration file can be modified using the following commands:

aslmcli config id <option preset> set|delete|show|initialize|update

The command update allows the easy creation of option preset by simply providing the options in their command line format, e.g. :

aslmcli config id node_to_node update --url=https://10.25.0.4:9092 --username=node_user --password=node_pass [email protected]:'{"precalculate_job_size":true}' --transfer=node [email protected]:'{"url":"https://10.25.0.8:9092","username":"node_user2","password":"node_pass2"}'
  • This creates a option preset node_to_node with all provided options.

The command set allows setting individual options in a option preset.

The command initialize, like update allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a Structured Value.

A good practice is to not manually edit this file and use modification commands. If necessary, the configuration file can be edited (or simply consulted) with:

$ aslmcli config open

A full terminal based overview of the configuration can be displayed using:

$ aslmcli config over

Format

The configuration file is a hash in a YAML file. Example:

config:
  version: 0.3.7
  loglevel: debug
default:
  faspex: myfaspparams
myfaspparams:
  url: https://faspex.my.org/aspera/faspex
  username: admin
  password: MyPassword
  • The configuration was created with CLI version 0.3.7
  • the log level is set to debug
  • the default option preset to load for plugin "faspex" is : myfaspparams
  • the option preset "myfaspparams" defines some parameters: the URL and credentials

Configuation is organized in option presets, like in .ini files. Each group has a name contains option name-value pairs.

Two option presets are reserved:

  • config is reserved for the global parameters of the aslmcli tool. It contains a special parameter: "version" showing the CLI version used to create this file. It is used to check compatibility.
  • default is reserved to define the default option preset name used for plugins.

The user may create as many option presets as needed. For instance, a particular option preset can be created for a particular application instance and contain URL and credentials.

Values in the configuration also follow the Extended Value Syntax.

Note: if the user wants to use the Extended Value Syntax inside the configuration file, using the config id update command, the user shall use the @val: prefix. Example:

$ aslmcli config id my_files_org set private_key @val:@file:"$HOME/.aspera/aslmcli/filesapikey"

This creates the option preset:

...
my_files_org:
  private_key: @file:"/Users/laurent/.aspera/aslmcli/filesapikey"
...

So, the key file will be read only at execution time, but not be embedded in the configuration file.

Options are loaded using this algorithm:

  • if option '--load-params=xxxx' is specified (or -Pxxxx), this reads the option preset specified from the configuration file.
    • else if option --no-default is specified, then dont load default
    • else it looks for the name of the default option preset in section "default" and loads it
  • environment variables are evaluated
  • command line options are evaluated

Parameters are evaluated in the order of command line.

To avoid loading the default option preset for a plugin, just specify a non existing configuration: -Pnone

On command line, words in parameter names are separated by a dash, in configuration file, separator is an underscore. E.g. --transfer-name on command line gives transfer_node in configuration file.

Note: before version 0.4.5, some keys could be ruby symbols, from 0.4.5 all keys are strings. To convert olver versions, remove the leading ":" in fronty of keys.

Examples

For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console, only username/password and url are required (either on command line, or from config file). Those can usually be provided on the command line:

$ aslmcli shares repo browse / --url=https://10.25.0.6 --username=john --password=4sp3ra 

This can also be provisioned in a config file:

1$ aslmcli config id shares06 set url https://10.25.0.6
2$ aslmcli config id shares06 set username john
3$ aslmcli config id shares06 set password 4sp3ra
4$ aslmcli config id default set shares shares06 
5$ aslmcli config show
6$ aslmcli shares repo browse /

The three first commands build a option preset. Note that this can also be done with one single command:

$ aslmcli config id shares06 init @json:'{"url":"https://10.25.0.6","username":"john","password":"4sp3ra"}'

The fourth command defines this option preset as the default option preset for the specified application ("shares"). The 5th command dumps the configuration file. Alternative option presets can be used with option "-P<option preset>" (or --load-params=<option preset>)

Eventually, the last command shows a call to the shares application using default parameters.

Plugins

The CLI tool uses a plugin mechanism. The first level command (just after aslmcli on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represent commands sent to a specific application. For instance, the plugin "faspex" allows operations on the application "Aspera Faspex".

Create your own plugin

$ mkdir -p ~/.aspera/aslmcli/plugins
$ cat<<EOF>~/.aspera/aslmcli/plugins/test.rb
require 'asperalm/cli/plugin'
module Asperalm
  module Cli
    module Plugins
      class Test < Plugin
        def declare_options; end
        def action_list; [];end
        def execute_action; puts "Hello World!"; end
      end # Test
    end # Plugins
  end # Cli
end # Asperalm
EOF

Learning Aspera Product APIs (REST)

This CLI uses REST APIs. To display HTTP calls, use argument -r or --rest-debug, this is useful to display exact content or HTTP requests and responses.

In order to get traces of execution, use argument : --log-level=debug

Graphical Interactions: Browser and Text Editor

Some actions may require the use of a graphical tool:

  • a browser for Aspera Files authentication
  • a text editor for configuration file edition

By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable. It is also possible to force the graphical mode with option --gui-mode :

  • --gui-mode=graphical forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
  • --gui-mode=text forces a text environment, the URL or file path to open is displayed on terminal.

Transfer Agents

Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (ascp).

This transfer can be done using on of the 3 following methods:

  • direct for a local execution of ascp
  • connect to make use of a local Connect Client
  • node to make use of a remote Aspera Transfer Node.

aslmcli standadizes on the use of a transfer-spec instead of raw ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.

FASPManager API based (command line)

By default the CLI will use the Aspera Connect Client FASP part, in this case it requires the installation of the Aspera Connect Client to be able to execute FASP based transfers. The CLI will try to automatically locate the Aspera Protocol (ascp). This is option: --transfer=ascp. Note that parameters are always provided with a transfer-spec.

Aspera Connect Client GUI

By specifying option: --transfer=connect, the CLI will start transfers in the Aspera Connect Client.

Aspera Node API : Node to node transfers

By specifying option: --transfer=node, the CLI will start transfers in an Aspera Transfer Server using the Node API.

If a default node has been configured in the configuration file, then this node is used by default else the parameter --transfer-node is required. The node specification shall be a hash table with three keys: url, username and password, corresponding to the URL of the node API and associated credentials (node user or access key).

The --transfer-node parameter can directly specify a pre-configured option preset : [email protected]:<psetname> or specified using the option syntax : [email protected]:'{"url":"https://...","username":"theuser","password":"thepass"}'

Transfer Specification

Some commands lead to file transfer (upload/download), all parameters necessary for this transfer is described in a transfer-spec (Transfer Specification), such as:

  • server address
  • transfer user name
  • credentials
  • file list
  • etc...

aslmcli builds a default transfer-spec internally, so it is not necessary to provide additional parameters on the command line for this transfer.

If needed, it is possible to modify or add any of the supported transfer-spec parameter using the ts option. The ts option accepts a Structured Value containing one or several transfer-spec parameters.

It is possible to specify ascp options when the transfer option is set to direct using the special transfer-spec parameter: EX_ascp_args. Example: [email protected]:'{"EX_ascp_args":["-l","100m"]}'.

The use of a transfer-spec instead of ascp parameters has the advantage of:

  • common to all Transfer Agent
  • not dependant on command line limitations (special characters...)

A transfer-spec is a Hash table, so it is described on the command line with the Extended Value Syntax. Keys are described in section Transfer Parameters.

Transfer Parameters

All standard transfer-spec parameter can be overloaded. To display parameter, run in debug mode (--log-level=debug). transfer-spec can also be saved/overridden in the config file.

(UNDER CONSTRUCTION ref)

  • F=Fasp Manager(local FASP execution)
  • N=remote node(node API)
  • C=Connect Client(web plugin)

Req/Def : Required or default value (- means emty)

Fields with EX_ prefix are specific to aslmcli in local mode.

arg: related ascp argument or env var suffix (PASS for ASPERA_SCP_PASS)

FieldReq/DefTypeFNCargDescription
directionRequiredstringYYY--modeDirection: "send" or "receive"
remote_hostRequiredstringYYY--hostIP or fully qualified domain name of the remote server
remote_userRequiredstringYYY--userRemote user. Default value is "xfer" on node or connect.
destination_rootRequiredstringYYYlast argDestination root directory.
title-stringNYY-Title of the transfer
tags-hashYYY--tags
--tags64
Metadata for transfer
token-stringYYYTOKEN
-W
Authorization token: Bearer, Basic or ATM
cookie-stringYYYCOOKIEMetadata for transfer (older,string)
remote_access_keyTODOstring?Node only?
source_root-string--source-prefix
--source-prefix64
Source root directory.(TODO: verify option)
fasp_port33001integerYYY-OSpecifies fasp (UDP) port.
ssh_port22 or 33001integerYYY-PSpecifies ssh (TCP) port.
rate_policyserver configstringYYY--policyValid literals include "low","fair","high" and "fixed".
symlink_policyfollowstringYYY--symbolic-linkscopy, follow, copy+force, skip. Default is follow. Handle source side symbolic links by following the link (follow), copying the link itself (copy), skipping (skip), or forcibly copying the link itself (copy+force).
target_rate_kbps-integerYYY-lSpecifies desired speed for the transfer.
min_rate_kbps0integerYYY-mSet the minimum transfer rate in kilobits per second.
ciphernonestringYYY-cin transit encryption type.
none, aes-128, aes-256
content_protectionencrypt
decrypt
string--file-crypt=encryption at rest
content_protection_password-stringPASSSpecifies a string password.
overwritediffstringYYY--overwriteOverwrite destination files with the source files of the same name.
never, always, diff, older, or diff+older
retry_durationstringTODOSpecifies how long to wait before retrying transfer. (e.g. "5min")
http_fallbackbool (node), integer-y
TODO
When true(1), attempts to perform an HTTP transfer if a fasp transfer cannot be performed.
create_dirbooleanYYY-dSpecifies whether to create new directories.
precalculate_job_sizesrv. def.booleanYYY--precalculate-job-sizeSpecifies whether to precalculate the job size.
delete_sourcebooleanY??
remove_after_transferbooleanY?Specifies whether to remove file after transfer.
remove_empty_directoriesbooleanY?Specifies whether to remove empty directories.
multi_session1integerNYN-CSpecifies how many parts the transfer is in.
multi_session_thresholdnullintegerNYN-in bytes
dgram_sizeintegerY-Zin bytes
compressionintegerascp4 only, 0 / 1?
read_threadsinteger-ascp4 only
write_threadsinteger-ascp4 only
use_ascp4falsebooleanY-specify version of protocol
pathssource files (dest)arraypositional
--file-list
--file-pair-list
Contains a path to the source (required) and a path to the destination.
http_fallback_portintegerY-tSpecifies http port.
https_fallback_portintegertodoSpecifies https port.
cipher_allowedstring-returned by node API. Valid literals include "aes-128" and "none".
target_rate_cap_kbpsN??-Returned by upload/download_setup node api.
rate_policy_allowed-returned by node API. Specifies most aggressive rate policy that is allowed. Valid literals include "low", "fair","high" and "fixed".
ssh_private_keystring-todo
remote_password-stringYYYPASSSSH session password
resume_policyfaspmgr:
none
other:
sparse_csum
stringYYY-knone,attrs,sparse_csum,full_csum
authenticationfaspmgr:
none
other:
sparse_csum
stringYYN-token: Aspera web keys are provided to allow transparent web based session initiation. on connect: password is not asked. Else, password is asked, and keys are not provided.
EX_ssh_key_value-stringYNNKEYPrivate key used for SSH authentication
EX_ssh_key_paths-arrayYNN-iUse public key authentication and specify the private key file
EX_fallback_key-stringYNN-YThe HTTPS transfer's key file name
EX_fallback_cert-stringYNN-IThe HTTPS certificate's file name
EX_at_rest_password-stringYNNFILEPASSPassphrase used for at rest encryption or decryption
EX_proxy_password-stringYNNPROXY_PASSTODO
EX_quiet-booleanYNN-qQuiet flag, disable progress display
EX_fasp_proxy_url-stringYNN--proxySpecify the address of the Aspera high-speed proxy server
EX_http_proxy_url-stringYNN-xSpecify the proxy server address used by HTTP Fallback
EX_ascp_args-arrayYNNsameAdd command line arguments to ascp
EX_http_transfer_jpeg0integerYNN-jHTTP transfers as JPEG file

Destination folder for transfers

The destination folder is set by aslmcli by default to:

  • . for downloads
  • / for uploads

It is specified by the transfer-spec parameter destination_root. As such, it can be modified with option: [email protected]:'{"destination_root":"<path>"}'. The option to_folder provides a convenient way to change this parameter: --to-folder=<path> and is equivalent.

Examples

  • Change target rate
[email protected]:'{"target_rate_kbps":500000}'
  • Override the FASP SSH port to a specific TCP port:
[email protected]:'{"ssh_port":12345}'
  • Force http fallback mode:
[email protected]:'{"http_fallback":"force"}'
  • Use multi-session transfer (when agent=node)
[email protected]:'{"multi_session":10,"multi_session_threshold":1}'

Exclusive execution

If it is required to run a command as a single instance, it is possible to protect with parameter: --lock-port=nnnn.

This opens a local TCP server port, and fails if this port is already used, providing a local lock.

This option is used when the tools is executed automatically, for instance with "preview" generatin.

Sample Commands

A non complete list of commands used in unit tests:

aslmcli
aslmcli --no-default node --url=my_url_here --username=my_username_here --password=my_password_here delete /500M.dat
aslmcli --no-default node --url=my_url_here --username=my_username_here --password=my_password_here upload --to-folder=/ 500M.dat [email protected]:'{"precalculate_job_size":true}' --transfer=node [email protected]:'{"url":"my_url_here","username":"my_username_here","password":"my_password_here"}' 
aslmcli --version)
aslmcli ats access_key create --cloud=aws --region=eu-west-1 [email protected]:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"sedemo-ireland","credentials":{"access_key_id":"AKIAIDSWKOSIM7XUVCJA","secret_access_key":"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"},"path":"/laurent"}}'
aslmcli ats access_key create --cloud=softlayer --region=ams [email protected]:'{"id":"testkey2","name":"laurent key","storage":{"type":"softlayer_swift","container":"laurent","credentials":{"api_key":"my_api_key_here","username":"my_username_here"},"path":"/"}}'
aslmcli ats access_key id testkey2 delete
aslmcli ats access_key id testkey2 node browse /
aslmcli ats access_key id testkey2 server
aslmcli ats access_key id testkey3 delete
aslmcli ats access_key list --fields=name,id,secret
aslmcli ats api_key list
aslmcli ats api_key repository list
aslmcli ats server id 1f412ae7-869a-445c-9c05-02ad16813be2
aslmcli ats server list clouds
aslmcli ats server list instance --cloud=aws --region=eu-west-1 
aslmcli ats server list provisioned
aslmcli ats subscriptions
aslmcli client connect id 'Aspera Connect for Windows' info
aslmcli client connect id 'Aspera Connect for Windows' links id 'Windows Installer' download --to-folder=.
aslmcli client connect id 'Aspera Connect for Windows' links list
aslmcli client connect list
aslmcli client location
aslmcli config id conf_name delete
aslmcli config id conf_name initialize @json:'{"p1":"v1","p2":"v2"}'
aslmcli config id conf_name set param value
aslmcli config id conf_name show
aslmcli config id conf_name update --p1=v1 --p2=v2
aslmcli config id default set shares conf_name
aslmcli config list
aslmcli config open
aslmcli config overview
aslmcli console transfer current list  --insecure=yes
aslmcli erb README.erb.md > README.md
aslmcli faspex package list --fields=delivery_id --format=csv --box=sent|tail -n 1) --box=sent
aslmcli faspex package list --insecure=yes
aslmcli faspex package send --load-params=reset --url=my_url_here --username=my_username_here --password=my_password_here --insecure=yes --note="my note" --title="my title" --recipient="[email protected]" ~/200KB.1
aslmcli faspex package send sample_file.bin --insecure=yes --note="my note" --title="my title" --recipient="[email protected]"
aslmcli faspex recv_publink 'https://faspex.mycompany.com/aspera/faspex/external_deliveries/78780?passcode=a003aaf2f53e3869126b908525084db6bebc7031' --insecure=yes
aslmcli files admin events
aslmcli files admin resource node id 5560 do browse / --secret=my_secret_here
aslmcli files admin resource workspace list
aslmcli files admin set_client_key ERuzXGuPA @file:$(APIKEY)
aslmcli files package list
aslmcli files package recv VleoMSrlA
aslmcli files package send sample_file.bin --note="my note" --title="my title" --recipient="[email protected]"
aslmcli files repo browse /
aslmcli files repo download /200KB.1 --to-folder=sample_dest_folder --download=node
aslmcli files repo download /200KB.1 --to-folder=sample_dest_folder --transfer=connect
aslmcli files repo upload sample_file.bin --to-folder=/
aslmcli node async id 1 counters 
aslmcli node async id 1 summary 
aslmcli node async list
aslmcli node browse / --insecure=yes
aslmcli node delete sample_dest_folder200KB.1 --insecure=yes
aslmcli node download sample_dest_folder200KB.1 --to-folder=sample_dest_folder --insecure=yes
aslmcli node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
aslmcli node service id service1 delete
aslmcli node service list
aslmcli node upload sample_file.bin --to-folder=sample_dest_folder --insecure=yes
aslmcli orchestrator info
aslmcli orchestrator plugins
aslmcli orchestrator processes
aslmcli orchestrator workflow id 10 inputs
aslmcli orchestrator workflow id 10 start [email protected]:'{"Param":"laurent"}'
aslmcli orchestrator workflow id 10 start [email protected]:'{"Param":"laurent"}' --result=ResultStep:Complete_status_message
aslmcli orchestrator workflow id 10 status
aslmcli orchestrator workflow list
aslmcli orchestrator workflow status
aslmcli preview events --skip-types=office
aslmcli preview scan --skip-types=office --log-level=info
aslmcli preview test ~/Documents/Samples/anatomic-2k/TG18-CH/TG18-CH-2k-01.dcm --log-level=debug png --video=clips
aslmcli server browse /
aslmcli server cp /Upload/200KB.1 /Upload/200KB.2
aslmcli server delete /Upload/200KB.1
aslmcli server delete /Upload/to.delete
aslmcli server df
aslmcli server download /Upload/200KB.1 --to-folder=sample_dest_folder
aslmcli server du /
aslmcli server info
aslmcli server md5sum /Upload/200KB.1
aslmcli server mkdir /Upload/123
aslmcli server mv /Upload/200KB.2 /Upload/to.delete
aslmcli server rm /Upload/123
aslmcli server upload sample_file.bin --to-folder=/Upload
aslmcli shares repository browse / --insecure=yes
aslmcli shares repository delete /n8-sh1/200KB.1 --insecure=yes
aslmcli shares repository download /n8-sh1/200KB.1 --to-folder=sample_dest_folder --insecure=yes
aslmcli shares repository upload sample_file.bin --to-folder=/n8-sh1 --insecure=yes
aslmcli shares2 appinfo
aslmcli shares2 organization list
aslmcli shares2 project list --organization=Sport
aslmcli shares2 repository browse /
aslmcli shares2 userinfo

...and more

Usage

$ aslmcli -h
NAME
    aslmcli -- a command line tool for Aspera Applications (v0.6.8)

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
SYNOPSIS
    aslmcli COMMANDS [OPTIONS] [ARGS]

DESCRIPTION
    Use Aspera application to perform operations on command line.
    OAuth 2.0 is used for authentication in Files, Several authentication methods are provided.
    Documentation and examples: https://rubygems.org/gems/asperalm
    execute: aslmcli conf doc

COMMANDS
    First level commands: config
    Note that commands can be written shortened (provided it is unique).

OPTIONS
    Options begin with a '-' (minus), and value is provided on command line.
    Special values are supported beginning with special prefix, like: @val: @file: @env: @["base64", "json", "zlib", "ruby"]:.
    Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'

ARGS
    Some commands require mandatory arguments, e.g. a path.

OPTIONS: global
    -h, --help                       Show this message.
        --show-config                Display parameters used for the provided action.
    -g, --uiTYPE                     method to start browser: text, graphical
        --insecure=ENUM              do not validate HTTPS certificate: yes, no
        --flat-hash=ENUM             display hash values as additional keys: yes, no
        --log-level=ENUM             Log level: warn, debug, info, error, fatal, unknown
        --logger=ENUM                log method: stderr, stdout, syslog
        --format=ENUM                output format: table, ruby, json, jsonpp, yaml, csv
        --transfer=ENUM              type of transfer: direct, connect, node
        --config-file=VALUE          read parameters from file in YAML format, current=/Users/laurent/.aspera/aslmcli/config.yaml
    -P, --load-paramsVALUE           load the named configuration from current config file, use "none" to avoid loading the default configuration
        --fasp-folder=VALUE          specify where to find FASP (main folder), current=
        --transfer-node=VALUE        name of configuration used to transfer when using --transfer=node
        --fields=VALUE               comma separated list of fields, or ALL, or DEF
        --fasp-proxy=VALUE           URL of FASP proxy (dnat / dnats)
        --http-proxy=VALUE           URL of HTTP proxy (for http fallback)
    -r, --rest-debug                 more debug for HTTP calls
    -N, --no-default                 do not load default configuration
    -v, --version                    display version
        --ts=VALUE                   override transfer spec values (hash, use @json: prefix), current={}
        --to-folder=VALUE            destination folder for downloaded files, current=
        --lock-port=VALUE            prevent dual execution of a command, e.g. in cron
        --use-product=VALUE          which local product to use for ascp

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: shares
SUBCOMMANDS: repository, admin
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: node
SUBCOMMANDS: events, space, info, mkdir, mklink, mkfile, rename, delete, browse, upload, download, postprocess, stream, transfer, cleanup, forward, access_key, watch_folder, service, async, central
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --persistency=VALUE          persistency file (cleanup,forward)
        --filter-transfer=VALUE      Ruby expression for filter at transfer level (cleanup)
        --filter-file=VALUE          Ruby expression for filter at file level (cleanup)
        --transfer-filter=VALUE      JSON expression for filter on API request
        --parameters=VALUE           creation parameters (hash, use @json: prefix), current=

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: orchestrator
SUBCOMMANDS: info, workflow, plugins, processes
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --params=VALUE               parameters hash table, use @json:{"param":"value"}
        --result=VALUE               specify result value as: 'work step:parameter'
        --synchronous=ENUM           work step:parameter expected as result: yes, no

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: ats
SUBCOMMANDS: server, api_key, subscriptions, access_key
OPTIONS:
        --ats-id=VALUE               ATS key identifier (ats_xxx)
        --params=VALUE               Parameters hash for access key (@json:)
        --cloud=VALUE                Cloud provider
        --region=VALUE               Cloud region

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: client
SUBCOMMANDS: installation, monitor, location, connect
OPTIONS:

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: faspex
SUBCOMMANDS: package, dropbox, recv_publink, source, me
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --recipient=VALUE            package recipient
        --title=VALUE                package title
        --note=VALUE                 package note
        --metadata=VALUE             package metadata hash value (use @json:)
        --source-name=VALUE          create package from remote source (by name)
        --box=ENUM                   package box: inbox, sent, archive

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: shares2
SUBCOMMANDS: repository, organization, project, team, share, appinfo, userinfo
OPTIONS:
        --organization=VALUE         organization
        --project=VALUE              project
        --share=VALUE                share

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: preview
SUBCOMMANDS: scan, events, folder, check, test
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --file-access=ENUM           how to read and write files in repository: local, remote
        --skip-types=VALUE           skip types in comma separated list
        --overwrite=ENUM             when to generate preview file: always, never, mtime
        --previews-folder=VALUE      preview folder in files
        --iteration-file=VALUE       path to iteration memory file
        --folder-reset-cache=ENUM    reset folder cache: no, header, read
        --temp-folder=VALUE          path to temp folder
        --skip-folders=VALUE         list of folder to skip
        --video=ENUM                 method to generate video: reencode, clips, preview
        --vid-offset-seconds=VALUE   generation parameter
        --vid-size=VALUE             generation parameter
        --vid-framecount=VALUE       generation parameter
        --vid-blendframes=VALUE      generation parameter
        --vid-framepause=VALUE       generation parameter
        --vid-fps=VALUE              generation parameter
        --vid-mp4-size-reencode=VALUE
                                     generation parameter
        --clips-offset-seconds=VALUE generation parameter
        --clips-size=VALUE           generation parameter
        --clips-length=VALUE         generation parameter
        --clips-count=VALUE          generation parameter
        --thumb-mp4-size=VALUE       generation parameter
        --thumb-img-size=VALUE       generation parameter
        --thumb-offset-fraction=VALUE
                                     generation parameter
        --validate-mime=ENUM         use magic number validation: no, yes
        --check-extension=ENUM       check extra file extensions: no, yes

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: server
SUBCOMMANDS: nodeadmin, userdata, configurator, download, upload, browse, delete, rename, ls, rm, mv, du, info, mkdir, cp, df, md5sum
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --ssh-keys=VALUE             PATH_ARRAY is @json:'["path1","path2"]'

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: files
SUBCOMMANDS: package, repository, faspexgw, admin
OPTIONS:
        --download-mode=ENUM         download mode: fasp, node_http
    -t, --authTYPE                   type of authentication: basic, web, jwt, url_token
        --bulk=ENUM                  download mode: no, yes
        --url=VALUE                  URL of application, e.g. http://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --private-key=VALUE          RSA private key PEM value for JWT (prefix file path with @val:@file:)
        --workspace=VALUE            name of workspace
        --recipient=VALUE            package recipient
        --title=VALUE                package title
        --note=VALUE                 package note
        --secret=VALUE               access key secret for node
        --query=VALUE                list filter (extended value: encode_www_form)

        --interactive=ENUM           use interactive input of missing params: yes, no
        --ask-options=ENUM           ask even optional options: yes, no
COMMAND: console
SUBCOMMANDS: transfer
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --filter-from=DATE           only after date
        --filter-to=DATE             only before date

Documentation : http://www.rubydoc.info/gems/asperalm


Note that actions and parameter values can be written in short form.

Applications

General: Application URL and Authentication

REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.

Those are using options:

  • url
  • username
  • password

Those can be provided using command line, parameter set, env var, see section above.

Aspera Files relies on Oauth, refer to the Aspera Files section.

Aspera Files

OAuth Authentication Preparation

Aspera Files requires the use of Oauth authentication. HTTP Basic authentication is not supported, instead a "Bearer" token is used to authenticate REST calls.

Bearer token are valid for a period of time, so aslmcli saves generated tokens in its configuration folder, and regenrate them when they have expired.

Several types of OAuth authentication are supported:

  • JSON Web Token (JWT) : authentication is secured by a private key (recommended)
  • Web based authentication : authentication is made by user in a browser
  • URL Token : external users authentication with url tokens

The authentication method is controled by option auth.

For all above methods, OAuth requires that the external client (aslmcli) is declared in Files using the admin interface. (see https://aspera.asperafiles.com/helpcenter/admin/organization/registering-an-api-client ).

Go to Admin View-Organization-API Clients-Create, and fill the API client creation form (here JWT):

  • Client Name: aslmcli
  • Redirect URIs: http://local.connectme.us:12345
  • Origins: localhost
  • Enable JWT Grant Type
  • Client can retrieve tokens for: All Users
  • Allowed keys: User and Global
  • Public Key: paste from the contents of the generated public key (PEM format, here file ~/.aspera/aslmcli/filesapikey.pub in next section)
  • Enable admin tokens

Files-admin-organization-apiclient-create

It is convenient to save several of those parameters in aslmcli configuration file, in order to not have to enter then for every command, so start with the creation of an option preset, lets call it: my_files_org.

Option preset can be created:

  • interactively:
$ aslmcli config id my_files_org ask url client_id client_secret
  • or one parameter at a time:
$ aslmcli config id my_files_org set url https://myorg.asperafiles.com
$ aslmcli config id my_files_org set client_id MyClIeNtKeY
$ aslmcli config id my_files_org set client_secret MyClIeNtSeCrEtMyClIeNtSeCrEt

To define this option preset as default for the Files application, execute:

$ aslmcli config id default set files my_files_org

JWT

If JWT is the chosen method, specify the authentication method:

$ aslmcli config id my_files_org set auth jwt

In order to use JWT for Aspera Files API client authentication, a private/public key pair must be generated (without passphrase) (TODO: add passphrase protection as option). This can be done using any of the following method:

  • using the CLI:
$ aslmcli config genkey ~/.aspera/aslmcli/filesapikey
  • ssh-keygen:
$ ssh-keygen -t rsa -f ~/.aspera/aslmcli/filesapikey -N ''
  • openssl

(on some openssl implementation (mac) there is option: -nodes (no DES))

$ APIKEY=~/.aspera/aslmcli/filesapikey
$ openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
$ openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
$ openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
$ rm -f ${APIKEY}.protected

To save the location of the private key in the option preset, execute:

$ aslmcli config id my_files_org set private_key @val:@file:~/.aspera/aslmcli/filesapikey

Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.

The JWT "subject", i.e. the Aspera Files user identifier (email) is provided with:

$ aslmcli config id my_files_org set username [email protected]

Web

(do not follow if JWT was selected). It is also possible to use a web browser based method to generate Bearer token, like done when using a web browser to access the Aspera Files application. In that case, no private key is needed, but the user us required to login using a URL (refer to this section) when Bearer token have expired, making this method not suitable for use in automated scripts.

Upon successful authentication, auth token are saved (cache) in local files, and can be used subsequently. Expired token are refreshed if possible (without user further authentication).

The redirection URI shall be provided (keep this value):

$ aslmcli config id my_files_org set redirect_uri http://local.connectme.us:12345

Example

$ aslmcli files repo browse /
Current Workspace: My Org (default)
:..............................:........:................:...........:......................:..............:
:             name             :  type  : recursive_size :   size    :    modified_time     : access_level :
:..............................:........:................:...........:......................:..............:
: Summer 2016 Training         : link   :                :           : 2016-07-25T15:21:22Z : edit         :
: Laurent Garage SE            : folder : 19316893       :           :                      : edit         :
: Italy Training               : folder : 312068540      :           :                      : edit         :
: Cheese pile.jpeg             : file   :                : 9824      : 2016-11-16T12:10:25Z : edit         :
: Aspera Video                 : folder : 122237276      :           :                      : edit         :
:..............................:........:................:...........:......................:..............:

Administration

The admin command allows several administrative tasks (and require admin privilege).

It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the admin resource command.

Bulk operations are possible using option bulk (yes,no(default)): currently: create only. In that case, the operation expects an Array of Hash instead of a simple Hash using the Extended Value Syntax.

Example

  • Bulk creation
aslmcli files admin res user create --bulk=yes @json:'[{"email":"[email protected]"},{"email":"[email protected]"}]'
:.......:.........:
:  id   : status  :
:.......:.........:
: 98398 : created :
: 98399 : created :
:.......:.........:
  • Find with filter and delete
$ aslmcli files admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email
:.......:........................:
:  id   :         email          :
:.......:........................:
: 98398 : [email protected] :
: 98399 : [email protected] :
:.......:........................:
$ thelist=$(echo $(aslmcli files admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email --field=id --format=csv)|tr ' ' ,)
$ echo $thelist
98398,98399
$ aslmcli files admin res user id @json:[$thelist] delete --bulk=yes
:.......:.........:
:  id   : status  :
:.......:.........:
: 98398 : deleted :
: 98399 : deleted :
:.......:.........:

Aspera Node (Transfer Server)

Simple Operations

It is possible to browse and transfer and various operations.

FASP Stream

It is possible to start a FASPStream session using the node API:

Use the "node stream create" command, then arguments are provided as a transfer-spec.

./bin/aslmcli node stream create [email protected]:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"XXXX"}' --load-params=stream

Watchfolder

Refer to Aspera documentation for watch folder creation.

aslmcli supports remote operations through the node API. Operations are:

  • Start watchd and watchfolderd services running as a system user having access to files
  • configure a watchfolder to define automated transfers
$ aslmcli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
$ aslmcli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
$ aslmcli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"Aspera123_"}}'

Transfer filter (deprecated)

special in node:

"transfer_filter"=>"t['status'].eql?('completed') and t['start_spec']['remote_user'].eql?('faspex')", :file_filter=>"f['status'].eql?('completed') and 0 != f['size'] and t['start_spec']['direction'].eql?('send')"

Example: SHOD to ATS

Access to a "Shares on Demand" (SHOD) server on AWS is provided by a partner. And we need to transfer files from this third party SHOD instance into our Azure BLOB storage. Simply create an "Aspera Transfer Service" instance (https://ts.asperasoft.com), which provides access to the node API. Then create a configuration for the "SHOD" instance in the configuration file: in section "shares", a configuration named: awsshod. Create another configuration for the Azure ATS instance: in section "node", named azureats. Then execute the following command:

aslmcli node download /share/sourcefile --to-folder=/destinationfolder --load-params=awsshod --transfer=node [email protected]:azureats

This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.

Client

The client plugin refers to the use of a local FASP client. It provides the following commands:

  • connect list : list connect client versions available on internet
  • installation list : list Aspera transfer products found locally
  • client location : list resources used for transfers

Examples

Download the installer for "Aspera Connect Client":

$ ./bin/aslmcli client connect list
:...............................................:......................................:..............:
:                      id                       :                title                 :   version    :
:...............................................:......................................:..............:
: urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E : Aspera Connect for Windows           : 3.7.0.138427 :
: urn:uuid:A3820D20-083E-11E2-892E-0800200C9A66 : Aspera Connect for Windows 64-bit    : 3.7.0.138427 :
: urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E : Aspera Connect for Windows XP        : 3.7.0.138427 :
: urn:uuid:55425020-083E-11E2-892E-0800200C9A66 : Aspera Connect for Windows XP 64-bit : 3.7.0.138427 :
: urn:uuid:D8629AD2-6898-4811-A46F-2AF386531BFF : Aspera Connect for Mac Intel 10.6    : 3.6.1.111259 :
: urn:uuid:D8629AD2-6898-4811-A46F-2AF386531BFF : Aspera Connect for Mac Intel         : 3.7.0.138427 :
: urn:uuid:213C9370-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 32          : 3.6.2.117442 :
: urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 : Aspera Connect for Linux 64          : 3.7.2.141527 :
:...............................................:......................................:..............:
$ aslmcli client connect id 'Aspera Connect for Mac Intel 10.6' links list
:.............................................:..........................:.......................................................................:..........:...............:
:                    title                    :           type           :                                 href                                  : hreflang :      rel      :
:.............................................:..........................:.......................................................................:..........:...............:
: Mac Intel Installer                         : application/octet-stream : bin/AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg                     : en       : enclosure     :
: Aspera Connect for Mac HTML Documentation   : text/html                :                                                                       : en       : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/ja-jp/pdf/Connect_User_3.7.0_OSX_ja-jp.pdf              : ja-jp    : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/en/pdf/Connect_User_3.7.0_OSX.pdf                       : en       : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/es-es/pdf/Connect_User_3.7.0_OSX_es-es.pdf              : es-es    : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/fr-fr/pdf/Connect_User_3.7.0_OSX_fr-fr.pdf              : fr-fr    : documentation :
: Aspera Connect PDF Documentation for Mac OS : application/pdf          : docs/user/osx/zh-cn/pdf/Connect_User_3.7.0_OSX_zh-cn.pdf              : zh-cn    : documentation :
: Aspera Connect for Mac Release Notes        : text/html                : http://www.asperasoft.com/en/release_notes/default_1/release_notes_54 : en       : release-notes :
:.............................................:..........................:.......................................................................:..........:...............:
$ aslmcli client connect id 'Aspera Connect for Mac Intel 10.6' links id 'Mac Intel Installer' download --to-folder=.
downloaded: AsperaConnect-3.6.1.111259-mac-intel-10.6.dmg

IBM Aspera High Speed Transfer Server

Works at FASP level (SSH/ascp/ascmd).

Example

One can test the "server" application using the well known demo server:

$ aslmcli config id aspera_demo_server update --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=demoaspera
$ aslmcli config id default set server aspera_demo_server 
$ aslmcli server browse /aspera-test-dir-large
$ aslmcli server download /aspera-test-dir-large/200MB

This creates a option preset "aspera_demo_server" and set it as default for application "server"

IBM Aspera Faspex

remote sources

Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this, the node API is used, for this it is required to add a section ":storage" that links a storage name to a node config and sub path. Example:

my_faspex_conf:
  url: https://10.25.0.3/aspera/faspex
  username: admin
  password: MyPassword
  storage:
    testlaurent:
      node: my_faspex_node
      :path: /myfiles
my_faspex_node:
  url: https://10.25.0.3:9092
  username: node_faspex
  password: MyPassword

In this example, a faspex storage named "testlaurent" exists in Faspex, and is located under the docroot in "/myfiles" (this must be the same as configured in Faspex). The node configuration name is "my_faspex_node" here.

Aspera Transfer Service

Aka Aspera Files, Aspera on Cloud...

First time use

Using the ATS requires an "Aspera ID" (https://id.asperasoft.com/) and have a subscription associated to it.

On first execution, the user is asked to login to Aspera ID using a web browser. This creates an "ats_id" identifier (stored in a cache file).

When only one ats_id is created, it is taken by default. Else it shall be specified with --ats-id or using a option preset.

Note: APIs are described here: https://ts.asperasoft.com/ats-guide/getting-started-guide/# guide_transfers_create_ak

Examples

Example: create access key on softlayer:

aslmcli ats access_key create --cloud=softlayer --region=ams [email protected]:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"value","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'

Example: create access key on AWS:

aslmcli ats access_key create --cloud=aws --region=eu-west-1 [email protected]:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"my/secret/here"},"path":"/laurent"}}'

Example: create access key on Azure SAS:

aslmcli ats access_key create --cloud=azure --region=eastus [email protected]:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://xxxx.blob.core.windows.net/..."},"path":"/"}}'

Example: create access key on Azure:

aslmcli ats access_key create --cloud=azure --region=eastus [email protected]:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'

delete all my access keys:

for k in $(aslmcli ats access_key list --field=id --format=csv);do aslmcli ats access_key id $k delete;done

Preview

The preview plugin provides generation of previews for Aspera Files.

The tool requires the following external tools:

  • ImageMagick : convert composite
  • OptiPNG : optipng
  • FFmpeg : ffmpeg ffprobe
  • Libreoffice : libreoffice

Preview Command

The preview plugin allows generation of preview files for Aspera Files for on-premise nodes. (thumbnails and video previews)

The preview generator creates/updates a preview for files located on an access key main "storage root". Several candidate detection methods are supported.

This version requires ES 3.7.4+

This is related to:

https://aspera.asperafiles.com/helpcenter/admin/organization/installing-files-preview-maker

Configuration

Like any aslmcli commands, parameters can be passed on command line or using a configuration option preset. Example using a option preset:

$ aslmcli config id default set preview my_files_access_key
$ aslmcli config id my_files_access_key update --url=https://localhost:9092 --username=my_access_key --password=my_secret

Once can check if the access key is well configured using:

$ aslmcli -Pmy_files_access_key node browse /

Execution

The tool intentionally supports only a "finite execution time" mode, i.e. it needs to be run regularly to create or update preview files. I.e. it does not implement a loop. Running preview generation on a regular basis shall be done using the operating system scheduler (e.g. Cron on Linux, or Task Scheduler on Windows). To prevent parallel execution of the task, one can use either the protection offered by the scheduler, if there is one, or use the parameter: --lock-port=12345.

Candidate detection

The tool will find candidates for preview generation using three commands:

  • events : only recently uploaded files will be tested
  • scan : deeply scan all files under the access key's "storage root"
  • folder : same as scan, but only on the specified folder's "file identifier"

Creation/Update

Once candidate are selected, once candidates are selected, a preview is always generated if it does not exist already, else if a preview already exist, it will be generated using one of three overwrite method:

  • always : preview is always generated, even if it already exists and is newer than original
  • never : preview is generated only if it does not exist already
  • mtime : preview is generated only if the original file is newer than the existing

Access to original files and preview creation

Standard open source tools are used to create thumnails and video previews. Those tools require that original files are accessible in the local file system and also write generated files on the local file system. The tool provides 2 ways to read and write files with the parameter: --file-access

If the preview generator is run on a system that has direct access to the file system, then the value local can be used.

If the preview generator does not have access to files on the file system (it is remote, no mount, or is an object storage), then the original file is first downloaded, then the result is uploaded, use method remote.

Options

To skip some folders, use the option : --skip-folders, note that it expects a list of path starting with slash, use the @json: notation, example:

$ aslmcli preview scan [email protected]:'["/not_here"]'

Examples

on command line:

aslmcli preview event --skip-types=office --file-access=remote --overwrite=always --iteration-file=/tmp/restart.txt --lock-port=12345

with crontab:

2-59 * * * * su -s /bin/bash - xfer -c 'timeout 10m aslmcli preview event --skip-types=office --lock-port=12345 --log-level=info --logger=syslog --iteration-file=/tmp/preview_restart.txt'
0 * * * *    su -s /bin/bash - xfer -c 'timeout 30m aslmcli preview scan  --skip-types=office --lock-port=12345 --log-level=info --logger=syslog'

External tools: Linux

Here shown on Redhat/CentOS

  • Imagemagick and optipng:
yum install -y ImageMagick optipng
  • FFmpeg
pushd /tmp
wget http://johnvansickle.com/ffmpeg/releases/ffmpeg-release-64bit-static.tar.xz
mkdir -p /opt/
cd /opt/
tar xJvf /tmp/ffmpeg-release-64bit-static.tar.xz
ln -s ffmpeg-* ffmpeg
ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin
popd
  • Libreoffice
yum install libreoffice
  • Xvfb

Although libreoffice is run headless, older versions may require an X server. If you get error running libreoffice headless, then install Xvfb:

yum install libreoffice Xvfb
cat<<EOF>/etc/init.d/xvfb 
# !/bin/bash
# chkconfig: 345 95 50
# description: Starts xvfb on display 42 for headless Libreoffice
if [ -z "\$1" ]; then
  echo "\`basename \$0\` {start|stop}"
  exit
fi
case "\$1" in
start) /usr/bin/Xvfb :42 -screen 0 1280x1024x8 -extension RANDR&;;
stop) killall Xvfb;;
esac
EOF
chkconfig xvfb on
service xvfb start

Tool: asession

This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: asession.

It aims at simplifying the startup of a FASP session from a programmatic stand point as formating a transfer-spec is:

  • common to Aspera Node API (HTTP POST /ops/transfer)
  • common to Aspera Connect API (browser javascript startTransfer)
  • easy to generate by using any third party language specific JSON library

This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.

The tool expect one single argument: a transfer-spec.

If not argument is provided, it assumes a value of: @json:@stdin, i.e. a JSON formated transfer-spec on stdin.

Note that if JSON is the format, one has to specify @json: to tell the tool to decode the hash using JSON.

During execution, it generates all low level events, one per line, in JSON format on stdout.

Comparison:

feature/toolasessionascpFaspManager
language integrationanyanyC/C++
C#/.net
Go
Python
java
additional components to ascpRuby
Asperalm
-library
(headers)
startupJSON on stdin
(standard APIs:
JSON.generate
Process.spawn)
command line argumentsAPI
eventsJSON on stdoutnone by default
or need to open management port
and proprietary text syntax
callback

Examples of use:

  • Shell
MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}'

echo "${MY_TSPEC}"|asession
$ asession -h
USAGE
    asession <session information>
    <session information> is a JSON formatted transfer spec, starting with "@json:".
    The value can be either:
    o the JSON description itself
    o @stdin, if the JSON is provided from stdin
    o @file:<path>, if the JSON is provided from a file
Note: debug information can be placed on STDERR, using the "EX_loglevel" parameter in transfer spec (debug=0)
EXAMPLES
    asession @json:'{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}'
    echo '{"remote_host":...}'|asession @json:@stdin


Module: Asperalm

The main classes are part of the module: Asperalm::Fasp. It allows starting and monitoring transfers. It can be considered as a FASPManager class for Ruby.

History

When I joined Aspera, there was only one CLI: ascp, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the ascp tool, I thought of an extended tool : eascp.pl which was accepting all ascp options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).

There were a few pitfalls:

  • The tool was written in the aging perl language while most Aspera application products (but the Transfer Server) are written in ruby.
  • The tool was only for transfers, but not able to call other products APIs

So, it evolved into aslmcli:

  • portable: works on platforms supporting ruby (and ascp)
  • easy to install with the gem utility
  • supports transfers with multiple Transfer Agents, that's why transfer parameters moved from ascp command line to transfer-spec (more reliable , more standard)
  • ruby is consistent with other Aspera products

BUGS

This is best effort code without official support, dont expect full capabilities. This code is not supported by IBM/Aspera. You can contact the author for bugs or features.

If you get message: "OpenSSH keys only supported if ED25519 is available" this means that you do not have ruby support for ED25519 SSH keys. You may either install the suggested Gems, or remove your ed25519 key from your .ssh folder to solve the issue.

TODO

Contribution

Create your own plugin !

Send comments !