Remote server executor plugin for Embulk

Embulk executor plugin to run Embulk tasks on remote servers.

Overview

  • Plugin type: executor

Notes

  • It's still very experimental version, so might change its spec without notification.
  • It might have some issues or bugs. I would appreciate it if you use this and give me reports/feedback!

Configuration

  • hosts: List of remote servers (hostname or hostname:port, default port is 30001). If not specified, the executor runs as the local mode, which starts an Embulk server on its own process (array of string)
  • timeout_seconds: Timeout seconds of the whole execution (integer, default: 3600)
  • tls: Enable to connect server over TLS (boolean, default: false)
  • key_store_p12: Information of a P12 file used as a KeyManager to register your client certificate. It would be needed when client auth mode is enabled on Embulk server.
    • path: Path of the file (string, required)
    • password: Password of the file (string, required)
  • trust_store_p12: Information of a P12 file used as a TrustManager to register a CA certificate. It would be needed when Embulk server uses unknown CA's certificate.
    • path: Path of the file (string, required)
    • password: Password of the file (string, required)

Example

exec:
  type: remoteserver
  hosts:
    - embulk-server1.local
    - embulk-server2.local:30002
  timeout_seconds: 86400

Embulk server

The server receives requests from client (Embulk) and run Embulk tasks, then returns results to client. It communicates with clients via TCP 30001 port.

Running Embulk server as a Docker container

The image is hosted by DockerHub.
You can try running Embulk server by the following command. 

$ docker run --rm -p 30001:30001 kamatama41/embulk-executor-remoteserver

Configure Embulk server

There are some environment variables to configure the server

  • BIND_ADDRESS: Bind address of the server (default: 0.0.0.0)
  • PORT: Port number to listen (default: 30001)
  • ENABLE_TLS: Try to connect to client via TLS if true (default: false)
  • ENABLE_TLS_CLIENT_AUTH: Require client authentication if true (default: false)
  • KEY_P12_PATH: Path of the P12 file used as a KeyManager
  • KEY_P12_PASSWORD: Password of the Key P12 file
  • TRUST_P12_PATH: Path of the P12 file used as a TrustManager
  • TRUST_P12_PASSWORD: Password of the Trust P12 file

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously