sqsrun
A Generics Worker Executor Service for Amazon SQS. It polls SQS queue and run command or script when a message is received. The message is removed only when the command succeeded.
Architecture
-
sqsrun receives a message from the queue.
-
sqsrun executes a command or runs a script.
-
if the process takes long time, sqsrun extends visibility timeout. this is repeated until it is killed.
-
if the process takes more long time, sqsrun kills the process.
-
-
if it succeeded, sqsrun removes the message from the queue.
-
if it failed, sqsrun set the visibility timeout to 0 and expect to be retried.
Usage
Usage: sqsrun [options] [-- <ARGV-for-exec-or-run>]
-k, --key-id ID AWS Access Key ID
-s, --secret-key KEY AWS Secret Access Key
-q, --queue NAME SQS queue name
-t, --timeout SEC SQS visibility timeout (default: 30)
--create Create new queue
--delete Delete a queue
--attr Key=Value Set attribute
--push MESSAGE Push maessage to the queue
--list List queues
--configure PATH.yaml Write configuration file
--exec COMMAND Execute command
--run SCRIPT.rb Run method named 'run' defined in the script
--env K=V Set environment variable
-e, --extend-timeout SEC Threashold time before extending visibility timeout (default: timeout * 3/4)
-x, --kill-timeout SEC Threashold time before killing process (default: timeout * 10)
-X, --kill-retry SEC Threashold time before retrying killing process (default: 60)
-i, --interval SEC Polling interval (default: 1)
-U, --release-on-fail Releases lock if task failed so that other node can retry immediately
-d, --daemon PIDFILE Daemonize (default: foreground)
-f, --file PATH.yaml Read configuration file
One of –list, –create, –delete, –attr, –push, –configure, –exec or –run is required. The behavior of the commands is described below:
list
Show list of queues. -k and -s options are required.
Example:
$ sqsrun -k KEY_ID -s SEC_KEY --list
create
Create new queue. -k, -s and -q options are required.
Example:
$ sqsrun -k KEY_ID -s SEC_KEY --create -q myqueue
delete
Delete a queue. -k, -s and -q options are required. Note that messages in the queue will be lost.
Example:
$ sqsrun -k KEY_ID -s SEC_KEY --delete -q myqueue
attr
Set a attribute on a queue. -k, -s and -q options are required.
Example:
$ sqsrun -k KEY_ID -s SEC_KEY -q myqueue --attr MessageRetentionPeriod=3600
push
Push a message to the queue. -k, -s and -q options are required.
Example:
$ sqsrun -k KEY_ID -s SEC_KEY -q myqueue --push '{"a":"b"}'
configure
Write configuration file and exit. Written configuration file can be used with -f option:
Example:
## create sqsrun.yaml file
$ sqsrun --configure sqsrun.yaml -k KEY_ID -s SEC_KEY -q myqueue --run myrun.rb -- my run args
## run sqsrun using the configuration file
$ sqsrun -f sqsrun.yaml
exec
Execute a command when a message is received. Body of the message is passed to the stdin. The command have to exit with status code 0 when it succeeded.
-k, -s and -q options are required.
Example:
#!/usr/bin/env ruby
require 'json'
js = JSON.load(STDIN.read)
puts "received: #{js.inspect}"
# $ sqsrun -k AWS_KEY_ID -s AWS_SEC_KEY -q SQS_NAME --exec ./this_file
When the kill timeout (-x, –klill-timeout) is elapsed, SIGTERM signal will be sent to the child process. The signal will be repeated every few seconds (-X, –kill-retry).
run
This is same as ‘exec’ except that this calls a method named ‘run’ defined in the file instead of executing the file. Body of the message is passed to the argument. It is assumed it succeeded if the method doesn’t any raise errors.
Example:
require 'json'
def run(msg)
js = JSON.load(msg)
puts "received: #{js.inspect}"
end
# optionally you can define terminate method
# which will be called when kill timeout is elapsed.
def terminate
end
# $ sqsrun -k AWS_KEY_ID -s AWS_SEC_KEY -q SQS_NAME --run ./this_file.rb
When the kill timeout (-x, –klill-timeout) is elapsed, terminate() method will be called (if it is defined). It will be repeated every few seconds (-X, –kill-retry).