remote_syslog Ruby daemon & sender
Lightweight Ruby daemon to tail one or more log files and transmit UDP syslog messages to a remote syslog host (centralized log aggregation).
remote_syslog generates UDP packets itself instead of depending on a system syslog daemon, so its configuration doesn't affect system-wide logging - syslog is just the transport.
Uses:
- collecting logs from servers & daemons which don't natively support syslog
- when reconfiguring the system logger is less convenient than a purpose-built daemon (e.g., automated app deployments)
- aggregating files not generated by daemons (e.g., package manager logs)
The library can also be used to generate one-off log messages from Ruby code.
Tested with the hosted log management service Papertrail and should work for transmitting to any syslog server.
Installation
Install the gem, which includes a binary called "remote_syslog":
$ [sudo] gem install remote_syslog
Optionally, create a log_files.yml with the log file paths to read and the host/port to log to (see examples/log_files.yml.example). These can also be specified as arguments to the remote_syslog daemon. More below.
Usage
$ remote_syslog -h
Usage: remote_syslog [options] <path to add'l log 1> .. <path to add'l log n>
Example: remote_syslog -c configs/logs.yml -p 12345 /var/log/mysqld.log
Options:
-c, --configfile PATH Path to config (/etc/log_files.yml)
-d, --dest-host HOSTNAME Destination syslog hostname or IP (logs.papertrailapp.com)
-p, --dest-port PORT Destination syslog port (514)
-D, --no-detach Don't daemonize and detach from the terminal
-f, --facility FACILITY Facility (user)
--hostname HOST Local hostname to send from
-P, --pid-dir DIRECTORY Directory to write .pid file in (/var/run/)
--pid-file FILENAME PID filename (<program name>.pid)
--parse-syslog Parse file as syslog-formatted file
-s, --severity SEVERITY Severity (notice)
--strip-color Strip color codes
--tls Connect via TCP with TLS
-h, --help Show this message
Example
Daemonize, collecting from files mentioned in ./config/logs.yml as well as
/var/log/mysqld.log:
$ remote_syslog -c configs/logs.yml -p 12345 /var/log/mysqld.log
Stay attached to the terminal, look for and use /etc/log_files.yml if it
exists, write PID to /tmp/remote_syslog.pid, and send with facility local0:
$ remote_syslog -d a.server.com -f local0 -P /tmp /var/log/mysqld.log
remote_syslog will daemonize by default. A sample init file is in the gem as remote_syslog.init.d. You may be able to:
$ cp examples/remote_syslog.init.d /etc/init.d/remote_syslog
Sending messages securely
If the receiving system supports sending syslog over TCP with TLS, you can
pass the --tls option when running remote_syslog:
$ remote_syslog --tls -p 1234 /var/log/mysqld.log
Configuration
By default, the gem looks for a configuration in /etc/log_files.yml.
The gem comes with a sample config. Optionally:
$ cp examples/log_files.yml.example /etc/log_files.yml
log_files.yml has filenames to log from (as an array) and hostname and port to log to (as a hash). Wildcards are supported using * and standard shell globbing. Filenames given on the command line are additive to those in the config file.
Only 1 destination server is supported; the command-line argument wins.
files: [/var/log/httpd/access_log, /var/log/httpd/error_log, /var/log/mysqld.log, /var/run/mysqld/mysqld-slow.log]
destination:
host: logs.papertrailapp.com
port: 12345
Optional: Parse fields from messages written by syslogd
This is not needed for most configurations.
In cases where logs from multiple programs are in the same file (for example,
/var/log/messages), the log line may include text that is not part of the
log message, like a timestamp, hostname, or program name. remote_syslog can
parse the program, hostname, and/or message text so that the message has
accurate metadata.
To do that, add an optional top-level configuration option parse_fields
with the name of a predefined regex (by remote_syslog) or a regex string. To
use the predefined regex for standard syslog messages, provide:
parse_fields: syslog
The syslog regex is (\w+ \d+ \S+) (\S+) ([^:]+): (.*). It parses this:
Jul 18 08:25:08 hostname programname[1234]: The log message
Or provide parse_fields: rfc3339 to parse high-precision RFC 3339
timestamps like:
2011-07-16T08:25:08.651413-07:00 hostname programname[1234]: The log message
Or provide your own regex that includes these 4 backreferences, in order: timestamp, system name, program name, message. Match and return empty strings for any empty positions where the log value should be ignored. For example, in the log:
something-meaningless The log message
You could ignore the first word, returning 3 empty values then the log message with:
parse_fields: "something-meaningless ()()()(.*)"
Per-file parsing is not supported. Run multiple instances.
Optional: Run multiple instances
Run multiple instances to support more than one message-specific file format (concurrently) or to specify distinct syslog hostnames. To do so, provide an alternative PID filename as a command-line option to additional instance(s), such as:
--pid-file remote_syslog_2.pid
Reporting bugs
- See whether the issue has already been reported: https://github.com/papertrail/remote_syslog/issues/
- If you don't find one, create an issue with a repro case.
Contributing
Once you've made your great commits: