Sherlog Holmes

Less data containing useful information is way better than lots of data containing a mess.

Don't you hate thousands of lines in a log blowing up with your troubleshooting? Lots of useless data that you have to filter just to turn that 300 MB of madness into a 30 KB of useful information. If you need something that can rip off useless entries so you can have a clue about what is going on with that application, you should give Sherlog Holmes a try.

Installation

Add this line to your application's Gemfile:

gem 'sherlog_holmes'

And then execute:

$ bundle

Or install it yourself as:

$ gem install sherlog-holmes

Concepts

Sherlog works by grabbing every line of a log and parsing it into a simple structure containing:

  • Time
  • Level
  • Category
  • Origin
  • Message
  • Exception
  • Stacktrace

You need to supply a regular expression that maps those fields in order to match your log entry. Here is an example:

(<?level>\w+)\s(<?category>\s+)\s(<?message>.+)

Notice that you don't need to define every field, just the ones shown in your log.

Patterns for exception and stacktrace should be defined separately. The exception pattern is used only in the message field. Here is a complete example of a pattern configuration:

jboss:
  entry: (?<time>[0-9,.:]+)\s+(?<level>\w+)\s+\[(?<category>\S+)\]\s\((?<origin>[^)]+)\)?\s?(?<message>.+)
  exception: (?<exception>\w+(\.\w+)+(Exception|Error))
  stacktrace: ^(\s+at)|(Caused by\:)|(\s+\.{3}\s\d+\smore)

The configuration should contain a unique id and at least a pattern for the log entry. Place you configuration file in a *.yml file inside your $HOME/.sherlog/patterns directory and you're ready to go!

Usage

Shelog Holmes provides the command line tool sherlog. You can use this to pass a log, the filters you need to apply and the process that needs to be executed (like showing the filtered entries or counting the exceptions):

Config Options

-p, --patterns FILE

Additionally to having definitions in your $HOME/.sherlog directory, you can pass a definition file from anywhere in your machine and Sherlog will scan and register the definitions.

--encode ENCODE

This sets the encode to use while reading the log file.

-t, --type TYPE

This will manually set the patterns definitions. If you don't specify this option, Sherlog will try the mapped ones with the first input line.

Filter Options

-c, --category EXPRESSION

This will filter entries using the category field. You can use the wildcard * here.

-l, --level EXPRESSION

This will filter entries using the level field. You can use the wildcard * here.

-o, --origin EXPRESSION

This will filter entries using the origin field. You can use the wildcard * here.

-m, --message EXPRESSION

This will filter entries using the message field. You can use the wildcard * here.

-e, --exception EXPRESSION

This will filter entries using the exception field. You can use the wildcard * here.

NOTICE: the expressions are case sensitive, wildcards can be used at start, end or both

--any-exception

This will filter entries with exceptions, regardless the kind.

Logical Options

--and

This will use the AND operation to connect the next filter. This is the default operation.

--or

This will use the OR operation to connect the next filter.

--not

This will negate the next filter.

sherlog --level WARN --or --not --level INFO --and --any-exception

This is equivalent to:

(WARN || ! INFO) && EXCEPTION

NOTICE: try not to do fuzzy logics with this operators

Operation Options

--print

This will instruct Sherlog to print every filtered entry. This is useful to reduce that crazy log file into a sane one.

$ sherlog --level ERROR --print crazy-log-file.log > sane-log-file.log

--no-stacktrace

This will instruct Sherlog to not print stacktraces for entries. This only has effect if used with --print.

--count GROUPS...

Set this and Sherlog will count the number of entries per level, category, origin or exception. The possible parameters are (separated by a ,):

  • levels: counts the number of entries per level
  • categories: counts the number of entries per category
  • origins: counts the number of entries per origin
  • exception: counts the number of entries per exception
  • all: counts all groups
$ sherlog --count levels,categories log-file.log

Built-in Patterns

Currently, Sherlog has the following patterns:

  • jboss: matches Wildfly | EAP logs

License

The gem is available as open source under the terms of the MIT License.