Sherlog Holmes
Less data containing useful information is way better than lots of data containing a mess.
Don't you hate thousands of lines in a log blowing up with your troubleshooting? Lots of useless data that you have to filter just to turn that 300 MB of madness into a 30 KB of useful information. If you need something that can rip off useless entries so you can have a clue about what is going on with that application, you should give Sherlog Holmes a try.
Installation
Add this line to your application's Gemfile:
gem 'sherlog_holmes'
And then execute:
$ bundle
Or install it yourself as:
$ gem install sherlog-holmes
Concepts
Sherlog works by grabbing every line of a log and parsing it into a simple structure containing:
- Time
- Level
- Category
- Origin
- Message
- Exception
- Stacktrace
You need to supply a regular expression that maps those fields in order to match your log entry. Here is an example:
(<?level>\w+)\s(<?category>\s+)\s(<?message>.+)
Notice that you don't need to define every field, just the ones shown in your log.
Patterns for exception and stacktrace should be defined separately. The exception pattern is used only in the message field. Here is a complete example of a pattern configuration:
jboss:
entry: (?<time>[0-9,.:]+)\s+(?<level>\w+)\s+\[(?<category>\S+)\]\s\((?<origin>[^)]+)\)?\s?(?<message>.+)
exception: (?<exception>\w+(\.\w+)+(Exception|Error))
stacktrace: ^(\s+at)|(Caused by\:)|(\s+\.{3}\s\d+\smore)
The configuration should contain a unique id and at least a pattern for the log entry. Place you configuration file in a *.yml
file inside your $HOME/.sherlog/patterns
directory and you're ready to go!
Usage
Shelog Holmes provides the command line tool sherlog
. You can use this to pass a log, the filters you need to apply and the process that needs to be executed (like showing the filtered entries or counting the exceptions):
Config Options
-p, --patterns FILE
Additionally to having definitions in your $HOME/.sherlog
directory, you can pass a definition file from anywhere in your machine and Sherlog will scan and register the definitions.
--encode ENCODE
This sets the encode to use while reading the log file.
-t, --type TYPE
This will manually set the patterns definitions. If you don't specify this option, Sherlog will try the mapped ones with the first input line.
Filter Options
-c, --category EXPRESSION
This will filter entries using the category field. You can use the wildcard *
here.
-l, --level EXPRESSION
This will filter entries using the level field. You can use the wildcard *
here.
-o, --origin EXPRESSION
This will filter entries using the origin field. You can use the wildcard *
here.
-m, --message EXPRESSION
This will filter entries using the message field. You can use the wildcard *
here.
-e, --exception EXPRESSION
This will filter entries using the exception field. You can use the wildcard *
here.
NOTICE: the expressions are case sensitive, wildcards can be used at start, end or both
--any-exception
This will filter entries with exceptions, regardless the kind.
Logical Options
--and
This will use the AND operation to connect the next filter. This is the default operation.
--or
This will use the OR operation to connect the next filter.
--not
This will negate the next filter.
sherlog --level WARN --or --not --level INFO --and --any-exception
This is equivalent to:
(WARN || ! INFO) && EXCEPTION
NOTICE: try not to do fuzzy logics with this operators
Operation Options
--print
This will instruct Sherlog to print every filtered entry. This is useful to reduce that crazy log file into a sane one.
$ sherlog --level ERROR --print crazy-log-file.log > sane-log-file.log
--no-stacktrace
This will instruct Sherlog to not print stacktraces for entries. This only has effect if used with --print
.
--count GROUPS...
Set this and Sherlog will count the number of entries per level, category, origin or exception. The possible parameters are (separated by a ,
):
levels
: counts the number of entries per levelcategories
: counts the number of entries per categoryorigins
: counts the number of entries per originexception
: counts the number of entries per exceptionall
: counts all groups
$ sherlog --count levels,categories log-file.log
Built-in Patterns
Currently, Sherlog has the following patterns:
jboss
: matches Wildfly | EAP logs
License
The gem is available as open source under the terms of the MIT License.