Module: LogStash
- Defined in:
- lib/logstash/filters/dissect.rb
Overview
The Dissect filter is a kind of split operation. Unlike a regular split operation where one delimiter is applied to the whole string, this operation applies a set of delimiters # to a string value. + Dissect does not use regular expressions and is very fast. + However, if the structure of your text varies from line to line then Grok is more suitable. + There is a hybrid case where Dissect can be used to de-structure the section of the line that is reliably repeated and then Grok can be used on the remaining field values with # more regex predictability and less overall work to do. +
A set of fields and delimiters is called a dissection.
The dissection is described using a set of %{} sections:
....
%a - %b - %c
....
A field is the text from % to } inclusive.
A delimiter is the text between } and % characters.
[NOTE]
delimiters can't contain these }{% characters.
The config might look like this: .... filter { dissect { mapping => { "message" => "%{ts} %{+ts} %{+ts} %{src} %{} %prog: %{msg}" } } } .... When dissecting a string from left to right, text is captured upto the first delimiter - this captured text is stored in the first field. This is repeated for each field/# delimiter pair thereafter until the last delimiter is reached, then the remaining text is stored in the last field. +
The Key: +
The key is the text between the %{` and `}, exclusive of the ?, +, & prefixes and the ordinal suffix. +
%{?aaa} - key is aaa +
%{+bbb/3} - key is bbb +
%{&ccc} - key is ccc +
Normal field notation: +
The found value is added to the Event using the key. +
%{some_field} - a normal field has no prefix or suffix
Skip field notation: +
The found value is stored internally but not added to the Event. +
The key, if supplied, is prefixed with a ?.
%{} is an empty skip field.
%{?foo} is a named skip field.
Append field notation: +
The value is appended to another value or stored if its the first field seen. +
The key is prefixed with a +. +
The final value is stored in the Event using the key. +
[NOTE]
The delimiter found before the field is appended with the value. + If no delimiter is found before the field, a single space character is used.
%{+some_field} is an append field. +
%{+some_field/2} is an append field with an order modifier.
An order modifier, /digits, allows one to reorder the append sequence. +
e.g. for a text of 1 2 3 go, this %{+a/2} %{+a/1} %{+a/4} %{+a/3} will build a key/value of a => 2 1 go 3 +
Append fields without an order modifier will append in declared order. +
e.g. for a text of 1 2 3 go, this %{a} %{b} %{+a} will build two key/values of a => 1 3 go, b => 2 +
Indirect field notation: +
The found value is added to the Event using the found value of another field as the key. +
The key is prefixed with a &. +
%{&some_field} - an indirect field where the key is indirectly sourced from the value of some_field. +
e.g. for a text of error: some_error, some_description, this error: %{?err}, %{&err} will build a key/value of some_error => some_description.
[NOTE] for append and indirect field the key can refer to a field that already exists in the event before dissection.
[NOTE] use a Skip field if you do not want the indirection key/value stored.
e.g. for a text of google: 77.98, this %{?a}: %{&a} will build a key/value of google => 77.98.
[NOTE]
append and indirect cannot be combined and will fail validation. +
%{+&something} - will add a value to the &something key, probably not the intended outcome. +
%{&+something} will add a value to the +something key, again probably unintended. +
Delimiter repetition: +
In the source text if a field has variable width padded with delimiters, the padding will be ignored. +
e.g. for texts of:
....
00000043 ViewReceiver I
000000b3 Peer I
....
with a dissection of %{a} %{b} %{c}; the padding is ignored, event.get([c]) -> "I"
[NOTE]
You probably want to use this filter inside an if block. +
This ensures that the event contains a field value with a suitable structure for the dissection.
For example... .... filter { if [type] == "syslog" or "syslog" in [tags] { dissect { mapping => { "message" => "%{ts} %{+ts} %{+ts} %{src} %{} %prog: %{msg}" } } } } ....
Defined Under Namespace
Modules: Filters