fluent-plugin-tag-normaliser

Tag-normaliser is a fluentd plugin to help re-tag logs with Kubernetes metadata coming from fluent-bit kubernetes filter.

Installation

RubyGems

$ gem install fluent-plugin-tag-normaliser

Bundler

Add following line to your Gemfile:

gem "fluent-plugin-tag-normaliser"

And then execute:

$ bundle

Configuration

You only need to specify the format option.

<match example.tag**>
  @type tag_normaliser
  format cluster1.${namespace_name}.${pod_name}.${labels.app}
</match>
Parameter Description Default
key_prefix Prefix used to access record attributes kubernetes
format Format to rewrite tag to. You can access record with ${record_name} expression. Nested attributes available via . separator: ${labels.app} ""
unknown Fallback value for missing record attribute "unknown

Available fluent-bit provided kubernetes attributes

Parameter Description Example
pod_name Pod name understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n
container_name Container name inside the Pod nginx-logging-demo
namespace_name Namespace name default
pod_id Kubernetes UUID for Pod 1f50d309-45a6-11e9-b795-025000000001
labels Kubernetes Pod labels. This is a nested map. You can access nested attributes via . {"app":"nginx-logging-demo", "pod-template-hash":"7dcdcfdcd7" }
host Node hostname the Pod runs on docker-desktop
docker_id Docker UUID of the container 3a38148aa37aa3...

Example

Configuration

<match example.tag**>
  @type tag_normaliser
  format cluster1.${namespace_name}.${pod_name}.${labels.app}
</match>

Input

tag = "kubernetes.var.log.containers.nginx-s6fvr_default_nginx-889831b33146bb9ec28a5700442b53c6cdf4c445fb5b71c0c48cfa27aaa00212.log"

{  
   "log":"10.1.0.1 - - [13/Mar/2019:15:42:31 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"kube-probe/1.13\" \"-\"\n",
   "kubernetes":{
      "pod_name":"understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n",
      "namespace_name":"default",
      "pod_id":"1f50d309-45a6-11e9-b795-025000000001",
      "labels":{  
         "app":"nginx-logging-demo",
         "pod-template-hash":"7dcdcfdcd7",
         "release":"understood-butterfly"
      },
      "host":"docker-desktop",
      "container_name":"nginx-logging-demo",
      "docker_id":"3a38148aa37aa30e6e2df96af95cbda7a47b0428689bb4152413f4be25532fda"
   }
}

Output

tag = "cluster1.default.understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n.nginx-logging-demo",

You can use the plugin without Kubernetes labels just set key_prefix to empty string.

  • Copyright(c) 2019- Banzai Cloud
  • License
    • Apache License, Version 2.0