Class: LogStash::Filters::Translate

Inherits:
Base
  • Object
show all
Defined in:
lib/logstash/filters/translate.rb

Overview

A general search and replace tool which uses a configured hash and/or a YAML file to determine replacement values.

The dictionary entries can be specified in one of two ways: First, the “dictionary” configuration item may contain a hash representing the mapping. Second, an external YAML file (readable by logstash) may be specified in the “dictionary_path” configuration item. These two methods may not be used in conjunction; it will produce an error.

Operationally, if the event field specified in the “field” configuration matches the EXACT contents of a dictionary entry key (or matches a regex if “regex” configuration item has been enabled), the field’s value will be substituted with the matched key’s value from the dictionary.

By default, the translate filter will replace the contents of the maching event field (in-place). However, by using the “destination” configuration item, you may also specify a target event field to populate with the new translated value.

Alternatively, for simple string search and replacements for just a few values you might consider using the gsub function of the mutate filter.

Instance Method Summary collapse

Instance Method Details

#filter(event) ⇒ Object 



149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
 
# File 'lib/logstash/filters/translate.rb', line 149

def filter(event)
  return unless filter?(event)

  if @dictionary_path
    if @next_refresh < Time.now
      load_yaml
      @next_refresh = Time.now + @refresh_interval
      @logger.info("refreshing dictionary file")
    end
  end
  
  return unless event.include?(@field) # Skip translation in case event does not have @event field.
  return if event.include?(@destination) and not @override # Skip translation in case @destination field already exists and @override is disabled.

  begin
    #If source field is array use first value and make sure source value is string
    source = event[@field].is_a?(Array) ? event[@field].first.to_s : event[@field].to_s
    matched = false
    if @exact
      if @regex
        key = @dictionary.keys.detect{|k| source.match(Regexp.new(k))}
        if key
          event[@destination] = @dictionary[key]
          matched = true
        end
      elsif @dictionary.include?(source)
        event[@destination] = @dictionary[source]
        matched = true
      end
    else 
      translation = source.gsub(Regexp.union(@dictionary.keys), @dictionary)
      if source != translation
        event[@destination] = translation.force_encoding(Encoding::UTF_8)
        matched = true
      end
    end

    if not matched and @fallback
      event[@destination] = @fallback
      matched = true
    end
    filter_matched(event) if matched or @field == @destination
  rescue Exception => e
    @logger.error("Something went wrong when attempting to translate from dictionary", :exception => e, :field => @field, :event => event)
  end
end

#load_yaml(registering = false) ⇒ Object 



131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/logstash/filters/translate.rb', line 131

def load_yaml(registering=false)
  if !File.exists?(@dictionary_path)
    @logger.warn("dictionary file read failure, continuing with old dictionary", :path => @dictionary_path)
    return
  end

  begin
    @dictionary.merge!(YAML.load_file(@dictionary_path))
  rescue Exception => e
    if registering
      raise "#{self.class.name}: Bad Syntax in dictionary file #{@dictionary_path}"
    else
      @logger.warn("#{self.class.name}: Bad Syntax in dictionary file, continuing with old dictionary", :dictionary_path => @dictionary_path)
    end
  end
end

#register ⇒ Object 



115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
# File 'lib/logstash/filters/translate.rb', line 115

def register
  if @dictionary_path
    @next_refresh = Time.now + @refresh_interval
    registering = true
    load_yaml(registering)
  end
  
  @logger.debug? and @logger.debug("#{self.class.name}: Dictionary - ", :dictionary => @dictionary)
  if @exact
    @logger.debug? and @logger.debug("#{self.class.name}: Dictionary translation method - Exact")
  else
    @logger.debug? and @logger.debug("#{self.class.name}: Dictionary translation method - Fuzzy")
  end
end