Class: LogstashWriter
- Inherits:
-
Object
- Object
- LogstashWriter
- Defined in:
- lib/logstash_writer.rb
Overview
Write messages to a logstash server.
Flings events, represented as JSON objects, to logstash using the
json_lines
codec (over TCP). Doesn't do any munging or modification of
the event data given to it, other than adding a @timestamp
field if
it doesn't already exist.
We support highly-available logstash installations by means of multiple address records, or via SRV records. See the docs for .new for details as to the valid formats for the server.
Constant Summary collapse
- INITIAL_RETRY_WAIT =
How long, in seconds, to pause the first time an error is encountered. Each successive error will cause a longer wait, so as to prevent thundering herds.
0.5
Instance Method Summary collapse
-
#force_disconnect! ⇒ NilClass
Disconnect from the currently-active server.
-
#initialize(server_name:, logger: Logger.new("/dev/null"), backlog: 1_000, metrics_registry: Prometheus::Client::Registry.new, metrics_prefix: :logstash_writer) ⇒ LogstashWriter
constructor
Create a new logstash writer.
-
#run ⇒ Object
Send events.
-
#send_event(e) ⇒ NilClass
Add an event to the queue, to be sent to logstash.
-
#shutdown ⇒ Object
Tell the LogstashWriter to flush its queue and terminate operation.
-
#start! ⇒ NilClass
Start sending events.
-
#stop! ⇒ NilClass
Stop the worker thread.
Constructor Details
#initialize(server_name:, logger: Logger.new("/dev/null"), backlog: 1_000, metrics_registry: Prometheus::Client::Registry.new, metrics_prefix: :logstash_writer) ⇒ LogstashWriter
Create a new logstash writer.
Once the object is created, you're ready to give it messages by calling #send_event. No messages will actually be delivered to logstash, though, until you call #run.
If multiple addresses are returned from an A/AAAA resolution, or multiple SRV records, then the records will all be tried in random order (for A/AAAA records) or in line with the standard rules for weight and priority (for SRV records).
75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
# File 'lib/logstash_writer.rb', line 75 def initialize(server_name:, logger: Logger.new("/dev/null"), backlog: 1_000, metrics_registry: Prometheus::Client::Registry.new, metrics_prefix: :logstash_writer) @server_name, @logger, @backlog = server_name, logger, backlog @metrics = { received: metrics_registry.counter(:"#{metrics_prefix}_events_received_total", docstring: "The number of logstash events which have been submitted for delivery"), sent: metrics_registry.counter(:"#{metrics_prefix}_events_written_total", docstring: "The number of logstash events which have been delivered to the logstash server", labels: %i{server}), queue_size: metrics_registry.gauge(:"#{metrics_prefix}_queue_size", docstring: "The number of events currently in the queue to be sent"), dropped: metrics_registry.counter(:"#{metrics_prefix}_events_dropped_total", docstring: "The number of events which have been dropped from the queue"), lag: metrics_registry.gauge(:"#{metrics_prefix}_last_sent_event_time_seconds", docstring: "When the last event successfully sent to logstash was originally received"), connected: metrics_registry.gauge(:"#{metrics_prefix}_connected_to_server", docstring: "Boolean flag indicating whether we are currently connected to a logstash server", labels: %i{server}), connect_exception: metrics_registry.counter(:"#{metrics_prefix}_connect_exceptions_total", docstring: "The number of exceptions that have occurred whilst attempting to connect to a logstash server", labels: %i{server class}), write_exception: metrics_registry.counter(:"#{metrics_prefix}_write_exceptions_total", docstring: "The number of exceptions that have occurred whilst attempting to write an event to a logstash server", labels: %i{server class}), write_loop_exception: metrics_registry.counter(:"#{metrics_prefix}_write_loop_exceptions_total", docstring: "The number of exceptions that have occurred in the writing loop", labels: %i{class}), write_loop_ok: metrics_registry.gauge(:"#{metrics_prefix}_write_loop_ok", docstring: "Boolean flag indicating whether the writing loop is currently operating correctly, or is in a post-apocalyptic hellscape of never-ending exceptions"), } @metrics[:lag].set(0) @metrics[:queue_size].set(0) metrics_registry.gauge(:"#{metrics_prefix}_queue_max", docstring: "The maximum size of the event queue").set(backlog) # We can't use a stdlib Queue object because we need to re-push items # onto the front of the queue in case of error @queue = [] @queue_mutex = Mutex.new @queue_cv = ConditionVariable.new @socket_mutex = Mutex.new @worker_mutex = Mutex.new end |
Instance Method Details
#force_disconnect! ⇒ NilClass
Disconnect from the currently-active server.
In certain circumstances, you may wish to force the writer to stop
sending messages to the currently-connected logstash server, and
re-resolve the server_name
to get new a new address to talk to.
Calling this method will cause that to happen.
227 228 229 230 231 232 233 234 235 236 237 |
# File 'lib/logstash_writer.rb', line 227 def force_disconnect! @socket_mutex.synchronize do return if @current_target.nil? @logger.info("LogstashWriter") { "Forced disconnect from #{@current_target.describe_peer}" } @current_target.close @current_target = nil end nil end |
#run ⇒ Object
Send events.
Does not return until #shutdown
is called (in another thread).
145 146 147 148 149 150 151 |
# File 'lib/logstash_writer.rb', line 145 def run @queue_mutex.synchronize do @terminate = false end write_loop end |
#send_event(e) ⇒ NilClass
Add an event to the queue, to be sent to logstash. Actual event delivery will happen in a worker thread that is started with
run. If the event does not have a @timestamp
field, it will
be added set to the current time.
118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
# File 'lib/logstash_writer.rb', line 118 def send_event(e) unless e.is_a?(Hash) raise ArgumentError, "Event must be a hash" end unless e.has_key?(:@timestamp) || e.has_key?("@timestamp") e[:@timestamp] = Time.now.utc.strftime("%FT%T.%NZ") end @queue_mutex.synchronize do @queue << { content: e, arrival_timestamp: Time.now } while @queue.length > @backlog @queue.shift stat_dropped end @queue_cv.signal stat_received end nil end |
#shutdown ⇒ Object
Tell the LogstashWriter to flush its queue and terminate operation.
Returns immediately.
157 158 159 160 161 162 163 164 165 166 |
# File 'lib/logstash_writer.rb', line 157 def shutdown #:nocov: @worker_mutex.synchronize do @queue_mutex.synchronize do @terminate = true @queue_cv.signal end end #:nocov: end |
#start! ⇒ NilClass
Start sending events.
This method will return almost immediately, and actual event transmission will commence in a separate thread.
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 |
# File 'lib/logstash_writer.rb', line 175 def start! @worker_mutex.synchronize do if @worker_thread.nil? @queue_mutex.synchronize do @terminate = false end @worker_thread = Thread.new do write_loop end end end nil end |
#stop! ⇒ NilClass
Stop the worker thread.
Politely ask the worker thread to please finish up once it's finished sending all messages that have been queued. This will return once the worker thread has finished.
199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 |
# File 'lib/logstash_writer.rb', line 199 def stop! @worker_mutex.synchronize do if @worker_thread @queue_mutex.synchronize do @terminate = true @queue_cv.signal end begin @worker_thread.join unless @worker_thread == Thread.current rescue Exception => ex @logger.error("LogstashWriter") { (["Worker thread terminated with exception: #{ex.} (#{ex.class})"] + ex.backtrace).join("\n ") } end @worker_thread = nil end end nil end |