Class: Webhookdb::LoggedWebhook

Inherits:
Object
  • Object
show all
Includes:
Appydays::Configurable
Defined in:
lib/webhookdb/logged_webhook.rb

Defined Under Namespace

Classes: Resilient

Constant Summary collapse

DELETE_UNOWNED = 
14.days
DELETE_SUCCESSES = 
90.days
TRUNCATE_SUCCESSES = 
7.days
DELETE_FAILURES = 
90.days
TRUNCATE_FAILURES = 
30.days
RETRY_HEADER =

When we retry a request, set this so we know not to re-log it.

"Whdb-Logged-Webhook-Retry"
NONOVERRIDABLE_HEADERS =

When we retry a request, these headers must come from the Ruby client, NOT the original request.

[
  "Accept-Encoding",
  "Accept",
  "Host",
  "Version",
].to_set
WEBHOST_HEADERS =

These headers have been added by Heroku/our web host, so should not be part of the retry.

[
  "Connection",
  "Connect-Time",
  "X-Request-Id",
  "X-Forwarded-For",
  "X-Request-Start",
  "Total-Route-Time",
  "X-Forwarded-Port",
  "X-Forwarded-Proto",
  "Via",
].to_set

Class Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Class Attribute Details

.available_resilient_database_urls ⇒ Object

Returns the value of attribute available_resilient_database_urls.



11
12
13
 
# File 'lib/webhookdb/logged_webhook.rb', line 11

def available_resilient_database_urls
  @available_resilient_database_urls
end

Class Method Details

.resilient_insert(**kwargs) ⇒ Object

Insert the logged webhook, and fall back to inserting into the configured available_resilient_database_urls. If none are inserted successfully, raise the error; otherwise, swallow the insert error and more on.

Note that these resilient inserts are MUCH slower than normal inserts; they require a separate database connection, CREATE TABLE call, etc. But it’s a reasonable way to handle when the database is down.



176
177
178
 
# File 'lib/webhookdb/logged_webhook.rb', line 176

def self.resilient_insert(**kwargs)
  Resilient.new.insert(kwargs)
end

.resilient_replay ⇒ Object

Replay and delete all rows in the resilient database tables.



181
182
183
 
# File 'lib/webhookdb/logged_webhook.rb', line 181

def self.resilient_replay
  Resilient.new.replay
end

.retry_logs(instances, truncate_successful: false) ⇒ Object

Send instances back in ‘through the front door’ of this API. Return is a partition of [logs with 2xx responses, others]. Generally you can safely call ‘truncate_logs(result)`, or pass in (truncate_successful: true).



118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
 
# File 'lib/webhookdb/logged_webhook.rb', line 118

def self.retry_logs(instances, truncate_successful: false)
  successes, failures = instances.partition do |lw|
    uri = URI(Webhookdb.api_url + lw.request_path)
    req = Net::HTTP::Post.new(uri.path, {"Content-Type" => "application/json"})
    req.body = lw.request_body
    # This is going to have these headers:
    # ["content-type", "accept-encoding", "accept", "user-agent", "host"]
    # We want to keep all of these, except if user-agent or content-type were set
    # in the original request; then we want to use those.
    # Additionally, there are a whole set of headers we'll find on our webserver
    # that are added by our web platform, which we do NOT want to include.
    lw.request_headers.each do |k, v|
      next if Webhookdb::LoggedWebhook::WEBHOST_HEADERS.include?(k)
      next if Webhookdb::LoggedWebhook::NONOVERRIDABLE_HEADERS.include?(k)
      req[k] = v
    end
    req[Webhookdb::LoggedWebhook::RETRY_HEADER] = lw.id
    resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https") do |http|
      http.request(req)
    end
    resp.code.to_i < 400
  end
  self.truncate_logs(*successes) if truncate_successful
  return successes, failures
end

.trim(now: Time.now) ⇒ Object

Trim logged webhooks to keep this table to a reasonable size. The current trim algorithm and rationale is:

  • Logs that belong to inserts that were not part of an org are for our internal use only. They usually indicate an integration that was misconfigured, or is for an org that doesn’t exist. We keep these around for 2 weeks (they are always errors since they have no org). Ideally we investigate and remove them before that. We may need to ‘block’ certain opaque ids from being logged in the future, if for example we cannot get a client to turn off a misconfigured webhook.

  • Successful webhooks get their contents (request body and headers) truncated after 7 days (but the webhook row remains). Usually we don’t need to worry about these so in theory we can avoid logging verbose info at all.

  • Successful webhooks are deleted entirely after 90 days. Truncated webhooks are useful for statistics, but we can remove them earlier in the future.

  • Failed webhooks get their contents truncated after 30 days, but the webhook row remains. We have a longer truncation date so we have more time to investigate.

  • Error webhooks are deleted entirely after 90 days.



86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
 
# File 'lib/webhookdb/logged_webhook.rb', line 86

def self.trim(now: Time.now)
  owned = self.exclude(organization_id: nil)
  unowned = self.where(organization_id: nil)
  successes = owned.where { response_status < 400 }
  failures = owned.where { response_status >= 400 }
  # NOTE: This code is tightly coupled with indices created in 050_logged_webhooks_indices.rb
  # We create a separate index for each operation; the indices (5 in total) cover the full combination of:
  # - rows without an organization (idx 1)
  # - rows with an organization
  #   - rows already truncated
  #     - rows with status < 400 (idx 2)
  #     - rows with status >= 400 (idx 3)
  #   - rows not truncated
  #     - rows with status < 400 (idx 4)
  #     - rows with status >= 400 (idx 5)
  # Note that we only delete already-truncated rows so we can keep our indices smaller;
  # since deletion ages are always older than truncation ages, this should not be a problem.

  # Delete old unowned
  unowned.where { inserted_at < now - DELETE_UNOWNED }.delete
  # Delete successes first so they don't have to be truncated
  successes.where { inserted_at < now - DELETE_SUCCESSES }.exclude(truncated_at: nil).delete
  self.truncate_dataset(successes.where { inserted_at < now - TRUNCATE_SUCCESSES })
  # Delete failures
  failures.where { inserted_at < now - DELETE_FAILURES }.exclude(truncated_at: nil).delete
  self.truncate_dataset(failures.where { inserted_at < now - TRUNCATE_FAILURES })
end

.truncate_dataset(ds) ⇒ Object 



160
161
162
163
 
# File 'lib/webhookdb/logged_webhook.rb', line 160

def self.truncate_dataset(ds)
  ds = ds.where(truncated_at: nil)
  return ds.update(request_body: "", request_headers: "{}", truncated_at: Time.now)
end

.truncate_logs(*instances) ⇒ Object

Truncate the logs id’ed by the given instances. Instances are NOT modified; you need to .refresh to see truncated values.



155
156
157
158
 
# File 'lib/webhookdb/logged_webhook.rb', line 155

def self.truncate_logs(*instances)
  ds = self.where(id: instances.map(&:id))
  return self.truncate_dataset(ds)
end

Instance Method Details

#replay_async ⇒ Object 



149
150
151
 
# File 'lib/webhookdb/logged_webhook.rb', line 149

def replay_async
  return self.publish_immediate("replay", self.id)
end

#retry_one(truncate_successful: false) ⇒ Object 



144
145
146
147
 
# File 'lib/webhookdb/logged_webhook.rb', line 144

def retry_one(truncate_successful: false)
  _, bad = self.class.retry_logs([self], truncate_successful:)
  return bad.empty?
end

#truncated? ⇒ Boolean

Returns:

  • (Boolean) 


165
166
167
 
# File 'lib/webhookdb/logged_webhook.rb', line 165

def truncated?
  return self.truncated_at ? true : false
end