Kubernetes::Health
This gem open a HTTP port for monitoring your rails app while it is running Migrates, Sidekiq and Puma.
Features
- Puma and Sidekiq metrics for autoscaling.
- Prometheus and JSON metrics (tested using https://github.com/zalando-incubator/kube-metrics-adapter and JSON format).
- add routes
/_readiness
,/_liveness
on Rails Stack. - add routes
/_readiness
,/_liveness
and/_metrics
as a puma plugin at another port to avoid problems when your app get busy. (code copied frompuma-metrics
gem). - add routes
/_readiness
and/_liveness
whilerake db:migrate
runs. (optional) - add routes
/_readiness
and/_liveness
whilesidekiq
runs. (optional) - add support to avoid parallel running of
rake db:migrate
while keep kubernetes waiting (PostgreSQL required). - allow custom checks for
/_readiness
and/_liveness
.
Installation
Add this line to your application's Gemfile:
gem 'kubernetes-health', '~> 3.6'
Enabling puma plugin
add in config/puma.rb
plugin 'kubernetes'
kubernetes_url 'tcp://0.0.0.0:9393'
In Kubernetes you need to configure your deployment readinessProbe
and livenessProbe
like this:
livenessProbe:
httpGet:
path: /_liveness
port: 9393
initialDelaySeconds: 30
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
readinessProbe:
httpGet:
path: /_readiness
port: 9393
initialDelaySeconds: 30
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
Setting failureThreshold
is import to avoid problems when app finish migrates and is starting the web process.
Enabling monitoring while rake db:migrate
runs
Your Dockerfile's entry script needs to run migrates before start your web app.
Add KUBERNETES_HEALTH_ENABLE_RACK_ON_MIGRATE=true
environment variable.
or add in your application.rb
.
# default: false
Kubernetes::Health::Config.enable_rack_on_migrate = true
The defined port at config/puma.rb
will be used.
Enabling monitoring for sidekiq
Add KUBERNETES_HEALTH_ENABLE_RACK_ON_SIDEKIQ=true
environment variable.
or add in your application.rb
.
# default: false
Kubernetes::Health::Config.enable_rack_on_sidekiq = true
The defined port at config/puma.rb
will be used.
How rake db:migrate
and sidekiq
monitoring works
It will run a RACK server for /_readiness
, /_liveness
and /_metrics
.
Avoiding migrations running in parallel and making kubernetes happy.
Rails already avoid migrations running in parallel, but it raise exceptions. This gem will just wait for other migrations without exit.
If you enable rack_on_migrate
together with this, kubernetes will just wait, avoiding erros.
Add KUBERNETES_HEALTH_ENABLE_LOCK_ON_MIGRATE=true
environment variable.
or add in your application.rb
.
# default: false
Kubernetes::Health::Config.enable_lock_on_migrate = true
Customizing locking
By default it is working for PostgreSQL, but you can customize it using a lambda:
Kubernetes::Health::Config.lock_or_wait = lambda {
ActiveRecord::Base.connection.execute 'select pg_advisory_lock(123456789123456789);'
}
Kubernetes::Health::Config.unlock = lambda {
ActiveRecord::Base.connection.execute 'select pg_advisory_unlock(123456789123456789);'
}
Customizing checks
It only works for routes in rails stack, they are not executed while rake db:migrate
runs.
I prefer do nothing else on liveness
to avoid unnecessary CrashLoopBackOff
status. params
is optional (request params).
Kubernetes::Health::Config.live_if = lambda { |params|
true
}
Ex. Check if PostgreSQL is reachable on readiness
indicating that credentials are setup right and keeps cache to avoid doing it a lot. params
is optional (request params).
Kubernetes::Health::Config.ready_if = lambda { |params|
return $kubernetes_health_test_db_connection if $kubernetes_health_test_db_connection
$kubernetes_health_test_db_connection = ActiveRecord::Base.connection.execute("SELECT 1").cmd_tuples == 1
}
Customizing routes
Kubernetes::Health::Config.route_liveness = '/liveness'
Kubernetes::Health::Config.route_readiness = '/readiness'
Kubernetes::Health::Config.route_metrics = '/metrics'
Response format
If you are using https://github.com/zalando-incubator/kube-metrics-adapter
you will want to use json
format.
Default is prometheus
.
Kubernetes::Health::Config.response_format = 'json'
Customizing requests logs
Kubernetes::Health::Config.request_log_callback = lambda { |req, http_code, content|
Rails.logger.debug "Kubernetes Health: Rack on Migrate - Request: Path: #{req.path_info} / Params: #{req.params} / HTTP Code: #{http_code}\n#{content}" rescue nil
}