FlippRubyKafka

Flipp's best practices for using Kafka producers and consumers. Built on top of Deimos, and also integrates DataDog metrics and APM.

See the Deimos gem for most configuration and usage.

Installation

Add this line to your application's Gemfile:

gem 'flipp-ruby-kafka', require: 'flipp_ruby_kafka'

And then execute:

$ bundle

Or install it yourself as:

$ gem install flipp-ruby-kafka

Versioning

We use semver for this gem. Any change in previous behavior is denoted with a bump in the minor version (0.4 -> 0.5). Patch versions are for bugfixes or new functionality which does not affect existing code. You should be locking your Gemfile to the minor version:

gem 'flipp-ruby-kafka', '~> 0.4', require: 'flipp_ruby_kafka'

Configuration

FlippRubyKafka comes with configuration out of the box for the Flipp Microservice Platform. Your application configuration should look something like this (in e.g. config/initializers/flipp_ruby_kafka.rb):

  FlippRubyKafka.configure do
    producers.schema_namespace 'com.flipp.my_app'
    # For integration testing - see below
    kafkateria_url ENV['KAFKATERIA_URL']
  end
  FlippRubyKafka.configure_datadog

FlippRubyKafka also changes the default for a number of settings to make your configuration smaller in the 80% case. The following are the setting defaults:

Config name	Default
logger	Clone of Rails logger
schema.backend	:avro_validation for development, :avro_schema_registry otherwise
schema.path	"#{Rails.root}/app/schemas"
schema.registry_url	`ENV['KAFKA_SCHEMA_REGISTRY'] \
consumers.report_lag	true for staging/production, false for development/test
consumers.fatal_error	true if ActiveRecord connection is not active
kafka.seed_brokers	`ENV['KAFKA_SSL_BROKERS'] \
kafka.ssl.enabled	true for staging/production, false for development/test
kafka.ssl.ca_cert	'/var/certs/kafka-broker-ca-public-cert'
kafka.ssl.client_cert	'/var/certs/kafka-client-public-key'
kafka.ssl.client_cert_key	'/var/certs/kafka-client-private-key-decrypted'
kafka.ssl.verify_hostname	false
kafkaeria_url	'http://localhost:3000'

Running consumers

FlippRubyKafka aliases rake deimos:start to rake phobos:start to support legacy applications.

Testing

FlippRubyKafka has helper methods for sending API requests to Kafkateria:

kafkateria_delete_topic('my-topic')

kafkateria_produce_messages(
  topic: 'my-topic',
  num_messages: 5,
  traits: %i(indexed),
  values: { merchant_id: 1 }
)

Make sure to set config.kafkateria_url when configuring FlippRubyKafka to enable this.

Additional Tools

FlippRubyKafka provides generators to hook into the platform build's features. These generate JSON config files to be used by the build script:

• rails g flipp_ruby_kafka:schema_validation will generate a schema_validation.json
• rails g flipp_ruby_kafka:topic_validation will generate a topic_validation.json

You can run both scripts in your build step - as long as the files are persisted to your workspace, you can use them in your deploy or in your separate validation steps.