Rdkafka

The rdkafka gem is a modern Kafka client library for Ruby based on librdkafka. It wraps the production-ready C client using the ffi gem and targets Kafka 0.10+ and Ruby 2.1+.

This gem only provides a high-level Kafka consumer. If you are running an older version of Kafka and/or need the legacy simple consumer we suggest using the Hermann gem.

Installation

This gem downloads and compiles librdkafka when it is installed. If you have any problems installing the gem please open an issue.

Usage

See the documentation for full details on how to use this gem. Two quick examples:

Consuming messages

“by

config = {

“bootstrap.servers” => “localhost:9092”,

“group.id” => “ruby-test” } consumer = Rdkafka::Config.new(config).consumer consumer.subscribe(“ruby-test-topic”)

consumer.each do |message| puts “Message received: #message” end

“

Producing messages

“by config = producer = Rdkafka::Config.new(config).producer

100.times do |i| puts “Producing message #i” producer.produce( topic: “ruby-test-topic”, payload: “Payload #i”, key: “Key #i” ).wait end

“

Development

For development we expect a local zookeeper and kafka instance to be running. Run bundle and cd ext && bundle exec rake && cd ... Then create the topics as expected in the specs: bundle exec rake create_topics.

You can then run bundle exec rspec to run the tests. To see rdkafka debug output:

“EBUG_PRODUCER=true bundle exec rspec DEBUG_CONSUMER=true bundle exec rspec

“

To see everything working run these in separate tabs:

“undle exec rake consume_messages bundle exec rake produce_messages

“