Rdkafka
The rdkafka
gem is a modern Kafka client library for Ruby based on
librdkafka.
It wraps the production-ready C client using the ffi
gem and targets Kafka 0.10+ and Ruby 2.1+.
This gem only provides a high-level Kafka consumer. If you are running an older version of Kafka and/or need the legacy simple consumer we suggest using the Hermann gem.
Installation
This gem downloads and compiles librdkafka when it is installed. If you have any problems installing the gem please open an issue.
Usage
See the documentation for full details on how to use this gem. Two quick examples:
Consuming messages
config = {
:"bootstrap.servers" => "localhost:9092",
:"group.id" => "ruby-test"
}
consumer = Rdkafka::Config.new(config).consumer
consumer.subscribe("ruby-test-topic")
consumer.each do ||
puts "Message received: #{}"
end
Producing messages
config = {:"bootstrap.servers" => "localhost:9092"}
producer = Rdkafka::Config.new(config).producer
100.times do |i|
puts "Producing message #{i}"
producer.produce(
topic: "ruby-test-topic",
payload: "Payload #{i}",
key: "Key #{i}"
).wait
end
Development
For development we expect a local zookeeper and kafka instance to be
running. Run bundle
and cd ext && bundle exec rake && cd ..
. Then
create the topics as expected in the specs: bundle exec rake create_topics
.
You can then run bundle exec rspec
to run the tests. To see rdkafka
debug output:
DEBUG_PRODUCER=true bundle exec rspec
DEBUG_CONSUMER=true bundle exec rspec
To see everything working run these in separate tabs:
bundle exec rake
bundle exec rake