Class: Kafka::Cluster
- Inherits:
-
Object
- Object
- Kafka::Cluster
- Defined in:
- lib/kafka/cluster.rb
Overview
A cluster represents the state of a Kafka cluster. It needs to be initialized with a non-empty list of seed brokers. The first seed broker that the cluster can connect to will be asked for the cluster metadata, which allows the cluster to map topic partitions to the current leader for those partitions.
Instance Method Summary collapse
-
#add_target_topics(topics) ⇒ nil
Adds a list of topics to the target list.
-
#clear_target_topics ⇒ nil
Clears the list of target topics.
- #disconnect ⇒ Object
- #get_group_coordinator(group_id:) ⇒ Object
-
#get_leader(topic, partition) ⇒ Broker
Finds the broker acting as the leader of the given topic and partition.
-
#initialize(seed_brokers:, broker_pool:, logger:) ⇒ Cluster
constructor
Initializes a Cluster with a set of seed brokers.
- #mark_as_stale! ⇒ Object
- #partitions_for(topic) ⇒ Object
- #refresh_metadata! ⇒ Object
- #refresh_metadata_if_necessary! ⇒ Object
- #resolve_offset(topic, partition, offset) ⇒ Object
- #resolve_offsets(topic, partitions, offset) ⇒ Object
- #topics ⇒ Object
Constructor Details
#initialize(seed_brokers:, broker_pool:, logger:) ⇒ Cluster
Initializes a Cluster with a set of seed brokers.
The cluster will try to fetch cluster metadata from one of the brokers.
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# File 'lib/kafka/cluster.rb', line 19 def initialize(seed_brokers:, broker_pool:, logger:) if seed_brokers.empty? raise ArgumentError, "At least one seed broker must be configured" end @logger = logger @seed_brokers = seed_brokers @broker_pool = broker_pool @cluster_info = nil @stale = true # This is the set of topics we need metadata for. If empty, metadata for # all topics will be fetched. @target_topics = Set.new end |
Instance Method Details
#add_target_topics(topics) ⇒ nil
Adds a list of topics to the target list. Only the topics on this list will be queried for metadata.
40 41 42 43 44 45 46 47 48 49 50 |
# File 'lib/kafka/cluster.rb', line 40 def add_target_topics(topics) new_topics = Set.new(topics) - @target_topics unless new_topics.empty? @logger.info "New topics added to target list: #{new_topics.to_a.join(', ')}" @target_topics.merge(new_topics) end end |
#clear_target_topics ⇒ nil
Clears the list of target topics.
56 57 58 59 |
# File 'lib/kafka/cluster.rb', line 56 def clear_target_topics @target_topics.clear end |
#disconnect ⇒ Object
184 185 186 |
# File 'lib/kafka/cluster.rb', line 184 def disconnect @broker_pool.close end |
#get_group_coordinator(group_id:) ⇒ Object
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
# File 'lib/kafka/cluster.rb', line 83 def get_group_coordinator(group_id:) @logger.debug "Getting group coordinator for `#{group_id}`" cluster_info.brokers.each do |broker_info| begin broker = connect_to_broker(broker_info.node_id) response = broker.find_group_coordinator(group_id: group_id) Protocol.handle_error(response.error_code) coordinator_id = response.coordinator_id @logger.debug "Coordinator for group `#{group_id}` is #{coordinator_id}. Connecting..." # It's possible that a new broker is introduced to the cluster and # becomes the coordinator before we have a chance to refresh_metadata. coordinator = begin connect_to_broker(coordinator_id) rescue Kafka::NoSuchBroker @logger.debug "Broker #{coordinator_id} missing from broker cache, refreshing" connect_to_broker(coordinator_id) end @logger.debug "Connected to coordinator: #{coordinator} for group `#{group_id}`" return coordinator rescue GroupCoordinatorNotAvailable @logger.debug "Coordinator not available; retrying in 1s" sleep 1 retry rescue ConnectionError => e @logger.error "Failed to get group coordinator info from #{broker}: #{e}" end end raise Kafka::Error, "Failed to find group coordinator" end |
#get_leader(topic, partition) ⇒ Broker
Finds the broker acting as the leader of the given topic and partition.
79 80 81 |
# File 'lib/kafka/cluster.rb', line 79 def get_leader(topic, partition) connect_to_broker(get_leader_id(topic, partition)) end |
#mark_as_stale! ⇒ Object
61 62 63 |
# File 'lib/kafka/cluster.rb', line 61 def mark_as_stale! @stale = true end |
#partitions_for(topic) ⇒ Object
124 125 126 127 128 129 130 131 |
# File 'lib/kafka/cluster.rb', line 124 def partitions_for(topic) add_target_topics([topic]) cluster_info.partitions_for(topic) rescue Kafka::ProtocolError mark_as_stale! raise end |
#refresh_metadata! ⇒ Object
65 66 67 68 |
# File 'lib/kafka/cluster.rb', line 65 def @cluster_info = nil cluster_info end |
#refresh_metadata_if_necessary! ⇒ Object
70 71 72 |
# File 'lib/kafka/cluster.rb', line 70 def if @stale end |
#resolve_offset(topic, partition, offset) ⇒ Object
176 177 178 |
# File 'lib/kafka/cluster.rb', line 176 def resolve_offset(topic, partition, offset) resolve_offsets(topic, [partition], offset).fetch(partition) end |
#resolve_offsets(topic, partitions, offset) ⇒ Object
133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 |
# File 'lib/kafka/cluster.rb', line 133 def resolve_offsets(topic, partitions, offset) add_target_topics([topic]) partitions_by_broker = partitions.each_with_object({}) {|partition, hsh| broker = get_leader(topic, partition) hsh[broker] ||= [] hsh[broker] << partition } if offset == :earliest offset = -2 elsif offset == :latest offset = -1 end offsets = {} partitions_by_broker.each do |broker, broker_partitions| response = broker.list_offsets( topics: { topic => broker_partitions.map {|partition| { partition: partition, time: offset, max_offsets: 1, } } } ) broker_partitions.each do |partition| offsets[partition] = response.offset_for(topic, partition) end end offsets rescue Kafka::ProtocolError mark_as_stale! raise end |
#topics ⇒ Object
180 181 182 |
# File 'lib/kafka/cluster.rb', line 180 def topics cluster_info.topics.map(&:topic_name) end |