Class: Jetpants::Pool

Inherits:
Object show all
Includes:
CallbackHandler
Defined in:
lib/jetpants/pool.rb

Overview

a Pool represents a group of database instances (Jetpants::DB objects).

The default implementation assumes that a Pool contains:

  • 1 master

  • 0 or more slaves, falling into one of these categories:

    • active slaves (actively taking production read queries)

    • standby slaves (for HA, promotable if a master or active slave fails + used to clone new replacements)

    • backup slaves (dedicated for backups and background jobs, never put into prod, potentially different hardware spec)

Plugins may of course override this extensively, to support different topologies, such as master-master trees.

Many of these methods are only useful in conjunction with an asset-tracker / configuration-generator plugin

Direct Known Subclasses

Shard

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from CallbackHandler

included

Constructor Details

#initialize(name, master) ⇒ Pool

Returns a new instance of Pool.



52
53
54
55
56
57
58
59
 
# File 'lib/jetpants/pool.rb', line 52

def initialize(name, master)
  @name = name
  @slave_name = false
  @aliases = []
  @master = master.to_db
  @master_read_weight = 0
  @active_slave_weights = {}
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(name, *args, &block) ⇒ Object

Jetpants::Pool proxies missing methods to the pool’s @master Jetpants::DB instance.



309
310
311
312
313
314
315
 
# File 'lib/jetpants/pool.rb', line 309

def method_missing(name, *args, &block)
  if @master.respond_to? name
    @master.send name, *args, &block
  else
    super
  end
end

Instance Attribute Details

#active_slave_weights ⇒ Object (readonly)

Hash mapping DB object => weight, for active (read) slaves. Default weight is 100. Safe to leave at default if your app framework doesn’t support different weights for individual read slaves. Weights have no effect inside Jetpants, but any asset tracker / config generator plugin can carry them through to the config file.



44
45
46
 
# File 'lib/jetpants/pool.rb', line 44

def active_slave_weights
  @active_slave_weights
end

#aliases ⇒ Object (readonly)

Array of strings containing other equivalent names for this pool



30
31
32
 
# File 'lib/jetpants/pool.rb', line 30

def aliases
  @aliases
end

#master ⇒ Object (readonly)

Jetpants::DB object that is the pool’s master



27
28
29
 
# File 'lib/jetpants/pool.rb', line 27

def master
  @master
end

#master_read_weight ⇒ Object

If the master also receives read queries, this stores its weight. Set to 0 if the master does not receive read queries (which is the default). This has no effect inside of Jetpants, but can be used by an asset tracker / config generator plugin to carry the value through to the config file.



50
51
52
 
# File 'lib/jetpants/pool.rb', line 50

def master_read_weight
  @master_read_weight
end

#name ⇒ Object (readonly)

human-readable String name of pool



24
25
26
 
# File 'lib/jetpants/pool.rb', line 24

def name
  @name
end

#slave_name ⇒ Object

Can be used to store a name that refers to just the active_slaves, for instance if your framework isn’t smart enough to know about master/slave relationships. Safe to leave as nil otherwise. Has no effect in Jetpants, but an asset tracker / config generator plugin may include this in the generated config file.



37
38
39
 
# File 'lib/jetpants/pool.rb', line 37

def slave_name
  @slave_name
end

Instance Method Details

#active_slaves ⇒ Object

Returns an array of Jetpants::DB objects. Active slaves are ones that receive read queries from your application.



75
76
77
 
# File 'lib/jetpants/pool.rb', line 75

def active_slaves
  @master.slaves.select {|sl| @active_slave_weights[sl]}
end

#add_alias(name) ⇒ Object

Informs this pool that it has an alias. A pool may have any number of aliases.



143
144
145
146
147
148
149
150
 
# File 'lib/jetpants/pool.rb', line 143

def add_alias(name)
  if @aliases.include? name
    false
  else
    @aliases << name
    true
  end
end

#backup_slaves ⇒ Object

Returns an array of Jetpants::DB objects. Backup slaves are never promoted to active or master. They are for dedicated backup purposes. They may be a different/cheaper hardware spec than other slaves.



90
91
92
 
# File 'lib/jetpants/pool.rb', line 90

def backup_slaves
  @master.slaves.reject {|sl| @active_slave_weights[sl] || !sl.for_backups?}
end

#before_sync_configuration ⇒ Object

Callback to ensure that a sync’ed pool is already in Topology.pools



285
286
287
288
289
 
# File 'lib/jetpants/pool.rb', line 285

def before_sync_configuration
  unless Jetpants.topology.pools.include? self
    Jetpants.topology.pools << self
  end
end

#has_active_slave(slave_db, weight = 100) ⇒ Object

Informs Jetpants that slave_db is an active slave. Potentially used by plugins, such as in Topology at start-up time.



102
103
104
105
106
 
# File 'lib/jetpants/pool.rb', line 102

def has_active_slave(slave_db, weight=100)
  slave_db = slave_db.to_db
  raise "Attempt to mark a DB as its own active slave" if slave_db == @master
  @active_slave_weights[slave_db] = weight
end

#mark_slave_active(slave_db, weight = 100) ⇒ Object

Turns a standby slave into an active slave, giving it the specified read weight. Syncs the pool’s configuration afterwards. It’s up to your asset tracker plugin to actually do something with this information.



111
112
113
114
115
 
# File 'lib/jetpants/pool.rb', line 111

def mark_slave_active(slave_db, weight=100)
  raise "Attempt to make a backup slave be an active slave" if slave_db.for_backups?
  has_active_slave slave_db, weight
  sync_configuration
end

#mark_slave_standby(slave_db) ⇒ Object

Turns an active slave into a standby slave. Syncs the pool’s configuration afterwards. It’s up to your asset tracker plugin to actually do something with this information.



119
120
121
122
123
124
 
# File 'lib/jetpants/pool.rb', line 119

def mark_slave_standby(slave_db)
  slave_db = slave_db.to_db
  raise "Cannot call mark_slave_standby on a master" if slave_db == @master
  @active_slave_weights.delete(slave_db)
  sync_configuration
end

#master_promotion!(promoted) ⇒ Object

Demotes the pool’s existing master, promoting a slave in its place.



194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/jetpants/pool.rb', line 194

def master_promotion!(promoted)
  demoted = @master
  raise "Demoted node is already the master of this pool!" if demoted == promoted
  raise "Promoted host is not in the right pool!" unless demoted.slaves.include?(promoted)
  
  output "Preparing to demote master #{demoted} and promote #{promoted} in its place."
  
  # If demoted machine is available, confirm it is read-only and binlog isn't moving,
  # and then wait for slaves to catch up to this position
  if demoted.running?
    demoted.enable_read_only! unless demoted.read_only?
    raise "Unable to enable global read-only mode on demoted machine" unless demoted.read_only?
    coordinates = demoted.binlog_coordinates
    raise "Demoted machine still taking writes (from superuser or replication?) despite being read-only" unless coordinates == demoted.binlog_coordinates
    demoted.slaves.concurrent_each do |s|
      while true do
        sleep 1
        break if s.repl_binlog_coordinates == coordinates
        output "Still catching up to coordinates of demoted master"
      end
    end
  
  # Demoted machine not available -- wait for slaves' binlogs to stop moving
  else
    demoted.slaves.concurrent_each do |s|
      progress = s.repl_binlog_coordinates
      while true do
        sleep 1
        break if s.repl_binlog_coordinates == progress
        s.output "Still catching up on replication"
      end
    end
  end
  
  # Stop replication on all slaves
  replicas = demoted.slaves.dup
  replicas.each do |s|
    s.pause_replication if s.replicating?
  end
  raise "Unable to stop replication on all slaves" if replicas.any? {|s| s.replicating?}
  
  user, password = promoted.replication_credentials.values
  log,  position = promoted.binlog_coordinates
  
  # reset slave on promoted, and make sure read_only is disabled
  promoted.disable_replication!
  promoted.disable_read_only!
  
  # gather our new replicas
  replicas.delete promoted
  replicas << demoted if demoted.running?
  
  # perform promotion
  replicas.each do |r|
    r.change_master_to promoted, user: user, password: password, log_file: log, log_pos: position
  end

  # ensure our replicas are configured correctly by comparing our staged values to current values of replicas
  promoted_replication_config = {
    master_host: promoted.ip,
    master_user: user,
    master_log_file:  log,
    exec_master_log_pos: position.to_s
  }
  replicas.each do |r|
    promoted_replication_config.each do |option, value|
      raise "Unexpected slave status value for #{option} in replica #{r} after promotion" unless r.slave_status[option] == value
    end
    r.resume_replication unless r.replicating?
  end
  
  # Update the pool
  # Note: if the demoted machine is not available, plugin may need to implement an
  # after_master_promotion! method which handles this case in configuration tracker
  @active_slave_weights.delete promoted # if promoting an active slave, remove it from read pool
  @master = promoted
  sync_configuration
  Jetpants.topology.write_config
  
  output "Promotion complete. Pool master is now #{promoted}."
  
  replicas.all? {|r| r.replicating?}
end

#nodes ⇒ Object

returns a flat array of all Jetpants::DB objects in the pool: the master and all slaves of all types.



96
97
98
 
# File 'lib/jetpants/pool.rb', line 96

def nodes
  [master, slaves].flatten.compact
end

#output(str) ⇒ Object

Displays the provided output, along with information about the current time, and self (the name of this Pool)



298
299
300
301
302
303
304
305
306
 
# File 'lib/jetpants/pool.rb', line 298

def output(str)
  str = str.to_s.strip
  str = nil if str && str.length == 0
  str ||= "Completed (no output)"
  output = Time.now.strftime("%H:%M:%S") + " [#{self}] "
  output << str
  print output + "\n"
  output
end

#remove_slave!(slave_db) ⇒ Object

Remove a slave from a pool entirely. This is destructive, ie, it does a RESET SLAVE on the db. Note that a plugin may want to override this (or implement after_remove_slave!) to actually sync the change to an asset tracker, depending on how the plugin implements Pool#sync_configuration. (If the implementation makes sync_configuration work by iterating over the pool’s current slaves to update their status/role/pool, it won’t see any slaves that have been removed, and therefore won’t update them.)



133
134
135
136
137
138
139
140
 
# File 'lib/jetpants/pool.rb', line 133

def remove_slave!(slave_db)
  raise "Slave is not in this pool" unless slave_db.pool == self
  slave_db.disable_monitoring
  slave_db.stop_replication
  slave_db.repl_binlog_coordinates # displays how far we replicated, in case you need to roll back this change manually
  slave_db.disable_replication!
  sync_configuration # may or may not be sufficient -- see note above.
end

#respond_to?(name, include_private = false) ⇒ Boolean

Returns:

  • (Boolean) 


317
318
319
 
# File 'lib/jetpants/pool.rb', line 317

def respond_to?(name, include_private=false)
  super || @master.respond_to?(name)
end

#slaves(type = false) ⇒ Object

Returns all slaves, or pass in :active, :standby, or :backup to receive slaves just of a particular type



63
64
65
66
67
68
69
70
71
 
# File 'lib/jetpants/pool.rb', line 63

def slaves(type=false)
  case type
  when :active  then active_slaves
  when :standby then standby_slaves
  when :backup  then backup_slaves
  when false    then @master.slaves
  else []
  end
end

#standby_slaves ⇒ Object

Returns an array of Jetpants::DB objects. Standby slaves do not receive queries from your application. These are for high availability. They can be turned into active slaves or even the master, and can also be used for cloning additional slaves.



83
84
85
 
# File 'lib/jetpants/pool.rb', line 83

def standby_slaves
  @master.slaves.reject {|sl| @active_slave_weights[sl] || sl.for_backups?}
end

#summary(extended_info = false) ⇒ Object

Displays a summary of the pool’s members. This outputs immediately instead of returning a string, so that you can invoke something like:

Jetpants.topology.pools.each &:summary

to easily display a summary.



156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
 
# File 'lib/jetpants/pool.rb', line 156

def summary(extended_info=false)
  probe

  alias_text = @aliases.count > 0 ? '  (aliases: ' + @aliases.join(', ') + ')' : ''
  data_size = @master.running? ? "[#{master.data_set_size(true)}GB]" : ''
  state_text = (respond_to?(:state) && state != :ready ? "  (state: #{state})" : '')
  print "#{name}#{alias_text}#{state_text}  #{data_size}\n"
  
  if extended_info
    details = {}
    nodes.concurrent_each do |s|
      if !s.running?
        details[s] = {coordinates: ['unknown'], lag: 'N/A'}
      elsif s == @master
        details[s] = {coordinates: s.binlog_coordinates(false), lag: 'N/A'}
      else
        lag = s.seconds_behind_master
        lag_str = lag.nil? ? 'NULL' : lag.to_s + 's'
        details[s] = {coordinates: s.repl_binlog_coordinates(false), lag: lag_str}
      end
    end
  end
  
  binlog_pos = extended_info ? details[@master][:coordinates].join(':') : ''
  print "\tmaster          = %-15s %-30s %s\n" % [@master.ip, @master.hostname, binlog_pos]
  
  [:active, :standby, :backup].each do |type|
    slave_list = slaves(type)
    slave_list.sort.each_with_index do |s, i|
      binlog_pos = extended_info ? details[s][:coordinates].join(':') : ''
      slave_lag = extended_info ? "lag=#{details[s][:lag]}" : ''
      print "\t%-7s slave #{i + 1} = %-15s %-30s %-26s %s\n" % [type, s.ip, s.hostname, binlog_pos, slave_lag]
    end
  end
  true
end

#sync_configuration ⇒ Object

Informs your asset tracker about any changes in the pool’s state or members. Plugins should override this, or use before_sync_configuration / after_sync_configuration callbacks, to provide an implementation of this method.



281
282
 
# File 'lib/jetpants/pool.rb', line 281

def sync_configuration
end

#to_s ⇒ Object

Returns the name of the pool.



292
293
294
 
# File 'lib/jetpants/pool.rb', line 292

def to_s
  @name
end