Top Level Namespace

Defined Under Namespace

Modules: Adhd, AdhdRESTServer, ProxyToServer, YAML Classes: Array, ContentDoc, ContentShard, Hash, Node, NodeDB, Object, ShardRange, ShardRangeDB

Instance Method Summary collapse

Instance Method Details

#assign_nodes_to_shards(node_list, shard_range_list, replication_factor) ⇒ Object

This is an automatic way to allocate shards to nodes that just arrive in the networks, as well as re-allocate shards if nodes become unavailable or leave the network.

NOTE: How to build skynet (Part III)

The invariant we try to impose on the list of nodes part of a shard
is that there should be at least replication_factor available nodes
in it. At the same time we try to keep the list stable over nodes
joining and leaving. To achieve this we hash in sequence the name of
each node with the name of the shard. We sort this list by hash, and
choose the first n nodes such that at least replication_factor nodes
are available. Then we chose the first available node as the master
for that shard.



250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
 
# File 'lib/adhd/node_manager.rb', line 250

def assign_nodes_to_shards(node_list, shard_range_list, replication_factor)

  shard_range_list.each do |shard_range|
    # Sort all nodes using consistent hashing
    sorted_nodes = node_list.sort_by {|node| MD5.new("#{node.name}||#{shard_range.shard_db_name}").to_s}
    avail = 0
    master = nil
    shard_node_list = []
    sorted_nodes.each do |node|
      shard_node_list << node
      if node.status == "RUNNING"
        master = node if !master # Chose the first available to be the master
        avail += 1
        break if avail == replication_factor # We have enough available nodes
      end
    end

    # Now put this list in the shard_range and save it
    # but only if there were changes
    new_master = master.name if master
    new_node_list = shard_node_list.map {|node| node.name}

    if !(new_master == shard_range.master_node) or !(new_node_list == shard_range.node_list)
      shard_range.master_node = master.name if master
      shard_range.node_list = shard_node_list.map {|node| node.name}
      shard_range.save
    end
  end

end

#parse_config(file) ⇒ Object

require ‘socket’



12
13
14
 
# File 'bin/adhd', line 12

def parse_config(file)
  @config = YAML.load_openstruct(File.read(file))
end