Class: MogileFS::Admin

Inherits:
Client
  • Object
show all
Defined in:
lib/mogilefs/admin.rb

Overview

MogileFS administration client, this has little real-world usage and is considered a work-in-progress

Instance Attribute Summary

Attributes inherited from Client

#backend, #hosts

Instance Method Summary collapse

Methods inherited from Client

#err, #errstr, #initialize, #readonly?, #reload

Constructor Details

This class inherits a constructor from MogileFS::Client

Instance Method Details

#change_device_state(host, device, state) ⇒ Object

Changes the device status of device on host to state which can be ‘alive’, ‘down’, or ‘dead’.

Raises:



298
299
300
301
 
# File 'lib/mogilefs/admin.rb', line 298

def change_device_state(host, device, state)
  raise MogileFS::ReadOnlyError if readonly?
  ! @backend.set_state(:host => host, :device => device, :state => state).nil?
end

#change_device_weight(host, device, weight) ⇒ Object

Changes the device weight of device on host to weight. weight should be a non-negative Integer. Devices with higher weight values are more likely to be chosen for reads and writes.

Raises:



307
308
309
310
311
 
# File 'lib/mogilefs/admin.rb', line 307

def change_device_weight(host, device, weight)
  raise MogileFS::ReadOnlyError if readonly?
  opts = { :host => host, :device => device, :weight => weight }
  ! @backend.set_weight(opts).nil?
end

#clean(count, prefix, res, underscore = true, to_i = [], want = nil) ⇒ Object

Turns the response res from the backend into an Array of Hashes from 1 to res[count]. If underscore is true then a ‘_’ character is assumed between the prefix and the hash key value.

res = {"host1_remoteroot"=>"/mnt/mogilefs/rur-1",
       "host1_hostname"=>"rur-1",
       "host1_hostid"=>"1",
       "host1_http_get_port"=>"",
       "host1_altip"=>"",
       "hosts"=>"1",
       "host1_hostip"=>"",
       "host1_http_port"=>"",
       "host1_status"=>"alive",
       "host1_altmask"=>""}
admin.clean 'hosts', 'host', res

Returns:

[{"status"=>"alive",
  "http_get_port"=>nil,
  "http_port"=>7600,
  "hostid"=>1,
  "hostip"=>"192.168.1.3",
  "hostname"=>"rur-1",
  "altip"=>"",
  "altmask"=>""}]



388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
 
# File 'lib/mogilefs/admin.rb', line 388

def clean(count, prefix, res, underscore = true, to_i = [], want = nil)
  empty = ""
  underscore = underscore ? '_' : empty

  # convert array to hash for O(1) lookups
  to_i = to_i.inject({}) { |m,k| m[k] = m }
  if want
    (1..res[count].to_i).map do |i|
      row = {}
      want.each do |k|
        v = res["#{prefix}#{i}#{underscore}#{k}"] or next
        row[k] = to_i.include?(k) ? (empty == v ? nil : v.to_i) : v
      end
      row
    end
  else
    keys = res.keys
    (1..res[count].to_i).map do |i|
      re = /^#{prefix}#{i}#{underscore}/
      row = {}
      keys.grep(re).each do |k|
        v = res[k]
        k = k.sub(re, empty)
        row[k] = to_i.include?(k) ? (empty == v ? nil : v.to_i) : v
      end
      row
    end
  end
end

#clear_cache ⇒ Object

Clears the tracker caches. Not implemented in all versions of MogileFS



324
325
326
 
# File 'lib/mogilefs/admin.rb', line 324

def clear_cache
  @backend.clear_cache
end

#create_class(domain, klass, policy) ⇒ Object

Creates a new class in domain named klass with policy for replication. Raises on failure.



228
229
230
 
# File 'lib/mogilefs/admin.rb', line 228

def create_class(domain, klass, policy)
  modify_class(domain, klass, policy, :create)
end

#create_device(host, devid, opts = {}) ⇒ Object

Creates device with Integer devid on host host may be an integer for hostid or String for hostname

Raises:



277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
 
# File 'lib/mogilefs/admin.rb', line 277

def create_device(host, devid, opts = {})
  raise MogileFS::ReadOnlyError if readonly?
  opts = opts.dup

  case host
  when Integer
    opts[:hostid] = host
  when String
    opts[:hostname] = host
  else
    raise ArgumentError, "host=#{host.inspect} is not a String or Integer"
  end

  opts[:devid] = devid
  ! @backend.create_device(opts).nil?
end

#create_domain(domain) ⇒ Object

Creates a new domain named domain. Returns nil if creation failed.

Raises:



209
210
211
212
213
 
# File 'lib/mogilefs/admin.rb', line 209

def create_domain(domain)
  raise MogileFS::ReadOnlyError if readonly?
  res = @backend.create_domain :domain => domain
  res ? res['domain'] : nil
end

#create_host(host, args = {}) ⇒ Object

Creates a new host named host. args must contain :ip and :port. Returns true if successful, false if not.

Raises:

  • (ArgumentError) 


252
253
254
255
256
257
 
# File 'lib/mogilefs/admin.rb', line 252

def create_host(host, args = {})
  raise ArgumentError, "Must specify ip and port" unless \
    args.include? :ip and args.include? :port

  modify_host(host, args, 'create')
end

#delete_class(domain, klass) ⇒ Object

Removes class klass from domain. Returns true if successful. Raises on failure



244
245
246
 
# File 'lib/mogilefs/admin.rb', line 244

def delete_class(domain, klass)
  ! @backend.delete_class(:domain => domain, :class => klass).nil?
end

#delete_domain(domain) ⇒ Object

Deletes domain. Returns true if successful, raises MogileFS::Backend::DomainNotFoundError if not

Raises:



219
220
221
222
 
# File 'lib/mogilefs/admin.rb', line 219

def delete_domain(domain)
  raise MogileFS::ReadOnlyError if readonly?
  ! @backend.delete_domain(:domain => domain).nil?
end

#delete_host(host) ⇒ Object

Deletes host host. Returns nil on failure.

Raises:



269
270
271
272
 
# File 'lib/mogilefs/admin.rb', line 269

def delete_host(host)
  raise MogileFS::ReadOnlyError if readonly?
  ! @backend.delete_host(:host => host).nil?
end

#each_fid ⇒ Object

Enumerates fids using #list_fids. Returns the number of valid fids processed



11
12
13
14
15
16
17
18
19
20
 
# File 'lib/mogilefs/admin.rb', line 11

def each_fid
  low = -1
  rv = 0
  begin
    fids = list_fids(low)
    fids.each { |fid| yield fid }
    rv += fids.size
  end while last = fids[-1] and low = last["fid"]
  rv
end

#get_devices(devid = nil) ⇒ Object

Returns an Array of device status Hashes. If devid is given only that device is returned.

admin.get_devices 1

Returns:

[{"status"=>"alive",
  "mb_asof"=>nil,
  "mb_free"=>666000,
  "devid"=>1,
  "hostid"=>1,
  "mb_used"=>666,
  "utilization"=>0.0,
  "reject_bad_md5"=>false,
  "observed_state"=>"writeable",
  "mb_total"=>666666}]



65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/mogilefs/admin.rb', line 65

def get_devices(devid = nil)
  to_i = %w(mb_asof mb_free mb_used mb_total devid weight hostid)
  want = %w(status observed_state).concat(to_i)
  rv = @backend.get_devices(devid ? { :devid => devid } : {})
  rv = clean('devices', 'dev', rv, true, to_i, want)
  ostates = Hash[%w(readable writeable unreachable).map! { |f| [f,f] }]

  rv.each do |row|
    u = row["utilization"] and
      row["utilization"] = nil == u ? nil : u.to_f

    # maps "" to nil (for dead devices)
    row["observed_state"] = ostates[row["observed_state"]]

    # be sure we do not set this at all for pre-2.60 MogileFS-Server
    case row["reject_bad_md5"]
    when "1"
      row["reject_bad_md5"] = true
    when "0"
      row["reject_bad_md5"] = false
    end
  end
end

#get_domains ⇒ Object

Returns the domains and classes, and their policies present in the mogilefs.

admin.get_domains

Returns (on newer MogileFS servers):

{
  "test" => {
    "default" => {
      "mindevcount" => 2,
      "replpolicy" => "MultipleHosts()",
      "hashtype => nil,
    }
  }
}

Returns (on older MogileFS servers without replication policies):

{"test"=>{"normal"=>3, "default"=>2}}



177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
 
# File 'lib/mogilefs/admin.rb', line 177

def get_domains
  res = @backend.get_domains
  have_replpolicy = false

  domains = {}
  to_i = %w(mindevcount)
  want = %w(name replpolicy hashtype mindevcount)
  (1..res['domains'].to_i).each do |i|
    domain = clean("domain#{i}classes", "domain#{i}class", res, false, to_i,
                   want)
    tmp = domains[res["domain#{i}"]] = {}
    domain.each do |d|
      tmp[d.delete("name")] = d
      have_replpolicy ||= d.include?("replpolicy")
    end
  end

  # only for MogileFS 1.x?, maybe we can drop support for this...
  unless have_replpolicy
    domains.each do |namespace, class_data|
      class_data.each do |class_name, data|
        class_data[class_name] = data["mindevcount"]
      end
    end
  end

  domains
end

#get_hosts(hostid = nil) ⇒ Object

Returns an Array of host status Hashes. If hostid is given only that host is returned.

admin.get_hosts 1

Returns:

[{"status"=>"alive",
  "http_get_port"=>nil,
  "http_port"=>"",
  "hostid"=>"1",
  "hostip"=>"",
  "hostname"=>"rur-1",
  "altip"=>"",
  "altmask"=>""}]



39
40
41
42
43
44
 
# File 'lib/mogilefs/admin.rb', line 39

def get_hosts(hostid = nil)
  to_i = %w(hostid http_port http_get_port)
  want = %w(status hostip hostname altip altmask).concat(to_i)
  rv = @backend.get_hosts(hostid ? { :hostid => hostid } : {})
  clean('hosts', 'host', rv, true, to_i, want)
end

#get_stats(type = 'all') ⇒ Object

Returns a statistics structure representing the state of mogilefs.

*** This command no longer works with recent versions of MogileFS *** *** Use mogstats(1) from the MogileFS::Utils package on CPAN ***

admin.get_stats

Returns:

{"fids"=>{"max"=>"99", "count"=>"2"},
 "device"=>
  [{"status"=>"alive", "files"=>"2", "id"=>"1", "host"=>"rur-1"},
   {"status"=>"alive", "files"=>"2", "id"=>"2", "host"=>"rur-2"}],
 "replication"=>
  [{"files"=>"2", "class"=>"normal", "devcount"=>"2", "domain"=>"test"}],
 "file"=>[{"files"=>"2", "class"=>"normal", "domain"=>"test"}]}



135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/mogilefs/admin.rb', line 135

def get_stats(type = 'all')
  res = @backend.stats type => 1
  stats = {}

  stats['device'] = clean 'devicescount', 'devices', res, false
  stats['file'] = clean 'filescount', 'files', res, false
  stats['replication'] = clean 'replicationcount', 'replication', res, false

  if res['fidmax'] or res['fidcount'] then
    stats['fids'] = {
      'max' => res['fidmax'].to_i,
      'count' => res['fidcount'].to_i
    }
  end

  stats.delete 'device' if stats['device'].empty?
  stats.delete 'file' if stats['file'].empty?
  stats.delete 'replication' if stats['replication'].empty?

  stats
end

#list_fids(from_fid, count = 100) ⇒ Object

Returns an Array of fid Hashes from from_fid, limited to count

admin.list_fids 0, 100

Returns:

[{"fid"=>99,
  "class"=>"normal",
  "domain"=>"test",
  "devcount"=>2,
  "length"=>4,
  "key"=>"file_key"},
 {"fid"=>82,
  "class"=>"normal",
  "devcount"=>2,
  "domain"=>"test",
  "length"=>9,
  "key"=>"new_new_key"}]



109
110
111
112
113
114
115
 
# File 'lib/mogilefs/admin.rb', line 109

def list_fids(from_fid, count = 100)
  to_i = %w(fid devcount length)
  want = %w(domain class key).concat(to_i)
  rv = @backend.list_fids(:from => from_fid, :to => count)
  # :to is now :count internally in mogilefsd
  clean('fid_count', 'fid_', rv, true, to_i, want)
end

#modify_class(domain, klass, policy, action) ⇒ Object

Modifies klass on domain to store files on mindevcount devices via action. Returns the class name if successful, raises if not

Raises:



334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
 
# File 'lib/mogilefs/admin.rb', line 334

def modify_class(domain, klass, policy, action)
  raise MogileFS::ReadOnlyError if readonly?
  args = { :domain => domain, :class => klass }
  case policy
  when Integer
    args[:mindevcount] = policy
  when String
    args[:replpolicy] = policy
  when Hash
    args.merge!(policy)
  else
    raise ArgumentError,
         "policy=#{policy.inspect} not understood for #{action}_class"
  end
  @backend.__send__("#{action}_class", args)["class"]
end

#modify_host(host, args = {}, action = 'create') ⇒ Object

Modifies host using args via action. Returns true if successful, false if not.



355
356
357
358
 
# File 'lib/mogilefs/admin.rb', line 355

def modify_host(host, args = {}, action = 'create')
  args[:host] = host
  ! @backend.__send__("#{action}_host", args).nil?
end

#replicate_now ⇒ Object

reschedules all deferred replication, returns a hash with the number of files rescheduled:

admin.replicate_now => { "count" => 5 }



317
318
319
320
321
 
# File 'lib/mogilefs/admin.rb', line 317

def replicate_now
  rv = @backend.replicate_now
  rv["count"] = rv["count"].to_i
  rv
end

#update_class(domain, klass, policy) ⇒ Object

Updates class klass in domain with policy for replication. Raises on failure.



236
237
238
 
# File 'lib/mogilefs/admin.rb', line 236

def update_class(domain, klass, policy)
  modify_class(domain, klass, policy, :update)
end

#update_host(host, args = {}) ⇒ Object

Updates host with args. Returns true if successful, false if not.



262
263
264
 
# File 'lib/mogilefs/admin.rb', line 262

def update_host(host, args = {})
  modify_host(host, args, 'update')
end