Class: Higgs::Utils::BackupManager

Inherits:
Object
  • Object
show all
Defined in:
lib/higgs/utils/bman.rb

Overview

backup manager

requirements for online-backup

these parameters should be required when Higgs::StorageManager of backup target is opened.

:jlog_rotate_max

value is 0. rotated journal logs shuold be preserved.

:remote_services_uri

value is "druby://host:port". journal log rotation remote service should be enabled.

online-backup

online-backup is controlled by higgs_backup command that is the front end of Higgs::Utils::BackupManager.

simple online-backup is like this…

% higgs_backup -v -f foo -t backup_dir -u druby://localhost:17320
2007-11-21 22:33:08.127 [18215]: **** START BACKUP SCENARIO ****
2007-11-21 22:33:08.129 [18215]: connect to remote services: druby://localhost:17320
2007-11-21 22:33:13.227 [18215]: DRb service started.
2007-11-21 22:33:13.233 [18215]: start index backup.
2007-11-21 22:33:13.724 [18215]: completed index backup.
2007-11-21 22:33:13.725 [18215]: start data backup.
2007-11-21 22:44:09.738 [18215]: completed data backup.
2007-11-21 22:44:09.763 [18215]: start journal log rotation.
2007-11-21 22:44:10.092 [18215]: completed journal log rotation.
2007-11-21 22:44:10.200 [18215]: start journal logs backup.
2007-11-21 22:44:10.339 [18215]: completed journal logs backup.
2007-11-21 22:44:10.340 [18215]: start backup storage recovery.
2007-11-21 22:44:11.101 [18215]: completed backup storage recovery.
2007-11-21 22:44:11.103 [18215]: start backup storage verify.
2007-11-21 22:58:04.552 [18215]: completed backup storage verify.
2007-11-21 22:58:04.581 [18215]: start journal logs clean of from-storage.
2007-11-21 22:58:04.638 [18215]: completed jounal logs clean of from-storage.
2007-11-21 22:58:04.640 [18215]: start journal logs clean of to-storage.
2007-11-21 22:58:04.668 [18215]: completed jounal logs clean of to-storage.
2007-11-21 22:58:04.669 [18215]: **** COMPLETED BACKUP SCENARIO ****

online-backup scenario includes these processes.

  1. index backup. see Higgs::Utils::BackupManager#backup_index.

  2. data backup. see Higgs::Utils::BackupManager#backup_data.

  3. journal log rotation. see Higgs::Utils::BackupManager#rotate_jlog.

  4. journal logs backup. see Higgs::Utils::BackupManager#backup_jlog.

  5. backup storage recovery. see Higgs::Utils::BackupManager#recover.

  6. backup storage verify. see Higgs::Utils::BackupManager#verify.

  7. journal logs clean of from-storage. see Higgs::Utils::BackupManager#clean_jlog_from.

  8. journal logs clean of to-storage. see Higgs::Utils::BackupManager#clean_jlog_to.

restore from online-backup

simple restore is like this…

% higgs_backup -f images -t ~/misc/photon/dat/1 -v restore
2007-10-08 20:56:07.048 [18133]: **** START RESTORE SCENARIO ****
2007-10-08 20:56:07.066 [18133]: start storage files restore.
2007-10-08 21:09:49.614 [18133]: completed storage files restore.
2007-10-08 21:09:49.614 [18133]: start restored storage recovery.
2007-10-08 21:09:51.090 [18133]: completed restored storage recovery.
2007-10-08 21:09:51.093 [18133]: start restored storage verify.
2007-10-08 21:13:26.521 [18133]: completed restored storage verify.
2007-10-08 21:13:26.521 [18133]: **** COMPLETED RESTORE SCENARIO ****

restore scenario includes these processes.

  1. storage files restore. see Higgs::Utils::BackupManager#restore_files.

  2. restored storage recovery. see Higgs::Utils::BackupManager#restore_recover.

  3. restored storage verify. see Higgs::Utils::BackupManager#restore_verify.

command-line options

% higgs_backup --help
Usage: higgs_backup [OPTIONs] [COMMANDs]
COMMANDs:
    online_backup
    index
    data
    rotate
    jlog
    recover
    verify
    clean_from
    clean_to
    restore
    restore_files
    restore_recover
    restore_verify
OPTIONs:
    -f, --from=BACKUP_TARGET_STORAGE
    -t, --to-dir=DIR_TO_BACKUP
    -n, --to-name=NAME_TO_BACKUP
    -U, --remote-services-uri=URI
    -v, --verbose, --[no-]verbose
        --verbose-level=LEVEL

COMMANDs

select a process of online-backup. COMMANDs for online-backup are these.

online_backup

default. run online-backup scenario. see Higgs::Utils::BackupManager#online_backup.

index

index backup. see Higgs::Utils::BackupManager#backup_index.

data

data backup. see Higgs::Utils::BackupManager#backup_data.

rotate

journal log rotation. see Higgs::Utils::BackupManager#rotate_jlog.

jlog

journal logs backup. see Higgs::Utils::BackupManager#backup_jlog.

recover

backup storage recovery. see Higgs::Utils::BackupManager#recover.

verify

backup storage verify. see Higgs::Utils::BackupManager#verify.

clean_from

journal logs clean. see Higgs::Utils::BackupManager#clean_jlog_from.

clean_to

journal logs clean. see Higgs::Utils::BackupManager#clean_jlog_to.

COMMANDs for restore are these.

restore

run restore scenario. see Higgs::Utils::BackupManager#restore.

restore_files

storage files restore. see Higgs::Utils::BackupManager#restore_files.

restore_recover

restored storage recovery. see Higgs::Utils::BackupManager#restore_recover.

restore_verify

restored storage verify. see Higgs::Utils::BackupManager#restore_verify.

OPTION: --from=BACKUP_TARGET_STORAGE

BACKUP_TARGET_STORAGE is the name of backup target storage.

OPTION: --to-dir=DIR_TO_BACKUP

backuped storage is copied to the directory of DIR_TO_BACKUP.

OPTION: --to-name=NAME_TO_BACKUP

NAME_TO_BACKUP is the name of backuped storage. if this option is omitted then NAME_TO_BACKUP is the same as BACKUP_TARGET_STORAGE.

OPTION: --remote-services-uri=URI

access point for journal log rotation remote service. URI is the same as :remote_services_uri when Higgs::StorageManager is opened.

OPTION: --verbose

verbose level up.

OPTION: --no-verbose

verbose level down.

OPTION: --verbose-level=LEVEL

set verbose level to LEVEL.

Constant Summary collapse

CVS_ID =

for ident(1)

'$Id: bman.rb 841 2008-12-24 09:23:20Z toki $'

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ BackupManager

Returns a new instance of BackupManager.



169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/higgs/utils/bman.rb', line 169

def initialize(options={})
  @from = options[:from]
  to_dir = options[:to_dir]
  to_name = options[:to_name] || (@from && File.basename(@from))
  @to = File.join(to_dir, to_name) if (to_dir && to_name)
  @remote_services_uri = options[:remote_services_uri]
  if (options.key? :drb_service_autostart) then
    @drb_service_autostart = options[:drb_service_autostart]
  else
    @drb_service_autostart = true
  end
  @verbose = options[:verbose] || 0
  @out = options[:out] || STDOUT
end

Instance Method Details

#backup_data ⇒ Object 



231
232
233
234
235
236
237
238
239
240
241
242
 
# File 'lib/higgs/utils/bman.rb', line 231

def backup_data
  unless (@from) then
    raise 'required from_storage'
  end
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start data backup.') if (@verbose >= 1)
  FileUtils.cp("#{@from}.tar", "#{@to}.tar", :preserve => true, :verbose => @verbose >= 2)
  @out << log('completed data backup.') if (@verbose >= 1)
  nil
end

#backup_index ⇒ Object 



216
217
218
219
220
221
222
223
224
225
226
227
228
229
 
# File 'lib/higgs/utils/bman.rb', line 216

def backup_index
  unless (@from) then
    raise 'required from_storage'
  end
  unless (@to) then
    raise 'required to_storage'
  end
  connect_service
  @out << log('start index backup.') if (@verbose >= 1)
  @out << log("save to #{@to}.idx") if (@verbose >= 2)
  @jlog_rotate_service.call(File.expand_path(@to) + '.idx')
  @out << log('completed index backup.') if (@verbose >= 1)
  nil
end

#backup_jlog ⇒ Object 



252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
 
# File 'lib/higgs/utils/bman.rb', line 252

def backup_jlog
  unless (@from) then
    raise 'required from_storage'
  end
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start journal logs backup.') if (@verbose >= 1)
  for path in Storage.rotated_entries("#{@from}.jlog")
    path =~ /\.jlog\.\d+$/ or raise "mismatch jlog name: #{path}"
    unless (JournalLogger.has_eof_mark? path) then
      raise "broken journal log: #{path}"
    end
    ext = $&
    FileUtils.cp(path, "#{@to}#{ext}", :preserve => true, :verbose => @verbose >= 2)
  end
  @out << log('completed journal logs backup.') if (@verbose >= 1)
  nil
end

#clean_jlog_from ⇒ Object 



297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
 
# File 'lib/higgs/utils/bman.rb', line 297

def clean_jlog_from
  unless (@from) then
    raise 'required from_storage'
  end
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start journal logs clean of from-storage.') if (@verbose >= 1)
  for to_jlog in Storage.rotated_entries("#{@to}.jlog")
    to_jlog =~ /\.jlog\.\d+$/ or raise "mismatch jlog name: #{to_jlog}"
    ext = $&
    from_jlog = @from + ext
    if (File.exist? from_jlog) then
      FileUtils.rm(from_jlog, :verbose => @verbose >= 2)
    end
  end
  @out << log('completed jounal logs clean of from-storage.') if (@verbose >= 1)
  nil
end

#clean_jlog_to ⇒ Object 



317
318
319
320
321
322
323
324
325
326
327
 
# File 'lib/higgs/utils/bman.rb', line 317

def clean_jlog_to
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start journal logs clean of to-storage.') if (@verbose >= 1)
  for to_jlog in Storage.rotated_entries("#{@to}.jlog")
    FileUtils.rm(to_jlog, :verbose => @verbose >= 2)
  end
  @out << log('completed jounal logs clean of to-storage.') if (@verbose >= 1)
  nil
end

#online_backup ⇒ Object

run online backup scenario



330
331
332
333
334
335
336
337
338
339
340
341
342
 
# File 'lib/higgs/utils/bman.rb', line 330

def online_backup
  @out << log('**** START BACKUP SCENARIO ****') if (@verbose >= 1)
  backup_index
  backup_data
  rotate_jlog
  backup_jlog
  recover
  verify
  clean_jlog_from
  clean_jlog_to
  @out << log('**** COMPLETED BACKUP SCENARIO ****') if (@verbose >= 1)
  nil
end

#recover ⇒ Object 



272
273
274
275
276
277
278
279
280
 
# File 'lib/higgs/utils/bman.rb', line 272

def recover
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start backup storage recovery.') if (@verbose >= 1)
  Storage.recover(@to, @out, @verbose - 1)
  @out << log('completed backup storage recovery.') if (@verbose >= 1)
  nil
end

#restore ⇒ Object

run restore scenario



393
394
395
396
397
398
399
400
 
# File 'lib/higgs/utils/bman.rb', line 393

def restore
  @out << log('**** START RESTORE SCENARIO ****') if (@verbose >= 1)
  restore_files
  restore_recover
  restore_verify
  @out << log('**** COMPLETED RESTORE SCENARIO ****') if (@verbose >= 1)
  nil
end

#restore_files ⇒ Object 



344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
 
# File 'lib/higgs/utils/bman.rb', line 344

def restore_files
  unless (@from) then
    raise 'required from_storage'
  end
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start storage files restore.') if (@verbose >= 1)
  FileLock.open("#{@from}.lock") {|flock|
    flock.synchronize{
      FileUtils.cp("#{@to}.idx", "#{@from}.idx", :preserve => true, :verbose => @verbose >= 2)
      FileUtils.cp("#{@to}.tar", "#{@from}.tar", :preserve => true, :verbose => @verbose >= 2)
      for path in Storage.rotated_entries("#{@to}.jlog")
        path =~ /\.jlog\.\d+$/ or raise "mismatch jlog name: #{path}"
        ext = $&
        FileUtils.cp(path, "#{@from}#{ext}", :preserve => true, :verbose => @verbose >= 2)
      end
    }
  }
  @out << log('completed storage files restore.') if (@verbose >= 1)
  nil
end

#restore_recover ⇒ Object 



367
368
369
370
371
372
373
374
375
 
# File 'lib/higgs/utils/bman.rb', line 367

def restore_recover
  unless (@from) then
    raise 'required from_storage'
  end
  @out << log('start restored storage recovery.') if (@verbose >= 1)
  Storage.recover(@from, @out, @verbose - 1)
  @out << log('completed restored storage recovery.') if (@verbose >= 1)
  nil
end

#restore_verify ⇒ Object 



377
378
379
380
381
382
383
384
385
386
387
388
389
390
 
# File 'lib/higgs/utils/bman.rb', line 377

def restore_verify
  unless (@from) then
    raise 'required from_storage'
  end
  @out << log('start restored storage verify.') if (@verbose >= 1)
  st = Storage.new(@from)   # read-write open for recovery
  begin
    st.verify(@out, @verbose - 1)
  ensure
    st.shutdown
  end
  @out << log('completed restored storage verify.') if (@verbose >= 1)
  nil
end

#rotate_jlog ⇒ Object 



244
245
246
247
248
249
250
 
# File 'lib/higgs/utils/bman.rb', line 244

def rotate_jlog
  connect_service
  @out << log('start journal log rotation.') if (@verbose >= 1)
  @jlog_rotate_service.call(true)
  @out << log('completed journal log rotation.') if (@verbose >= 1)
  nil
end

#verify ⇒ Object 



282
283
284
285
286
287
288
289
290
291
292
293
294
295
 
# File 'lib/higgs/utils/bman.rb', line 282

def verify
  unless (@to) then
    raise 'required to_storage'
  end
  @out << log('start backup storage verify.') if (@verbose >= 1)
  st = Storage.new(@to, :read_only => true)
  begin
    st.verify(@out, @verbose - 1)
  ensure
    st.shutdown
  end
  @out << log('completed backup storage verify.') if (@verbose >= 1)
  nil
end