Class: Replication::BagitBag

Inherits:
Object
  • Object
show all
Defined in:
lib/replication/bagit_bag.rb

Overview

Note:

Copyright (c) 2014 by The Board of Trustees of the Leland Stanford Junior University. All rights reserved. See LICENSE for details.

A BagIt bag contains a structured copy of a digital object for storage, transfer, or replication This class can be used to create, parse, or validate a bag instance

See Also:

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.create_bag(pathname) ⇒ BagitBag

Returns Initialize a new bag, create home and payload folders, write bagit.txt file.

Parameters:

  • pathname (Pathname, String) —

    The location of the bag home directory

Returns:

  • (BagitBag) —

    Initialize a new bag, create home and payload folders, write bagit.txt file



16
17
18
19
20
21
22
 
# File 'lib/replication/bagit_bag.rb', line 16

def BagitBag.create_bag(pathname)
  bag = BagitBag.new
  bag.bag_pathname = pathname
  bag.payload_pathname.mkpath
  bag.write_bagit_txt
  bag
end

.open_bag(pathname) ⇒ BagitBag

Returns Initialize a new bag, create home and payload folders, write bagit.txt file.

Parameters:

  • pathname (Pathname, String) —

    The location of the bag home directory

Returns:

  • (BagitBag) —

    Initialize a new bag, create home and payload folders, write bagit.txt file



26
27
28
29
30
31
32
33
 
# File 'lib/replication/bagit_bag.rb', line 26

def BagitBag.open_bag(pathname)
  bag = BagitBag.new
  bag.bag_pathname = pathname
  raise "No bag found at #{bag.bag_pathname}" unless bag.bag_pathname.exist?
  bagit_txt = bag.bag_pathname.join("bagit.txt")
  raise "No bagit.txt file found at #{bagit_txt}" unless bagit_txt.exist?
  bag
end

Instance Method Details

#add_data_prefix(file_fixity_hash) ⇒ Hash<String,FileFixity>

Returns A revised hash with file_id paths prefixed with 'data/'.

Parameters:

  • file_fixity_hash (Hash<String,FileFixity>) —

    key is file_id, values are Fixity objects containing checksums

Returns:

  • (Hash<String,FileFixity>) —

    A revised hash with file_id paths prefixed with 'data/'



110
111
112
113
114
115
116
117
 
# File 'lib/replication/bagit_bag.rb', line 110

def add_data_prefix(file_fixity_hash)
  new_hash = Hash.new
  file_fixity_hash.values.each do |fixity|
    fixity.file_id = "data/#{fixity.file_id}"
    new_hash[fixity.file_id] = fixity
  end
  new_hash
end

#add_dir_to_payload(link_mode, source_dir) ⇒ Pathname

Returns Generate file_fixity_hash and send it to #add_files_to_payload.

Parameters:

  • link_mode (Symbol) —

    Specifies whether to :copy, :link, or :symlink the files to the payload directory

  • source_dir (Pathname) —

    The source location of the directory whose contents are to be bagged

Returns:

  • (Pathname) —

    Generate file_fixity_hash and send it to #add_files_to_payload



87
88
89
90
91
 
# File 'lib/replication/bagit_bag.rb', line 87

def add_dir_to_payload (link_mode, source_dir)
  file_fixity_hash = Fixity.generate_checksums(source_dir, source_dir.find ,bag_checksum_types)
  add_files_to_payload(link_mode, source_dir, file_fixity_hash)
  payload_pathname
end

#add_files_to_payload(link_mode, source_basepath, file_fixity_hash) ⇒ Pathname

Returns Copy or link the files specified in the file_fixity_hash to the payload directory, then update the payload manifest files.

Parameters:

  • link_mode (Symbol) —

    Specifies whether to :copy, :link, or :symlink the files to the payload directory

  • source_basepath (Pathname) —

    The source location of the directory whose contents are to be ingested

  • file_fixity_hash (Hash<String,FileFixity>) —

    The list of files (with fixity data) to be added to the payload

Returns:

  • (Pathname) —

    Copy or link the files specified in the file_fixity_hash to the payload directory, then update the payload manifest files



98
99
100
101
102
103
104
105
106
 
# File 'lib/replication/bagit_bag.rb', line 98

def add_files_to_payload(link_mode, source_basepath, file_fixity_hash)
  file_fixity_hash.keys.each do |file_id|
    source_pathname = source_basepath.join(file_id)
    target_pathname = payload_pathname.join(file_id)
    copy_file(link_mode, source_pathname, target_pathname)
  end
  write_manifest_checksums('manifest', add_data_prefix(file_fixity_hash))
  payload_pathname
end

#add_payload_tarfile(tarfile_id, source_fullpath, source_basepath) ⇒ Tarfile

Returns Create a tar archive of a directory into the payload directory, generating checksums in parallel processes and recording those checksums in the payload manifests.

Parameters:

  • source_fullpath (Pathname, String) —

    The location of the directory whose content will be tarred

  • source_basepath (Pathname, String) —

    The location of the directory to change to before doing the tar create

Returns:

  • (Tarfile) —

    Create a tar archive of a directory into the payload directory, generating checksums in parallel processes and recording those checksums in the payload manifests



142
143
144
145
146
147
148
149
150
151
152
 
# File 'lib/replication/bagit_bag.rb', line 142

def add_payload_tarfile(tarfile_id,source_fullpath, source_basepath)
  tarfile = Tarfile.new
  tarfile.source_basepath = Pathname(source_basepath)
  tarfile.source_fullpath = Pathname(source_fullpath)
  tarfile.tarfile_basepath = payload_pathname
  tarfile.tarfile_fullpath = payload_pathname.join("#{tarfile_id}")
  tarfile.create_tarfile
  file_fixity_hash = Fixity.generate_checksums(bag_pathname,[tarfile.tarfile_fullpath],bag_checksum_types)
  write_manifest_checksums('manifest', file_fixity_hash)
  tarfile
end

#bag_checksum_types ⇒ Array<Symbol>

Returns The list of checksum types to be used when generating fixity data.

Returns:

  • (Array<Symbol>) —

    The list of checksum types to be used when generating fixity data



74
75
76
 
# File 'lib/replication/bagit_bag.rb', line 74

def bag_checksum_types
  @bag_checksum_types ||= Fixity.default_checksum_types
end

#bag_checksum_types=(*types) ⇒ Void

Returns Set the list of checksum types to be used when generating fixity data.

Parameters:

  • types (Object) —

    The list of checksum types to be used when generating fixity data

Returns:

  • (Void) —

    Set the list of checksum types to be used when generating fixity data



80
81
82
 
# File 'lib/replication/bagit_bag.rb', line 80

def bag_checksum_types=(*types)
  @bag_checksum_types = Fixity.validate_checksum_types(*types)
end

#bag_pathname ⇒ Pathname

Returns The location of the bag home directory.

Returns:

  • (Pathname) —

    The location of the bag home directory



36
37
38
 
# File 'lib/replication/bagit_bag.rb', line 36

def bag_pathname
  @bag_pathname
end

#bag_pathname=(pathname) ⇒ Void

Returns Set the location of the bag home directory.

Parameters:

  • pathname (Pathname, String) —

    The location of the bag home directory

Returns:

  • (Void) —

    Set the location of the bag home directory



42
43
44
 
# File 'lib/replication/bagit_bag.rb', line 42

def bag_pathname=(pathname)
  @bag_pathname = Pathname(pathname)
end

#bag_payload_size ⇒ Hash<Symbol,Integer>

Returns A hash contining the payload size in bytes, and the number of files, derived from the payload directory contents.

Returns:

  • (Hash<Symbol,Integer>) —

    A hash contining the payload size in bytes, and the number of files, derived from the payload directory contents



168
169
170
171
172
173
174
 
# File 'lib/replication/bagit_bag.rb', line 168

def bag_payload_size
  payload_pathname.find.select{|f| f.file?}.inject({bytes: 0, files: 0}) do |hash,file|
    hash[:bytes] += file.size
    hash[:files] += 1
    hash
  end
end

#bag_size_human(bytes) ⇒ String

Returns Human-readable rendition of the total payload size.

Parameters:

  • bytes (Integer) —

    The total number of bytes in the payload

Returns:

  • (String) —

    Human-readable rendition of the total payload size



178
179
180
181
182
183
184
185
186
187
188
189
190
 
# File 'lib/replication/bagit_bag.rb', line 178

def bag_size_human(bytes)
  count = 0
  size = bytes
  while ( size >= 1024 and count < 4 )
    size /= 1024.0
    count += 1
  end
  if (count == 0)
    return sprintf("%d B", size)
  else
    return sprintf("%.2f %s", size, %w[B KB MB GB TB][count] )
  end
end

#copy_file(link_mode, source_pathname, target_pathname) ⇒ Pathname

Returns link or copy the specified file from source location to the target location.

Parameters:

  • link_mode (Symbol) —

    Specifies whether to :copy, :link, or :symlink the files to the payload directory

  • source_pathname (Pathname) —

    The source location of the file to be ingested

  • target_pathname (Pathname) —

    The location of the directory in which to place the file

Returns:

  • (Pathname) —

    link or copy the specified file from source location to the target location



123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/replication/bagit_bag.rb', line 123

def copy_file(link_mode, source_pathname, target_pathname)
  target_pathname.parent.mkpath
  case link_mode
    when :copy, nil
      FileUtils.copy(source_pathname.to_s, target_pathname.to_s) # automatically dereferences symlinks
    when :link
      FileUtils.link(source_pathname.to_s, target_pathname.to_s) #, :force => true (false is default)
    when :symlink
      FileUtils.symlink(source_pathname.to_s, target_pathname.to_s) #, :force => true (false is default)
    else
      raise "Invalid link_mode: #{link_mode}, expected one of [:copy,:link,:symlink]"
  end
  target_pathname
end

#generate_payload_checksums ⇒ Hash<String,FileFixity>

Returns create hash containing ids and checksums for all files in the bag's payload.

Returns:

  • (Hash<String,FileFixity>) —

    create hash containing ids and checksums for all files in the bag's payload



232
233
234
235
236
237
 
# File 'lib/replication/bagit_bag.rb', line 232

def generate_payload_checksums
  # get list of all files in the data directory
  path_list = payload_pathname.find
  # generate checksums, but use bag home dir as the base directory for file ids (per bagit spec)
  Fixity.generate_checksums(bag_pathname, path_list, bag_checksum_types)
end

#generate_tagfile_checksums ⇒ Hash<String,FileFixity>

Returns create hash containing ids and checksums for all files in the bag's root directory.

Returns:

  • (Hash<String,FileFixity>) —

    create hash containing ids and checksums for all files in the bag's root directory



224
225
226
227
228
229
 
# File 'lib/replication/bagit_bag.rb', line 224

def generate_tagfile_checksums
  # get list of all files in the bag home dir, except those starting with 'tagmanifest'
  tagfiles = bag_pathname.children.reject{|file| file.basename.to_s.start_with?('tagmanifest')}
  # generate checksums, using bag home dir as the base directory for file ids (per bagit spec)
  Fixity.generate_checksums(bag_pathname, tagfiles, bag_checksum_types )
end

#info_payload_size ⇒ Hash<Symbol,Integer>

Returns A hash contining the payload size in bytes, and the number of files, derived from the Payload-Oxum property.

Returns:

  • (Hash<Symbol,Integer>) —

    A hash contining the payload size in bytes, and the number of files, derived from the Payload-Oxum property



206
207
208
209
210
211
 
# File 'lib/replication/bagit_bag.rb', line 206

def info_payload_size
  info = read_bag_info_txt
  size_array = info['Payload-Oxum'].split('.')
  size_hash = {:bytes => size_array[0].to_i, :files => size_array[1].to_i}
  size_hash
end

#manifest_diff(manifest_fixity_hash, bag_fixity_hash) ⇒ Hash

Returns A report of the differences between the fixity data from the manifest files against the values measured by digesting the files.

Parameters:

  • manifest_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the manifest files

  • bag_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the actual files

Returns:

  • (Hash) —

    A report of the differences between the fixity data from the manifest files against the values measured by digesting the files



313
314
315
316
317
318
319
320
321
322
323
 
# File 'lib/replication/bagit_bag.rb', line 313

def manifest_diff(manifest_fixity_hash, bag_fixity_hash)
  diff = Hash.new
  (manifest_fixity_hash.keys | bag_fixity_hash.keys).each do |file_id|
    manifest_fixity = manifest_fixity_hash[file_id] || FileFixity.new(file_id: file_id)
    bag_fixity = bag_fixity_hash[file_id] || FileFixity.new(file_id: file_id)
    if manifest_fixity != bag_fixity
      diff[file_id] = manifest_fixity.diff(bag_fixity,'manifest','bag')
    end
  end
  diff
end

#payload_pathname ⇒ Pathname

Returns The location of the bag data directory.

Returns:

  • (Pathname) —

    The location of the bag data directory



47
48
49
 
# File 'lib/replication/bagit_bag.rb', line 47

def payload_pathname
  bag_pathname.join('data')
end

#read_bag_info_txt ⇒ Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile

Returns Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile.

Returns:

  • (Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile) —

    Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile



193
194
195
196
197
198
199
200
201
202
 
# File 'lib/replication/bagit_bag.rb', line 193

def read_bag_info_txt
  properties = Hash.new
  bag_info = bag_pathname.join("bag-info.txt")
  bag_info.readlines.each do |line|
    line.chomp!.strip!
    key,value = line.split(':',2)
    properties[key.strip] = value.strip if value
  end
  properties
end

#read_bagit_txt ⇒ Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile

Returns Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile.

Returns:

  • (Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile) —

    Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile



62
63
64
65
66
67
68
69
70
71
 
# File 'lib/replication/bagit_bag.rb', line 62

def read_bagit_txt
  properties = Hash.new
  bagit_txt = bag_pathname.join("bagit.txt")
  bagit_txt.readlines.each do |line|
    line.chomp!.strip!
    key,value = line.split(':',2)
    properties[key.strip] = value.strip if value
  end
  properties
end

#read_manifest_files(manifest_type) ⇒ Hash<String,FileFixity>

Returns A hash containing file ids and fixity data derived from the manifest files.

Parameters:

  • manifest_type (String) —

    The type of manifest file ('manifest' or 'tagmanifest') to be read

Returns:

  • (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the manifest files



260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/replication/bagit_bag.rb', line 260

def read_manifest_files(manifest_type)
 file_fixity_hash = Hash.new
 checksum_type_list = Array.new
 Fixity.valid_checksum_ids.each do |checksum_type|
   manifest_pathname = bag_pathname.join("#{manifest_type}-#{checksum_type}.txt")
   if manifest_pathname.file?
     checksum_type_list << checksum_type
     manifest_pathname.readlines.each do |line|
       line.chomp!.strip!
       checksum,file_id = line.split(/[\s*]+/,2)
       file_fixity = file_fixity_hash[file_id] || FileFixity.new(file_id: file_id)
       file_fixity.set_checksum(checksum_type,checksum)
       file_fixity_hash[file_id] = file_fixity
     end
   end
 end
 self.bag_checksum_types = self.bag_checksum_types | checksum_type_list
 file_fixity_hash
end

#verify_bag ⇒ Boolean

Returns Validate the bag containing the digital object.

Returns:

  • (Boolean) —

    Validate the bag containing the digital object



326
327
328
329
330
331
332
 
# File 'lib/replication/bagit_bag.rb', line 326

def verify_bag
  verify_bag_structure
  verify_tagfile_manifests
  verify_payload_size
  verify_payload_manifests
  true
end

#verify_bag_structure ⇒ Boolean

Returns Test the existence of expected files, return true if files exist, raise exception if not.

Returns:

  • (Boolean) —

    Test the existence of expected files, return true if files exist, raise exception if not



335
336
337
338
339
340
 
# File 'lib/replication/bagit_bag.rb', line 335

def verify_bag_structure
  required_files = ['data','bagit.txt','bag-info.txt','manifest-sha256.txt','tagmanifest-sha256.txt']
  required_files.each{|filename| verify_pathname(bag_pathname.join(filename))}
  optional_files = []
  true
end

#verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash) ⇒ Boolean

Compare fixity data from the manifest files against the values measured by digesting the files, returning true if equal or false if not equal

Parameters:

  • manifest_type (String) —

    The type of manifest file ('manifest' or 'tagmanifest') to be read

  • manifest_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the manifest files

  • bag_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the actual files

Returns:

  • (Boolean) —

    Compare fixity data from the manifest files against the values measured by digesting the files, returning true if equal or false if not equal



301
302
303
304
305
306
307
 
# File 'lib/replication/bagit_bag.rb', line 301

def verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash)
  diff = manifest_diff(manifest_fixity_hash, bag_fixity_hash)
  if diff.size > 0
    raise "Failed #{manifest_type} verification! Differences: \n#{diff.inspect}"
  end
  true
end

#verify_pathname(pathname) ⇒ Boolean

Returns Test the existence of the specified path. Return true if file exists, raise exception if not.

Parameters:

  • pathname (Pathname) —

    The file whose existence should be verified

Returns:

  • (Boolean) —

    Test the existence of the specified path. Return true if file exists, raise exception if not



344
345
346
347
 
# File 'lib/replication/bagit_bag.rb', line 344

def verify_pathname(pathname)
  raise "#{pathname.basename} not found at #{pathname}" unless pathname.exist?
  true
end

#verify_payload_manifests ⇒ Boolean

Returns Compare fixity data from the payload manifest files against the values measured by digesting the files.

Returns:

  • (Boolean) —

    Compare fixity data from the payload manifest files against the values measured by digesting the files



289
290
291
292
293
294
 
# File 'lib/replication/bagit_bag.rb', line 289

def verify_payload_manifests
  manifest_type = 'manifest'
  manifest_fixity_hash = read_manifest_files(manifest_type)
  bag_fixity_hash = generate_payload_checksums
  verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash)
end

#verify_payload_size ⇒ Boolean

Returns Compare the actual measured payload size against the value recorded in bag-info.txt.

Returns:

  • (Boolean) —

    Compare the actual measured payload size against the value recorded in bag-info.txt



214
215
216
217
218
219
220
221
 
# File 'lib/replication/bagit_bag.rb', line 214

def verify_payload_size
  info_size = info_payload_size
  bag_size = bag_payload_size
  if info_size != bag_size
    raise "Failed payload size verification! Expected: #{info_size}, Found: #{bag_size}"
  end
  true
end

#verify_tagfile_manifests ⇒ Boolean

Returns Compare fixity data from the tag manifest files against the values measured by digesting the files.

Returns:

  • (Boolean) —

    Compare fixity data from the tag manifest files against the values measured by digesting the files



281
282
283
284
285
286
 
# File 'lib/replication/bagit_bag.rb', line 281

def verify_tagfile_manifests
  manifest_type = 'tagmanifest'
  manifest_fixity_hash = read_manifest_files(manifest_type)
  bag_fixity_hash = generate_tagfile_checksums
  verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash)
end

#write_bag_info_txt ⇒ Pathname

Returns Generate the bag-info.txt tag file to record the payload size.

Returns:

  • (Pathname) —

    Generate the bag-info.txt tag file to record the payload size



155
156
157
158
159
160
161
162
163
164
 
# File 'lib/replication/bagit_bag.rb', line 155

def write_bag_info_txt
  payload_size = bag_payload_size
  bag_info_txt = bag_pathname.join("bag-info.txt")
  bag_info_txt.open('w') do |f|
    f.puts "External-Identifier: #{bag_pathname.basename}"
    f.puts "Payload-Oxum: #{payload_size[:bytes]}.#{payload_size[:files]}"
    f.puts "Bag-Size: #{bag_size_human(payload_size[:bytes])}"
  end
  bag_info_txt
end

#write_bagit_txt ⇒ Pathname

Returns Generate the bagit.txt tag file.

Returns:

  • (Pathname) —

    Generate the bagit.txt tag file



52
53
54
55
56
57
58
59
 
# File 'lib/replication/bagit_bag.rb', line 52

def write_bagit_txt
  bagit_txt = bag_pathname.join("bagit.txt")
  bagit_txt.open('w') do |f|
   f.puts "Tag-File-Character-Encoding: UTF-8"
   f.puts "BagIt-Version: 0.97"
  end
  bagit_txt
end

#write_manifest_checksums(manifest_type, file_fixity_hash, open_mode = 'a') ⇒ Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash

Returns Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash.

Parameters:

  • manifest_type (String) —

    The type of manifest file ('manifest' or 'tagmanifest') to be updated

  • file_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data

  • open_mode (String) (defaults to: 'a') —

    The file open mode (default is 'a')

Returns:

  • (Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash) —

    Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash



243
244
245
246
247
248
249
250
251
252
253
254
255
256
 
# File 'lib/replication/bagit_bag.rb', line 243

def write_manifest_checksums(manifest_type, file_fixity_hash, open_mode='a')
  manifests = Hash.new
  self.bag_checksum_types.each do |checksum_type|
    manifest_pathname = bag_pathname.join("#{manifest_type}-#{checksum_type}.txt")
    manifest_file = manifest_pathname.open(open_mode)
    file_fixity_hash.values.each do |fixity|
      checksum = fixity.get_checksum(checksum_type)
      manifest_file.puts("#{checksum} #{fixity.file_id}") if checksum
    end
    manifest_file.close
    manifests[checksum_type] = manifest_pathname
  end
  manifests
end