Class: Replication::BagitBag

Inherits:
Object
  • Object
show all
Defined in:
lib/replication/bagit_bag.rb

Overview

Note:

Copyright (c) 2014 by The Board of Trustees of the Leland Stanford Junior University. All rights reserved. See LICENSE for details.

A BagIt bag contains a structured copy of a digital object for storage, transfer, or replication This class can be used to create, parse, or validate a bag instance

See Also:

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.create_bag(pathname) ⇒ BagitBag

Returns Initialize a new bag, create home and payload folders, write bagit.txt file.

Parameters:

  • pathname (Pathname, String) —

    The location of the bag home directory

Returns:

  • (BagitBag) —

    Initialize a new bag, create home and payload folders, write bagit.txt file



16
17
18
19
20
21
22
 
# File 'lib/replication/bagit_bag.rb', line 16

def BagitBag.create_bag(pathname)
  bag = BagitBag.new
  bag.bag_pathname = pathname
  bag.payload_pathname.mkpath
  bag.write_bagit_txt
  bag
end

.open_bag(pathname) ⇒ BagitBag

Returns Initialize a new bag, create home and payload folders, write bagit.txt file.

Parameters:

  • pathname (Pathname, String) —

    The location of the bag home directory

Returns:

  • (BagitBag) —

    Initialize a new bag, create home and payload folders, write bagit.txt file



26
27
28
29
30
31
32
33
 
# File 'lib/replication/bagit_bag.rb', line 26

def BagitBag.open_bag(pathname)
  bag = BagitBag.new
  bag.bag_pathname = pathname
  raise "No bag found at #{bag.bag_pathname}" unless bag.bag_pathname.exist?
  bagit_txt = bag.bag_pathname.join("bagit.txt")
  raise "No bagit.txt file found at #{bagit_txt}" unless bagit_txt.exist?
  bag
end

Instance Method Details

#add_payload_dir(link_mode, source_dir) ⇒ Pathname

Returns Generate file_fixity_hash and send it to #add_payload_files.

Parameters:

  • link_mode (Symbol) —

    Specifies whether to :copy, :link, or :symlink the files to the payload directory

  • source_dir (Pathname) —

    The source location of the directory whose contents are to be ingested

Returns:

  • (Pathname) —

    Generate file_fixity_hash and send it to #add_payload_files



87
88
89
90
91
 
# File 'lib/replication/bagit_bag.rb', line 87

def add_payload_dir (link_mode, source_dir)
  file_fixity_hash = Fixity.generate_checksums(source_dir, nil ,bag_checksum_types)
  add_payload_files(link_mode, source_dir, file_fixity_hash)
  payload_pathname
end

#add_payload_files(link_mode, source_basepath, file_fixity_hash) ⇒ Pathname

Returns Copy or link the files specified in the file_fixity_hash to the payload directory, then update the payload manifest files.

Parameters:

  • link_mode (Symbol) —

    Specifies whether to :copy, :link, or :symlink the files to the payload directory

  • source_basepath (Pathname) —

    The source location of the directory whose contents are to be ingested

  • file_fixity_hash (Hash<String,FileFixity>) —

    The list of files (with fixity data) to be added to the payload

Returns:

  • (Pathname) —

    Copy or link the files specified in the file_fixity_hash to the payload directory, then update the payload manifest files



98
99
100
101
102
103
104
105
106
 
# File 'lib/replication/bagit_bag.rb', line 98

def add_payload_files(link_mode, source_basepath, file_fixity_hash)
  file_fixity_hash.keys.each do |file_id|
    source_pathname = source_basepath.join(file_id)
    target_pathname = payload_pathname.join(file_id)
    copy_file(link_mode, source_pathname, target_pathname)
  end
  write_manifest_checksums('manifest', file_fixity_hash)
  payload_pathname
end

#add_payload_tarfile(tarfile_id, source_fullpath, source_basepath) ⇒ Tarfile

Returns Create a tar archive of a directory into the payload directory, generating checksums in parallel processes and recording those checksums in the payload manifests.

Parameters:

  • source_fullpath (Pathname, String) —

    The location of the directory whose content will be tarred

  • source_basepath (Pathname, String) —

    The location of the directory to change to before doing the tar create

Returns:

  • (Tarfile) —

    Create a tar archive of a directory into the payload directory, generating checksums in parallel processes and recording those checksums in the payload manifests



131
132
133
134
135
136
137
138
139
140
141
 
# File 'lib/replication/bagit_bag.rb', line 131

def add_payload_tarfile(tarfile_id,source_fullpath, source_basepath)
  tarfile = Tarfile.new
  tarfile.source_basepath = Pathname(source_basepath)
  tarfile.source_fullpath = Pathname(source_fullpath)
  tarfile.tarfile_basepath = payload_pathname
  tarfile.tarfile_fullpath = payload_pathname.join("#{tarfile_id}")
  tarfile.create_tarfile
  file_fixity_hash = Fixity.generate_checksums(tarfile.tarfile_basepath,[tarfile.tarfile_fullpath],bag_checksum_types)
  write_manifest_checksums('manifest', file_fixity_hash)
  tarfile
end

#bag_checksum_types ⇒ Array<Symbol>

Returns The list of checksum types to be used when generating fixity data.

Returns:

  • (Array<Symbol>) —

    The list of checksum types to be used when generating fixity data



74
75
76
 
# File 'lib/replication/bagit_bag.rb', line 74

def bag_checksum_types
  @bag_checksum_types ||= Fixity.default_checksum_types
end

#bag_checksum_types=(*types) ⇒ Void

Returns Set the list of checksum types to be used when generating fixity data.

Parameters:

  • types (Object) —

    The list of checksum types to be used when generating fixity data

Returns:

  • (Void) —

    Set the list of checksum types to be used when generating fixity data



80
81
82
 
# File 'lib/replication/bagit_bag.rb', line 80

def bag_checksum_types=(*types)
  @bag_checksum_types = Fixity.validate_checksum_types(*types)
end

#bag_pathname ⇒ Pathname

Returns The location of the bag home directory.

Returns:

  • (Pathname) —

    The location of the bag home directory



36
37
38
 
# File 'lib/replication/bagit_bag.rb', line 36

def bag_pathname
  @bag_pathname
end

#bag_pathname=(pathname) ⇒ Void

Returns Set the location of the bag home directory.

Parameters:

  • pathname (Pathname, String) —

    The location of the bag home directory

Returns:

  • (Void) —

    Set the location of the bag home directory



42
43
44
 
# File 'lib/replication/bagit_bag.rb', line 42

def bag_pathname=(pathname)
  @bag_pathname = Pathname(pathname)
end

#bag_payload_size ⇒ Hash<Symbol,Integer>

Returns A hash contining the payload size in bytes, and the number of files, derived from the payload directory contents.

Returns:

  • (Hash<Symbol,Integer>) —

    A hash contining the payload size in bytes, and the number of files, derived from the payload directory contents



157
158
159
160
161
162
163
 
# File 'lib/replication/bagit_bag.rb', line 157

def bag_payload_size
  payload_pathname.find.select{|f| f.file?}.inject({bytes: 0, files: 0}) do |hash,file|
    hash[:bytes] += file.size
    hash[:files] += 1
    hash
  end
end

#bag_size_human(bytes) ⇒ String

Returns Human-readable rendition of the total payload size.

Parameters:

  • bytes (Integer) —

    The total number of bytes in the payload

Returns:

  • (String) —

    Human-readable rendition of the total payload size



167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/replication/bagit_bag.rb', line 167

def bag_size_human(bytes)
  count = 0
  size = bytes
  while ( size >= 1024 and count < 4 )
    size /= 1024.0
    count += 1
  end
  if (count == 0)
    return sprintf("%d B", size)
  else
    return sprintf("%.2f %s", size, %w[B KB MB GB TB][count] )
  end
end

#copy_file(link_mode, source_pathname, target_pathname) ⇒ Pathname

Returns link or copy the specified file from source location to the target location.

Parameters:

  • link_mode (Symbol) —

    Specifies whether to :copy, :link, or :symlink the files to the payload directory

  • source_pathname (Pathname) —

    The source location of the file to be ingested

  • target_pathname (Pathname) —

    The location of the directory in which to place the file

Returns:

  • (Pathname) —

    link or copy the specified file from source location to the target location



112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/replication/bagit_bag.rb', line 112

def copy_file(link_mode, source_pathname, target_pathname)
  target_pathname.parent.mkpath
  case link_mode
    when :copy, nil
      FileUtils.copy(source_pathname.to_s, target_pathname.to_s) # automatically dereferences symlinks
    when :link
      FileUtils.link(source_pathname.to_s, target_pathname.to_s) #, :force => true (false is default)
    when :symlink
      FileUtils.symlink(source_pathname.to_s, target_pathname.to_s) #, :force => true (false is default)
    else
      raise "Invalid link_mode: #{link_mode}, expected one of [:copy,:link,:symlink]"
  end
  target_pathname
end

#generate_payload_checksums ⇒ Hash<String,FileFixity>

Returns create hash containing ids and checksums for all files in the bag's payload.

Returns:

  • (Hash<String,FileFixity>) —

    create hash containing ids and checksums for all files in the bag's payload



219
220
221
 
# File 'lib/replication/bagit_bag.rb', line 219

def generate_payload_checksums
  Fixity.generate_checksums(payload_pathname, nil, bag_checksum_types)
end

#generate_tagfile_checksums ⇒ Hash<String,FileFixity>

Returns create hash containing ids and checksums for all files in the bag's root directory.

Returns:

  • (Hash<String,FileFixity>) —

    create hash containing ids and checksums for all files in the bag's root directory



213
214
215
216
 
# File 'lib/replication/bagit_bag.rb', line 213

def generate_tagfile_checksums
  tagfiles = bag_pathname.children.reject{|file| file.basename.to_s.start_with?('tagmanifest')}
  Fixity.generate_checksums(bag_pathname, tagfiles, bag_checksum_types )
end

#info_payload_size ⇒ Hash<Symbol,Integer>

Returns A hash contining the payload size in bytes, and the number of files, derived from the Payload-Oxum property.

Returns:

  • (Hash<Symbol,Integer>) —

    A hash contining the payload size in bytes, and the number of files, derived from the Payload-Oxum property



195
196
197
198
199
200
 
# File 'lib/replication/bagit_bag.rb', line 195

def info_payload_size
  info = read_bag_info_txt
  size_array = info['Payload-Oxum'].split('.')
  size_hash = {:bytes => size_array[0].to_i, :files => size_array[1].to_i}
  size_hash
end

#manifest_diff(manifest_fixity_hash, bag_fixity_hash) ⇒ Hash

Returns A report of the differences between the fixity data from the manifest files against the values measured by digesting the files.

Parameters:

  • manifest_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the manifest files

  • bag_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the actual files

Returns:

  • (Hash) —

    A report of the differences between the fixity data from the manifest files against the values measured by digesting the files



297
298
299
300
301
302
303
304
305
306
307
 
# File 'lib/replication/bagit_bag.rb', line 297

def manifest_diff(manifest_fixity_hash, bag_fixity_hash)
  diff = Hash.new
  (manifest_fixity_hash.keys | bag_fixity_hash.keys).each do |file_id|
    manifest_fixity = manifest_fixity_hash[file_id] || FileFixity.new(file_id: file_id)
    bag_fixity = bag_fixity_hash[file_id] || FileFixity.new(file_id: file_id)
    if manifest_fixity != bag_fixity
      diff[file_id] = manifest_fixity.diff(bag_fixity,'manifest','bag')
    end
  end
  diff
end

#payload_pathname ⇒ Pathname

Returns The location of the bag data directory.

Returns:

  • (Pathname) —

    The location of the bag data directory



47
48
49
 
# File 'lib/replication/bagit_bag.rb', line 47

def payload_pathname
  bag_pathname.join('data')
end

#read_bag_info_txt ⇒ Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile

Returns Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile.

Returns:

  • (Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile) —

    Hash<String,String] A hash containing the properties documented in the bag-info.txt tagfile



182
183
184
185
186
187
188
189
190
191
 
# File 'lib/replication/bagit_bag.rb', line 182

def read_bag_info_txt
  properties = Hash.new
  bag_info = bag_pathname.join("bag-info.txt")
  bag_info.readlines.each do |line|
    line.chomp!.strip!
    key,value = line.split(':',2)
    properties[key.strip] = value.strip if value
  end
  properties
end

#read_bagit_txt ⇒ Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile

Returns Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile.

Returns:

  • (Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile) —

    Hash<String,String] A hash containing the properties documented in the bagit.txt tagfile



62
63
64
65
66
67
68
69
70
71
 
# File 'lib/replication/bagit_bag.rb', line 62

def read_bagit_txt
  properties = Hash.new
  bagit_txt = bag_pathname.join("bagit.txt")
  bagit_txt.readlines.each do |line|
    line.chomp!.strip!
    key,value = line.split(':',2)
    properties[key.strip] = value.strip if value
  end
  properties
end

#read_manifest_files(manifest_type) ⇒ Hash<String,FileFixity>

Returns A hash containing file ids and fixity data derived from the manifest files.

Parameters:

  • manifest_type (String) —

    The type of manifest file ('manifest' or 'tagmanifest') to be read

Returns:

  • (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the manifest files



244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
 
# File 'lib/replication/bagit_bag.rb', line 244

def read_manifest_files(manifest_type)
 file_fixity_hash = Hash.new
 checksum_type_list = Array.new
 Fixity.valid_checksum_ids.each do |checksum_type|
   manifest_pathname = bag_pathname.join("#{manifest_type}-#{checksum_type}.txt")
   if manifest_pathname.file?
     checksum_type_list << checksum_type
     manifest_pathname.readlines.each do |line|
       line.chomp!.strip!
       checksum,file_id = line.split(/[\s*]+/,2)
       file_fixity = file_fixity_hash[file_id] || FileFixity.new(file_id: file_id)
       file_fixity.set_checksum(checksum_type,checksum)
       file_fixity_hash[file_id] = file_fixity
     end
   end
 end
 self.bag_checksum_types = self.bag_checksum_types | checksum_type_list
 file_fixity_hash
end

#verify_bag ⇒ Boolean

Returns Validate the bag containing the digital object.

Returns:

  • (Boolean) —

    Validate the bag containing the digital object



310
311
312
313
314
315
316
 
# File 'lib/replication/bagit_bag.rb', line 310

def verify_bag
  verify_bag_structure
  verify_tagfile_manifests
  verify_payload_size
  verify_payload_manifests
  true
end

#verify_bag_structure ⇒ Boolean

Returns Test the existence of expected files, return true if files exist, raise exception if not.

Returns:

  • (Boolean) —

    Test the existence of expected files, return true if files exist, raise exception if not



319
320
321
322
323
324
 
# File 'lib/replication/bagit_bag.rb', line 319

def verify_bag_structure
  required_files = ['data','bagit.txt','bag-info.txt','manifest-sha256.txt','tagmanifest-sha256.txt']
  required_files.each{|filename| verify_pathname(bag_pathname.join(filename))}
  optional_files = []
  true
end

#verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash) ⇒ Boolean

Compare fixity data from the manifest files against the values measured by digesting the files, returning true if equal or false if not equal

Parameters:

  • manifest_type (String) —

    The type of manifest file ('manifest' or 'tagmanifest') to be read

  • manifest_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the manifest files

  • bag_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data derived from the actual files

Returns:

  • (Boolean) —

    Compare fixity data from the manifest files against the values measured by digesting the files, returning true if equal or false if not equal



285
286
287
288
289
290
291
 
# File 'lib/replication/bagit_bag.rb', line 285

def verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash)
  diff = manifest_diff(manifest_fixity_hash, bag_fixity_hash)
  if diff.size > 0
    raise "Failed #{manifest_type} verification! Differences: \n#{diff.inspect}"
  end
  true
end

#verify_pathname(pathname) ⇒ Boolean

Returns Test the existence of the specified path. Return true if file exists, raise exception if not.

Parameters:

  • pathname (Pathname) —

    The file whose existence should be verified

Returns:

  • (Boolean) —

    Test the existence of the specified path. Return true if file exists, raise exception if not



328
329
330
331
 
# File 'lib/replication/bagit_bag.rb', line 328

def verify_pathname(pathname)
  raise "#{pathname.basename} not found at #{pathname}" unless pathname.exist?
  true
end

#verify_payload_manifests ⇒ Boolean

Returns Compare fixity data from the payload manifest files against the values measured by digesting the files.

Returns:

  • (Boolean) —

    Compare fixity data from the payload manifest files against the values measured by digesting the files



273
274
275
276
277
278
 
# File 'lib/replication/bagit_bag.rb', line 273

def verify_payload_manifests
  manifest_type = 'manifest'
  manifest_fixity_hash = read_manifest_files(manifest_type)
  bag_fixity_hash = generate_payload_checksums
  verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash)
end

#verify_payload_size ⇒ Boolean

Returns Compare the actual measured payload size against the value recorded in bag-info.txt.

Returns:

  • (Boolean) —

    Compare the actual measured payload size against the value recorded in bag-info.txt



203
204
205
206
207
208
209
210
 
# File 'lib/replication/bagit_bag.rb', line 203

def verify_payload_size
  info_size = info_payload_size
  bag_size = bag_payload_size
  if info_size != bag_size
    raise "Failed payload size verification! Expected: #{info_size}, Found: #{bag_size}"
  end
  true
end

#verify_tagfile_manifests ⇒ Boolean

Returns Compare fixity data from the tag manifest files against the values measured by digesting the files.

Returns:

  • (Boolean) —

    Compare fixity data from the tag manifest files against the values measured by digesting the files



265
266
267
268
269
270
 
# File 'lib/replication/bagit_bag.rb', line 265

def verify_tagfile_manifests
  manifest_type = 'tagmanifest'
  manifest_fixity_hash = read_manifest_files(manifest_type)
  bag_fixity_hash = generate_tagfile_checksums
  verify_manifests(manifest_type, manifest_fixity_hash, bag_fixity_hash)
end

#write_bag_info_txt ⇒ Pathname

Returns Generate the bag-info.txt tag file to record the payload size.

Returns:

  • (Pathname) —

    Generate the bag-info.txt tag file to record the payload size



144
145
146
147
148
149
150
151
152
153
 
# File 'lib/replication/bagit_bag.rb', line 144

def write_bag_info_txt
  payload_size = bag_payload_size
  bag_info_txt = bag_pathname.join("bag-info.txt")
  bag_info_txt.open('w') do |f|
    f.puts "External-Identifier: #{bag_pathname.basename}"
    f.puts "Payload-Oxum: #{payload_size[:bytes]}.#{payload_size[:files]}"
    f.puts "Bag-Size: #{bag_size_human(payload_size[:bytes])}"
  end
  bag_info_txt
end

#write_bagit_txt ⇒ Pathname

Returns Generate the bagit.txt tag file.

Returns:

  • (Pathname) —

    Generate the bagit.txt tag file



52
53
54
55
56
57
58
59
 
# File 'lib/replication/bagit_bag.rb', line 52

def write_bagit_txt
  bagit_txt = bag_pathname.join("bagit.txt")
  bagit_txt.open('w') do |f|
   f.puts "Tag-File-Character-Encoding: UTF-8"
   f.puts "BagIt-Version: 0.97"
  end
  bagit_txt
end

#write_manifest_checksums(manifest_type, file_fixity_hash, open_mode = 'a') ⇒ Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash

Returns Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash.

Parameters:

  • manifest_type (String) —

    The type of manifest file ('manifest' or 'tagmanifest') to be updated

  • file_fixity_hash (Hash<String,FileFixity>) —

    A hash containing file ids and fixity data

  • open_mode (String) (defaults to: 'a') —

    The file open mode (default is 'a')

Returns:

  • (Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash) —

    Hash<Symbol,Pathname] Update each of the manifests with data from the file_fixity_hash



227
228
229
230
231
232
233
234
235
236
237
238
239
240
 
# File 'lib/replication/bagit_bag.rb', line 227

def write_manifest_checksums(manifest_type, file_fixity_hash, open_mode='a')
  manifests = Hash.new
  self.bag_checksum_types.each do |checksum_type|
    manifest_pathname = bag_pathname.join("#{manifest_type}-#{checksum_type}.txt")
    manifest_file = manifest_pathname.open(open_mode)
    file_fixity_hash.values.each do |fixity|
      checksum = fixity.get_checksum(checksum_type)
      manifest_file.puts("#{checksum} #{fixity.file_id}") if checksum
    end
    manifest_file.close
    manifests[checksum_type] = manifest_pathname
  end
  manifests
end