Class: PEROBS::FlatFileBlobHeader
- Inherits:
-
Object
- Object
- PEROBS::FlatFileBlobHeader
- Defined in:
- lib/perobs/FlatFileBlobHeader.rb
Overview
The FlatFile blob header has the following structure:
1 Byte: Flags byte.
Bit 0: 0 deleted entry, 1 valid entry
Bit 1: 0 reserved, must be 0
Bit 2: 0 uncompressed data, 1 compressed data
Bit 3: 0 current entry, 1 outdated entry
Bit 4 - 7: reserved, must be 0
8 bytes: Length of the data blob in bytes 8 bytes: ID of the value in the data blob 4 bytes: CRC32 checksum of the data blob
If the bit 0 of the flags byte is 0, only the length is valid. The blob is empty. Only of bit 0 is set then entry is valid.
Constant Summary collapse
- FORMAT =
The ‘pack()’ format of the header.
'CQQL'
- LENGTH =
The length of the header in bytes.
25
- VALID_FLAG_BIT =
0
- COMPRESSED_FLAG_BIT =
2
- OUTDATED_FLAG_BIT =
3
Instance Attribute Summary collapse
-
#addr ⇒ Object
readonly
Returns the value of attribute addr.
-
#corruption_start ⇒ Object
Returns the value of attribute corruption_start.
-
#crc ⇒ Object
readonly
Returns the value of attribute crc.
-
#flags ⇒ Object
readonly
Returns the value of attribute flags.
-
#id ⇒ Object
readonly
Returns the value of attribute id.
-
#length ⇒ Object
readonly
Returns the value of attribute length.
Class Method Summary collapse
-
.read(file, addr = nil, id = nil) ⇒ Object
Read the header from the given File.
Instance Method Summary collapse
-
#clear_flags ⇒ Object
Reset all the flags bit to 0.
-
#initialize(file, addr, flags, length, id, crc) ⇒ FlatFileBlobHeader
constructor
Create a new FlatFileBlobHeader with the given flags, length, id and crc.
-
#is_compressed? ⇒ Boolean
Return true if the blob contains compressed data.
-
#is_outdated? ⇒ Boolean
Return true if the blob contains outdated data.
-
#is_valid? ⇒ Boolean
Return true if the header is for a non-empty blob.
-
#set_outdated_flag ⇒ Object
Set the outdated bit.
-
#write ⇒ Object
Write the header to a given File.
Constructor Details
#initialize(file, addr, flags, length, id, crc) ⇒ FlatFileBlobHeader
Create a new FlatFileBlobHeader with the given flags, length, id and crc.
66 67 68 69 70 71 72 73 74 75 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 66 def initialize(file, addr, flags, length, id, crc) @file = file @addr = addr @flags = flags @length = length @id = id @crc = crc # This is only set if the header is preceded by a corrupted blob. @corruption_start = nil end |
Instance Attribute Details
#addr ⇒ Object (readonly)
Returns the value of attribute addr.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def addr @addr end |
#corruption_start ⇒ Object
Returns the value of attribute corruption_start.
57 58 59 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 57 def corruption_start @corruption_start end |
#crc ⇒ Object (readonly)
Returns the value of attribute crc.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def crc @crc end |
#flags ⇒ Object (readonly)
Returns the value of attribute flags.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def flags @flags end |
#id ⇒ Object (readonly)
Returns the value of attribute id.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def id @id end |
#length ⇒ Object (readonly)
Returns the value of attribute length.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def length @length end |
Class Method Details
.read(file, addr = nil, id = nil) ⇒ Object
Read the header from the given File.
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 85 def FlatFileBlobHeader::read(file, addr = nil, id = nil) # If an address was specified we expect the read to always succeed. If # no address is specified and we can't read the header we generate an # error message but it is not fatal. errors_are_fatal = !addr.nil? mode = :searching_next_header addr = file.pos unless addr buf = nil corruption_start = nil loop do buf_with_crc = nil begin file.seek(addr) buf_with_crc = file.read(LENGTH) rescue IOError => e if errors_are_fatal PEROBS.log.fatal "Cannot read blob header in flat file DB at " + "address #{addr}: #{e.}" else PEROBS.log.error "Cannot read blob header in flat file DB: " + e. return nil end end # Did we read anything? if buf_with_crc.nil? if errors_are_fatal PEROBS.log.fatal "Cannot read blob header " + "#{id ? "for ID #{id} " : ''}at address #{addr}" else # We have reached the end of the file. return nil end end # Did we get the full header? if buf_with_crc.length != LENGTH PEROBS.log.error "Incomplete FlatFileBlobHeader: Only " + "#{buf_with_crc.length} " + "bytes of #{LENGTH} could be read " "#{id ? "for ID #{id} " : ''}at address #{addr}" return nil end # Check the CRC of the header buf = buf_with_crc[0..-5] crc = buf_with_crc[-4..-1].unpack('L')[0] if (read_crc = Zlib.crc32(buf, 0)) == crc # We have found a valid header. if corruption_start PEROBS.log.error "FlatFile corruption ends at #{addr}. " + "#{addr - corruption_start} bytes skipped. Some data may " + "not be recoverable." end break else if errors_are_fatal PEROBS.log.fatal "FlatFile Header CRC mismatch at address " + "#{addr}. Header CRC is #{'%08x' % read_crc} but should be " + "#{'%08x' % crc}." else if corruption_start.nil? PEROBS.log.error "FlatFile corruption found. The FlatFile " + "Header CRC mismatch at address #{addr}. Header CRC is " + "#{'%08x' % read_crc} but should be #{'%08x' % crc}. Trying " + "to find the next header." corruption_start = addr end # The blob file is corrupted. There is no valid header at the # current position in the file. We now try to find the next valid # header by iterating over the remainder of the file advanding one # byte with each step until we hit the end of the file or find the # next valid header. addr += 1 end end end header = FlatFileBlobHeader.new(file, addr, *buf.unpack(FORMAT)) if corruption_start header.corruption_start = corruption_start end if id && header.id != id PEROBS.log.fatal "Mismatch between FlatFile index and blob file " + "found. FlatFile has entry with ID #{header.id} at address " + "#{addr}. Index has ID #{id} for this address." end return header end |
Instance Method Details
#clear_flags ⇒ Object
Reset all the flags bit to 0. This marks the blob as invalid.
195 196 197 198 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 195 def clear_flags @flags = 0 write end |
#is_compressed? ⇒ Boolean
Return true if the blob contains compressed data.
206 207 208 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 206 def is_compressed? bit_set?(COMPRESSED_FLAG_BIT) end |
#is_outdated? ⇒ Boolean
Return true if the blob contains outdated data.
218 219 220 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 218 def is_outdated? bit_set?(OUTDATED_FLAG_BIT) end |
#is_valid? ⇒ Boolean
Return true if the header is for a non-empty blob.
201 202 203 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 201 def is_valid? bit_set?(VALID_FLAG_BIT) end |
#set_outdated_flag ⇒ Object
Set the outdated bit. The entry will be invalid as soon as the current transaction has been completed.
212 213 214 215 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 212 def set_outdated_flag set_flag(OUTDATED_FLAG_BIT) write end |
#write ⇒ Object
Write the header to a given File.
182 183 184 185 186 187 188 189 190 191 192 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 182 def write begin buf = [ @flags, @length, @id, @crc].pack(FORMAT) crc = Zlib.crc32(buf, 0) @file.seek(@addr) @file.write(buf + [ crc ].pack('L')) rescue IOError => e PEROBS.log.fatal "Cannot write blob header into flat file DB: " + e. end end |