Class: PEROBS::FlatFileBlobHeader
- Inherits:
-
Object
- Object
- PEROBS::FlatFileBlobHeader
- Defined in:
- lib/perobs/FlatFileBlobHeader.rb
Overview
The FlatFile blob header has the following structure:
1 Byte: Flags byte.
Bit 0: 0 deleted entry, 1 valid entry
Bit 1: 0 reserved, must be 0
Bit 2: 0 uncompressed data, 1 compressed data
Bit 3: 0 current entry, 1 outdated entry
Bit 4 - 7: reserved, must be 0
8 bytes: Length of the data blob in bytes 8 bytes: ID of the value in the data blob 4 bytes: CRC32 checksum of the data blob
If the bit 0 of the flags byte is 0, only the length is valid. The blob is empty. Only of bit 0 is set then entry is valid.
Constant Summary collapse
- FORMAT =
The ‘pack()’ format of the header.
'CQQL'
- LENGTH =
The length of the header in bytes.
25
- VALID_FLAG_BIT =
0
- COMPRESSED_FLAG_BIT =
2
- OUTDATED_FLAG_BIT =
3
Instance Attribute Summary collapse
-
#addr ⇒ Object
readonly
Returns the value of attribute addr.
-
#corruption_start ⇒ Object
Returns the value of attribute corruption_start.
-
#crc ⇒ Object
readonly
Returns the value of attribute crc.
-
#flags ⇒ Object
readonly
Returns the value of attribute flags.
-
#id ⇒ Object
readonly
Returns the value of attribute id.
-
#length ⇒ Object
readonly
Returns the value of attribute length.
Class Method Summary collapse
-
.read(file, addr = nil, id = nil) ⇒ Object
Read the header from the given File.
Instance Method Summary collapse
-
#clear_flags ⇒ Object
Reset all the flags bit to 0.
-
#initialize(file, addr, flags, length, id, crc) ⇒ FlatFileBlobHeader
constructor
Create a new FlatFileBlobHeader with the given flags, length, id and crc.
-
#is_compressed? ⇒ Boolean
Return true if the blob contains compressed data.
-
#is_outdated? ⇒ Boolean
Return true if the blob contains outdated data.
-
#is_valid? ⇒ Boolean
Return true if the header is for a non-empty blob.
-
#set_outdated_flag ⇒ Object
Set the outdated bit.
-
#write ⇒ Object
Write the header to a given File.
Constructor Details
#initialize(file, addr, flags, length, id, crc) ⇒ FlatFileBlobHeader
Create a new FlatFileBlobHeader with the given flags, length, id and crc.
66 67 68 69 70 71 72 73 74 75 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 66 def initialize(file, addr, flags, length, id, crc) @file = file @addr = addr @flags = flags @length = length @id = id @crc = crc # This is only set if the header is preceded by a corrupted blob. @corruption_start = nil end |
Instance Attribute Details
#addr ⇒ Object (readonly)
Returns the value of attribute addr.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def addr @addr end |
#corruption_start ⇒ Object
Returns the value of attribute corruption_start.
57 58 59 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 57 def corruption_start @corruption_start end |
#crc ⇒ Object (readonly)
Returns the value of attribute crc.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def crc @crc end |
#flags ⇒ Object (readonly)
Returns the value of attribute flags.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def flags @flags end |
#id ⇒ Object (readonly)
Returns the value of attribute id.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def id @id end |
#length ⇒ Object (readonly)
Returns the value of attribute length.
56 57 58 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56 def length @length end |
Class Method Details
.read(file, addr = nil, id = nil) ⇒ Object
Read the header from the given File.
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 85 def FlatFileBlobHeader::read(file, addr = nil, id = nil) # If an address was specified we expect the read to always succeed. If # no address is specified and we can't read the header we generate an # error message but it is not fatal. errors_are_fatal = !addr.nil? mode = :searching_next_header addr = file.pos unless addr buf = nil corruption_start = nil loop do buf_with_crc = nil begin file.seek(addr) buf_with_crc = file.read(LENGTH) rescue IOError => e if errors_are_fatal PEROBS.log.fatal "Cannot read blob header in flat file DB at " + "address #{addr}: #{e.}" else PEROBS.log.error "Cannot read blob header in flat file DB: " + e. return nil end end # Did we read anything? if buf_with_crc.nil? if errors_are_fatal PEROBS.log.fatal "Cannot read blob header " + "#{id ? "for ID #{id} " : ''}at address #{addr}" else if corruption_start PEROBS.log.error "Corruption found at end of blob file at " + "address #{addr}" end # We have reached the end of the file. return nil end end # Did we get the full header? if buf_with_crc.length != LENGTH msg = "Incomplete FlatFileBlobHeader: Only " + "#{buf_with_crc.length} " + "bytes of #{LENGTH} could be read " "#{id ? "for ID #{id} " : ''}at address #{addr}" if errors_are_fatal PEROBS.log.fatal msg else PEROBS.log.error msg end return nil end # Check the CRC of the header buf = buf_with_crc[0..-5] crc = buf_with_crc[-4..-1].unpack('L')[0] if (read_crc = Zlib.crc32(buf, 0)) == crc # We have found a valid header. if corruption_start PEROBS.log.error "FlatFile corruption ends at #{addr}. " + "#{addr - corruption_start} bytes skipped. Some data may " + "not be recoverable." end break else if errors_are_fatal PEROBS.log.fatal "FlatFile Header CRC mismatch at address " + "#{addr}. Header CRC is #{'%08x' % read_crc} but should be " + "#{'%08x' % crc}." else if corruption_start.nil? if errors_are_fatal PEROBS.log.fatal "FlatFile corruption found. The FlatFile " + "Header CRC mismatch at address #{addr}. Header CRC is " + "#{'%08x' % read_crc} but should be #{'%08x' % crc}." else PEROBS.log.error "FlatFile corruption found. The FlatFile " + "Header CRC mismatch at address #{addr}. Header CRC is " + "#{'%08x' % read_crc} but should be #{'%08x' % crc}. " + "Trying to find the next header." end corruption_start = addr end # The blob file is corrupted. There is no valid header at the # current position in the file. We now try to find the next valid # header by iterating over the remainder of the file advanding one # byte with each step until we hit the end of the file or find the # next valid header. addr += 1 end end end header = FlatFileBlobHeader.new(file, addr, *buf.unpack(FORMAT)) if corruption_start header.corruption_start = corruption_start end if id && header.id != id PEROBS.log.fatal "Mismatch between FlatFile index and blob file " + "found. FlatFile has entry with ID #{header.id} at address " + "#{addr}. Index has ID #{id} for this address." end return header end |
Instance Method Details
#clear_flags ⇒ Object
Reset all the flags bit to 0. This marks the blob as invalid.
210 211 212 213 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 210 def clear_flags @flags = 0 write end |
#is_compressed? ⇒ Boolean
Return true if the blob contains compressed data.
221 222 223 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 221 def is_compressed? bit_set?(COMPRESSED_FLAG_BIT) end |
#is_outdated? ⇒ Boolean
Return true if the blob contains outdated data.
233 234 235 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 233 def is_outdated? bit_set?(OUTDATED_FLAG_BIT) end |
#is_valid? ⇒ Boolean
Return true if the header is for a non-empty blob.
216 217 218 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 216 def is_valid? bit_set?(VALID_FLAG_BIT) end |
#set_outdated_flag ⇒ Object
Set the outdated bit. The entry will be invalid as soon as the current transaction has been completed.
227 228 229 230 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 227 def set_outdated_flag set_flag(OUTDATED_FLAG_BIT) write end |
#write ⇒ Object
Write the header to a given File.
197 198 199 200 201 202 203 204 205 206 207 |
# File 'lib/perobs/FlatFileBlobHeader.rb', line 197 def write begin buf = [ @flags, @length, @id, @crc].pack(FORMAT) crc = Zlib.crc32(buf, 0) @file.seek(@addr) @file.write(buf + [ crc ].pack('L')) rescue IOError => e PEROBS.log.fatal "Cannot write blob header into flat file DB: " + e. end end |