Class: PEROBS::FlatFileBlobHeader

Inherits:
Object
  • Object
show all
Defined in:
lib/perobs/FlatFileBlobHeader.rb

Overview

The FlatFile blob header has the following structure:

1 Byte: Flags byte.

Bit 0: 0 deleted entry, 1 valid entry
Bit 1: 0 reserved, must be 0
Bit 2: 0 uncompressed data, 1 compressed data
Bit 3: 0 current entry, 1 outdated entry
Bit 4 - 7: reserved, must be 0

8 bytes: Length of the data blob in bytes 8 bytes: ID of the value in the data blob 4 bytes: CRC32 checksum of the data blob

If the bit 0 of the flags byte is 0, only the length is valid. The blob is empty. Only of bit 0 is set then entry is valid.

Constant Summary collapse

FORMAT =

The ‘pack()’ format of the header.

'CQQL'
LENGTH =

The length of the header in bytes.

25
VALID_FLAG_BIT = 
0
COMPRESSED_FLAG_BIT = 
2
OUTDATED_FLAG_BIT = 
3

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(file, addr, flags, length, id, crc) ⇒ FlatFileBlobHeader

Create a new FlatFileBlobHeader with the given flags, length, id and crc.

Parameters:

  • file (File) —

    the FlatFile that contains the header

  • addr (Integer) —

    the offset address of the header in the file

  • flags (Integer) —

    8 bit number, see above

  • length (Integer) —

    length of the header in bytes

  • id (Integer) —

    ID of the blob entry

  • crc (Integer) —

    CRC32 checksum of the blob entry



66
67
68
69
70
71
72
73
74
75
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 66

def initialize(file, addr, flags, length, id, crc)
  @file = file
  @addr = addr
  @flags = flags
  @length = length
  @id = id
  @crc = crc
  # This is only set if the header is preceded by a corrupted blob.
  @corruption_start = nil
end

Instance Attribute Details

#addr ⇒ Object (readonly)

Returns the value of attribute addr.



56
57
58
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56

def addr
  @addr
end

#corruption_start ⇒ Object

Returns the value of attribute corruption_start.



57
58
59
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 57

def corruption_start
  @corruption_start
end

#crc ⇒ Object (readonly)

Returns the value of attribute crc.



56
57
58
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56

def crc
  @crc
end

#flags ⇒ Object (readonly)

Returns the value of attribute flags.



56
57
58
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56

def flags
  @flags
end

#id ⇒ Object (readonly)

Returns the value of attribute id.



56
57
58
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56

def id
  @id
end

#length ⇒ Object (readonly)

Returns the value of attribute length.



56
57
58
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 56

def length
  @length
end

Class Method Details

.read(file, addr = nil, id = nil) ⇒ Object

Read the header from the given File.

Parameters:

  • file (File)
  • addr (Integer) (defaults to: nil) —

    address in the file to start reading. If no address is specified use the current position in the file.

  • id (Integer) (defaults to: nil) —

    Optional ID that the header should have. If no id is specified there is no check against the actual ID done.

Returns:

  • FlatFileBlobHeader or nil if there are no more blobs to read in the file.



85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 85

def FlatFileBlobHeader::read(file, addr = nil, id = nil)
  # If an address was specified we expect the read to always succeed. If
  # no address is specified and we can't read the header we generate an
  # error message but it is not fatal.
  errors_are_fatal = !addr.nil?

  mode = :searching_next_header
  addr = file.pos unless addr
  buf = nil
  corruption_start = nil

  loop do
    buf_with_crc = nil
    begin
      file.seek(addr)
      buf_with_crc = file.read(LENGTH)
    rescue IOError => e
      if errors_are_fatal
        PEROBS.log.fatal "Cannot read blob header in flat file DB at " +
          "address #{addr}: #{e.message}"
      else
        PEROBS.log.error "Cannot read blob header in flat file DB: " +
          e.message
        return nil
      end
    end

    # Did we read anything?
    if buf_with_crc.nil?
      if errors_are_fatal
        PEROBS.log.fatal "Cannot read blob header " +
          "#{id ? "for ID #{id} " : ''}at address #{addr}"
      else
        # We have reached the end of the file.
        return nil
      end
    end

    # Did we get the full header?
    if buf_with_crc.length != LENGTH
      PEROBS.log.error "Incomplete FlatFileBlobHeader: Only " +
        "#{buf_with_crc.length} " +
        "bytes of #{LENGTH} could be read "
      "#{id ? "for ID #{id} " : ''}at address #{addr}"
      return nil
    end

    # Check the CRC of the header
    buf = buf_with_crc[0..-5]
    crc = buf_with_crc[-4..-1].unpack('L')[0]

    if (read_crc = Zlib.crc32(buf, 0)) == crc
      # We have found a valid header.
      if corruption_start
        PEROBS.log.error "FlatFile corruption ends at #{addr}. " +
          "#{addr - corruption_start} bytes skipped. Some data may " +
          "not be recoverable."
      end
      break
    else
      if errors_are_fatal
        PEROBS.log.fatal "FlatFile Header CRC mismatch at address " +
          "#{addr}. Header CRC is #{'%08x' % read_crc} but should be " +
          "#{'%08x' % crc}."
      else
        if corruption_start.nil?
          PEROBS.log.error "FlatFile corruption found. The FlatFile " +
            "Header CRC mismatch at address #{addr}. Header CRC is " +
            "#{'%08x' % read_crc} but should be #{'%08x' % crc}. Trying " +
            "to find the next header."
          corruption_start = addr
        end
        # The blob file is corrupted. There is no valid header at the
        # current position in the file. We now try to find the next valid
        # header by iterating over the remainder of the file advanding one
        # byte with each step until we hit the end of the file or find the
        # next valid header.
        addr += 1
      end
    end
  end

  header = FlatFileBlobHeader.new(file, addr, *buf.unpack(FORMAT))
  if corruption_start
    header.corruption_start = corruption_start
  end

  if id && header.id != id
    PEROBS.log.fatal "Mismatch between FlatFile index and blob file " +
      "found. FlatFile has entry with ID #{header.id} at address " +
      "#{addr}. Index has ID #{id} for this address."
  end

  return header
end

Instance Method Details

#clear_flags ⇒ Object

Reset all the flags bit to 0. This marks the blob as invalid.



195
196
197
198
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 195

def clear_flags
  @flags = 0
  write
end

#is_compressed? ⇒ Boolean

Return true if the blob contains compressed data.

Returns:

  • (Boolean) 


206
207
208
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 206

def is_compressed?
  bit_set?(COMPRESSED_FLAG_BIT)
end

#is_outdated? ⇒ Boolean

Return true if the blob contains outdated data.

Returns:

  • (Boolean) 


218
219
220
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 218

def is_outdated?
  bit_set?(OUTDATED_FLAG_BIT)
end

#is_valid? ⇒ Boolean

Return true if the header is for a non-empty blob.

Returns:

  • (Boolean) 


201
202
203
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 201

def is_valid?
  bit_set?(VALID_FLAG_BIT)
end

#set_outdated_flag ⇒ Object

Set the outdated bit. The entry will be invalid as soon as the current transaction has been completed.



212
213
214
215
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 212

def set_outdated_flag
  set_flag(OUTDATED_FLAG_BIT)
  write
end

#write ⇒ Object

Write the header to a given File.



182
183
184
185
186
187
188
189
190
191
192
 
# File 'lib/perobs/FlatFileBlobHeader.rb', line 182

def write
  begin
    buf = [ @flags, @length, @id, @crc].pack(FORMAT)
    crc = Zlib.crc32(buf, 0)
    @file.seek(@addr)
    @file.write(buf + [ crc ].pack('L'))
  rescue IOError => e
    PEROBS.log.fatal "Cannot write blob header into flat file DB: " +
      e.message
  end
end