Class: Kaitai::Struct::Stream

Inherits:
Object
  • Object
show all
Defined in:
lib/kaitai/struct/struct.rb

Overview

Kaitai::Struct::Stream is an implementation of Kaitai Stream API for Ruby. It’s implemented as a wrapper for generic IO objects.

It provides a wide variety of simple methods to read (parse) binary representations of primitive types, such as integer and floating point numbers, byte arrays and strings, and also provides stream positioning / navigation methods with unified cross-language and cross-toolkit semantics.

Typically, end users won’t access Kaitai Stream class manually, but would describe a binary structure format using .ksy language and then would use Kaitai Struct compiler to generate source code in desired target language. That code, in turn, would use this class and API to do the actual parsing job.

Defined Under Namespace

Classes: UnexpectedDataError

Stream positioning collapse

Integer numbers collapse

Floating point numbers collapse

Unaligned bit values collapse

Byte arrays collapse

Byte array processing collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(arg) ⇒ Stream

Constructs new Kaitai Stream object.

Parameters:

  • arg (String, IO, StringIO, SubIO) —

    if String, it will be used as byte array to read data from; if IO (or StringIO, or SubIO), if will be used literally as the source of data



104
105
106
107
108
109
110
111
112
113
 
# File 'lib/kaitai/struct/struct.rb', line 104

def initialize(arg)
  if arg.is_a?(String)
    @_io = StringIO.new(arg)
  elsif arg.is_a?(IO) or arg.is_a?(StringIO) or arg.is_a?(SubIO)
    @_io = arg
  else
    raise TypeError.new('can be initialized with IO, StringIO, SubIO or String only')
  end
  align_to_byte
end

Class Method Details

.bytes_strip_right(bytes, pad_byte) ⇒ Object 



457
458
459
460
461
462
463
464
 
# File 'lib/kaitai/struct/struct.rb', line 457

def self.bytes_strip_right(bytes, pad_byte)
  new_len = bytes.length
  while new_len > 0 and bytes.getbyte(new_len - 1) == pad_byte
    new_len -= 1
  end

  bytes[0, new_len]
end

.bytes_terminate(bytes, term, include_term) ⇒ Object 



466
467
468
469
470
471
472
473
 
# File 'lib/kaitai/struct/struct.rb', line 466

def self.bytes_terminate(bytes, term, include_term)
  term_index = bytes.index(term.chr)
  if term_index.nil?
    bytes.dup
  else
    bytes[0, term_index + (include_term ? 1 : 0)]
  end
end

.bytes_terminate_multi(bytes, term, include_term) ⇒ Object 



475
476
477
478
479
480
481
482
483
484
485
486
487
488
 
# File 'lib/kaitai/struct/struct.rb', line 475

def self.bytes_terminate_multi(bytes, term, include_term)
  unit_size = term.bytesize
  search_index = bytes.index(term)
  loop {
    if search_index.nil?
      return bytes.dup
    end
    mod = search_index % unit_size
    if mod == 0
      return bytes[0, search_index + (include_term ? unit_size : 0)]
    end
    search_index = bytes.index(term, search_index + (unit_size - mod))
  }
end

.open(filename) ⇒ Object

Convenience method to create a Kaitai Stream object, opening a local file with a given filename.

Parameters:

  • filename (String) —

    local file to open



119
120
121
 
# File 'lib/kaitai/struct/struct.rb', line 119

def self.open(filename)
  self.new(File.open(filename, 'rb:ASCII-8BIT'))
end

.process_rotate_left(data, amount, group_size) ⇒ String

Performs a circular left rotation shift for a given buffer by a given amount of bits, using groups of groupSize bytes each time. Right circular rotation should be performed using this procedure with corrected amount.

Parameters:

  • data (String) —

    source data to process

  • amount (Fixnum) —

    number of bits to shift by

  • group_size (Fixnum) —

    number of bytes per group to shift

Returns:

  • (String) —

    copy of source array with requested shift applied

Raises:

  • (NotImplementedError) 


547
548
549
550
551
552
553
554
555
556
557
558
559
560
 
# File 'lib/kaitai/struct/struct.rb', line 547

def self.process_rotate_left(data, amount, group_size)
  raise NotImplementedError.new("unable to rotate group #{group_size} bytes yet") unless group_size == 1

  mask = group_size * 8 - 1
  anti_amount = -amount & mask

  # NB: actually, left bit shift (<<) in Ruby would have required
  # truncation to type_bits size (i.e. something like "& 0xff" for
  # group_size == 8), but we can skip this one, because later these
  # number would be packed with Array#pack, which will do truncation
  # anyway

  data.bytes.map { |x| (x << amount) | (x >> anti_amount) }.pack('C*')
end

.process_xor_many(data, key) ⇒ String

Performs a XOR processing with given data, XORing every byte of input with a key array, repeating key array many times, if necessary (i.e. if data array is longer than key array). Uses pure Ruby implementation suggested by [Thomas Leitner](github.com/gettalong), borrowed from github.com/fny/xorcist/blob/master/bin/benchmark

Parameters:

  • data (String) —

    data to process

  • key (String) —

    array of bytes to XOR with

Returns:

  • (String) —

    processed data



523
524
525
526
527
528
529
530
531
532
533
534
535
536
 
# File 'lib/kaitai/struct/struct.rb', line 523

def self.process_xor_many(data, key)
  out = data.dup
  kl = key.length
  ki = 0
  i = 0
  max = data.length
  while i < max
    out.setbyte(i, data.getbyte(i) ^ key.getbyte(ki))
    ki += 1
    ki = 0 if ki >= kl
    i += 1
  end
  out
end

.process_xor_one(data, key) ⇒ String

Performs a XOR processing with given data, XORing every byte of input with a single given value. Uses pure Ruby implementation suggested by [Thomas Leitner](github.com/gettalong), borrowed from github.com/fny/xorcist/blob/master/bin/benchmark

Parameters:

  • data (String) —

    data to process

  • key (Fixnum) —

    value to XOR with

Returns:

  • (String) —

    processed data



502
503
504
505
506
507
508
509
510
511
 
# File 'lib/kaitai/struct/struct.rb', line 502

def self.process_xor_one(data, key)
  out = data.dup
  i = 0
  max = data.length
  while i < max
    out.setbyte(i, data.getbyte(i) ^ key)
    i += 1
  end
  out
end

.resolve_enum(enum_map, value) ⇒ Object

Resolves value using enum: if the value is not found in the map, we’ll just use literal value per se.



589
590
591
 
# File 'lib/kaitai/struct/struct.rb', line 589

def self.resolve_enum(enum_map, value)
  enum_map[value] || value
end

Instance Method Details

#align_to_byte ⇒ Object 



271
272
273
274
 
# File 'lib/kaitai/struct/struct.rb', line 271

def align_to_byte
  @bits_left = 0
  @bits = 0
end

#close ⇒ Object

Closes underlying IO object.



125
126
127
128
129
130
131
 
# File 'lib/kaitai/struct/struct.rb', line 125

def close
  # NOTE: `unless @_io.closed?` is only needed in Ruby 2.2 and below. Ruby 2.3
  # and later versions no longer raise `IOError: closed stream` when
  # `StringIO#close` is called a second time, see
  # https://github.com/ruby/ruby/commit/2e02f2dfd2dab936e7cd9a68d46bd910c5d184e5
  @_io.close unless @_io.closed?
end

#ensure_fixed_contents(expected) ⇒ String

Deprecated.

Unused since Kaitai Struct compiler 0.9. It is only available for backward compatibility and will be removed in the future. KSC 0.9 and later versions raise ValidationNotEqualError instead.

Reads next len bytes from the stream and ensures that they match expected fixed byte array. If they differ, throws a UnexpectedDataError runtime exception.

Parameters:

  • expected (String) —

    contents to be expected

Returns:

  • (String) —

    read bytes as byte array, which are guaranteed to equal to expected

Raises:



446
447
448
449
450
451
452
453
454
455
 
# File 'lib/kaitai/struct/struct.rb', line 446

def ensure_fixed_contents(expected)
  Internal.warn_deprecated(
    'method Stream#ensure_fixed_contents is deprecated since 0.9, ' \
    'explicitly raise ValidationNotEqualError from an `if` statement instead'
  )
  len = expected.bytesize
  actual = @_io.read(len)
  raise UnexpectedDataError.new(actual, expected) if actual != expected
  actual
end

#eof? ⇒ true, false

Check if stream pointer is at the end of stream.

Returns:

  • (true, false) —

    true if we are located at the end of the stream



138
 
# File 'lib/kaitai/struct/struct.rb', line 138

def eof?; @_io.eof? and @bits_left == 0; end

#pos ⇒ Fixnum

Get current position of a stream pointer.

Returns:

  • (Fixnum) —

    pointer position, number of bytes from the beginning of the stream



148
 
# File 'lib/kaitai/struct/struct.rb', line 148

def pos; @_io.pos; end

#read_bits_int(n) ⇒ Object

Deprecated.

Unused since Kaitai Struct compiler 0.9. It is only available for backward compatibility and will be removed in the future. KSC 0.9 and later versions use #read_bits_int_be instead.



310
311
312
313
314
315
316
 
# File 'lib/kaitai/struct/struct.rb', line 310

def read_bits_int(n)
  Internal.warn_deprecated(
    'method Stream#read_bits_int is deprecated since 0.9, ' \
    'use Stream#read_bits_int_be instead'
  )
  read_bits_int_be(n)
end

#read_bits_int_be(n) ⇒ Object 



276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
 
# File 'lib/kaitai/struct/struct.rb', line 276

def read_bits_int_be(n)
  res = 0

  bits_needed = n - @bits_left
  @bits_left = -bits_needed % 8

  if bits_needed > 0
    # 1 bit  => 1 byte
    # 8 bits => 1 byte
    # 9 bits => 2 bytes
    bytes_needed = ((bits_needed - 1) / 8) + 1 # `ceil(bits_needed / 8)`
    buf = read_bytes(bytes_needed)
    buf.each_byte { |byte|
      res = res << 8 | byte
    }

    new_bits = res
    res = res >> @bits_left | @bits << bits_needed
    @bits = new_bits # will be masked at the end of the function
  else
    res = @bits >> -bits_needed # shift unneeded bits out
  end

  mask = (1 << @bits_left) - 1 # `@bits_left` is in range 0..7
  @bits &= mask

  res
end

#read_bits_int_le(n) ⇒ Object 



318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
 
# File 'lib/kaitai/struct/struct.rb', line 318

def read_bits_int_le(n)
  res = 0
  bits_needed = n - @bits_left

  if bits_needed > 0
    # 1 bit  => 1 byte
    # 8 bits => 1 byte
    # 9 bits => 2 bytes
    bytes_needed = ((bits_needed - 1) / 8) + 1 # `ceil(bits_needed / 8)`
    buf = read_bytes(bytes_needed)
    i = 0
    buf.each_byte { |byte|
      res |= byte << (i * 8)
      i += 1
    }

    new_bits = res >> bits_needed
    res = res << @bits_left | @bits
    @bits = new_bits
  else
    res = @bits
    @bits >>= n
  end

  @bits_left = -bits_needed % 8

  mask = (1 << n) - 1 # no problem with this in Ruby (arbitrary precision integers)
  res &= mask
  return res
end

#read_bytes(n) ⇒ String

Reads designated number of bytes from the stream.

Parameters:

  • n (Fixnum) —

    number of bytes to read

Returns:

  • (String) —

    read bytes as byte array

Raises:

  • (EOFError) —

    if there were less bytes than requested available in the stream



359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
 
# File 'lib/kaitai/struct/struct.rb', line 359

def read_bytes(n)
  if n.nil?
    # This `read(0)` call is only used to raise `IOError: not opened for reading`
    # if the stream is closed. This ensures identical behavior to the `substream`
    # method.
    @_io.read(0)
    raise TypeError.new('no implicit conversion from nil to integer')
  end

  r = @_io.read(n)
  rl = r ? r.bytesize : 0
  n = n.to_int
  if rl < n
    begin
      @_io.seek(@_io.pos - rl)
    rescue Errno::ESPIPE
      # We have a non-seekable stream, so we can't go back to the
      # previous position - that's fine.
    end
    raise EOFError.new("attempted to read #{n} bytes, got only #{rl}")
  end
  r
end

#read_bytes_full ⇒ String

Reads all the remaining bytes in a stream as byte array.

Returns:

  • (String) —

    all remaining bytes in a stream as byte array



386
387
388
 
# File 'lib/kaitai/struct/struct.rb', line 386

def read_bytes_full
  @_io.read
end

#read_bytes_term(term, include_term, consume_term, eos_error) ⇒ Object 



390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
 
# File 'lib/kaitai/struct/struct.rb', line 390

def read_bytes_term(term, include_term, consume_term, eos_error)
  term_byte = term.chr
  r = String.new
  loop {
    c = @_io.getc
    if c.nil?
      if eos_error
        raise EOFError.new("end of stream reached, but no terminator #{term} found")
      end

      return r
    end
    if c == term_byte
      r << c if include_term
      @_io.seek(@_io.pos - 1) unless consume_term
      return r
    end
    r << c
  }
end

#read_bytes_term_multi(term, include_term, consume_term, eos_error) ⇒ Object 



411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
 
# File 'lib/kaitai/struct/struct.rb', line 411

def read_bytes_term_multi(term, include_term, consume_term, eos_error)
  unit_size = term.bytesize
  r = String.new
  loop {
    c = @_io.read(unit_size) || ''
    if c.bytesize < unit_size
      if eos_error
        raise EOFError.new("end of stream reached, but no terminator #{term} found")
      end

      r << c
      return r
    end
    if c == term
      r << c if include_term
      @_io.seek(@_io.pos - unit_size) unless consume_term
      return r
    end
    r << c
  }
end

#read_f4be ⇒ Object

Big-endian



247
248
249
 
# File 'lib/kaitai/struct/struct.rb', line 247

def read_f4be
  read_bytes(4).unpack('g')[0]
end

#read_f4le ⇒ Object

Little-endian



259
260
261
 
# File 'lib/kaitai/struct/struct.rb', line 259

def read_f4le
  read_bytes(4).unpack('e')[0]
end

#read_f8be ⇒ Object 



251
252
253
 
# File 'lib/kaitai/struct/struct.rb', line 251

def read_f8be
  read_bytes(8).unpack('G')[0]
end

#read_f8le ⇒ Object 



263
264
265
 
# File 'lib/kaitai/struct/struct.rb', line 263

def read_f8le
  read_bytes(8).unpack('E')[0]
end

#read_s1 ⇒ Object

Signed



163
164
165
 
# File 'lib/kaitai/struct/struct.rb', line 163

def read_s1
  read_bytes(1).unpack('c')[0]
end

#read_s2be ⇒ Object

.….….….….….….….….….….….….….….….….….… Big-endian .….….….….….….….….….….….….….….….….….…



171
172
173
 
# File 'lib/kaitai/struct/struct.rb', line 171

def read_s2be
  read_bytes(2).unpack('s>')[0]
end

#read_s2le ⇒ Object

.….….….….….….….….….….….….….….….….….… Little-endian .….….….….….….….….….….….….….….….….….…



187
188
189
 
# File 'lib/kaitai/struct/struct.rb', line 187

def read_s2le
  read_bytes(2).unpack('s<')[0]
end

#read_s4be ⇒ Object 



175
176
177
 
# File 'lib/kaitai/struct/struct.rb', line 175

def read_s4be
  read_bytes(4).unpack('l>')[0]
end

#read_s4le ⇒ Object 



191
192
193
 
# File 'lib/kaitai/struct/struct.rb', line 191

def read_s4le
  read_bytes(4).unpack('l<')[0]
end

#read_s8be ⇒ Object 



179
180
181
 
# File 'lib/kaitai/struct/struct.rb', line 179

def read_s8be
  read_bytes(8).unpack('q>')[0]
end

#read_s8le ⇒ Object 



195
196
197
 
# File 'lib/kaitai/struct/struct.rb', line 195

def read_s8le
  read_bytes(8).unpack('q<')[0]
end

#read_u1 ⇒ Object

Unsigned



203
204
205
 
# File 'lib/kaitai/struct/struct.rb', line 203

def read_u1
  read_bytes(1).unpack('C')[0]
end

#read_u2be ⇒ Object

.….….….….….….….….….….….….….….….….….… Big-endian .….….….….….….….….….….….….….….….….….…



211
212
213
 
# File 'lib/kaitai/struct/struct.rb', line 211

def read_u2be
  read_bytes(2).unpack('S>')[0]
end

#read_u2le ⇒ Object

.….….….….….….….….….….….….….….….….….… Little-endian .….….….….….….….….….….….….….….….….….…



227
228
229
 
# File 'lib/kaitai/struct/struct.rb', line 227

def read_u2le
  read_bytes(2).unpack('S<')[0]
end

#read_u4be ⇒ Object 



215
216
217
 
# File 'lib/kaitai/struct/struct.rb', line 215

def read_u4be
  read_bytes(4).unpack('L>')[0]
end

#read_u4le ⇒ Object 



231
232
233
 
# File 'lib/kaitai/struct/struct.rb', line 231

def read_u4le
  read_bytes(4).unpack('L<')[0]
end

#read_u8be ⇒ Object 



219
220
221
 
# File 'lib/kaitai/struct/struct.rb', line 219

def read_u8be
  read_bytes(8).unpack('Q>')[0]
end

#read_u8le ⇒ Object 



235
236
237
 
# File 'lib/kaitai/struct/struct.rb', line 235

def read_u8le
  read_bytes(8).unpack('Q<')[0]
end

#seek(x) ⇒ Object

Set stream pointer to designated position.

Parameters:

  • x (Fixnum) —

    new position (offset in bytes from the beginning of the stream)



143
 
# File 'lib/kaitai/struct/struct.rb', line 143

def seek(x); @_io.seek(x); end

#size ⇒ Fixnum

Get total size of the stream in bytes.

Returns:

  • (Fixnum) —

    size of the stream in bytes



153
 
# File 'lib/kaitai/struct/struct.rb', line 153

def size; @_io.size; end

#substream(n) ⇒ Stream

Reserves next n bytes from current stream as a Kaitai::Struct::Stream substream. Substream has its own pointer and addressing in the range of [0, n) bytes. This stream’s pointer is advanced to the position right after this substream.

Parameters:

  • n (Fixnum) —

    number of bytes to reserve for a substream

Returns:

  • (Stream) —

    substream covering n bytes from the current position

Raises:

  • (IOError) 


572
573
574
575
576
577
578
579
580
581
582
583
584
 
# File 'lib/kaitai/struct/struct.rb', line 572

def substream(n)
  raise IOError.new('not opened for reading') if @_io.closed?

  n = Internal.num2long(n)
  raise ArgumentError.new("negative length #{n} given") if n < 0

  rl = [0, @_io.size - @_io.pos].max
  raise EOFError.new("attempted to read #{n} bytes, got only #{rl}") if rl < n

  sub = Stream.new(SubIO.new(@_io, @_io.pos, n))
  @_io.seek(@_io.pos + n)
  sub
end