Class: Mapi::Pst::BlockParser

Inherits:

Object

• Object
• Mapi::Pst::BlockParser

Includes:

Types::Constants

Defined in:

lib/mapi/pst.rb

Overview

the job of this class, is to take a desc record, and be able to enumerate through the mapi properties of the associated thing.

corresponds to

• _pst_parse_block

• _pst_process (in some ways. although perhaps thats more the Item::Properties#add_property)

Direct Known Subclasses

AttachmentTable, RawPropertyStore, RawPropertyStoreTable, RecipientTable

Constant Summary

TYPES =

{
  0xbc => 1,
  0x7c => 2,
  # type 3 is removed. an artifact of not handling the indirect blocks properly in libpst.
}

PR_SUBJECT =

PropertySet::TAGS.find { |num, (name, type)| name == 'PR_SUBJECT' }.first.hex

PR_BODY_HTML =

PropertySet::TAGS.find { |num, (name, type)| name == 'PR_BODY_HTML' }.first.hex

IMMEDIATE_TYPES =

[
  PT_SHORT, PT_LONG, PT_BOOLEAN
]

INDIRECT_TYPES =

[
  PT_DOUBLE, PT_OBJECT,
  0x0014, # whats this? probably something like PT_LONGLONG, given the correspondence with the
          # ole variant types. (= VT_I8)
  PT_STRING8, PT_UNICODE, # unicode isn't in libpst, but added here for outlook 2003 down the track
  PT_SYSTIME,
  0x0048, # another unknown
  0x0102, # this is PT_BINARY vs PT_CLSID
  #0x1003, # these are vector types, but they're commented out for now because i'd expect that
  #0x1014, # there's extra decoding needed that i'm not doing. (probably just need a simple
  #        # PT_* => unpack string mapping for the immediate types, and just do unpack('V*') etc
  #0x101e,
  #0x1102
]

ID2_ATTACHMENTS =

0x671

ID2_RECIPIENTS =

0x692

USE_MAIN_DATA =

Targeting main data, not sub

-1

Instance Attribute Summary

• #data_chunks ⇒ Hash<Integer, String> readonly

  HID to data block.

• #node ⇒ NodePtr readonly

Instance Method Summary

• #get_data_array(offset) ⇒ Array<String>
• #get_data_indirect(offset) ⇒ String

  based on the value of offset, return either some data from buf, or some data from the id2 chain id2, where offset is some key into a lookup table that is stored as the id2 chain.

• #get_data_indirect_io(offset) ⇒ StringIO

  Resolve data pointed by HNID.

• #handle_indirect_values(key, type, value) ⇒ Object
• #initialize(node, local_node_id = USE_MAIN_DATA) ⇒ BlockParser constructor

  A new instance of BlockParser.

• #load_page_header(data, page_index) ⇒ Object

  Parse HNPAGEHDR / HNBITMAPHDR.

• #load_root_header(data) ⇒ Object

  Parse HNHDR.

Constructor Details

#initialize(node, local_node_id = USE_MAIN_DATA) ⇒ BlockParser

Returns a new instance of BlockParser.

Parameters:

• node (NodePtr)
• local_node_id (Integer) (defaults to: USE_MAIN_DATA)

# File 'lib/mapi/pst.rb', line 1002

def initialize node, local_node_id = USE_MAIN_DATA
  #raise FormatError, "unable to get associated index record for #{node.inspect}" unless node.block
  @node = node
  @data_chunks = {}

  data_array = (local_node_id == USE_MAIN_DATA) ? node.read_main_array : (node.read_sub_array local_node_id)

  data_array.each_with_index { |data, index|
    # see https://docs.microsoft.com/en-us/openspecs/office_file_formats/ms-pst/a3fa280c-eba3-434f-86e4-b95141b3c7b1
    if index == 0
      load_root_header data
    else
      load_page_header data, index
    end
  }

  # now, we may have multiple different blocks
end

Instance Attribute Details

#data_chunks ⇒ Hash<Integer, String> (readonly)

Returns HID to data block.

Returns:

• (Hash<Integer, String>) —

  HID to data block

# File 'lib/mapi/pst.rb', line 998

def data_chunks
  @data_chunks
end

#node ⇒ NodePtr (readonly)

Returns:

• (NodePtr)

# File 'lib/mapi/pst.rb', line 994

def node
  @node
end

Instance Method Details

#get_data_array(offset) ⇒ Array<String>

Parameters:

• offset (Integer)

Returns:

• (Array<String>)

# File 'lib/mapi/pst.rb', line 1107

def get_data_array offset
  raise "offset must be Integer" unless Integer === offset

  if offset == 0
    nil
  elsif (offset & 0x1f) != 0
    # this is NID (node)
    node.read_sub_array(offset)
  else
    # this is HID (heap)
    [data_chunks[offset]]
  end
end

#get_data_indirect(offset) ⇒ String

based on the value of offset, return either some data from buf, or some data from the id2 chain id2, where offset is some key into a lookup table that is stored as the id2 chain. i think i may need to create a BlockParser class that wraps up all this mess.

corresponds to:

• _pst_getBlockOffsetPointer

• _pst_getBlockOffset

Parameters:

• offset (Integer)

Returns:

• (String)

# File 'lib/mapi/pst.rb', line 1072

def get_data_indirect offset
  raise "offset must be Integer" unless Integer === offset

  return get_data_indirect_io(offset).read
end

#get_data_indirect_io(offset) ⇒ StringIO

Resolve data pointed by HNID

Parameters:

• offset (Integer)

Returns:

• (StringIO)

See Also:

• https://docs.microsoft.com/en-us/openspecs/office_file_formats/ms-pst/7ac490ce-31af-4a75-97df-eb9d07a003fd

# File 'lib/mapi/pst.rb', line 1084

def get_data_indirect_io offset
  raise "offset must be Integer" unless Integer === offset

  if offset == 0
    nil
  elsif (offset & 0x1f) != 0
    # this is NID (node)
    data_array = node.read_sub_array(offset)
    raise "local node id #{offset} points multi page count #{data_array.count}, use get_data_array() instead" if data_array.count >= 2
    if data_array.empty?
      StringIO.new ""
    else
      StringIO.new data_array.first
    end
  else
    # this is HID (heap)
    StringIO.new data_chunks[offset]
  end
end

#handle_indirect_values(key, type, value) ⇒ Object

# File 'lib/mapi/pst.rb', line 1121

def handle_indirect_values key, type, value
  case type
  when PT_BOOLEAN
    value = value != 0
  when *IMMEDIATE_TYPES # not including PT_BOOLEAN which we just did above
    # no processing current applied (needed?).
  when *INDIRECT_TYPES
    # the value is a pointer
    if String === value # ie, value size > 4 above
      value = StringIO.new value
    else
      value = get_data_array(value)
      if value
        value = StringIO.new value.join("")
      end
    end
    # keep strings as immediate values for now, for compatability with how i set up
    # Msg::Properties::ENCODINGS
    if value
      if type == PT_STRING8
        value = node.pst.helper.convert_ansi_str value.read
      elsif type == PT_UNICODE
        value = Ole::Types::FROM_UTF16.iconv value.read
      end
    end
    # special subject handling
    if key == PR_BODY_HTML and value
      # to keep the msg code happy, which thinks body_html will be an io
      # although, in 2003 version, they are 0102 already
      value = StringIO.new value unless value.respond_to?(:read)
    end
    if key == PR_SUBJECT and String === value and value.length >= 2
      if value[0].ord == 1
        # This 2 chars header tell us how to omit subject prefix like `Yes: `, `Re: `, etc.
        # We need not to omit them.
        value = value[2..-1]
      end
=begin
      index = value =~ /^[A-Z]*:/ ? $~[0].length - 1 : nil
      unless ignore == 1 and offset == index
        warn 'something wrong with subject hack' 
        $x = [ignore, offset, value]
        require 'irb'
        IRB.start
        exit
      end
=end

=begin
new idea:

making sense of the \001\00[156] i've seen prefixing subject. i think its to do with the placement
of the ':', or the ' '. And perhaps an optimization to do with thread topic, and ignoring the prefixes
added by mailers. thread topic is equal to subject with all that crap removed.

can test by creating some mails with bizarre subjects.

subject="\001\005RE: blah blah"
subject="\001\001blah blah"
subject="\001\032Out of Office AutoReply: blah blah"
subject="\001\020Undeliverable: blah blah"

looks like it

=end


      # now what i think, is that perhaps, value[offset..-1] ...
      # or something like that should be stored as a special tag. ie, do a double yield
      # for this case. probably PR_CONVERSATION_TOPIC, in which case i'd write instead:
      # yield [PR_SUBJECT, ref_type, value]
      # yield [PR_CONVERSATION_TOPIC, ref_type, value[offset..-1]
      # next # to skip the yield.
    end

    # special handling for embedded objects
    # used for attach_data for attached messages. in which case attach_method should == 5,
    # for embedded object.
    if type == PT_OBJECT and value
      value = value.read if value.respond_to?(:read)
      id2, unknown = value.unpack 'V2'
      io = get_data_indirect_io id2

      # hacky
      #desc2 = OpenStruct.new(:node => io, :pst => node.pst, :sub_block => node.sub_block, :children => [])
      # put nil instead of desc.list_index, otherwise the attachment is attached to itself ad infinitum.
      # should try and fix that FIXME
      # this shouldn't be done always. for an attached message, yes, but for an attached
      # meta file, for example, it shouldn't. difference between embedded_ole vs embedded_msg
      # really.
      # note that in the case where its a embedded ole, you actually get a regular serialized ole
      # object, so i need to create an ole storage object on a rangesioidxchain!
      # eg:
=begin
att.props.display_name # => "Picture (Metafile)"
io = att.props.attach_data
io.read(32).unpack('H*') # => ["d0cf11e0a1b11ae100000.... note the docfile signature.
# plug some missing rangesio holes:
def io.rewind; seek 0; end
def io.flush; raise IOError; end
ole = Ole::Storage.open io
puts ole.root.to_tree

- #<Dirent:"Root Entry">
|- #<Dirent:"\001Ole" size=20 data="\001\000\000\002\000...">
|- #<Dirent:"CONTENTS" size=65696 data="\327\315\306\232\000...">
\- #<Dirent:"\003MailStream" size=12 data="\001\000\000\000[...">
=end

      # until properly fixed, i have disabled this code here, so this will break
      # nested messages temporarily.
      #value = Item.new desc2, RawPropertyStore.new(desc2).to_a
      #desc2.list_index = nil
      value = io
    end
  # this is PT_MV_STRING8, i guess.
  # should probably have the 0x1000 flag, and do the or-ring.
  # example of 0x1102 is PR_OUTLOOK_2003_ENTRYIDS. less sure about that one.
  when 0x101e, 0x1102
    # example data:
    # 0x802b "\003\000\000\000\020\000\000\000\030\000\000\000#\000\000\000BusinessCompetitionFavorites"
    # this 0x802b would be an extended attribute for categories / keywords.
    value = get_data_indirect_io(value).read unless String === value
    num = value.unpack('V')[0]
    offsets = value[4, 4 * num].unpack("V#{num}")
    value = (offsets + [value.length]).to_enum(:each_cons, 2).map { |from, to| value[from...to] }
    value.map! { |str| StringIO.new str } if type == 0x1102
  when 0x101f
    value = get_data_indirect_io(value).read unless String === value
    num = value.unpack('V')[0]
    offsets = value[4, 4 * num].unpack("V#{num}")
    value = (offsets + [value.length]).to_enum(:each_cons, 2).map { |from, to| value[from...to] }
    value.map! { |str| Ole::Types::FROM_UTF16.iconv str }
  when 0x1003 # uint32 array
    value = get_data_indirect_io(value).read unless String === value
    # there is no count field
    value = value.unpack("V#{(value.length / 4)}")
  else
    name = Mapi::Types::DATA[type].first rescue nil
    warn '0x%04x %p' % [key, get_data_indirect_io(value).read]
    raise NotImplementedError, 'unsupported mapi property type - 0x%04x (%p)' % [type, name]
  end
  [key, type, value]
end

#load_page_header(data, page_index) ⇒ Object

Parse HNPAGEHDR / HNBITMAPHDR

Parameters:

• data (String)
• page_index (Integer)

See Also:

• https://docs.microsoft.com/en-us/openspecs/office_file_formats/ms-pst/9c34ecf8-36bc-45a1-a2df-ee35c6dc840a

# File 'lib/mapi/pst.rb', line 1028

def load_page_header data, page_index
  page_map = data.unpack('v').first

  # read HNPAGEMAP
  offsets_count = data[page_map, 2].unpack("v").first + 1
  offset_tables = data[page_map + 4, 2 * offsets_count].unpack("v#{offsets_count}")

  offset_tables.each_cons(2).to_a.each_with_index do |(from, to), index|
    # conver to HID
    @data_chunks[0x20 * (1 + index) + 65536 * page_index] = data[from, to - from]
  end
end

#load_root_header(data) ⇒ Object

Parse HNHDR

Raises:

• (FormatError)

See Also:

• https://docs.microsoft.com/en-us/openspecs/office_file_formats/ms-pst/8e4ae05c-3c24-4103-b7e5-ffef6f244834

# File 'lib/mapi/pst.rb', line 1045

def load_root_header data
  page_map, sig, @heap_type, @offset1 = data.unpack 'vCCVV'
  raise FormatError, 'invalid signature 0x%02x' % sig unless sig == 0xec
  raise FormatError, 'unknown block type signature 0x%02x' % @heap_type unless TYPES[@heap_type]
  @type = TYPES[@heap_type]

  # read HNPAGEMAP
  offsets_count = data[page_map, 2].unpack("v").first + 1
  offset_tables = data[page_map + 4, 2 * offsets_count].unpack("v#{offsets_count}")

  offset_tables.each_cons(2).to_a.each_with_index do |(from, to), index|
    # conver to HID
    @data_chunks[0x20 * (1 + index)] = data[from, to - from]
  end
end