Class: Wexpr::Expression

Inherits:
Object show all
Defined in:
lib/wexpr/expression.rb

Overview

A wexpr expression - based off of libWexpr’s API

Expression types:

  • :null - Null expression (null/nil)

  • :value - A value. Can be a number, quoted, or token. Must be UTF-8 safe.

  • :array - An array of items where order matters.

  • :map - An array of items where each pair is a key/value pair. Not ordered. Keys are required to be values.

  • :binarydata - A set of binary data, which is encoded in text format as base64.

  • :invalid - Invalid expression - not filled in or usable

Parse flags:

  • none currently specified

Write flags:

  • :humanReadable - If set, will add newlines and indentation to make it more readable.

Constant Summary collapse

START_BLOCK_COMMENT = 
";(--"
END_BLOCK_COMMENT = 
"--)"
SIZE_U8 =

these are for porting sizeof() like operations over nicely

1
SIZE_U16 = 
2
SIZE_U32 = 
4
SIZE_U64 = 
8
BIN_EXPRESSIONTYPE_NULL =

type codes for binary

0x00
BIN_EXPRESSIONTYPE_VALUE = 
0x01
BIN_EXPRESSIONTYPE_ARRAY = 
0x02
BIN_EXPRESSIONTYPE_MAP = 
0x03
BIN_EXPRESSIONTYPE_BINARYDATA = 
0x04
BIN_EXPRESSIONTYPE_INVALID = 
0xFF
BIN_BINARYDATA_RAW =

binarydata types

0x00
BIN_BINARYDATA_ZLIB =

no compression

0x01

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.create_from_binary_chunk(data) ⇒ Object

Creates an expression from a binary chunk.



72
73
74
75
76
77
78
79
 
# File 'lib/wexpr/expression.rb', line 72

def self.create_from_binary_chunk(data)
  expr = Expression.create_invalid()
  
  expr.p_parse_from_binary_chunk(data)
  # result unused: remaining part of buffer
  
  return expr
end

.create_from_ruby(variable) ⇒ Object

Create from a ruby variable. Will convert as: If a nil value, will be a null expression. If an Array, will turn into an array expression (and recurse). If a Hash, will turn into a map expression (and recurse). If a String, and is UTF-8 safe, will turn into a value expression. If a String, and is not UTF-8 safe, will turn into a binary expression. [this case is weird due to how ruby handles memory. Otherwise, we just use .to_s and store that. Not sure if best, but we want to catch things like numbers.



132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
 
# File 'lib/wexpr/expression.rb', line 132

def self.create_from_ruby(variable)
  case variable
    when NilClass
      return Expression.create_null()
      
    when Array
      e = Expression.create_null()
      e.change_type(:array)
      
      variable.each do |val|
        e.array_add_element_to_end(
          Expression.create_from_ruby(val)
        )
      end
      
      return e
      
    when Hash
      e = Expression.create_null()
      e.change_type(:map)
      
      variable.each do |k, v|
        ve = Expression.create_from_ruby(v)
        
        # key could technically be anything, so convert to string
        e.map_set_value_for_key(k.to_s, ve)
      end
      
      return e
      
    when String
      varC = +variable # + creates a dup mutable string if frozen
      isUTF8 = varC.force_encoding('UTF-8').valid_encoding?
      
      if isUTF8
        return Expression.create_value(varC)
      else
        return Expression.create_binary_representation(variable)
      end
      
    else
      # might be a number or something else, to_s it
      return Expression.create_value(variable.to_s)
  end
end

.create_from_string(str, parseFlags = []) ⇒ Object

Create an expression from a string.



31
32
33
 
# File 'lib/wexpr/expression.rb', line 31

def self.create_from_string(str, parseFlags=[])
  return self.create_from_string_with_external_reference_table(str, parseFlags, {})
end

.create_from_string_with_external_reference_table(str, parseFlags = [], referenceTable = {}) ⇒ Object

Create an expression from a string with an external reference table.



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/wexpr/expression.rb', line 38

def self.create_from_string_with_external_reference_table(str, parseFlags=[], referenceTable={})
  expr = Expression.new()
  expr.change_type(:invalid)
  
  parserState = PrivateParserState.new()
  parserState.externalReferenceMap=referenceTable
  
  strC = str.dup  # + creates a dup mutable string if frozen
  
  # check that the string is valid UTF-8
  if strC.force_encoding('UTF-8').valid_encoding?
    # now start parsing
    rest = expr.p_parse_from_string(
      strC, parseFlags, parserState
    )
    
    postRest = Expression.s_trim_front_of_string(rest, parserState)
    if postRest.size != 0
      raise ExtraDataAfterParsingRootError.new(parserState.line, parserState.column, "Extra data after parsing the root expression: #{postRest}")
    end
    
    if expr.type == :invalid
      raise EmptyStringError.new(parserState.line, parserState.column, "No expression found [remained invalid]")
    end
  else
    raise InvalidUTF8Error.new(0, 0, "Invalid UTF-8")
  end
  
  return expr
end

.create_invalid ⇒ Object

Creates an empty invalid expression.



84
85
86
87
88
 
# File 'lib/wexpr/expression.rb', line 84

def self.create_invalid()
  expr = Expression.new()
  expr.change_type(:invalid)
  return expr
end

.create_null ⇒ Object

Creates a null expression.



93
94
95
96
97
 
# File 'lib/wexpr/expression.rb', line 93

def self.create_null()
  expr = Expression.new()
  expr.change_type(:null)
  return expr
end

.create_value(val) ⇒ Object

Creates a value expression with the given string being the value.



102
103
104
105
106
107
108
109
110
 
# File 'lib/wexpr/expression.rb', line 102

def self.create_value(val)
  expr = Expression.create_null()
  if expr != nil
    expr.change_type(:value)
    expr.value_set(val)
  end
  
  return expr
end

.s_create_value_of_string(str, parserState) ⇒ Object

will copy out the value of the string to a new buffer, will parse out quotes as needed



642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
 
# File 'lib/wexpr/expression.rb', line 642

def self.s_create_value_of_string(str, parserState)
  # two pass:
  # first pass, get the length of the size
  # second pass, store the buffer
  
  bufferLength = 0
  isQuotedString = false
  isEscaped = false
  pos = 0 # position we're parsing at
  
  if str[0] == '"'
    isQuotedString = true
    pos += 1
  end
  
  while pos < str.size
    c = str[pos]
    
    if isQuotedString
      if isEscaped
        # we're in an escape. Is it valid?
        if self.s_is_escape_valid(c)
          bufferLength += 1 # counts
          isEscaped = false # escape ended
        else
          raise InvalidStringEscapeError.new(parserState.line, parserState.column, "Invalid escape found in the string")
        end
      else
        if c == '"'
          # end quote, part of us
          pos += 1
          break
        elsif c == "\\"
          # we're escaping
          isEscaped = true
        else
          # otherwise it's a character
          bufferLength += 1
        end
      end
    else
      # have we ended the word?
      if self.s_is_not_bareword_safe(c)
        # ended - not part of us
        break
      end
      
      # otherwise, its a character
      bufferLength += 1
    end
    
    pos += 1
  end
  
  if bufferLength == 0 and !isQuotedString # cannot have an empty barewords string
    raise EmptyStringError.new(parserState.line, parserState.column, "Was told to parse an empty string")
  end
  
  endVal = pos
  
  # we now know our buffer size and the string has been checked
  # ... not that we needed this in ruby
  buffer = ""
  writePos = 0
  pos = 0
  if isQuotedString
    pos = 1
  end
  
  while writePos < bufferLength
    c = str[pos]
  
    if isQuotedString
      if isEscaped
        escapedValue = self.s_value_for_escape(c)
        buffer[writePos] = escapedValue
        writePos += 1
        isEscaped = false
      else
        if c == "\\"
          # we're escaping
          isEscaped = true
        else
          # otherwise it's a character
          buffer[writePos] = c
          writePos += 1
        end
      end
      
    else
      # it's a character
      buffer[writePos] = c
      writePos += 1
    end
    
    # next character
    pos += 1
  end
  
  return buffer, endVal
end

.s_escape_for_value(c) ⇒ Object 



575
576
577
578
579
580
581
582
583
584
585
 
# File 'lib/wexpr/expression.rb', line 575

def self.s_escape_for_value(c)
  return case c
    when '"' then '"'
    when "\r" then "r"
    when "\n" then "n"
    when "\t" then "t"
    when "\\" then "\\"
  else
    0 # invalid escape
  end
end

.s_indent(indent) ⇒ Object

returns the indent for the given amount



770
771
772
 
# File 'lib/wexpr/expression.rb', line 770

def self.s_indent(indent)
  return "\t" * indent
end

.s_is_escape_valid(c) ⇒ Object 



555
556
557
 
# File 'lib/wexpr/expression.rb', line 555

def self.s_is_escape_valid(c)
  return (c == '"' || c == 'r' || c == 'n' || c == 't' || c == "\\")
end

.s_is_newline(c) ⇒ Object

——————– PRIVATE —————————-



531
532
533
 
# File 'lib/wexpr/expression.rb', line 531

def self.s_is_newline(c)
  return c == "\n"
end

.s_is_not_bareword_safe(c) ⇒ Object 



541
542
543
544
545
546
547
548
549
550
551
552
553
 
# File 'lib/wexpr/expression.rb', line 541

def self.s_is_not_bareword_safe(c)
  return (c == '*'               \
    or c == '#'                \
    or c == '@'                \
    or c == '(' or c == ')'    \
    or c == '[' or c == ']'    \
    or c == '^'                \
    or c == '<' or c == '>'    \
    or c == '"'                \
    or c == ';'                \
    or self.s_is_whitespace(c) \
  )
end

.s_is_whitespace(c) ⇒ Object 



535
536
537
538
539
 
# File 'lib/wexpr/expression.rb', line 535

def self.s_is_whitespace(c)
  # we put \r in whitespace and not newline so its counted as a column instead of a line, cause windows.
  # we dont support classic macos style newlines properly as a side effect.
  return (c == ' ' or c == "\t" or c == "\r" or self.s_is_newline(c))
end

.s_requires_escape(c) ⇒ Object 



571
572
573
 
# File 'lib/wexpr/expression.rb', line 571

def self.s_requires_escape(c)
  return (c == '"' || c == "\r" || c == "\n" || c == "\t" || c == "\\")
end

.s_trim_front_of_string(str, parserState) ⇒ Object

trims the given string by removing whitespace or comments from the beginning of the string



588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
 
# File 'lib/wexpr/expression.rb', line 588

def self.s_trim_front_of_string(str, parserState)
  while true
    if str.size == 0
      return str
    end
    
    first = str[0]
    
    # skip whitespace
    if self.s_is_whitespace(first)
      str = str[1..-1]
      
      if self.s_is_newline(first)
        parserState.line += 1
        parserState.column = 1
      else
        parserState.column += 1
      end
    
    # comment
    elsif first == ';'
      isTillNewline = true
      
      if str.size >= 4
        if str[0..3] == Expression::START_BLOCK_COMMENT
          isTillNewline = false
        end
      end
      
      endIndex = isTillNewline \
        ? str.index("\n") \
        : str.index(Expression::END_BLOCK_COMMENT)
      
      lengthToSkip = isTillNewline ? 1 : Expression::END_BLOCK_COMMENT.size
      
      # move forward columns/rows as needed
      parserState.move_forward_based_on_string(
        str[0 .. (endIndex == nil ? (str.size - 1) : (endIndex + lengthToSkip - 1))]
      )
      
      if endIndex == nil or endIndex > (str.size - lengthToSkip)
        str = "" # dead
      else # slice
        str = str[endIndex+lengthToSkip..-1] # skip the comment
      end
    else
      break
    end
  end
  
  return str
end

.s_value_for_escape(c) ⇒ Object 



559
560
561
562
563
564
565
566
567
568
569
 
# File 'lib/wexpr/expression.rb', line 559

def self.s_value_for_escape(c)
  return case c
    when '"' then '"'
    when 'r' then "\r"
    when 'n' then "\n"
    when 't' then "\t"
    when "\\" then "\\"
  else
    0 # invalid escape
  end
end

.s_wexpr_value_string_properties(ref) ⇒ Object

returns information about a string



745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
 
# File 'lib/wexpr/expression.rb', line 745

def self.s_wexpr_value_string_properties(ref)
  props = {
    :isbarewordsafe => true, # default to being safe
    :needsescaping => false # but we don't need escaping
  }
  
  ref.each_char do |c|
    # for now we cant escape so that stays false
    # bareword safe we're just check for a few symbols
    
    # see any symbols that makes it not bareword safe?
    if self.s_is_not_bareword_safe(c)
      props[:isbarewordsafe] = false
      break
    end
  end
  
  if ref.length == 0
    props[:isbarewordsafe] = false # empty string is not safe since that will be nothing
  end
  
  return props
end

.s_write_string_escaped(v, props) ⇒ Object

Will write the string escaped, and with quotes around it if needed.



1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
 
# File 'lib/wexpr/expression.rb', line 1176

def self.s_write_string_escaped(v, props)
  buf = ''

  if not props[:isbarewordsafe]
    buf += '"'
  end
    
  # do per character so we can escape as needed
  for c in v.chars
    if Expression.s_requires_escape(c)
      buf += "\\"
      buf += Expression.s_escape_for_value(c)
    else
      buf += c
    end
  end
    
  if not props[:isbarewordsafe]
    buf += '"'
  end

  return buf
end

Instance Method Details

#[](index) ⇒ Object

array operator for array and map



518
519
520
521
522
523
524
525
526
527
 
# File 'lib/wexpr/expression.rb', line 518

def [](index)
  if @type == :array
    return self.array_at(index)
  elsif @type == :map
    return self.map_value_for_key(index)
  end
  
  # wrong type
  return nil
end

#array_add_element_to_end(element) ⇒ Object

Add an element to the end of the array



400
401
402
403
404
405
406
 
# File 'lib/wexpr/expression.rb', line 400

def array_add_element_to_end(element)
  if @type != :array
    return nil
  end
  
  @array << element
end

#array_at(index) ⇒ Object

Return an object at the given index



389
390
391
392
393
394
395
 
# File 'lib/wexpr/expression.rb', line 389

def array_at(index)
  if @type != :array
    return nil
  end
  
  return @array[index]
end

#array_count ⇒ Object

Return the length of the array



378
379
380
381
382
383
384
 
# File 'lib/wexpr/expression.rb', line 378

def array_count()
  if @type != :array
    return nil
  end
  
  return @array.count
end

#binarydata ⇒ Object

Return the binary data of the expression. Will return nil if not binary data.



354
355
356
357
358
359
360
 
# File 'lib/wexpr/expression.rb', line 354

def binarydata()
  if @type != :binarydata
    return nil
  end
  
  return @binarydata
end

#binarydata_set(buffer) ⇒ Object

Set the binary data of the expression.



365
366
367
368
369
370
371
 
# File 'lib/wexpr/expression.rb', line 365

def binarydata_set(buffer)
  if @type != :binarydata
    return nil
  end
  
  @binarydata = buffer
end

#change_type(type) ⇒ Object

Change the type of the expression. Invalidates all data currently in the expression.



190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
 
# File 'lib/wexpr/expression.rb', line 190

def change_type(type)
  # first destroy
  @value=nil; remove_instance_variable(:@value)
  @binarydata=nil; remove_instance_variable(:@binarydata)
  @array=nil; remove_instance_variable(:@array)
  @map=nil; remove_instance_variable(:@map)
  
  # then set
  @type = type
  
  # then init
  if @type == :value
    @value = "".freeze
  elsif @type == :binarydata
    @binarydata = ""
  elsif @type == :array
    @array = []
  elsif @type == :map
    @map = {}
  end
end

#create_binary_representation ⇒ Object

Create the binary representation for the current chunk. This contains an expression chunk and all of its child chunks, but NOT the file header.



224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
 
# File 'lib/wexpr/expression.rb', line 224

def create_binary_representation()
  buf = ""
  
  # format is:
  # size, type, data
  
  if @type == :null
    # data is 0x0
    buf += UVLQ64::write(0)
    buf += [BIN_EXPRESSIONTYPE_NULL].pack('C')
    
  elsif @type == :value
    val = self.value
    valLength = val.bytes.size
    
    buf += UVLQ64::write(valLength)
    buf += [BIN_EXPRESSIONTYPE_VALUE].pack('C')
    buf += val.bytes.pack('C*')
    
  elsif @type == :array
    # first pass, figure out the total size
    sizeOfArrayContents = 0
    
    for i in 0 ... self.array_count()
      # TODO: dont create the entire representation twice - ability to ask for size
      child = self.array_at(i)
      childBuffer = child.create_binary_representation()
      
      sizeOfArrayContents += childBuffer.bytes.size
    end
    
    # header
    buf += UVLQ64::write(sizeOfArrayContents)
    buf += [BIN_EXPRESSIONTYPE_ARRAY].pack('C*')
    
    # data
    for i in 0 ... self.array_count()
      child = self.array_at(i)
      childBuffer = child.create_binary_representation()
      
      buf += childBuffer
    end
    
    # done
    
  elsif @type == :map
    # first pass, figure out the total size
    sizeOfMapContents = 0
    
    for i in 0 ... self.map_count()
      # TODO: dont create the entire representation twice - ability to ask for sizes
      mapKey = self.map_key_at(i)
      mapKeyLen = mapKey.bytes.size
      
      # write the map key as a new value
      keySizeSize = UVLQ64::byte_size(mapKeyLen)
      keySize = keySizeSize + SIZE_U8 + mapKeyLen
      sizeOfMapContents += keySize
      
      # write the map value
      mapValue = self.map_value_at(i)
      childBuffer = mapValue.create_binary_representation()
      sizeOfMapContents += childBuffer.bytes.size
    end
    
    # second pass, write the header and pairs
    buf += UVLQ64::write(sizeOfMapContents)
    buf += [BIN_EXPRESSIONTYPE_MAP].pack('C')
    
    for i in 0 ... self.map_count()
      mapKey = self.map_key_at(i)
      mapKeyLen = mapKey.bytes.size
      
      # write the map key as a new value
      buf += UVLQ64::write(mapKeyLen)
      buf += [BIN_EXPRESSIONTYPE_VALUE].pack('C')
      buf += mapKey.bytes.pack('C*')
      
      # write the map value
      mapValue = self.map_value_at(i)
      childBuffer = mapValue.create_binary_representation()
      
      buf += childBuffer
    end
    
    # done
  elsif @type == :binarydata
    binData = self.binarydata
    
    buf += UVLQ64::write(binData.bytes.size+1) # +1 for compression method
    buf += [BIN_EXPRESSIONTYPE_BINARYDATA].pack('C')
    buf += [BIN_BINARYDATA_RAW].pack('C') # for now we never compress
    buf += binData.bytes.pack('C*')
     
  else
    raise Exception.new(0, 0, "Unknown type")
  end
  
  return buf
end

#create_copy ⇒ Object

Create a copy of an expression. You own the copy - deep copy.



115
116
117
118
119
120
121
 
# File 'lib/wexpr/expression.rb', line 115

def create_copy()
  expr = Expression.create_null()
  
  expr.p_copy_from(self)
  
  return expr
end

#create_string_representation(indent, writeFlags = []) ⇒ Object

Create a string which represents the expression.



215
216
217
218
219
 
# File 'lib/wexpr/expression.rb', line 215

def create_string_representation(indent, writeFlags=[])
  newBuf = self.p_append_string_representation_to_buffer(writeFlags, indent, "")
  
  return newBuf
end

#map_count ⇒ Object

Return the number of elements in the map



413
414
415
416
417
418
419
 
# File 'lib/wexpr/expression.rb', line 413

def map_count()
  if @type != :map
    return nil
  end
  
  return @map.count
end

#map_key_at(index) ⇒ Object

Return the key at a given index



424
425
426
427
428
429
430
 
# File 'lib/wexpr/expression.rb', line 424

def map_key_at(index)
  if @type != :map
    return nil
  end
  
  return @map.keys[index]
end

#map_set_value_for_key(key, value) ⇒ Object

Set the value for a given key in the map. Will overwrite if already exists.



457
458
459
460
461
462
463
 
# File 'lib/wexpr/expression.rb', line 457

def map_set_value_for_key(key, value)
  if @type != :map
    return nil
  end
  
  @map[key] = value
end

#map_value_at(index) ⇒ Object

Return the value at a given index



435
436
437
438
439
440
441
 
# File 'lib/wexpr/expression.rb', line 435

def map_value_at(index)
  if @type != :map
    return nil
  end
  
  return @map.values[index]
end

#map_value_for_key(key) ⇒ Object

Return the value for a given key in the map



446
447
448
449
450
451
452
 
# File 'lib/wexpr/expression.rb', line 446

def map_value_for_key(key)
  if @type != :map
    return nil
  end
  
  return @map[key]
end

#p_append_string_representation_to_buffer(flags, indent, buffer) ⇒ Object

NOTE THESE BUFFERS ARE ACTUALLY MUTABLE

Human Readable notes: even though you pass an indent, we assume you’re already indented for the start of the object we assume this so that an object for example as a key-value will be writen in the correct spot. if it writes multiple lines, we will use the given indent to predict. it will end after writing all data, no newline generally at the end.



1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
 
# File 'lib/wexpr/expression.rb', line 1208

def p_append_string_representation_to_buffer(flags, indent, buffer)
  writeHumanReadable = flags.include?(:humanReadable)
  type = self.type()
  newBuf = buffer.clone
  
  if type == :null
    newBuf += "null"
    return newBuf
    
  elsif type == :value
    # value - always write directly
    v = self.value
    props = Expression.s_wexpr_value_string_properties(v)
    
    newBuf += Expression.s_write_string_escaped(v, props)
    
    return newBuf
    
  elsif type == :binarydata
    # binary data - encode as Base64
    v = Base64.strict_encode64(self.binarydata)
    
    newBuf += "<#{v}>"
    
    return newBuf
    
  elsif type == :array
    arraySize = self.array_count
    
    if arraySize == 0
      # straightforward : always empty structure
      newBuf += "#()"
      return newBuf
    end
    
    # otherwise we have items
    
    # array: human readable we'll write each one on its own line
    if writeHumanReadable
      newBuf += "#(\n"
    else
      newBuf += "#("
    end
    
    for i in 0..arraySize-1
      obj = self.array_at(i)
      
      # if human readable we need to indent the line, output the object, then add a newline
      if writeHumanReadable
        newBuf += Expression.s_indent(indent+1)
        
        # now add our normal
        newBuf = obj.p_append_string_representation_to_buffer(flags, indent+1, newBuf)
        
        # add the newline
        newBuf += "\n"
      else
        # if not human readable, we need to either output theo bject, or put a space then the object
        if i > 0
          # we need a space
          newBuf += " "
        end
        
        # now add our normal
        newBuf = obj.p_append_string_representation_to_buffer(flags, indent, newBuf)
      end
    end
    
    # done with the core of the array
    # if human readable, indent and add the end array
    # otherwise, just add the end array
    if writeHumanReadable
      newBuf += Expression.s_indent(indent)
    end
    newBuf += ")"
    
    # and done
    return newBuf
    
  elsif type == :map
    mapSize = self.map_count()
    
    if mapSize == 0
      # straightforward - always empty structure
      newBuf += "@()"
      return newBuf
    end
    
    # otherwise we have items
    
    # map : human readable we'll write each one on its own line
    if writeHumanReadable
      newBuf += "@(\n"
    else
      newBuf += "@("
    end
    
    for i in 0..mapSize-1
      key = self.map_key_at(i)
      if key == nil
        next # we shouldnt ever get an empty key, but its possible currently in the case of dereffing in a key for some reason : @([a]a b *[a] c)
      end
      
      value = self.map_value_at(i)
      
      # if human readable, indent the line, output the key, space, object, newline
      if writeHumanReadable
        newBuf += Expression.s_indent(indent+1)
        newBuf += Expression.s_write_string_escaped(key, Expression.s_wexpr_value_string_properties(key))
        newBuf += " "
        
        # add the value
        newBuf = value.p_append_string_representation_to_buffer(flags, indent+1, newBuf)
        
        # add the newline
        newBuf += "\n"
      else
        # if not human readable, just output with spaces as needed
        if i > 0
          # we need a space
          newBuf += " "
        end
        
        # now key, space, value
        newBuf += Expression.s_write_string_escaped(key, Expression.s_wexpr_value_string_properties(key))
        newBuf += " "
        
        newBuf = value.p_append_string_representation_to_buffer(flags, indent, newBuf)
      end
    end
    
    # done with the core of the map
    # if human readable, indent and add the end map
    # otherwise, just add the end map
    if writeHumanReadable
      newBuf += Expression.s_indent(indent)
    end
    
    newBuf += ")"
    
    # and done
    return newBuf

  else
    raise Exception.new(0,0,"p_append_string_representation_to_buffer - Unknown type to generate string for: {}", type)
  end
end

#p_copy_from(rhs) ⇒ Object

copy an expression into self. lhs should be null cause we dont cleanup ourself atm (ruby note: might not be true).



775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
 
# File 'lib/wexpr/expression.rb', line 775

def p_copy_from(rhs)
  # copy recursively
  case rhs.type
    when :value
      self.change_type(:value)
      self.value_set(rhs.value)
      
    when :binarydata
      self.change_type(:binarydata)
      self.binarydata_set(rhs.binarydata)
      
    when :array
      self.change_type(:array)
      
      c = rhs.array_count()
      
      for i in 0..c-1
        child = rhs.array_at(i)
        childCopy = child.create_copy()
      
        # add to our array
        self.array_add_element_to_end(childCopy)
      end
      
    when :map
      self.change_type(:map)
      
      c = rhs.map_count()
      
      for i in 0..c-1
        key = rhs.map_key_at(i)
        value = rhs.map_value_at(i)
        
        # add to our map
        self.map_set_value_for_key(key, value)
      end
      
    else
      # ignore - we dont handle this type
  end
end

#p_parse_from_binary_chunk(data) ⇒ Object

returns the part of the buffer remaining will load into self, setitng up everything. Assumes we’re empty/null to start.



819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
 
# File 'lib/wexpr/expression.rb', line 819

def p_parse_from_binary_chunk(data)
  if data.size < (1 + SIZE_U8) 
    raise BinaryChunkNotBigEnoughError.new(0, 0, "Chunk not big enough for header: size was #{data.size}")
  end
  
  size, dataNew = UVLQ64::read(data)
  sizeSize = (data.size - dataNew.size)
  chunkType = data[sizeSize].unpack('C')[0]
  readAmount = sizeSize + SIZE_U8
  
  if chunkType == BIN_EXPRESSIONTYPE_NULL
    # nothing more to do
    self.change_type(:null)
    
    return data[readAmount .. -1]
    
  elsif chunkType == BIN_EXPRESSIONTYPE_VALUE
    # data is the entire binary data
    self.change_type(:value)
    self.value_set(
      data[readAmount...readAmount+size]
    )
    
    readAmount += size
    
    return data[readAmount .. -1]
    
  elsif chunkType == BIN_EXPRESSIONTYPE_ARRAY
    # data is child chunks
    self.change_type(:array)
    
    curPos = 0
    
    # build children as needed
    while curPos < size
      # read a new element
      startSize = size-curPos
      
      childExpr = Expression.create_invalid()
      remaining = childExpr.p_parse_from_binary_chunk(
        data[readAmount+curPos...readAmount+curPos+startSize]
      )
      
      if remaining == nil
        # failure when parsing the array
        raise Exception.new(0, 0, "Failure when parsing array")
      end
      
      curPos += (startSize - remaining.size)
      
      # otherwise, add it
      self.array_add_element_to_end(childExpr)
    end
    
    readAmount += curPos
    return data[readAmount .. -1]
    
  elsif chunkType == BIN_EXPRESSIONTYPE_MAP
    # data is key, value chunks
    self.change_type(:map)
    
    curPos = 0
    
    # build children as needed
    while curPos < size
      # read a new key
      startSize = size - curPos
      
      keyExpression = Expression.create_invalid()
      remaining = keyExpression.p_parse_from_binary_chunk(
        data[readAmount+curPos ... readAmount+curPos+startSize]
      )
      
      if remaining == nil
        raise Exception.new(0, 0, "Failure when parsing map key")
      end
      
      keySize = startSize - remaining.size
      curPos += keySize
      
      # now parse the value
      valueExpr = Expression.create_invalid()
      remaining = valueExpr.p_parse_from_binary_chunk(
        remaining
      )
      
      if remaining == nil
        raise Exception.new(0, 0, "Failure when parsing map value")
      end
      
      curPos += (startSize - remaining.size - keySize)
      
      # now add it
      self.map_set_value_for_key(keyExpression.value, valueExpr)
    end
    
    readAmount += curPos
    return data[readAmount .. -1]
    
  elsif chunkType == BIN_EXPRESSIONTYPE_BINARYDATA
    # data is the entire binary data
    # first byte is the compression method
    compression = data[readAmount].unpack('C')[0]
    
    if compression == 0x0 # NONE
      # simple and raw
      self.change_type(:binarydata)
      self.binarydata_set(
        data[readAmount+1...readAmount+size]
      )
      
      readAmount += size
      return data[readAmount .. -1]
    else
      raise Exception.new(0, 0, "Unknown compression method for binary data")
    end
    
  else
    # unknown type
    raise BinaryChunkNotBigEnoughError.new(0, 0, "Unknown chunk type to read: was #{chunkType}")
  end
end

#p_parse_from_string(str, parseFlags, parserState) ⇒ Object

returns the part of the string remaining will load into self, setting up everything. Assumes we’re empty/null to start.



944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
 
# File 'lib/wexpr/expression.rb', line 944

def p_parse_from_string(str, parseFlags, parserState)
  if str.size == 0
    raise EmptyStringError(parserState.line, parserState.column, "Was told to parse an empty string")
  end
  
  # now we parse
  str = Expression.s_trim_front_of_string(str, parserState)
  
  if str.size == 0
    return "" # nothing left to parse
  end
  
  # start parsing types
  # if first two characters are #(, we're an array
  # if @( we're a map
  # if [] we're a ref
  # if <> we're a binary string
  # otherwise we're a value.
  if str.size >= 2 and str[0..1] == "#("
    # we're an array
    @type = :array
    @array = []
    
    # move our string forward
    str = str[2..-1]
    parserState.column += 2
    
    # continue building children as needed
    while true
      str = Expression.s_trim_front_of_string(str, parserState)
      
      if str.size == 0
        raise ArrayMissingEndParenError.new(parserState.line, parserState.column, "An array was missing its ending paren")
      end
      
      if str[0] == ")"
        break # done
      else
        # parse as a new expression
        newExpression = Expression.create_null()
        str = newExpression.p_parse_from_string(str, parseFlags, parserState)
        
        # otherwise, add it to our array
        @array << newExpression
      end
    end
    
    str = str[1..-1] # remove the end array
    parserState.column += 1
    
    # done with array
    return str
  elsif str.size >= 2 and str[0..1] == "@("
    # we're a map
    @type = :map
    @map = {}
    
    # move our string accordingly
    str = str[2..-1]
    parserState.column += 2
    
    # build our children as needed
    while true
      str = Expression.s_trim_front_of_string(str, parserState)
      
      if str.size == 0
        raise MapMissingEndParenError.new(parserState.line, parserState.column, "A Map was missing its ending paren")
      end
      
      if str.size >= 1 and str[0] == ")" # end map
        break # done
      else
        # parse as a new expression - we'll alternate keys and values
        # keep our previous position just in case the value is bad
        prevLine = parserState.line
        prevColumn = parserState.column
        
        keyExpression = Expression.create_null()
        str = keyExpression.p_parse_from_string(str, parseFlags, parserState)
        
        if keyExpression.type != :value
          raise MapKeyMustBeAValueError.new(prevLine, prevColumn, "Map keys must be a value")
        end
        
        valueExpression = Expression.create_invalid()
        str = valueExpression.p_parse_from_string(str, parseFlags, parserState)
        
        if valueExpression.type == :invalid
          # it wasn't filled in! no key found.
          # TODO: this changes the error from an upper level, so we'd need to catch and rethrow for this
          raise MapNoValueError.new(prevLine, prevColumn, "Map key must have a value")
        end
        
        # ok we have the key and the value
        self.map_set_value_for_key(keyExpression.value, valueExpression)
      end
    end
    
    # remove the end map
    str = str[1..-1]
    parserState.column += 1
    
    # done with map
    return str
    
  elsif str.size >= 1 and str[0] == "["
    # the current expression being processed is the one the attribute will be linked to.
    
    # process till the closing ]
    endingBracketIndex = str.index ']'
    if endingBracketIndex == nil
      raise ReferenceMissingEndBracketError.new(parserState.line, parserState.column, "A reference [] is missing its ending bracket")
    end
    
    refName = str[1..endingBracketIndex-1]
    
    # validate the contents
    invalidName = false
    for i in 0..refName.size-1
      v = refName[i]
      
      isAlpha = (v >= 'a' && v <= 'z') || (v >= 'A' && v <= 'Z')
      isNumber = (v >= '0' && v <= '9')
      isUnder = (v == '_')
      
      if i == 0 and (isAlpha || isUnder)
      elsif i != 0 and (isAlpha or isNumber or isUnder)
      else
        invalidName = true
        break
      end
    end
    
    if invalidName
      raise ReferenceInvalidNameError.new(parserState.line, parserState.column, "A reference doesn't have a valid name")
    end
    
    # move forward
    parserState.move_forward_based_on_string(str[0..endingBracketIndex+1-1])
    str = str[endingBracketIndex+1..-1]
    
    # continue parsing at the same level : stored the reference name
    resultString = self.p_parse_from_string(str, parseFlags, parserState)
    
    # now bind the ref - creating a copy of what was made. This will be used for the template.
    parserState.internalReferenceMap[refName] = self.create_copy()
    
    # and continue
    return resultString
    
  elsif str.size >= 2 and str[0..1] == "*["
    # parse the reference name
    endingBracketIndex = str.index ']'
    if endingBracketIndex == nil
      raise ReferenceInsertMissingEndBracketError.new(parserState.line, parserState.column, "A reference insert *[] is missing its ending bracket")
    end
    
    refName = str[2 .. endingBracketIndex-1]
    
    # move forward
    parserState.move_forward_based_on_string(
      str[0 .. endingBracketIndex+1-1]
    )
    
    str = str[endingBracketIndex+1 .. -1]
    
    referenceExpr = parserState.internalReferenceMap[refName]
    
    if referenceExpr == nil
      # try again with the external if we have it
      if parserState.externalReferenceMap != nil
        referenceExpr = parserState.externalReferenceMap[refName]
      end
    end
    
    if referenceExpr == nil
      # not found
      raise ReferenceUnknownReferenceError.new(parserState.line, parserState.column, "Tried to insert a reference, but couldn't find it.")
    end
    
    # copy this into ourself
    self.p_copy_from(referenceExpr)
    
    return str
    
  # null expressions will be treated as a value and then parsed seperately
    
  elsif str.size >= 1 and str[0] == "<"
    # look for the ending >
    endingQuote = str.index '>'
    if endingQuote == nil
      # not found
      raise BinaryDataNoEndingError.new(parserState.line, parserState.column, "Tried to find the ending > for binary data, but not found.")
    end

    # we require strict so we fail out on incorrect padding
    outBuf = Base64.strict_decode64(str[1 .. endingQuote-1]) # -1 for starting quote, ending was not part
    if outBuf == nil
      raise BinaryDataInvalidBase64Error.new(parserState.line, parserState.column, "Unable to decode the base64 data")
    end
    
    @type = :binarydata
    @binarydata = outBuf
    
    parserState.move_forward_based_on_string(str[0..endingQuote+1-1])
    
    return str[endingQuote+1 .. -1]
    
  elsif str.size >= 1 # its a value : must be at least one character
    val, endPos = Expression.s_create_value_of_string(str, parserState)
    
    # was it a null/nil string?
    if val == "nil" or val == "null"
      @type = :null
    else
      @type = :value
      @value = val.dup.freeze
    end
    
    parserState.move_forward_based_on_string(str[0..endPos-1])
    
    return str[endPos .. -1]
  end
  
  # otherwise, we have no idea what happened
  return ""
end

#to_ruby ⇒ Object

Convert to a ruby type. Types will convert as follows:

  • :null - Returns nil

  • :value - Returns the value as a string.

  • :array - Returns as an array.

  • :map - Returns as a hash.

  • :binarydata - Returns the binary data as a ruby string.

  • :invalid - Throws an exception



484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
 
# File 'lib/wexpr/expression.rb', line 484

def to_ruby()
  case @type
    when :null
      return nil
      
    when :value
      return @value
      
    when :array
      a=[]
      for i in 0..self.array_count-1
        a << self.array_at(i).to_ruby
      end
      return a
      
    when :map
      m={}
      for i in 0..self.map_count-1
        k = self.map_key_at(i)
        v = self.map_value_at(i)
        
        m[k] = v.to_ruby
      end
      
      return m
    when :binarydata
      # um, direct i guess - its raw/pure data
      return @binarydata
    else
      raise Exception.new(0,0,"Invalid type to convert to ruby: #{@type}")
  end
end

#to_s ⇒ Object

Output as a string - we do a compact style



470
471
472
 
# File 'lib/wexpr/expression.rb', line 470

def to_s()
  return self.create_string_representation(0, [])
end

#type ⇒ Object

Return the type of the expression



183
184
185
 
# File 'lib/wexpr/expression.rb', line 183

def type()
  return @type
end

#value ⇒ Object

Return the value of the expression. Will return nil if not a value



330
331
332
333
334
335
336
 
# File 'lib/wexpr/expression.rb', line 330

def value()
  if @type != :value
    return nil
  end
  
  return @value
end

#value_set(str) ⇒ Object

Set the value of the expression.



341
342
343
344
345
346
347
 
# File 'lib/wexpr/expression.rb', line 341

def value_set(str)
  if @type != :value
    return
  end
  
  @value = str.dup.freeze
end