Class: Tracksperanto::ShakeGrammar::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/import/shake_grammar/lexer.rb

Overview

Since Shake uses a C-like language for it’s scripts we rig up a very sloppy but concise C-like lexer to cope

Direct Known Subclasses

Catcher

Constant Summary collapse

STOP_TOKEN =

:nodoc:

:__stop
MAX_BUFFER_SIZE = 
32000
MAX_STACK_DEPTH = 
127

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(with_io, sentinel = nil, limit_to_one_stmt = false, stack_depth = 0) ⇒ Lexer

The first argument is the IO handle to the data of the Shake script. The second argument is a “sentinel” that is going to be passed to the downstream lexers instantiated for nested data structures. You can use the sentinel to collect data from child nodes for example.



25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/import/shake_grammar/lexer.rb', line 25

def initialize(with_io, sentinel = nil, limit_to_one_stmt = false, stack_depth = 0)
  # We parse byte by byte, but reading byte by byte is very slow. We therefore use a buffering reader
  # that will cache in chunks, and then read from there byte by byte.
  # This yields a substantial speedup (4.9 seconds for the test
  # as opposed to 7.9 without this). We do check for the proper class only once so that when we use nested lexers
  # we only wrap the passed IO once, and only if necessary.
  with_io = Bychar.wrap(with_io) unless with_io.respond_to?(:read_one_char)
  @io, @stack, @buf, @sentinel, @limit_to_one_stmt, @stack_depth  = with_io, [], '', sentinel, limit_to_one_stmt, stack_depth
  
  catch(STOP_TOKEN) do
    loop { parse }
  end
  
  @in_comment ? consume_comment! : consume_atom!
end

Instance Attribute Details

#sentinel ⇒ Object (readonly)

Access to the sentinel object



15
16
17
 
# File 'lib/import/shake_grammar/lexer.rb', line 15

def sentinel
  @sentinel
end

#stack ⇒ Object (readonly)

Parsed stack



12
13
14
 
# File 'lib/import/shake_grammar/lexer.rb', line 12

def stack
  @stack
end