Class: SDL4R::Parser::Tokenizer
- Inherits:
-
Object
- Object
- SDL4R::Parser::Tokenizer
- Defined in:
- lib/sdl4r/parser/tokenizer.rb
Overview
Tokenizer of the SDL parser
Constant Summary collapse
- TOKEN_TYPES =
:nodoc: all
[ :IDENTIFIER, # punctuation :COLON, :SEMICOLON, :EQUALS, :START_BLOCK, :END_BLOCK, # literals :STRING, :CHARACTER, :BOOLEAN, :NUMBER, :DATE, :TIME, :BINARY, :NULL ]
Instance Method Summary collapse
-
#close ⇒ Object
Closes this Tokenizer and its underlying
Reader
. -
#expecting_but_got(expecting, got, line, position) ⇒ Object
Close the reader and throw a SdlParseError using the format Was expecting X but got Y.
-
#initialize(io) ⇒ Tokenizer
constructor
Creates an SDL tokenizer on the specified
IO
. - #line ⇒ Object
- #line_no ⇒ Object
-
#parse_error(description, line_no = nil, position = nil) ⇒ Object
Raises a SdlParseError.
- #pos ⇒ Object
-
#read_line_tokens ⇒ Object
Returns the next line as tokens or nil if the end of the stream has been reached.
Constructor Details
#initialize(io) ⇒ Tokenizer
Creates an SDL tokenizer on the specified IO
.
43 44 45 46 47 48 49 50 51 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 43 def initialize(io) raise ArgumentError, "io == nil" if io.nil? @reader = Parser::Reader.new(io) @token_start = 0 @starting_escaped_quote_line = false @tokens = nil @token_text = nil end |
Instance Method Details
#close ⇒ Object
Closes this Tokenizer and its underlying Reader
.
54 55 56 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 54 def close @reader.close end |
#expecting_but_got(expecting, got, line, position) ⇒ Object
Close the reader and throw a SdlParseError using the format Was expecting X but got Y.
71 72 73 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 71 def expecting_but_got(expecting, got, line, position) parse_error("Was expecting #{expecting} but got #{got}", line, position) end |
#line ⇒ Object
96 97 98 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 96 def line @reader.line end |
#line_no ⇒ Object
88 89 90 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 88 def line_no @reader.line_no end |
#parse_error(description, line_no = nil, position = nil) ⇒ Object
Raises a SdlParseError.
59 60 61 62 63 64 65 66 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 59 def parse_error(description, line_no = nil, position = nil) line_no = @reader.line_no unless line_no # @reader.pos points to the *next* position (thence the "pos - 1") position = @reader.pos - 1 unless position and position >= 0 # We add one because editors typically start with line 1 rather than 0... raise SdlParseError.new(description, line_no + 1, position + 1, @reader.line) end |
#pos ⇒ Object
92 93 94 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 92 def pos @reader.pos end |
#read_line_tokens ⇒ Object
Returns the next line as tokens or nil if the end of the stream has been reached. This method handles line continuations both within and outside String literals. The line of tokens is assigned to @tokens.
Returns a logical line as a list of Tokens.
81 82 83 84 85 86 |
# File 'lib/sdl4r/parser/tokenizer.rb', line 81 def read_line_tokens begin read_line_tokens_even_if_empty() end until @tokens.nil? or !@tokens.empty? return @tokens end |