Class: Cadenza::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/cadenza/lexer.rb

Overview

The Lexer class accepts in input IO object which it will parse simple Tokens from for use in a Parser class.

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Lexer

constructs a new parser and sets it to the position (0, 0)



9
10
11
12
 
# File 'lib/cadenza/lexer.rb', line 9

def initialize
  @line = 0
  @column = 0
end

Instance Method Details

#next_token ⇒ Object

Gets the next token and returns it. Tokens are two element arrays where the first element is one of the following symbols and the second is an instance of Token containing the value of the token.

valid tokens:

  • :VAR_OPEN - for opening an inject tag ex. “

  • :VAR_CLOSE - for closing an inject tag ex. “}”

  • :STMT_OPEN - for opening a control tag ex. “

  • :STMT_CLOSE - for closing a control tag ex. “%”

  • :TEXT_BLOCK - for a block of raw text

  • :OP_EQ - for an equivalence symbol ex. “==”

  • :OP_NEQ - for a nonequivalence symbol ex. “!=”

  • :OP_GEQ - for a greater than or equal to symbol ex. “>=”

  • :OP_LEQ - for a less than or equal to symbol ex. “<=”

  • :REAL - for a number with a decimal value ex. “123.45”

  • :INTEGER - for a number without a decimal value ex. “12345”

  • :STRING - for a string literal, either from single quotes or double quotes ex. “‘foo’”

  • :IDENTIFIER - for a variable name ex. “foo”

  • :IF - for the ‘if’ keyword

  • :UNLESS - for the ‘unless’ keyword

  • :ELSE - for the ‘else’ keyword

  • :ENDIF - for the ‘endif’ keyword

  • :ENDUNLESS - for the ‘endunless’ keyword

  • :FOR - for the ‘for’ keyword

  • :IN - for the ‘in’ keyword

  • :ENDFOR - for the ‘endfor’ keyword

  • :BLOCK - for the ‘block’ keyword

  • :ENDBLOCK - for the ‘endblock’ keyword

  • :EXTENDS - for the ‘extends’ keyword

  • :END - for the ‘end’ keyword

  • :AND - for the ‘and’ keyword

  • :OR - for the ‘or’ keyword

  • :NOT - for the ‘not’ keyword

if no tokens are left the return value will be [false, false]



67
68
69
70
71
72
73
 
# File 'lib/cadenza/lexer.rb', line 67

def next_token
  if @scanner.eos?
    [false, false]
  else
    send("scan_#{@context}")
  end
end

#position ⇒ Array

gives the current line and column counter as a two element array

Returns:

  • (Array) —

    the line and column



28
29
30
 
# File 'lib/cadenza/lexer.rb', line 28

def position
  [@line, @column]
end

#remaining_tokens ⇒ Array

returns an array of all remaining tokens left to be parsed. See #next_token for details regarding the definition of a token. The array will always end in [false, false].

Returns:

  • (Array) —

    a list of all remaining tokens



80
81
82
83
84
85
86
87
88
89
 
# File 'lib/cadenza/lexer.rb', line 80

def remaining_tokens
  result = []

  loop do
    result.push next_token
    break if result.last == [false, false]
  end

  result
end

#source=(source) ⇒ Object

assigns a new string to retrieve tokens and resets the line and column counters to (1, 1)

Parameters:

  • source (String) —

    the string from which to parse tokens



17
18
19
20
21
22
23
24
 
# File 'lib/cadenza/lexer.rb', line 17

def source=(source)
  @scanner = ::StringScanner.new(source || "")

  @line = 1
  @column = 1

  @context = :body
end