Class: Rley::Lexical::Token

Inherits:
Object
  • Object
show all
Defined in:
lib/rley/lexical/token.rb

Overview

In Rley, a (lexical) token is an object created by a lexer (tokenizer) and passed to the parser. Such token an object is created when a lexer detects that a sequence of characters(a lexeme) from the input stream is an instance of a terminal grammar symbol. Say, that in a particular language, the lexeme 'foo' is an occurrence of the terminal symbol IDENTIFIER. Then the lexer will return a Token object that states the fact that 'foo' is indeed an IDENTIFIER. Basically, a Token is a pair (lexeme, terminal): it asserts that a given lexeme is an instance of given terminal symbol.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(theLexeme, aTerminal, aPosition) ⇒ Token

Constructor.

Parameters:

  • theLexeme (String) —

    the lexeme (= piece of text from input)

  • aTerminal (Syntax::Terminal, String) —

    The terminal symbol corresponding to the lexeme.



38
39
40
41
42
43
44
 
# File 'lib/rley/lexical/token.rb', line 38

def initialize(theLexeme, aTerminal, aPosition)
  raise 'Internal error: nil terminal symbol detected' if aTerminal.nil?

  @lexeme = theLexeme
  @terminal = aTerminal
  @position = aPosition
end

Instance Attribute Details

#lexeme ⇒ String (readonly)

The sequence of character(s) from the input stream that is an occurrence of the related terminal symbol.

Returns:

  • (String) —

    Input substring that is an instance of the terminal.



26
27
28
 
# File 'lib/rley/lexical/token.rb', line 26

def lexeme
  @lexeme
end

#position ⇒ Position (readonly)

Returns The position of the lexeme in the source file.

Returns:

  • (Position) —

    The position of the lexeme in the source file.



32
33
34
 
# File 'lib/rley/lexical/token.rb', line 32

def position
  @position
end

#terminal ⇒ String (readonly)

Returns The name of terminal symbol matching the lexeme.

Returns:

  • (String) —

    The name of terminal symbol matching the lexeme.



29
30
31
 
# File 'lib/rley/lexical/token.rb', line 29

def terminal
  @terminal
end