Class: Rley::Lexical::Token

Inherits:
Object
  • Object
show all
Defined in:
lib/rley/lexical/token.rb

Overview

In Rley, a (lexical) token is an object created by a lexer (tokenizer) and passed to the parser. Such token an object is created when a lexer detects that a sequence of characters(a lexeme) from the input stream is an instance of a terminal grammar symbol. Say, that in a particular language, the lexeme 'foo' is an occurrence of the terminal symbol IDENTIFIER. Then the lexer will return a Token object that states the fact that 'foo' is indeed an IDENTIFIER. Basically, a Token is a pair (lexeme, terminal): it asserts that a given lexeme is an instance of given terminal symbol.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(theLexeme, aTerminal) ⇒ Token

Constructor.

Parameters:

  • theLexeme (String) —

    the lexeme (= piece of text from input)

  • aTerminal (Syntax::Terminal) —

    The terminal symbol corresponding to the lexeme.



24
25
26
27
 
# File 'lib/rley/lexical/token.rb', line 24

def initialize(theLexeme, aTerminal)
  @lexeme = theLexeme
  @terminal = aTerminal
end

Instance Attribute Details

#lexeme ⇒ String (readonly)

The sequence of character(s) from the input stream that is an occurrence of the related terminal symbol.

Returns:

  • (String) —

    Input substring that is an instance of the terminal.



16
17
18
 
# File 'lib/rley/lexical/token.rb', line 16

def lexeme
  @lexeme
end

#terminal ⇒ Syntax::Terminal (readonly)

Returns Terminal symbol corresponding to the lexeme.

Returns:



19
20
21
 
# File 'lib/rley/lexical/token.rb', line 19

def terminal
  @terminal
end