Class: Dendroid::Lexical::Token

Inherits:
Object
  • Object
show all
Defined in:
lib/dendroid/lexical/token.rb

Overview

A (lexical) token is an object created by a tokenizer (lexer) and passed to the parser. Such token object is created when a lexer detects that a sequence of characters(a lexeme) from the input stream is an instance of a terminal grammar symbol. Say, that in a particular language, the lexeme ‘foo’ is an occurrence of the terminal symbol IDENTIFIER. Then the lexer will return a Token object that states the fact that ‘foo’ is indeed an IDENTIFIER. Basically, a Token is a pair (lexeme, terminal): it asserts that a given piece of text is an instance of given terminal symbol.

Direct Known Subclasses

Literal

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(original, pos, symbol) ⇒ Token

Constructor.



34
35
36
37
38
 
# File 'lib/dendroid/lexical/token.rb', line 34

def initialize(original, pos, symbol)
  @source = original.dup
  @position = pos
  @terminal = symbol
end

Instance Attribute Details

#position ⇒ TokenPosition (readonly) 



24
25
26
 
# File 'lib/dendroid/lexical/token.rb', line 24

def position
  @position
end

#source ⇒ String (readonly)

The sequence of character(s) from the input stream that is an occurrence of the related terminal symbol.



21
22
23
 
# File 'lib/dendroid/lexical/token.rb', line 21

def source
  @source
end

#terminal ⇒ String (readonly) 



27
28
29
 
# File 'lib/dendroid/lexical/token.rb', line 27

def terminal
  @terminal
end

Instance Method Details

#pos_to_s ⇒ String 



41
42
43
 
# File 'lib/dendroid/lexical/token.rb', line 41

def pos_to_s
  position.to_s
end