Class: Dendroid::Lexical::Token

Inherits:
Object
  • Object
show all
Defined in:
lib/dendroid/lexical/token.rb

Overview

A (lexical) token is an object created by a tokenizer (lexer) and passed to the parser. Such token object is created when a lexer detects that a sequence of characters(a lexeme) from the input stream is an instance of a terminal grammar symbol. Say, that in a particular language, the lexeme ‘foo’ is an occurrence of the terminal symbol IDENTIFIER. Then the lexer will return a Token object that states the fact that ‘foo’ is indeed an IDENTIFIER. Basically, a Token is a pair (lexeme, terminal): it asserts that a given piece of text is an instance of given terminal symbol.

Direct Known Subclasses

Literal

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(original, pos, symbol) ⇒ Token

Constructor.

Parameters:



34
35
36
37
38
 
# File 'lib/dendroid/lexical/token.rb', line 34

def initialize(original, pos, symbol)
  @source = original.dup
  @position = pos
  @terminal = symbol
end

Instance Attribute Details

#position ⇒ TokenPosition (readonly)

Returns The position -in “editor” coordinates- of the text in the source file.

Returns:

  • (TokenPosition) —

    The position -in “editor” coordinates- of the text in the source file.



24
25
26
 
# File 'lib/dendroid/lexical/token.rb', line 24

def position
  @position
end

#source ⇒ String (readonly)

The sequence of character(s) from the input stream that is an occurrence of the related terminal symbol.

Returns:

  • (String) —

    Input substring that is an instance of the terminal.



21
22
23
 
# File 'lib/dendroid/lexical/token.rb', line 21

def source
  @source
end

#terminal ⇒ String (readonly)

Returns The name of terminal symbol matching the text.

Returns:

  • (String) —

    The name of terminal symbol matching the text.



27
28
29
 
# File 'lib/dendroid/lexical/token.rb', line 27

def terminal
  @terminal
end

Instance Method Details

#literal? ⇒ Boolean

Returns true if the token is a literal (has a value associated with is).

Returns:

  • (Boolean) —

    true if the token is a literal (has a value associated with is)



46
47
48
 
# File 'lib/dendroid/lexical/token.rb', line 46

def literal?
  false
end

#pos_to_s ⇒ String

Returns The text representation of the token position.

Returns:

  • (String) —

    The text representation of the token position



41
42
43
 
# File 'lib/dendroid/lexical/token.rb', line 41

def pos_to_s
  position.to_s
end