Class: Token

Inherits:
Object show all
Defined in:
lib/antlr4/Token.rb

Overview

A token has properties: text, type, line, character position in the line (so we can ignore tabs), token channel, index, and source from which we obtained this token.

Direct Known Subclasses

CommonToken, RuleTagToken

Constant Summary collapse

INVALID_TYPE = 
0
EPSILON =

During lookahead operations, this “token” signifies we hit rule end ATN state and did not follow it despite needing to.

-2
MIN_USER_TOKEN_TYPE = 
1
EOF = 
-1
# All tokens go to the parser (unless skip() is called in that rule)
# on a particular "channel".  The parser tunes to a particular channel
# so that whitespace etc... can go to the parser on a "hidden" channel.
DEFAULT_CHANNEL =

All tokens go to the parser (unless skip() is called in that rule) on a particular “channel”. The parser tunes to a particular channel so that whitespace etc… can go to the parser on a “hidden” channel.

0
HIDDEN_CHANNEL =

Anything on different channel than DEFAULT_CHANNEL is not parsed by parser.

1

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Token

we obtained this token.



26
27
28
29
30
31
32
33
34
35
 
# File 'lib/antlr4/Token.rb', line 26

def initialize()
    self.source = nil
    self.type = nil       # token type of the token
    self.channel = nil    # The parser ignores everything not on DEFAULT_CHANNEL
    self.start = -1       # optional; return -1 if not implemented.
    self.stop = -1        # optional; return -1 if not implemented.
    self.tokenIndex = nil # from 0..n-1 of the token object in the input stream
    self.line = nil       # line=1..n of the 1st character
    self.column = nil     # beginning of the line at which it occurs, 0..n-1
end

Instance Attribute Details

#channel ⇒ Object

Returns the value of attribute channel.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def channel
  @channel
end

#column ⇒ Object

Returns the value of attribute column.



22
23
24
 
# File 'lib/antlr4/Token.rb', line 22

def column
  @column
end

#line ⇒ Object

Returns the value of attribute line.



22
23
24
 
# File 'lib/antlr4/Token.rb', line 22

def line
  @line
end

#source ⇒ Object

Returns the value of attribute source.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def source
  @source
end

#start ⇒ Object

Returns the value of attribute start.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def start
  @start
end

#stop ⇒ Object

Returns the value of attribute stop.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def stop
  @stop
end

#text ⇒ Object

Returns the value of attribute text.



22
23
24
 
# File 'lib/antlr4/Token.rb', line 22

def text
  @text
end

#tokenIndex ⇒ Object

Returns the value of attribute tokenIndex.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def tokenIndex
  @tokenIndex
end

#type ⇒ Object

Returns the value of attribute type.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def type
  @type
end

Instance Method Details

#getInputStream ⇒ Object 



48
49
50
 
# File 'lib/antlr4/Token.rb', line 48

def getInputStream()
    return self.source[1]
end

#getTokenSource ⇒ Object

Explicitly set the text for this token. If text is not null, then #getText will return this value rather than extracting the text from the input.

should be obtained from the input along with the start and stop indexes of the token.

Parameters:

  • text —

    The explicit text of the token, or null if the text



44
45
46
 
# File 'lib/antlr4/Token.rb', line 44

def getTokenSource()
    return self.source[0]
end