Class: Token

Inherits:
Object show all
Defined in:
lib/antlr4/Token.rb

Overview

A token has properties: text, type, line, character position in the line (so we can ignore tabs), token channel, index, and source from which we obtained this token.

Direct Known Subclasses

CommonToken, RuleTagToken

Constant Summary collapse

INVALID_TYPE = 
0
EPSILON =

During lookahead operations, this “token” signifies we hit rule end ATN state and did not follow it despite needing to.

-2
MIN_USER_TOKEN_TYPE = 
1
EOF = 
-1
# All tokens go to the parser (unless skip() is called in that rule)
# on a particular "channel".  The parser tunes to a particular channel
# so that whitespace etc... can go to the parser on a "hidden" channel.
DEFAULT_CHANNEL =

All tokens go to the parser (unless skip() is called in that rule) on a particular “channel”. The parser tunes to a particular channel so that whitespace etc… can go to the parser on a “hidden” channel.

0
HIDDEN_CHANNEL =

Anything on different channel than DEFAULT_CHANNEL is not parsed by parser.

1

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Token

we obtained this token.



26
27
28
29
30
31
32
33
34
35
 
# File 'lib/antlr4/Token.rb', line 26

def initialize()
    self.source = nil
    self.type = nil       # token type of the token
    self.channel = nil    # The parser ignores everything not on DEFAULT_CHANNEL
    self.start = -1       # optional; return -1 if not implemented.
    self.stop = -1        # optional; return -1 if not implemented.
    self.tokenIndex = nil # from 0..n-1 of the token object in the input stream
    self.line = nil       # line=1..n of the 1st character
    self.column = nil     # beginning of the line at which it occurs, 0..n-1
end

Instance Attribute Details

#channel ⇒ Object

Returns the value of attribute channel.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def channel
  @channel
end

#column ⇒ Object

Returns the value of attribute column.



22
23
24
 
# File 'lib/antlr4/Token.rb', line 22

def column
  @column
end

#line ⇒ Object

Returns the value of attribute line.



22
23
24
 
# File 'lib/antlr4/Token.rb', line 22

def line
  @line
end

#source ⇒ Object

Returns the value of attribute source.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def source
  @source
end

#start ⇒ Object

Returns the value of attribute start.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def start
  @start
end

#stop ⇒ Object

Returns the value of attribute stop.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def stop
  @stop
end

#text ⇒ Object

Returns the value of attribute text.



22
23
24
 
# File 'lib/antlr4/Token.rb', line 22

def text
  @text
end

#tokenIndex ⇒ Object

Returns the value of attribute tokenIndex.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def tokenIndex
  @tokenIndex
end

#type ⇒ Object

Returns the value of attribute type.



21
22
23
 
# File 'lib/antlr4/Token.rb', line 21

def type
  @type
end

Instance Method Details

#getInputStream ⇒ Object 



48
49
50
 
# File 'lib/antlr4/Token.rb', line 48

def getInputStream()
    return self.source[1]
end

#getTokenSource ⇒ Object

Explicitly set the text for this token. If text is not null, then #getText will return this value rather than extracting the text from the input.

should be obtained from the input along with the start and stop indexes of the token.



44
45
46
 
# File 'lib/antlr4/Token.rb', line 44

def getTokenSource()
    return self.source[0]
end