Module: ANTLR3::Token

Includes:
Constants, Comparable
Included in:
CommonToken
Defined in:
lib/antlr3/token.rb

Overview

At a minimum, tokens are data structures that bind together a chunk of text and a corresponding type symbol, which categorizes/characterizes the content of the text. Tokens also usually carry information about their location in the input, such as absolute character index, line number, and position within the line (or column).

Furthermore, ANTLR tokens are assigned a “channel” number, an extra degree of categorization that groups things on a larger scale. Parsers will usually ignore tokens that have channel value 99 (the HIDDEN_CHANNEL), so you can keep things like comment and white space huddled together with neighboring tokens, effectively ignoring them without discarding them.

ANTLR tokens also keep a reference to the source stream from which they originated. Token streams will also provide an index value for the token, which indicates the position of the token relative to other tokens in the stream, starting at zero. For example, the 22nd token pulled from a lexer by CommonTokenStream will have index value 21.

Token as an Interface

This library provides a token implementation (see CommonToken). Additionally, you may write your own token class as long as you provide methods that give access to the attributes expected by a token. Even though most of the ANTLR library tries to use duck-typing techniques instead of pure object-oriented type checking, it’s a good idea to include this ANTLR3::Token into your customized token class.

Constant Summary

Constants included from Constants

Constants::BUILT_IN_TOKEN_NAMES, Constants::DEFAULT, Constants::DOWN, Constants::EOF, Constants::EOF_TOKEN, Constants::EOR_TOKEN_TYPE, Constants::HIDDEN, Constants::INVALID, Constants::INVALID_NODE, Constants::INVALID_TOKEN, Constants::MEMO_RULE_FAILED, Constants::MEMO_RULE_UNKNOWN, Constants::MIN_TOKEN_TYPE, Constants::SKIP_TOKEN, Constants::UP

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#channel ⇒ Object

the integer value of the channel to which the token is assigned



85
86
87
 
# File 'lib/antlr3/token.rb', line 85

def channel
  @channel
end

#column ⇒ Object

the text’s starting position in the line within the source (indexed starting at 0)



82
83
84
 
# File 'lib/antlr3/token.rb', line 82

def column
  @column
end

#index ⇒ Object Also known as: token_index

the index of the token with respect to other the other tokens produced during lexing



88
89
90
 
# File 'lib/antlr3/token.rb', line 88

def index
  @index
end

#input ⇒ Object Also known as: input_stream

a reference to the input stream from which the token was extracted



91
92
93
 
# File 'lib/antlr3/token.rb', line 91

def input
  @input
end

#line ⇒ Object

the text’s starting line number within the source (indexed starting at 1)



79
80
81
 
# File 'lib/antlr3/token.rb', line 79

def line
  @line
end

#start ⇒ Object

the absolute character index in the input at which the text starts



94
95
96
 
# File 'lib/antlr3/token.rb', line 94

def start
  @start
end

#stop ⇒ Object

the absolute character index in the input at which the text ends



97
98
99
 
# File 'lib/antlr3/token.rb', line 97

def stop
  @stop
end

#text ⇒ Object

the token’s associated chunk of text



73
74
75
 
# File 'lib/antlr3/token.rb', line 73

def text
  @text
end

#type ⇒ Object

the integer value associated with the token’s type



76
77
78
 
# File 'lib/antlr3/token.rb', line 76

def type
  @type
end

Instance Method Details

#<=>(tk2) ⇒ Object

Tokens are comparable by their stream index values



130
131
132
 
# File 'lib/antlr3/token.rb', line 130

def <=> tk2
  index <=> tk2.index
end

#=~(obj) ⇒ Object

The match operator has been implemented to match against several different attributes of a token for convenience in quick scripts

Examples:

Match against an integer token type constant

token =~ VARIABLE_NAME   => true/false

Match against a token type name as a Symbol

token =~ :FLOAT          => true/false

Match the token text against a Regular Expression

token =~ /^@[a-z_]\w*$/i

Compare the token’s text to a string

token =~ "class"


117
118
119
120
121
122
123
124
125
 
# File 'lib/antlr3/token.rb', line 117

def =~ obj
  case obj
  when Integer then type == obj
  when Symbol then name == obj.to_s
  when Regexp then obj =~ text
  when String then text == obj
  else super
  end
end

#concrete? ⇒ Boolean

Returns:

  • (Boolean) 


146
147
148
 
# File 'lib/antlr3/token.rb', line 146

def concrete?
  input && start && stop ? true : false
end

#hidden? ⇒ Boolean

Returns:

  • (Boolean) 


162
163
164
 
# File 'lib/antlr3/token.rb', line 162

def hidden?
  channel == HIDDEN_CHANNEL
end

#hide! ⇒ Object

Sets the token’s channel value to HIDDEN_CHANNEL



173
174
175
 
# File 'lib/antlr3/token.rb', line 173

def hide!
  self.channel = HIDDEN_CHANNEL
end

#imaginary? ⇒ Boolean

Returns:

  • (Boolean) 


150
151
152
 
# File 'lib/antlr3/token.rb', line 150

def imaginary?
  input && start && stop ? false : true
end

#initialize_copy(orig) ⇒ Object 



134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/antlr3/token.rb', line 134

def initialize_copy( orig )
  self.index   = -1
  self.type    = orig.type
  self.channel = orig.channel
  self.text    = orig.text.clone if orig.text
  self.start   = orig.start
  self.stop    = orig.stop
  self.line    = orig.line
  self.column  = orig.column
  self.input   = orig.input
end

#inspect ⇒ Object 



177
178
179
180
181
182
183
184
185
186
187
188
 
# File 'lib/antlr3/token.rb', line 177

def inspect
  text_inspect    = text  ? "[#{ text.inspect }] " : ' '
  text_position   = line > 0  ? "@ line #{ line } col #{ column } " : ''
  stream_position = start ? "(#{ range.inspect })" : ''
  
  front =  index >= 0 ? "#{ index } " : ''
  rep = front << name << text_inspect <<
              text_position << stream_position
  rep.strip!
  channel == DEFAULT_CHANNEL or rep << " (#{ channel.to_s })"
  return( rep )
end

#name ⇒ Object 



154
155
156
 
# File 'lib/antlr3/token.rb', line 154

def name
  token_name( type )
end

#pretty_print(printer) ⇒ Object 



190
191
192
 
# File 'lib/antlr3/token.rb', line 190

def pretty_print( printer )
  printer.text( inspect )
end

#range ⇒ Object 



194
195
196
 
# File 'lib/antlr3/token.rb', line 194

def range
  start..stop rescue nil
end

#source_name ⇒ Object 



158
159
160
 
# File 'lib/antlr3/token.rb', line 158

def source_name
  i = input and i.source_name
end

#source_text ⇒ Object 



166
167
168
 
# File 'lib/antlr3/token.rb', line 166

def source_text
  concrete? ? input.substring( start, stop ) : text
end

#to_i ⇒ Object 



198
199
200
 
# File 'lib/antlr3/token.rb', line 198

def to_i
  index.to_i
end

#to_s ⇒ Object 



202
203
204
 
# File 'lib/antlr3/token.rb', line 202

def to_s
  text.to_s
end