Class: ANTLR3::CommonToken

Inherits:
Struct
  • Object
show all
Includes:
Token
Defined in:
lib/antlr3/token.rb,
lib/antlr3/token.rb

Overview

The base class for the standard implementation of Token. It is implemented as a simple Struct as tokens are basically simple data structures binding together a bunch of different information and Structs are slightly faster than a standard Object with accessor methods implementation.

By default, ANTLR generated ruby code will provide a customized subclass of CommonToken to track token-type names efficiently for debugging, inspection, and general utility. Thus code generated for a standard combo lexer-parser grammar named XYZ will have a base module named XYZ and a customized CommonToken subclass named XYZ::Token.

Here is the token structure attribute list in order:

  • type

  • channel

  • text

  • input

  • start

  • stop

  • index

  • line

  • column

Constant Summary

DEFAULT_VALUES = 
{ 
  :channel => DEFAULT_CHANNEL,
  :index   => -1,
  :line    =>  0,
  :column  => -1
}.freeze

Constants included from Constants

ANTLR3::Constants::BUILT_IN_TOKEN_NAMES, ANTLR3::Constants::DEFAULT, ANTLR3::Constants::DOWN, ANTLR3::Constants::EOF, ANTLR3::Constants::EOF_TOKEN, ANTLR3::Constants::EOR_TOKEN_TYPE, ANTLR3::Constants::HIDDEN, ANTLR3::Constants::INVALID_TOKEN, ANTLR3::Constants::INVALID_TOKEN_TYPE, ANTLR3::Constants::MEMO_RULE_FAILED, ANTLR3::Constants::MEMO_RULE_UNKNOWN, ANTLR3::Constants::MIN_TOKEN_TYPE, ANTLR3::Constants::SKIP_TOKEN, ANTLR3::Constants::UP

Instance Attribute Summary (collapse)

Class Method Summary (collapse)

Instance Method Summary (collapse)

Methods included from Token

#<=>, #=~, #concrete?, #hidden?, #hide!, #imaginary?, #initialize_copy, #inspect, #name, #pretty_print, #range, #source_name, #source_text, #to_i, #to_s

Constructor Details

- (CommonToken) initialize(type = nil, channel = DEFAULT_CHANNEL, text = nil, input = nil, start = nil, stop = nil, index = 1, line = 0, column = 1)

A new instance of CommonToken



270
271
272
273
274
275
276
277
 
# File 'lib/antlr3/token.rb', line 270

def initialize( type = nil, channel = DEFAULT_CHANNEL, text = nil,
               input = nil, start = nil, stop = nil, index = -1,
               line = 0, column = -1 )
  super
  block_given? and yield( self )
  self.text.nil? && self.start && self.stop and
    self.text = self.input.substring( self.start, self.stop )
end

Instance Attribute Details

- (Object) channel

Returns the value of attribute channel

Returns:

  • (Object) —

    the current value of channel



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def channel
  @channel
end

- (Object) column

Returns the value of attribute column

Returns:

  • (Object) —

    the current value of column



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def column
  @column
end

- (Object) index Also known as: token_index

Returns the value of attribute index

Returns:

  • (Object) —

    the current value of index



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def index
  @index
end

- (Object) input Also known as: input_stream

Returns the value of attribute input

Returns:

  • (Object) —

    the current value of input



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def input
  @input
end

- (Object) line

Returns the value of attribute line

Returns:

  • (Object) —

    the current value of line



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def line
  @line
end

- (Object) start

Returns the value of attribute start

Returns:

  • (Object) —

    the current value of start



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def start
  @start
end

- (Object) stop

Returns the value of attribute stop

Returns:

  • (Object) —

    the current value of stop



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def stop
  @stop
end

- (Object) text

Returns the value of attribute text

Returns:

  • (Object) —

    the current value of text



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def text
  @text
end

- (Object) type

Returns the value of attribute type

Returns:

  • (Object) —

    the current value of type



213
214
215
 
# File 'lib/antlr3/token.rb', line 213

def type
  @type
end

Class Method Details

+ (Object) create(fields = {}) 



256
257
258
259
260
 
# File 'lib/antlr3/token.rb', line 256

def self.create( fields = {} )
  fields = DEFAULT_VALUES.merge( fields )
  args = members.map { |name| fields[ name.to_sym ] }
  new( *args )
end

+ (Object) from_token(token)

allows you to make a copy of a token with a different class



263
264
265
266
267
268
 
# File 'lib/antlr3/token.rb', line 263

def self.from_token( token )
  new( 
    token.type,  token.channel, token.text ? token.text.clone : nil,
    token.input, token.start,   token.stop, -1, token.line, token.column
  )
end

+ (Object) token_name(type) 



252
253
254
 
# File 'lib/antlr3/token.rb', line 252

def self.token_name( type )
  BUILT_IN_TOKEN_NAMES[ type ]
end