Module: ANTLR3::TokenStream

Extended by:
ClassMacros
Includes:
Stream
Included in:
CommonTokenStream
Defined in:
lib/antlr3/streams.rb

Overview

TokenStream further extends the abstract-ish base mixin Stream to add methods specific to navigating token sequences. Thus, it serves as an imitation of the Java interface for token-based streams, which are used by many different components in ANTLR, including parsers and tree parsers.

Token Streams

Token streams wrap a sequence of token objects produced by some token source, usually a lexer. They provide the operations required by higher-level recognizers, such as parsers and tree parsers for navigating through the sequence of tokens. Unlike simple character-based streams, such as StringStream, token-based streams have an additional level of complexity because they must manage the task of “tuning” to a specific token channel.

One of the main advantages of ANTLR-based recognition is the token channel feature, which allows you to hold on to all tokens of interest while only presenting a specific set of interesting tokens to a parser. For example, if you need to hide whitespace and comments from a parser, but hang on to them for some other purpose, you have the lexer assign the comments and whitespace to channel value HIDDEN as it creates the tokens.

When you create a token stream, you can tune it to some specific channel value. Then, all peek, look, and consume operations only yield tokens that have the same value for channel. The stream skips over any non-matching tokens in between.

The TokenStream Interface

In addition to the abstract methods and attribute methods provided by the base Stream module, TokenStream adds a number of additional method implementation requirements and attributes.

Constant Summary

Constants included from Constants

Constants::BUILT_IN_TOKEN_NAMES, Constants::DEFAULT, Constants::DOWN, Constants::EOF, Constants::EOF_TOKEN, Constants::EOR_TOKEN_TYPE, Constants::HIDDEN, Constants::INVALID, Constants::INVALID_NODE, Constants::INVALID_TOKEN, Constants::MEMO_RULE_FAILED, Constants::MEMO_RULE_UNKNOWN, Constants::MIN_TOKEN_TYPE, Constants::SKIP_TOKEN, Constants::UP

Instance Attribute Summary collapse

Attributes included from Stream

#size, #source_name

Instance Method Summary collapse

Methods included from Stream

#consume, #index, #look, #mark, #peek, #release, #rewind, #seek

Instance Attribute Details

#channel ⇒ Object

the integer channel value to which the stream is “tuned”



326
327
328
 
# File 'lib/antlr3/streams.rb', line 326

def channel
  @channel
end

#last_marker ⇒ Object (readonly)

expected to return the value of the last marker produced by a call to stream.mark



318
319
320
 
# File 'lib/antlr3/streams.rb', line 318

def last_marker
  @last_marker
end

#position ⇒ Object (readonly)

expected to return the integer index of the stream cursor



322
323
324
 
# File 'lib/antlr3/streams.rb', line 322

def position
  @position
end

#token_source ⇒ Object (readonly)

expected to return the token source object (such as a lexer) from which all tokens in the stream were retreived



313
314
315
 
# File 'lib/antlr3/streams.rb', line 313

def token_source
  @token_source
end

Instance Method Details

#at ⇒ Object

:method: at( i ) return the stream symbol at index i



337
 
# File 'lib/antlr3/streams.rb', line 337

abstract :at

#to_s ⇒ Object

:method: to_s(start=0,stop=tokens.length-1) should take the tokens between start and stop in the sequence, extract their text and return the concatenation of all the text chunks



332
 
# File 'lib/antlr3/streams.rb', line 332

abstract :to_s