Method: ANTLR3::CommonTokenStream#initialize

Defined in:
lib/antlr3/streams.rb

#initialize(token_source, options = {}) ⇒ CommonTokenStream

constructs a new token stream using the token_source provided. token_source is usually a lexer, but can be any object that implements next_token and includes ANTLR3::TokenSource.

If a block is provided, each token harvested will be yielded and if the block returns a nil or false value, the token will not be added to the stream – it will be discarded.

Options

:channel

The channel value the stream should be tuned to initially

:source_name

The source name (file name) attribute of the stream

Example

# create a new token stream that is tuned to channel :comment, and
# discard all WHITE_SPACE tokens
ANTLR3::CommonTokenStream.new(lexer, :channel => :comment) do |token|
  token.name != 'WHITE_SPACE'
end



780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
 
# File 'lib/antlr3/streams.rb', line 780

def initialize( token_source, options = {} )
  case token_source
  when CommonTokenStream
    # this is useful in cases where you want to convert a CommonTokenStream
    # to a RewriteTokenStream or other variation of the standard token stream
    stream = token_source
    @token_source = stream.token_source
    @channel = options.fetch( :channel ) { stream.channel or DEFAULT_CHANNEL }
    @source_name = options.fetch( :source_name ) { stream.source_name }
    tokens = stream.tokens.map { | t | t.dup }
  else
    @token_source = token_source
    @channel = options.fetch( :channel, DEFAULT_CHANNEL )
    @source_name = options.fetch( :source_name ) {  @token_source.source_name rescue nil }
    tokens = @token_source.to_a
  end
  @last_marker = nil
  @tokens = block_given? ? tokens.select { | t | yield( t, self ) } : tokens
  @tokens.each_with_index { |t, i| t.index = i }
  @position = 
    if first_token = @tokens.find { |t| t.channel == @channel }
      @tokens.index( first_token )
    else @tokens.length
    end
end