Module: ANTLR3::TokenSource

Extended by:
ClassMacros
Includes:
Constants, Enumerable
Included in:
Lexer
Defined in:
lib/antlr3/token.rb

Overview

TokenSource is a simple mixin module that demands an implementation of the method #next_token. In return, it defines methods #next and #each, which provide basic iterator methods for token generators. Furthermore, it includes Enumerable to provide the standard Ruby iteration methods to token generators, like lexers.

Constant Summary

Constants included from Constants

Constants::BUILT_IN_TOKEN_NAMES, Constants::DEFAULT, Constants::DOWN, Constants::EOF, Constants::EOF_TOKEN, Constants::EOR_TOKEN_TYPE, Constants::HIDDEN, Constants::INVALID, Constants::INVALID_NODE, Constants::INVALID_TOKEN, Constants::MEMO_RULE_FAILED, Constants::MEMO_RULE_UNKNOWN, Constants::MIN_TOKEN_TYPE, Constants::SKIP_TOKEN, Constants::UP

Instance Method Summary collapse

Instance Method Details

#each ⇒ Object 



319
320
321
322
323
324
325
 
# File 'lib/antlr3/token.rb', line 319

def each
  block_given? or return enum_for( :each )
  while token = next_token and token.type != EOF
    yield( token )
  end
  return self
end

#next ⇒ Object

Raises:

  • (StopIteration) 


313
314
315
316
317
 
# File 'lib/antlr3/token.rb', line 313

def next
  token = next_token()
  raise StopIteration if token.nil? || token.type == EOF
  return token
end

#to_stream(options = {}) ⇒ Object 



327
328
329
330
331
332
333
 
# File 'lib/antlr3/token.rb', line 327

def to_stream( options = {} )
  if block_given?
    CommonTokenStream.new( self, options ) { | t, stream | yield( t, stream ) }
  else
    CommonTokenStream.new( self, options )
  end
end