Class: LanguageParser::Tokenizer

Inherits:

Object

Object
LanguageParser::Tokenizer

show all

Defined in:: lib/cgialib/lp/Tokenizer.rb

Overview

class : Tokenizer

The base class for all Tokenizers. The Tokenizers are meant to be specialized to parse the text of various languages into token streams that consist of CodeToken, WhitespaceToken, and CommentToken objects

Direct Known Subclasses

CTokenizer, SQLTokenizer

Instance Attribute Summary collapse

#tokens ⇒ Object readonly

The ordered array of tokens.

Instance Method Summary collapse

#initialize ⇒ Tokenizer constructor

initialize().
#parse(text) ⇒ Object

parse( text ).

Constructor Details

#initialize ⇒ `Tokenizer`

initialize()

Constructs the Tokenizer base class



66
67
68

# File 'lib/cgialib/lp/Tokenizer.rb', line 66

def initialize( )
  @tokens = TokenStream.new()
end

Instance Attribute Details

#tokens ⇒ `Object` (readonly)

The ordered array of tokens



70
71
72

# File 'lib/cgialib/lp/Tokenizer.rb', line 70

def tokens
  @tokens
end

Instance Method Details

#parse(text) ⇒ `Object`

parse( text )

text - The string of code to be turned into tokens

This should be overridden by all derived classes. It is meant to parse the text into tokens. Which should be stored in @tokens.



80
81

# File 'lib/cgialib/lp/Tokenizer.rb', line 80

def parse( text )
end