Class: LanguageParser::CTokenizer

Inherits:
Tokenizer
  • Object
show all
Defined in:
lib/cgialib/lp/CTokenizer.rb

Overview

class : CTokenizer

The main entry class that parses C text into a set of tokens

Instance Attribute Summary

Attributes inherited from Tokenizer

#tokens

Instance Method Summary collapse

Methods inherited from Tokenizer

#initialize

Constructor Details

This class inherits a constructor from LanguageParser::Tokenizer

Instance Method Details

#parse(text) ⇒ Object

parse( text )

text - The C text

Parses the C text string into tokens



419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
 
# File 'lib/cgialib/lp/CTokenizer.rb', line 419

def parse( text )
  
  # Set the current state to the normal state
  
  @state = CT_NormalState.new( method( :newstate ), method( :addtoken ) )
  
  # Iterate through the text
  
  index = 0
  
  while index < text.length
  
    # Dispatch the character to the current state
  
    if ( @state.next( text[ index ].chr() ) )
  
      index += 1
  
    end
  
  end
  
end