Class: LanguageParser::CT_WhitespaceTokenizer

Inherits:
CT_State
  • Object
show all
Defined in:
lib/cgialib/lp/CTokenizer.rb

Overview

CT_WhitespaceTokenizer

Handles whitespace in the character stream

Instance Method Summary collapse

Constructor Details

#initialize(newstate, addtoken) ⇒ CT_WhitespaceTokenizer

initialize( newstate, addtoken )

newstate - A method to be called to change state addtoken - The method to be called to add a token



239
240
241
242
243
244
245
246
247
 
# File 'lib/cgialib/lp/CTokenizer.rb', line 239

def initialize( newstate, addtoken )
  
  super( newstate, addtoken )
  
  # Initialize the text buffer to blank
  
  @text = ""
  
end

Instance Method Details

#next(ch) ⇒ Object

next( ch )

ch - The character

Handles the character in the parsing stream



255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
 
# File 'lib/cgialib/lp/CTokenizer.rb', line 255

def next( ch )
  
  if ( ch =~ /\s/ )
  
    # If the character is whitespace add it to
    # the buffer
  
    @text += ch
    return true
  
  else
  
    # Otherwise return to the normal state and 
    # add the token
  
    @addtoken.call( WhitespaceToken.new( @text ) )
    @newstate.call( CT_NormalState )
  
    # Return false because we want the tokenizer
    # to re-run on the current character
  
    return false
  
  end
  
end