Class: Electr::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/electr/parser/tokenizer.rb

Overview

The tokenizer is the first step of the parser. It explodes a code in its constituants (token), ie # ‘sqrt(3+2)` becomes the following list of tokens:

  • ‘sqrt`

  • ‘(`

  • ‘3`

  • ‘+`

  • ‘2`

  • ‘)`

Instance Method Summary collapse

Constructor Details

#initialize(string) ⇒ Tokenizer

Create a new Tokenizer for a specific code.

string - The String code to tokenize.



18
19
20
21
22
23
24
 
# File 'lib/electr/parser/tokenizer.rb', line 18

def initialize(string)
  @index = 0
  @token = ''
  @look_ahead = ''
  @codeline = string
  forward_look_ahead
end

Instance Method Details

#has_more_token? ⇒ Boolean

Public: Check if the tokenizer is able to give another token.

Returns Boolean true if there is another waiting token.

Returns:

  • (Boolean) 


29
30
31
 
# File 'lib/electr/parser/tokenizer.rb', line 29

def has_more_token?
  @index <= @codeline.size
end

#next_token ⇒ Object

Public: Get the next token.

Returns a String token.



36
37
38
39
40
41
 
# File 'lib/electr/parser/tokenizer.rb', line 36

def next_token
  ret = produce_next_token
  skip_white
  @token = ''
  ret
end