Class: Pinpoint::Format::Tokenizer

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/pinpoint/format/tokenizer.rb

Instance Method Summary collapse

Constructor Details

#initialize(string_with_tokens) ⇒ Tokenizer

Public: Initializes a Tokenizer with a given String



17
18
19
 
# File 'lib/pinpoint/format/tokenizer.rb', line 17

def initialize(string_with_tokens)
  self.tokenable = string_with_tokens
end

Instance Method Details

#each ⇒ Object

Public: From the beginning of the tokenable String, it reads each token and passes it to the block.

Yields each successive Token to the given block

Raises subclasses of Pinpoint::Format::ParseError upon various syntax

errors

Returns nothing



32
33
34
35
36
37
38
 
# File 'lib/pinpoint/format/tokenizer.rb', line 32

def each
  tokenable.reset

  while current_token = next_token
    yield current_token
  end
end

#to_token_list ⇒ Object

Public: Wraps the Array of Tokens in a TokenList

Returns a TokenList



45
46
47
 
# File 'lib/pinpoint/format/tokenizer.rb', line 45

def to_token_list
  TokenList.new(to_a)
end