Class: REHTML::Tokenizer

Inherits:

Object

• Object
• REHTML::Tokenizer

Defined in:

lib/rehtml/tokenizer.rb

Instance Method Summary

• #initialize(html) ⇒ Tokenizer constructor

  Create a new Tokenizer for the given text.

• #next ⇒ Object

  Return the next token in the sequence, or nil if there are no more tokens in the stream.

Constructor Details

#initialize(html) ⇒ Tokenizer

Create a new Tokenizer for the given text.

# File 'lib/rehtml/tokenizer.rb', line 17

def initialize(html)
  @scanner = Scanner.new(html)
  @bpos = 0
end

Instance Method Details

#next ⇒ Object

Return the next token in the sequence, or nil if there are no more tokens in the stream.

# File 'lib/rehtml/tokenizer.rb', line 24

def next
  return nil if @scanner.eos?
  add_parse_info(@scanner.check(/<\S/) ? scan_element : scan_text)
end