Class: Shoes::Highlighter::Syntax::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/shoes/highlighter/common.rb

Overview

The base class of all tokenizers. It sets up the scanner and manages the looping until all tokens have been extracted. It also provides convenience methods to make sure adjacent tokens of identical groups are returned as a single token.

Direct Known Subclasses

Ruby

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#chunk ⇒ Object (readonly)

The current chunk of text being accumulated



35
36
37
 
# File 'lib/shoes/highlighter/common.rb', line 35

def chunk
  @chunk
end

#group ⇒ Object (readonly)

The current group being processed by the tokenizer



32
33
34
 
# File 'lib/shoes/highlighter/common.rb', line 32

def group
  @group
end

Instance Method Details

#finish ⇒ Object

Finish tokenizing. This flushes the buffer, yielding any remaining text to the client.



55
56
57
58
 
# File 'lib/shoes/highlighter/common.rb', line 55

def finish
  start_group nil
  teardown
end

#option(opt) ⇒ Object

Get the value of the specified option.



87
88
89
 
# File 'lib/shoes/highlighter/common.rb', line 87

def option(opt)
  @options ? @options[opt] : nil
end

#set(opts = {}) ⇒ Object

Specify a set of tokenizer-specific options. Each tokenizer may (or may not) publish any options, but if a tokenizer does those options may be used to specify optional behavior.



82
83
84
 
# File 'lib/shoes/highlighter/common.rb', line 82

def set(opts = {})
  (@options ||= {}).update opts
end

#setup ⇒ Object

Subclasses may override this method to provide implementation-specific setup logic.



50
51
 
# File 'lib/shoes/highlighter/common.rb', line 50

def setup
end

#start(text, &block) ⇒ Object

Start tokenizing. This sets up the state in preparation for tokenization, such as creating a new scanner for the text and saving the callback block. The block will be invoked for each token extracted.



40
41
42
43
44
45
46
 
# File 'lib/shoes/highlighter/common.rb', line 40

def start(text, &block)
  @chunk = ""
  @group = :normal
  @callback = block
  @text = StringScanner.new(text)
  setup
end

#step ⇒ Object

Subclasses must implement this method, which is called for each iteration of the tokenization process. This method may extract multiple tokens.



67
68
69
 
# File 'lib/shoes/highlighter/common.rb', line 67

def step
  fail NotImplementedError, "subclasses must implement #step"
end

#teardown ⇒ Object

Subclasses may override this method to provide implementation-specific teardown logic.



62
63
 
# File 'lib/shoes/highlighter/common.rb', line 62

def teardown
end

#tokenize(text, &block) ⇒ Object

Begins tokenizing the given text, calling #step until the text has been exhausted.



73
74
75
76
77
 
# File 'lib/shoes/highlighter/common.rb', line 73

def tokenize(text, &block)
  start text, &block
  step until @text.eos?
  finish
end