Class: Bbcode::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/bbcode/tokenizer.rb

Overview

Scans a string and converts it to a stream of bbcode tokens.

Constant Summary collapse

BBCODE_TAG_PATTERN = 
/\[(\/?)([a-z0-9_-]*)(\s*=?(?:(?:\s*(?:(?:[a-z0-9_-]+)|(?<=\=))\s*[:=]\s*)?(?:"[^"\\]*(?:\\[\s\S][^"\\]*)*"|'[^'\\]*(?:\\[\s\S][^'\\]*)*'|[^\]\s,]+|(?<=,)(?=\s*,))\s*,?\s*)*)\]/i
ATTRIBUTE_PATTERN = 
/(?:\s*(?:([a-z0-9_-]+)|^)\s*[:=]\s*)?("[^"\\]*(?:\\[\s\S][^"\\]*)*"|'[^'\\]*(?:\\[\s\S][^'\\]*)*'|[^\]\s,]+|(?<=,)(?=\s*,))\s*,?/i
UNESCAPE_PATTERN = 
/\\(.)/

Instance Method Summary collapse

Instance Method Details

#parse_attributes_string(attributes_string) ⇒ Object 



8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
# File 'lib/bbcode/tokenizer.rb', line 8

def parse_attributes_string( attributes_string )
	attrs = HashWithIndifferentAccess.new
	return attrs if attributes_string.nil?

	next_anonymous_key = -1
	attributes_string.scan ATTRIBUTE_PATTERN do |key, value|
		skip_value = key.blank? && value.blank?
		key = next_anonymous_key+=1 if key.blank?
		unless skip_value
			value = value[1...-1].gsub UNESCAPE_PATTERN, "\\1" if value[0] == value[-1] && ["'", '"'].include?(value[0])
			attrs[key] = value
		end
	end

	return attrs
end

#tokenize(document, handler) ⇒ Object

Parses the document as BBCode-formatted text and calls block with bbcode events.

The handler will have the following methods called:

  • .text text A text-event with an additional parameter containing the actual text.

  • .start_element element_name, element_arguments An element-event with 2 additional parameters: The element name as a symbol and the element attributes as a hash. This events indicate the start of the element.

  • .end_element element_name An element-event indicating the end of an element. Optionally, the element_name is added as a parameter. If no parameter is present, it is assumed to be the last started element.

Note that :start_element and :end_element are not guaranteed to be called evenly or in the “correct” order. You must match correct start- and end tags yourself to create the elements.

Also note that :text events are not guaranteed to match the whole text. In some cases, the text might be separated to multiple :text events, even though there are no nodes in between.



47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/bbcode/tokenizer.rb', line 47

def tokenize(document, handler)
	while !(match = BBCODE_TAG_PATTERN.match(document)).nil?
		offset = match.begin(0)
		elem_source = match[0]

		handler.text document[0...offset] unless offset == 0

		elem_is_closing_tag = match[1]=='/'
		elem_name = (match[2].length > 0 && match[2].to_sym) || nil
		elem_attr_string = (match[3].length > 0 && match[3]) || nil

		if (elem_is_closing_tag && !elem_attr_string) || (!elem_is_closing_tag && elem_name)
			if !elem_is_closing_tag
				handler.start_element elem_name, parse_attributes_string(elem_attr_string), elem_source
			else
				handler.end_element elem_name, elem_source
			end
		else
			handler.text elem_source
		end

		document = document[(offset+elem_source.length)..-1]
	end

	handler.text document unless document.length == 0
end