Class: Landescape::Tokenizer
- Inherits:
-
Object
- Object
- Landescape::Tokenizer
- Defined in:
- lib/landescape/tokenizer.rb
Overview
Tokenize: “e[1;2fHi” to [‘e[1;2f’, ‘H’, ‘i’]]
Instance Attribute Summary collapse
-
#result ⇒ Object
readonly
Returns the value of attribute result.
-
#source ⇒ Object
readonly
Returns the value of attribute source.
Class Method Summary collapse
Instance Method Summary collapse
-
#initialize(source) ⇒ Tokenizer
constructor
A new instance of Tokenizer.
- #start ⇒ Object
- #stop ⇒ Object
Constructor Details
#initialize(source) ⇒ Tokenizer
Returns a new instance of Tokenizer.
12 13 14 15 16 17 18 19 20 21 22 23 24 |
# File 'lib/landescape/tokenizer.rb', line 12 def initialize(source) @result = Queue.new @source = if source.respond_to?(:getc) source else StringIO.new(source.to_s) end at_exit do @source.close unless @source.closed? end end |
Instance Attribute Details
#result ⇒ Object (readonly)
Returns the value of attribute result.
10 11 12 |
# File 'lib/landescape/tokenizer.rb', line 10 def result @result end |
#source ⇒ Object (readonly)
Returns the value of attribute source.
10 11 12 |
# File 'lib/landescape/tokenizer.rb', line 10 def source @source end |
Class Method Details
.start(source) ⇒ Object
5 6 7 |
# File 'lib/landescape/tokenizer.rb', line 5 def start(source) new(source).tap(&:start) end |
Instance Method Details
#start ⇒ Object
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
# File 'lib/landescape/tokenizer.rb', line 26 def start @tokenize_thread = Thread.fork { while char = read_char tokens = case char when nil then Thread.exit # StringIO#getc # => nil (EOF) when /\e/ then escape char else char end Array(tokens).each do |token| result.push token end end } end |
#stop ⇒ Object
43 44 45 |
# File 'lib/landescape/tokenizer.rb', line 43 def stop @tokenize_thread.exit if @tokenize_thread.alive? end |