Class: Moonstone::Tokenizer

Inherits:

Lucene::Analysis::Tokenizer

Object
Lucene::Analysis::Tokenizer
Moonstone::Tokenizer

Includes:: Lucene::Analysis

Defined in:: lib/moonstone/tokenizer.rb

Instance Method Summary collapse

#initialize(reader) ⇒ Tokenizer constructor

A new instance of Tokenizer.
#next(token = nil) ⇒ Object

No, this is not terribly useful.

Constructor Details

#initialize(reader) ⇒ `Tokenizer`

Returns a new instance of Tokenizer.



6
7
8

# File 'lib/moonstone/tokenizer.rb', line 6

def initialize(reader)
  @reader = java.io.BufferedReader.new(reader)
end

Instance Method Details

#next(token = nil) ⇒ `Object`

No, this is not terribly useful. Subclass me already.

# File 'lib/moonstone/tokenizer.rb', line 11

def next(token=nil)
  token = (token ? token.clear :  Token.new)
  token.set_term_text @reader.read_line
  token.set_start_offset 1
  token.set_end_offset 1
end