Class: Ferret::Analysis::TokenStream

Inherits:
Object
  • Object
show all
Defined in:
lib/ferret/analysis/token_stream.rb

Overview

A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text.

This is an abstract class. Concrete subclasses are:

  • Tokenizer, a TokenStream whose input is a Reader; and

  • TokenFilter, a TokenStream whose input is another TokenStream.

Direct Known Subclasses

TokenFilter, Tokenizer

Instance Method Summary collapse

Instance Method Details

#close ⇒ Object

Releases resources associated with this stream.

Raises:

  • (NotImplementedError) 


15
16
17
 
# File 'lib/ferret/analysis/token_stream.rb', line 15

def close
  raise NotImplementedError
end

#each ⇒ Object

Iterates through the tokens in the field



20
21
22
23
24
 
# File 'lib/ferret/analysis/token_stream.rb', line 20

def each # :yields: token
  while (n = self.next())
    yield n
  end
end

#next ⇒ Object

Returns the next token in the stream, or null at EOS.

Raises:

  • (NotImplementedError) 


10
11
12
 
# File 'lib/ferret/analysis/token_stream.rb', line 10

def next
  raise NotImplementedError
end