Class: Ferret::Analysis::TokenStream
- Inherits:
-
Object
- Object
- Ferret::Analysis::TokenStream
- Defined in:
- ext/r_analysis.c
Overview
Summary
A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text.
This is an abstract class. Concrete subclasses are:
- Tokenizer
-
a TokenStream whose input is a string
- TokenFilter
-
a TokenStream whose input is another TokenStream