Class: Ferret::Analysis::LetterTokenizer

Inherits:

RegExpTokenizer

• Object
• TokenStream
• Tokenizer
• RegExpTokenizer
• Ferret::Analysis::LetterTokenizer

Defined in:

lib/ferret/analysis/tokenizers.rb

Overview

A LetterTokenizer is a tokenizer that divides text at non-letters. That’s to say, it defines tokens as maximal strings of adjacent letters, as defined by the regular expression _/[]+/_.

Direct Known Subclasses

LowerCaseTokenizer

Method Summary

Methods inherited from RegExpTokenizer

#close, #initialize, #next

Methods inherited from Tokenizer

#close

Methods inherited from TokenStream

#close, #each, #next

Constructor Details

This class inherits a constructor from Ferret::Analysis::RegExpTokenizer