Class: Ferret::Analysis::AsciiStandardTokenizer

Inherits:
Object
  • Object
show all
Defined in:
ext/r_analysis.c

Overview

Summary

The standard tokenizer is an advanced tokenizer which tokenizes most words correctly as well as tokenizing things like email addresses, web addresses, phone numbers, etc.

Example

"Dave's résumé, at http://www.davebalmain.com/ 1234"
  => ["Dave's", "r", "sum", "at", "http://www.davebalmain.com", "1234"]