Class: SClust::Util::TokenizerFilter

Inherits:
Filter
  • Object
show all
Defined in:
lib/sclust/util/filters.rb

Direct Known Subclasses

DocumentTokenizer

Instance Method Summary collapse

Methods inherited from Filter

#after, #apply, #before, #initialize

Constructor Details

This class inherits a constructor from SClust::Util::Filter

Instance Method Details

#filter(document) ⇒ Object 



163
164
165
 
# File 'lib/sclust/util/filters.rb', line 163

def filter(document)
    document.split(/[\s,\.\t!\?\(\)\{\}\[\]\t\r\n";':]+/m)
end