Class: Bishop::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/bishop.rb

Instance Method Summary collapse

Instance Method Details

#tokenize(item) ⇒ Object 



44
45
46
47
48
 
# File 'lib/bishop.rb', line 44

def tokenize( item )
  item.split( /\s+/ ).map do |i|
    token = i.split( /\-/ ).map { |token| token.downcase.gsub( /\W/, "" ) }.join( "-" )
  end.reject { |t| t == "" || t == "-" }
end