Class: Droonga::Searcher::QuerySearcher::MeCabTokenizer

Inherits:

Object

• Object
• Droonga::Searcher::QuerySearcher::MeCabTokenizer

Defined in:

lib/droonga/searcher/mecab_filter.rb

Instance Method Summary

• #initialize ⇒ MeCabTokenizer constructor

  A new instance of MeCabTokenizer.

• #tokenize(text) ⇒ Object

Constructor Details

#initialize ⇒ MeCabTokenizer

Returns a new instance of MeCabTokenizer.

# File 'lib/droonga/searcher/mecab_filter.rb', line 24

def initialize
  @mecab = MeCab::Tagger.new("-Owakati")
end

Instance Method Details

#tokenize(text) ⇒ Object

# File 'lib/droonga/searcher/mecab_filter.rb', line 28

def tokenize(text)
  tokens = @mecab.parse(text).force_encoding("utf-8").split(/\s+/)
  tokens.reject do |token|
    token.empty?
  end
end