Class: TextRank::TokenFilter::Stopwords

Inherits:
Object
  • Object
show all
Defined in:
lib/text_rank/token_filter/stopwords.rb

Overview

Token filter to remove common stop word tokens

Example

Stopwords.new.filter!(%w[
  but for what purpose was the earth formed to drive us mad
])
=> ["purpose", "earth", "formed", "drive", "mad"]

Constant Summary collapse

STOP_WORDS =

Default English stop-word list.

Set.new(YAML.load_file(File.expand_path('stopwords.yml', __dir__)))

Instance Method Summary collapse

Instance Method Details

#filter!(tokens) ⇒ Array<String>

Perform the filter

Parameters:

  • tokens (Array<String>)

Returns:

  • (Array<String>) 


21
22
23
24
25
 
# File 'lib/text_rank/token_filter/stopwords.rb', line 21

def filter!(tokens)
  tokens.delete_if do |token|
    STOP_WORDS.include?(token.downcase)
  end
end