Class: CorpusProcessor::Token

Inherits:

Object

• Object
• CorpusProcessor::Token

Defined in:

lib/corpus-processor/token.rb

Overview

The internal representation of a token.

Tokens are extracted from original corpus and are defined by single words or punctuation.

They also contain a category, which is originated form the tagging in the corpus.

Instance Attribute Summary

• #category ⇒ Symbol readonly

  The type of the Token.

• #word ⇒ String readonly

  The word from text.

Instance Method Summary

• #==(other) ⇒ Object

  Determine equality of two Tokens.

• #initialize(word = '', category = nil) ⇒ Token constructor

  A new instance of Token.

Constructor Details

#initialize(word = '', category = nil) ⇒ Token

# File 'lib/corpus-processor/token.rb', line 20

def initialize word = '', category = nil
  self.word     = word
  self.category = category
end

Instance Attribute Details

#category ⇒ Symbol

# File 'lib/corpus-processor/token.rb', line 15

def category
  @category
end

#word ⇒ String

# File 'lib/corpus-processor/token.rb', line 11

def word
  @word
end

Instance Method Details

#==(other) ⇒ Object

Determine equality of two CorpusProcessor::Tokens.

# File 'lib/corpus-processor/token.rb', line 28

def ==(other)
  word == other.word && category == other.category
end