Class: TactfulTokenizer::Frag

Inherits:

Object

• Object
• TactfulTokenizer::Frag

Defined in:

lib/tactful_tokenizer.rb

Overview

A fragment is a potential sentence, but is based only on the existence of a period. The text “Here in the U.S. Senate we prefer to devour our friends.” will be split into “Here in the U.S.” and “Senate we prefer to devour our friends.”

Instance Attribute Summary

• #cleaned ⇒ Object

  orig = The original text of the fragment.

• #features ⇒ Object

  orig = The original text of the fragment.

• #next ⇒ Object

  orig = The original text of the fragment.

• #orig ⇒ Object

  orig = The original text of the fragment.

• #pred ⇒ Object

  orig = The original text of the fragment.

Instance Method Summary

• #clean(s) ⇒ Object

  Normalizes numbers and discards ambiguous punctuation.

• #initialize(orig = '') ⇒ Frag constructor

  Create a new fragment.

Constructor Details

#initialize(orig = '') ⇒ Frag

Create a new fragment.

# File 'lib/tactful_tokenizer.rb', line 193

def initialize(orig='')
    @orig = orig
    clean(orig)
    @next, @pred, @features = nil, nil, nil
end

Instance Attribute Details

#cleaned ⇒ Object

orig = The original text of the fragment. next = The next word following the fragment. cleaned = Array of the fragment’s words after cleaning. pred = Probability that the fragment is a sentence. features = Array of the fragment’s features.

# File 'lib/tactful_tokenizer.rb', line 190

def cleaned
  @cleaned
end

#features ⇒ Object

orig = The original text of the fragment. next = The next word following the fragment. cleaned = Array of the fragment’s words after cleaning. pred = Probability that the fragment is a sentence. features = Array of the fragment’s features.

# File 'lib/tactful_tokenizer.rb', line 190

def features
  @features
end

#next ⇒ Object

orig = The original text of the fragment. next = The next word following the fragment. cleaned = Array of the fragment’s words after cleaning. pred = Probability that the fragment is a sentence. features = Array of the fragment’s features.

# File 'lib/tactful_tokenizer.rb', line 190

def next
  @next
end

#orig ⇒ Object

orig = The original text of the fragment. next = The next word following the fragment. cleaned = Array of the fragment’s words after cleaning. pred = Probability that the fragment is a sentence. features = Array of the fragment’s features.

# File 'lib/tactful_tokenizer.rb', line 190

def orig
  @orig
end

#pred ⇒ Object

orig = The original text of the fragment. next = The next word following the fragment. cleaned = Array of the fragment’s words after cleaning. pred = Probability that the fragment is a sentence. features = Array of the fragment’s features.

# File 'lib/tactful_tokenizer.rb', line 190

def pred
  @pred
end

Instance Method Details

#clean(s) ⇒ Object

Normalizes numbers and discards ambiguous punctuation. And then splits into an array, because realistically only the last and first words are ever accessed.

# File 'lib/tactful_tokenizer.rb', line 201

def clean(s)
    @cleaned = String.new(s)
    tokenize(@cleaned)
    @cleaned.gsub!(/[.,\d]*\d/, '<NUM>')
    @cleaned.gsub!(/[^a-zA-Z0-9,.;:<>\-'\/$% ]/, '')
    @cleaned.gsub!('--', ' ')
    @cleaned = @cleaned.split
end