Class: Textoken::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/textoken/tokenizer.rb

Overview

This is the last step in the process Takes findings objects and makes finishing punctuation split Does not split if base.dont_split is true

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(base) ⇒ Tokenizer

Returns a new instance of Tokenizer.



8
9
10
11
12
 
# File 'lib/textoken/tokenizer.rb', line 8

def initialize(base)
  @base    = base
  @result  = []
  @col     = base.options.collection.length > 0 ? base.findings : base.text
end

Instance Attribute Details

#base ⇒ Object (readonly)

Returns the value of attribute base.



6
7
8
 
# File 'lib/textoken/tokenizer.rb', line 6

def base
  @base
end

#col ⇒ Object (readonly)

Returns the value of attribute col.



6
7
8
 
# File 'lib/textoken/tokenizer.rb', line 6

def col
  @col
end

#result ⇒ Object (readonly)

Returns the value of attribute result.



6
7
8
 
# File 'lib/textoken/tokenizer.rb', line 6

def result
  @result
end

Instance Method Details

#tokens ⇒ Object 



14
15
16
17
18
 
# File 'lib/textoken/tokenizer.rb', line 14

def tokens
  return col if base.dont_split
  split_punctuations
  @result
end