Class: ChunkerRuby::BaseSplitter

Inherits:
Object
  • Object
show all
Defined in:
lib/chunker_ruby/base_splitter.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(chunk_size: 1000, chunk_overlap: 200, **options) ⇒ BaseSplitter

Returns a new instance of BaseSplitter.

Raises:

  • (ArgumentError) 


7
8
9
10
11
12
13
14
 
# File 'lib/chunker_ruby/base_splitter.rb', line 7

def initialize(chunk_size: 1000, chunk_overlap: 200, **options)
  raise ArgumentError, "chunk_size must be positive" unless chunk_size > 0
  raise ArgumentError, "chunk_overlap must be non-negative" unless chunk_overlap >= 0
  raise ArgumentError, "chunk_overlap must be less than chunk_size" unless chunk_overlap < chunk_size

  @chunk_size = chunk_size
  @chunk_overlap = chunk_overlap
end

Instance Attribute Details

#chunk_overlap ⇒ Object (readonly)

Returns the value of attribute chunk_overlap.



5
6
7
 
# File 'lib/chunker_ruby/base_splitter.rb', line 5

def chunk_overlap
  @chunk_overlap
end

#chunk_size ⇒ Object (readonly)

Returns the value of attribute chunk_size.



5
6
7
 
# File 'lib/chunker_ruby/base_splitter.rb', line 5

def chunk_size
  @chunk_size
end

Instance Method Details

#split(text, metadata: {}) ⇒ Object

Raises:

  • (NotImplementedError) 


16
17
18
 
# File 'lib/chunker_ruby/base_splitter.rb', line 16

def split(text, metadata: {})
  raise NotImplementedError, "#{self.class}#split must be implemented"
end

#split_many(texts) ⇒ Object 



20
21
22
 
# File 'lib/chunker_ruby/base_splitter.rb', line 20

def split_many(texts)
  texts.flat_map.with_index { |t, i| split(t, metadata: { doc_index: i }) }
end