Class: TranslationDiff::Segmenters::Simple

Inherits:
Object
  • Object
show all
Defined in:
lib/translation_diff/segmenters/simple.rb

Overview

No runtime dependency, but deliberately conservative: every guard exists to turn a split off, never on.

Constant Summary collapse

TERMINATOR =

CJK terminators need no trailing whitespace and no guards -- those scripts have no case or abbreviations.

/[.!?…]+|[。！？]+/
CJK_TERMINATOR = 
/\A[。！？]/
ABBREVIATIONS =

Words that end in "." without ending a sentence, matched case-insensitively; extend for your own domain.

%w[
  т.е. т.д. т.п. см. рис. стр. гр. ул. г. руб. проф. акад. тыс. млн. млрд. им.
  Mr. Mrs. Ms. Dr. Prof. St. etc. e.g. i.e. vs. approx. No. im. fig. Fig. vol. p. pp.
].map(&:downcase).freeze
WORD_TAIL =

Strips leading punctuation like an opening quote so "Dr. Smith still guards on "Dr.".

/[\p{L}\p{N}.]+\z/

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.build(_config) ⇒ Object 



17
 
# File 'lib/translation_diff/segmenters/simple.rb', line 17

def self.build(_config) = new

Instance Method Details

#split_offsets(text, language: nil) ⇒ Object

language: is part of the shared segmenter contract but ignored here -- these rules are language-neutral. rubocop:disable-next Lint/UnusedMethodArgument



21
22
23
24
25
26
27
28
29
30
31
32
 
# File 'lib/translation_diff/segmenters/simple.rb', line 21

def split_offsets(text, language: nil)
  offsets = [0]
  position = 0

  while (match = TERMINATOR.match(text, position))
    boundary = boundary_for(text, match)
    offsets << boundary if boundary
    position = match.end(0)
  end

  offsets
end