Class: TranslationDiff::Segmenters::Pragmatic

Inherits:
Object
  • Object
show all
Defined in:
lib/translation_diff/segmenters/pragmatic.rb

Overview

Default segmenter: scores 76/80 on the Golden Rules corpus vs Simple's 47/80 (golden_rules_test.rb).

Defined Under Namespace

Classes: Error

Constant Summary collapse

DEFAULT_LANGUAGE =

Without a language, Russian mis-segments: it treats "Проф." as a full sentence and stops there.

"en".freeze
SINGLE_NEWLINE =

A lone "\n" is incidental source formatting, not a paragraph break; a run of two or more is left alone.

/(?<!\n)\n(?!\n)/

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.build(_config) ⇒ Object 



14
 
# File 'lib/translation_diff/segmenters/pragmatic.rb', line 14

def self.build(_config) = new

Instance Method Details

#split_offsets(text, language: nil) ⇒ Object 



16
17
18
19
20
21
22
23
24
25
26
 
# File 'lib/translation_diff/segmenters/pragmatic.rb', line 16

def split_offsets(text, language: nil)
  return [0] unless split_candidate?(text)

  shadow = shadow_newlines(text)
  sentences = segment(shadow, language)
  return [0] if sentences.size <= 1

  offsets = recover_offsets(shadow, sentences)
  assert_valid_offsets(text, offsets)
  offsets
end