Class: Analyzers::Utils::HumanLanguageDetector

Inherits:
Object
  • Object
show all
Defined in:
lib/crypto-toolbox/analyzers/utils/human_language_detector.rb

Overview

NOTE the implementation decisions are based on the result of benchmarks/language_detector.rb

Instance Method Summary collapse

Constructor Details

#initialize ⇒ HumanLanguageDetector

Returns a new instance of HumanLanguageDetector.



6
7
8
9
 
# File 'lib/crypto-toolbox/analyzers/utils/human_language_detector.rb', line 6

def initialize
  @spell_checker = ::Analyzers::Utils::SpellChecker.new
  @ascii_checker = ::Analyzers::Utils::AsciiLanguageDetector.new
end

Instance Method Details

#human_language?(buffer) ⇒ Boolean

Returns:

  • (Boolean) 


22
23
24
 
# File 'lib/crypto-toolbox/analyzers/utils/human_language_detector.rb', line 22

def human_language?(buffer)
  ascii_valid?(buffer) && spell_valid?(buffer)
end

#human_language_entries(buffers, spellcheck: true) ⇒ Object

NOTE: we dont use the human_language? method to be faster at processing and more idiomatic



13
14
15
16
17
18
19
20
 
# File 'lib/crypto-toolbox/analyzers/utils/human_language_detector.rb', line 13

def human_language_entries(buffers,spellcheck: true )
  filtered = buffers.select{|b| ascii_valid?(b) }
  if spellcheck
    buffers.select{|b| spell_valid?(b) }
  else
    filtered
  end
end