Module: Bringit::EncodingHelper

Extended by:
EncodingHelper
Included in:
Blame, Blob, Commit, Diff, EncodingHelper, Ref, Tree
Defined in:
lib/bringit/encoding_helper.rb

Constant Summary collapse

ENCODING_CONFIDENCE_THRESHOLD =

This threshold is carefully tweaked to prevent usage of encodings detected by CharlockHolmes with low confidence. If CharlockHolmes confidence is low, we’re better off sticking with utf8 encoding. Reason: git diff can return strings with invalid utf8 byte sequences if it truncates a diff in the middle of a multibyte character. In this case CharlockHolmes will try to guess the encoding and will likely suggest an obscure encoding with low confidence. There is a lot more info with this merge request: gitlab.com/gitlab-org/gitlab_git/merge_requests/77#note_4754193

40

Instance Method Summary collapse

Instance Method Details

#encode!(message) ⇒ Object 



16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
 
# File 'lib/bringit/encoding_helper.rb', line 16

def encode!(message)
  return nil unless message.respond_to? :force_encoding

  # if message is utf-8 encoding, just return it
  message.force_encoding("UTF-8")
  return message if message.valid_encoding?

  # return message if message type is binary
  detect = CharlockHolmes::EncodingDetector.detect(message)
  return message.force_encoding("BINARY") if detect && detect[:type] == :binary

  # force detected encoding if we have sufficient confidence.
  if detect && detect[:encoding] && detect[:confidence] > ENCODING_CONFIDENCE_THRESHOLD
    message.force_encoding(detect[:encoding])
  end

  # encode and clean the bad chars
  message.replace clean(message)
rescue
  encoding = detect ? detect[:encoding] : "unknown"
  "--broken encoding: #{encoding}"
end

#encode_utf8(message) ⇒ Object 



39
40
41
42
43
44
45
46
 
# File 'lib/bringit/encoding_helper.rb', line 39

def encode_utf8(message)
  detect = CharlockHolmes::EncodingDetector.detect(message)
  if detect
    CharlockHolmes::Converter.convert(message, detect[:encoding], 'UTF-8')
  else
    clean(message)
  end
end