Class: ActiveSupport::Multibyte::Handlers::UTF8Handler

Inherits:
Object
  • Object
show all
Defined in:
lib/active_support/multibyte/handlers/utf8_handler.rb

Overview

UTF8Handler implements Unicode aware operations for strings, these operations will be used by the Chars proxy when $KCODE is set to ‘UTF8’.

Direct Known Subclasses

UTF8HandlerProc

Constant Summary collapse

HANGUL_SBASE =

Hangul character boundaries and properties

0xAC00
HANGUL_LBASE = 
0x1100
HANGUL_VBASE = 
0x1161
HANGUL_TBASE = 
0x11A7
HANGUL_LCOUNT = 
19
HANGUL_VCOUNT = 
21
HANGUL_TCOUNT = 
28
HANGUL_NCOUNT = 
HANGUL_VCOUNT * HANGUL_TCOUNT
HANGUL_SCOUNT = 
11172
HANGUL_SLAST = 
HANGUL_SBASE + HANGUL_SCOUNT
HANGUL_JAMO_FIRST = 
0x1100
HANGUL_JAMO_LAST = 
0x11FF
UNICODE_WHITESPACE =

All the unicode whitespace

[
  (0x0009..0x000D).to_a,  # White_Space # Cc   [5] <control-0009>..<control-000D>
  0x0020,          # White_Space # Zs       SPACE
  0x0085,          # White_Space # Cc       <control-0085>
  0x00A0,          # White_Space # Zs       NO-BREAK SPACE
  0x1680,          # White_Space # Zs       OGHAM SPACE MARK
  0x180E,          # White_Space # Zs       MONGOLIAN VOWEL SEPARATOR
  (0x2000..0x200A).to_a, # White_Space # Zs  [11] EN QUAD..HAIR SPACE
  0x2028,          # White_Space # Zl       LINE SEPARATOR
  0x2029,          # White_Space # Zp       PARAGRAPH SEPARATOR
  0x202F,          # White_Space # Zs       NARROW NO-BREAK SPACE
  0x205F,          # White_Space # Zs       MEDIUM MATHEMATICAL SPACE
  0x3000,          # White_Space # Zs       IDEOGRAPHIC SPACE
].flatten.freeze
UNICODE_LEADERS_AND_TRAILERS =

BOM (byte order mark) can also be seen as whitespace, it’s a non-rendering character used to distinguish between little and big endian. This is not an issue in utf-8, so it must be ignored.

UNICODE_WHITESPACE + [65279]
UTF8_PAT =

Borrowed from the Kconv library by Shinji KONO - (also as seen on the W3C site)

/\A(?:
 [\x00-\x7f]                                     |
 [\xc2-\xdf] [\x80-\xbf]                         |
 \xe0        [\xa0-\xbf] [\x80-\xbf]             |
 [\xe1-\xef] [\x80-\xbf] [\x80-\xbf]             |
 \xf0        [\x90-\xbf] [\x80-\xbf] [\x80-\xbf] |
 [\xf1-\xf3] [\x80-\xbf] [\x80-\xbf] [\x80-\xbf] |
 \xf4        [\x80-\x8f] [\x80-\xbf] [\x80-\xbf]
)*\z/xn
UNICODE_TRAILERS_PAT = 
/(#{codepoints_to_pattern(UNICODE_LEADERS_AND_TRAILERS)})+\Z/
UNICODE_LEADERS_PAT = 
/\A(#{codepoints_to_pattern(UNICODE_LEADERS_AND_TRAILERS)})+/

Class Method Summary collapse

Class Method Details

.capitalize(str) ⇒ Object

Returns a copy of str with the first character converted to uppercase and the remainder to lowercase



191
192
193
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 191

def capitalize(str)
  upcase(slice(str, 0..0)) + downcase(slice(str, 1..-1) || '')
end

.codepoints_to_pattern(array_of_codepoints) ⇒ Object

Returns a regular expression pattern that matches the passed Unicode codepoints



114
115
116
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 114

def self.codepoints_to_pattern(array_of_codepoints) #:nodoc:
  array_of_codepoints.collect{ |e| [e].pack 'U*' }.join('|') 
end

.compose(str) ⇒ Object

Perform composition on the characters in the string



227
228
229
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 227

def compose(str)
  compose_codepoints u_unpack(str).pack('U*')
end

.consumes?(str) ⇒ Boolean

Checks if the string is valid UTF8.

Returns:

  • (Boolean) 


257
258
259
260
261
262
263
264
265
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 257

def consumes?(str)
  # Unpack is a little bit faster than regular expressions
  begin
    str.unpack('U*')
    true
  rescue ArgumentError
    false
  end
end

.decompose(str) ⇒ Object

Perform decomposition on the characters in the string



222
223
224
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 222

def decompose(str)
  decompose_codepoints(:canonical, u_unpack(str)).pack('U*')
end

.downcase(str) ⇒ Object

Convert characters in the string to lowercase



188
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 188

def downcase(str); to_case :lowercase_mapping, str; end

.g_length(str) ⇒ Object

Returns the number of grapheme clusters in the string. This method is very likely to be moved or renamed in future versions.



269
270
271
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 269

def g_length(str)
  g_unpack(str).length
end

.index(str, *args) ⇒ Object

Returns the position of the passed argument in the string, counting in codepoints



137
138
139
140
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 137

def index(str, *args)
  bidx = str.index(*args)
  bidx ? (u_unpack(str.slice(0...bidx)).size) : nil
end

.insert(str, offset, fragment) ⇒ Object

Inserts the passed string at specified codepoint offsets



127
128
129
130
131
132
133
134
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 127

def insert(str, offset, fragment)
  str.replace(
    u_unpack(str).insert(
      offset,
      u_unpack(fragment)
    ).flatten.pack('U*')
  )
end

.lstrip(str) ⇒ Object

Does Unicode-aware lstrip



148
149
150
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 148

def lstrip(str)
  str.gsub(UNICODE_LEADERS_PAT, '')
end

.normalize(str, form = ActiveSupport::Multibyte::DEFAULT_NORMALIZATION_FORM) ⇒ Object

Returns the KC normalization of the string by default. NFKC is considered the best normalization form for passing strings to databases and validations.

  • str: The string to perform normalization on.

  • form: The form you want to normalize in. Should be one of the following: :c, :kc, :d or :kd.



204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 204

def normalize(str, form=ActiveSupport::Multibyte::DEFAULT_NORMALIZATION_FORM)
  # See http://www.unicode.org/reports/tr15, Table 1
  codepoints = u_unpack(str)
  case form
    when :d
      reorder_characters(decompose_codepoints(:canonical, codepoints))
    when :c
      compose_codepoints reorder_characters(decompose_codepoints(:canonical, codepoints))
    when :kd
      reorder_characters(decompose_codepoints(:compatability, codepoints))
    when :kc
      compose_codepoints reorder_characters(decompose_codepoints(:compatability, codepoints))
    else
      raise ArgumentError, "#{form} is not a valid normalization variant", caller
  end.pack('U*')
end

.reverse(str) ⇒ Object

Reverses codepoints in the string.



164
165
166
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 164

def reverse(str)
  u_unpack(str).reverse.pack('U*')
end

.rstrip(str) ⇒ Object

Does Unicode-aware rstrip



143
144
145
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 143

def rstrip(str)
  str.gsub(UNICODE_TRAILERS_PAT, '')
end

.size(str) ⇒ Object Also known as: length

Returns the number of codepoints in the string



158
159
160
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 158

def size(str)
  u_unpack(str).size
end

.slice(str, *args) ⇒ Object Also known as: []

Implements Unicode-aware slice with codepoints. Slicing on one point returns the codepoints for that character.



170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 170

def slice(str, *args)
  if (args.size == 2 && args.first.is_a?(Range))
    raise TypeError, 'cannot convert Range into Integer' # Do as if we were native
  elsif args[0].kind_of? Range
    cps = u_unpack(str).slice(*args)
    cps.nil? ? nil : cps.pack('U*')
  elsif args.size == 1 && args[0].kind_of?(Numeric)
    u_unpack(str)[args[0]]
  else
    u_unpack(str).slice(*args).pack('U*')
  end
end

.strip(str) ⇒ Object

Removed leading and trailing whitespace



153
154
155
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 153

def strip(str)
  str.gsub(UNICODE_LEADERS_PAT, '').gsub(UNICODE_TRAILERS_PAT, '')
end

.tidy_bytes(str) ⇒ Object

Replaces all the non-utf-8 bytes by their iso-8859-1 or cp1252 equivalent resulting in a valid utf-8 string



274
275
276
277
278
279
280
281
282
283
284
285
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 274

def tidy_bytes(str)
  str.split(//u).map do |c|
    if !UTF8_PAT.match(c)
      n = c.unpack('C')[0]
      n < 128 ? n.chr :
      n < 160 ? [UCD.cp1252[n] || n].pack('U') :
      n < 192 ? "\xC2" + n.chr : "\xC3" + (n-64).chr
    else
      c
    end
  end.join
end

.translate_offset(str, byte_offset) ⇒ Object

Used to translate an offset from bytes to characters, for instance one received from a regular expression match



236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 236

def translate_offset(str, byte_offset)
  return 0 if str == ''
  return nil if byte_offset.nil?
  chunk = str[0..byte_offset]
  begin
    begin
      chunk.unpack('U*').length - 1
    rescue ArgumentError => e
      chunk = str[0..(byte_offset+=1)]
      # Stop retrying at the end of the string
      raise e unless byte_offset < chunk.length 
      # We damaged a character, retry
      retry
    end
  # Catch the ArgumentError so we can throw our own
  rescue ArgumentError 
    raise EncodingError.new('malformed UTF-8 character')
  end
end

.upcase(str) ⇒ Object

Convert characters in the string to uppercase



185
 
# File 'lib/active_support/multibyte/handlers/utf8_handler.rb', line 185

def upcase(str); to_case :uppercase_mapping, str; end