A simple wrapper around utf8proc for normalizing Unicode strings. Will use the
utf8proc shared library and headers installed on your system if they are available (Packages are available. OSX:
brew install utf8proc, Linux:
utf8proc-devel). Failing that, it will fall-back to compiling the library into the extension.
Currently supports UTF-8/ASCII string input and NFC, NFD, NFKC, NFKD, and NKFC-Casefold forms (US-ASCII strings return an unmodified or case-folded copy). Handles Unicode 9.0 and includes the current official full suite of 9.0 normalization tests.
Quick benchmarks against the UNF gem show
utf8_proc to be between the same speed (best-case) and ~2x slower (worst-case), averaging about 1.15 to 1.5x slower and improving on complex Unicode strings. (However, UNF currently only officially supports Unicode 6.0 and does not pass all 9.0 normalization tests.)
Add this line to your application's Gemfile:
And then execute:
Or install it yourself as:
$ gem install utf8_proc
YARD documentation is available at rubydoc.info
require "utf8_proc" # Canonical Decomposition, followed by Canonical Composition UTF8Proc.NFC(utf8_string) # Canonical Decomposition UTF8Proc.NFD(utf8_string) # Compatibility Decomposition, followed by Canonical Composition UTF8Proc.NFKC(utf8_string) # Compatibility Decomposition UTF8Proc.NFKD(utf8_string) # Compatibility Decomposition, followed by Canonical Composition with Case-folding UTF8Proc.NFKC_CF(utf8_string) # Second argument may be any of: [:nfc (default), :nfd, :nfkc, :nfkd, :nfkc_cf] UTF8Proc.normalize(utf8_string, form = :nfc) # Version string of loaded libutf8proc UTF8Proc::LIBRARY_VERSION # Add normalization methods directly to String class require "utf8_proc/core_ext/string" # This enables: "String".NFC "String".normalize(:nfc)
unf) on JRuby the gem will fall-back to using
java.text.normalizer. The interface remains the same.
Bug reports and pull requests are welcome on GitHub at https://github.com/nomoon/utf8_proc. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
The gem is available as open source under the terms of the MIT License.