Class: Twitter::TwitterText::Regex

Inherits:
Object
  • Object
show all
Defined in:
lib/twitter-text/regex.rb,
lib/twitter-text/emoji_regex.rb

Overview

A collection of regular expressions for parsing Tweet text. The regular expression list is frozen at load time to ensure immutability. These regular expressions are used throughout the TwitterText classes. Special care has been taken to make sure these reular expressions work with Tweets in all languages.

Defined Under Namespace

Classes: Emoji

Constant Summary collapse

REGEXEN =

:nodoc:

{}
TLDS = 
YAML.load_file(
  File.join(
    File.expand_path('../../..', __FILE__), # project root
    'lib', 'assets', 'tld_lib.yml'
  )
)
UNICODE_SPACES =

Space is more than %20, U+3000 for example is the full-width space used with Kanji. Provide a short-hand to access both the list of characters and a pattern suitible for use with String#split Taken from: ActiveSupport::Multibyte::Handlers::UTF8Handler::UNICODE_WHITESPACE

[
  (0x0009..0x000D).to_a,  # White_Space # Cc   [5] <control-0009>..<control-000D>
  0x0020,          # White_Space # Zs       SPACE
  0x0085,          # White_Space # Cc       <control-0085>
  0x00A0,          # White_Space # Zs       NO-BREAK SPACE
  0x1680,          # White_Space # Zs       OGHAM SPACE MARK
  0x180E,          # White_Space # Zs       MONGOLIAN VOWEL SEPARATOR
  (0x2000..0x200A).to_a, # White_Space # Zs  [11] EN QUAD..HAIR SPACE
  0x2028,          # White_Space # Zl       LINE SEPARATOR
  0x2029,          # White_Space # Zp       PARAGRAPH SEPARATOR
  0x202F,          # White_Space # Zs       NARROW NO-BREAK SPACE
  0x205F,          # White_Space # Zs       MEDIUM MATHEMATICAL SPACE
  0x3000,          # White_Space # Zs       IDEOGRAPHIC SPACE
].flatten.map{|c| [c].pack('U*')}.freeze
DIRECTIONAL_CHARACTERS = 
[
  0x061C,          # ARABIC LETTER MARK (ALM)
  0x200E,          # LEFT-TO-RIGHT MARK (LRM)
  0x200F,          # RIGHT-TO-LEFT MARK (RLM)
  0x202A,          # LEFT-TO-RIGHT EMBEDDING (LRE)
  0x202B,          # RIGHT-TO-LEFT EMBEDDING (RLE)
  0x202C,          # POP DIRECTIONAL FORMATTING (PDF)
  0x202D,          # LEFT-TO-RIGHT OVERRIDE (LRO)
  0x202E,          # RIGHT-TO-LEFT OVERRIDE (RLO)
  0x2066,          # LEFT-TO-RIGHT ISOLATE (LRI)
  0x2067,          # RIGHT-TO-LEFT ISOLATE (RLI)
  0x2068,          # FIRST STRONG ISOLATE (FSI)
  0x2069,          # POP DIRECTIONAL ISOLATE (PDI)
].map{|cp| [cp].pack('U')}.freeze
INVALID_CHARACTERS =

Character not allowed in Tweets

[
  0xFFFE, 0xFEFF, # BOM
  0xFFFF,         # Special
].map{|cp| [cp].pack('U') }.freeze
LATIN_ACCENTS =

Latin accented characters Excludes 0xd7 from the range (the multiplication sign, confusable with "x"). Also excludes 0xf7, the division sign

[
  regex_range(0xc0, 0xd6),
  regex_range(0xd8, 0xf6),
  regex_range(0xf8, 0xff),
  regex_range(0x0100, 0x024f),
  regex_range(0x0253, 0x0254),
  regex_range(0x0256, 0x0257),
  regex_range(0x0259),
  regex_range(0x025b),
  regex_range(0x0263),
  regex_range(0x0268),
  regex_range(0x026f),
  regex_range(0x0272),
  regex_range(0x0289),
  regex_range(0x028b),
  regex_range(0x02bb),
  regex_range(0x0300, 0x036f),
  regex_range(0x1e00, 0x1eff)
].join('').freeze
RTL_CHARACTERS = 
[
  regex_range(0x0600,0x06FF),
  regex_range(0x0750,0x077F),
  regex_range(0x0590,0x05FF),
  regex_range(0xFE70,0xFEFF)
].join('').freeze
PUNCTUATION_CHARS = 
'!"#$%&\'()*+,-./:;<=>?@\[\]^_\`{|}~'
SPACE_CHARS = 
" \t\n\x0B\f\r"
CTRL_CHARS = 
"\x00-\x1F\x7F"
HASHTAG_LETTERS_AND_MARKS =

Generated from unicode_regex/unicode_regex_groups.scala, more inclusive than Ruby's \pL\pM

"\\p{L}\\p{M}" +
"\u037f\u0528-\u052f\u08a0-\u08b2\u08e4-\u08ff\u0978\u0980\u0c00\u0c34\u0c81\u0d01\u0ede\u0edf" +
"\u10c7\u10cd\u10fd-\u10ff\u16f1-\u16f8\u17b4\u17b5\u191d\u191e\u1ab0-\u1abe\u1bab-\u1bad\u1bba-" +
"\u1bbf\u1cf3-\u1cf6\u1cf8\u1cf9\u1de7-\u1df5\u2cf2\u2cf3\u2d27\u2d2d\u2d66\u2d67\u9fcc\ua674-" +
"\ua67b\ua698-\ua69d\ua69f\ua792-\ua79f\ua7aa-\ua7ad\ua7b0\ua7b1\ua7f7-\ua7f9\ua9e0-\ua9ef\ua9fa-" +
"\ua9fe\uaa7c-\uaa7f\uaae0-\uaaef\uaaf2-\uaaf6\uab30-\uab5a\uab5c-\uab5f\uab64\uab65\uf870-\uf87f" +
"\uf882\uf884-\uf89f\uf8b8\uf8c1-\uf8d6\ufa2e\ufa2f\ufe27-\ufe2d\u{102e0}\u{1031f}\u{10350}-\u{1037a}" +
"\u{10500}-\u{10527}\u{10530}-\u{10563}\u{10600}-\u{10736}\u{10740}-\u{10755}\u{10760}-\u{10767}" +
"\u{10860}-\u{10876}\u{10880}-\u{1089e}\u{10980}-\u{109b7}\u{109be}\u{109bf}\u{10a80}-\u{10a9c}" +
"\u{10ac0}-\u{10ac7}\u{10ac9}-\u{10ae6}\u{10b80}-\u{10b91}\u{1107f}\u{110d0}-\u{110e8}\u{11100}-" +
"\u{11134}\u{11150}-\u{11173}\u{11176}\u{11180}-\u{111c4}\u{111da}\u{11200}-\u{11211}\u{11213}-" +
"\u{11237}\u{112b0}-\u{112ea}\u{11301}-\u{11303}\u{11305}-\u{1130c}\u{1130f}\u{11310}\u{11313}-" +
"\u{11328}\u{1132a}-\u{11330}\u{11332}\u{11333}\u{11335}-\u{11339}\u{1133c}-\u{11344}\u{11347}" +
"\u{11348}\u{1134b}-\u{1134d}\u{11357}\u{1135d}-\u{11363}\u{11366}-\u{1136c}\u{11370}-\u{11374}" +
"\u{11480}-\u{114c5}\u{114c7}\u{11580}-\u{115b5}\u{115b8}-\u{115c0}\u{11600}-\u{11640}\u{11644}" +
"\u{11680}-\u{116b7}\u{118a0}-\u{118df}\u{118ff}\u{11ac0}-\u{11af8}\u{1236f}-\u{12398}\u{16a40}-" +
"\u{16a5e}\u{16ad0}-\u{16aed}\u{16af0}-\u{16af4}\u{16b00}-\u{16b36}\u{16b40}-\u{16b43}\u{16b63}-" +
"\u{16b77}\u{16b7d}-\u{16b8f}\u{16f00}-\u{16f44}\u{16f50}-\u{16f7e}\u{16f8f}-\u{16f9f}\u{1bc00}-" +
"\u{1bc6a}\u{1bc70}-\u{1bc7c}\u{1bc80}-\u{1bc88}\u{1bc90}-\u{1bc99}\u{1bc9d}\u{1bc9e}\u{1e800}-" +
"\u{1e8c4}\u{1e8d0}-\u{1e8d6}\u{1ee00}-\u{1ee03}\u{1ee05}-\u{1ee1f}\u{1ee21}\u{1ee22}\u{1ee24}" +
"\u{1ee27}\u{1ee29}-\u{1ee32}\u{1ee34}-\u{1ee37}\u{1ee39}\u{1ee3b}\u{1ee42}\u{1ee47}\u{1ee49}" +
"\u{1ee4b}\u{1ee4d}-\u{1ee4f}\u{1ee51}\u{1ee52}\u{1ee54}\u{1ee57}\u{1ee59}\u{1ee5b}\u{1ee5d}\u{1ee5f}" +
"\u{1ee61}\u{1ee62}\u{1ee64}\u{1ee67}-\u{1ee6a}\u{1ee6c}-\u{1ee72}\u{1ee74}-\u{1ee77}\u{1ee79}-" +
"\u{1ee7c}\u{1ee7e}\u{1ee80}-\u{1ee89}\u{1ee8b}-\u{1ee9b}\u{1eea1}-\u{1eea3}\u{1eea5}-\u{1eea9}" +
"\u{1eeab}-\u{1eebb}"
HASHTAG_NUMERALS =

Generated from unicode_regex/unicode_regex_groups.scala, more inclusive than Ruby's \pNd

"\\p{Nd}" +
"\u0de6-\u0def\ua9f0-\ua9f9\u{110f0}-\u{110f9}\u{11136}-\u{1113f}\u{111d0}-\u{111d9}\u{112f0}-" +
"\u{112f9}\u{114d0}-\u{114d9}\u{11650}-\u{11659}\u{116c0}-\u{116c9}\u{118e0}-\u{118e9}\u{16a60}-" +
"\u{16a69}\u{16b50}-\u{16b59}"
HASHTAG_SPECIAL_CHARS = 
"_\u200c\u200d\ua67e\u05be\u05f3\u05f4\uff5e\u301c\u309b\u309c\u30a0\u30fb\u3003\u0f0b\u0f0c\u00b7"
HASHTAG_LETTERS_NUMERALS = 
"#{HASHTAG_LETTERS_AND_MARKS}#{HASHTAG_NUMERALS}#{HASHTAG_SPECIAL_CHARS}"
HASHTAG_LETTERS_NUMERALS_SET = 
"[#{HASHTAG_LETTERS_NUMERALS}]"
HASHTAG_LETTERS_SET = 
"[#{HASHTAG_LETTERS_AND_MARKS}]"
HASHTAG = 
/(\A|\ufe0e|\ufe0f|[^&#{HASHTAG_LETTERS_NUMERALS}])(#|＃)(?!\ufe0f|\u20e3)(#{HASHTAG_LETTERS_NUMERALS_SET}*#{HASHTAG_LETTERS_SET}#{HASHTAG_LETTERS_NUMERALS_SET}*)/io
DOMAIN_VALID_CHARS = 
"[^#{DIRECTIONAL_CHARACTERS.join('')}#{PUNCTUATION_CHARS}#{SPACE_CHARS}#{CTRL_CHARS}#{INVALID_CHARACTERS.join('')}#{UNICODE_SPACES.join('')}]"

Class Method Summary collapse

Class Method Details

.[](key) ⇒ Object

Return the regular expression for a given key. If the key is not a known symbol a nil will be returned.



376
377
378
 
# File 'lib/twitter-text/regex.rb', line 376

def self.[](key)
  REGEXEN[key]
end

.regex_range(from, to = nil) ⇒ Object

:nodoc:



18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
 
# File 'lib/twitter-text/regex.rb', line 18

def self.regex_range(from, to = nil) # :nodoc:
  if $RUBY_1_9
    if to
      "\\u{#{from.to_s(16).rjust(4, '0')}}-\\u{#{to.to_s(16).rjust(4, '0')}}"
    else
      "\\u{#{from.to_s(16).rjust(4, '0')}}"
    end
  else
    if to
      [from].pack('U') + '-' + [to].pack('U')
    else
      [from].pack('U')
    end
  end
end