Module: Moxml::Entity

Defined in:
lib/moxml/entity.rb,
lib/moxml/entity/restorer.rb,
lib/moxml/entity/reference.rb

Overview

Entity-reference round-trip pipeline.

XML libraries split into two camps: those that preserve named entity references natively (nokogiri) and those that expand or reject them at parse (ox, rexml, oga, libxml, leptris). Moxml bridges the camps with one pipeline, owned here and delegated to by Adapter::Base:

  1. Parse time — .preprocess_entities rewrites non-standard entity references to a two-character marker (see MARKER) so the reference survives native parsing verbatim. The five standard entities (& < > " ') are NOT converted.
  2. Read time — marker-bearing text is split into text and Entity::Reference nodes (adapters whose natives cannot hold references), or decoded in place (.decode_entities covers the five standard entities plus numeric character references).
  3. Serialize time — .restore_entities maps markers back to named references, including markers a native serializer already rendered as character references.

How references are STORED between parse and serialize is adapter-shaped (in-tree value objects for ox, attachment sequences for rexml/libxml, text markers for oga/leptris); the marker lifecycle and the Entity::Reference value type are the shared, single-sourced parts.

Defined Under Namespace

Classes: Reference, Restorer

Constant Summary collapse

MARKER =

Marker for adapters that resolve entities during parsing. U+FFFC (Object Replacement Character) + U+FEFF (BOM) is a two-character sentinel chosen because this exact sequence followed by a valid entity name pattern is vanishingly unlikely in real XML content.

"\u{FFFC}\u{FEFF}"
NAME_PATTERN = 
"[a-zA-Z_][\\w.:-]*"
NAME_RE = 
/&(#{NAME_PATTERN});/
MARKER_RE = 
/\u{FFFC}\u{FEFF}(#{NAME_PATTERN});/
SERIALIZED_MARKER_RE = 
/&#xFFFC;&#xFEFF;(#{NAME_PATTERN});/
STANDARD_ENTITIES = 
%w[amp lt gt quot apos].freeze
NON_STANDARD_ENTITY_RE =

One regex pass, no allocations: true when a NON-standard named entity exists somewhere. Documents carrying only the five predefined entities (the overwhelmingly common case) skip the marker gsub's full-buffer copy entirely.

/&(?!amp;|lt;|gt;|quot;|apos;)(#{NAME_PATTERN});/

Class Method Summary collapse

Class Method Details

.c_entity_probe ⇒ Object 



66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/moxml/entity.rb', line 66

def c_entity_probe
  if @c_entity_probe.nil?
    @c_entity_probe =
      if defined?(::Leptris::XML::FFI) &&
          ::Leptris::XML::FFI.respond_to?(:leptris_str_has_nonstandard_entity)
        ::Leptris::XML::FFI.method(:leptris_str_has_nonstandard_entity)
      else
        false
      end
  end
  @c_entity_probe || nil
end

.decode_entities(text) ⇒ Object

Resolve numeric (&#NN; / &#xNN;) and the five standard named (& < > " ') XML entity references in one single pass. The resulting characters are data; no further decoding is applied. Numeric refs producing invalid UTF-8 (NUL, surrogate halves, > U+10FFFF) are preserved verbatim to avoid silently emitting malformed bytes.



148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
 
# File 'lib/moxml/entity.rb', line 148

def decode_entities(text)
  return text unless text.is_a?(String) && text.include?("&")

  text.gsub(DECODE_RE) do
    if (named = ::Regexp.last_match(1))
      NAMED_DECODE_MAP[named]
    else
      code = ::Regexp.last_match(2) ? ::Regexp.last_match(2).to_i : ::Regexp.last_match(3).to_i(16)
      if code.zero? || code.between?(0xD800, 0xDFFF) || code > 0x10FFFF
        ::Regexp.last_match(0)
      else
        [code].pack("U")
      end
    end
  end
end

.nonstandard_entity?(str) ⇒ Boolean

True when a NON-standard named entity exists somewhere (the gsub would then be needed). The leptris binding's C probe (leptris-ruby#124) answers in one pass at memcmp speed; it is resolved lazily because adapters load on demand. Otherwise: the cheap & pre-filter, then the Ruby regex.

Returns:

  • (Boolean) 


57
58
59
60
61
62
63
64
 
# File 'lib/moxml/entity.rb', line 57

def nonstandard_entity?(str)
  probe = c_entity_probe
  if probe
    probe.call(str, str.bytesize) == 1
  else
    str.include?("&") && str.match?(NON_STANDARD_ENTITY_RE)
  end
end

.preprocess_entities(xml) ⇒ Object 



138
139
140
 
# File 'lib/moxml/entity.rb', line 138

def preprocess_entities(xml)
  preprocess_with_marker_flag(xml)[0]
end

.preprocess_with_marker_flag(xml) ⇒ Object

Replace non-standard entity references with markers before parsing. Always returns a UTF-8 encoded string, and reports whether the result can contain markers: the flag rides the same & scan, so callers needing it (leptris parse) avoid a second full-buffer multibyte probe.



96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/moxml/entity.rb', line 96

def preprocess_with_marker_flag(xml)
  return ["", false] if xml.nil?

  str = if xml.encoding == Encoding::BINARY
          # Binary strings are assumed to be UTF-8. If the bytes are
          # not valid UTF-8, fall back to encoding as UTF-8 with
          # replacement to avoid raising on gsub.
          dup = xml.dup.force_encoding("UTF-8")
          if dup.valid_encoding?
            dup
          else
            xml.dup.encode("UTF-8",
                           "ASCII-8BIT", invalid: :replace, undef: :replace)
          end
        elsif xml.encoding == Encoding::UTF_8
          xml
        else
          xml.encode("UTF-8")
        end
  # Fast path: no `&` means no entity references to mark — skip
  # the regex scan and string allocation entirely. The vast
  # majority of XML payloads contain no entity references.
  # Second fast path: only predefined entities — the gsub would
  # pass every match through unchanged, so skip its full-buffer
  # copy too. When the leptris binding is loaded, its C probe
  # (leptris-ruby#124) answers in one memcmp-class pass; the
  # binding loads lazily, so the probe is resolved on first use.
  return [str, false] unless nonstandard_entity?(str)

  marked = false
  processed = str.gsub(NAME_RE) do |match|
    name = ::Regexp.last_match(1)
    if STANDARD_ENTITIES.include?(name)
      match
    else
      marked = true
      "#{MARKER}#{name};"
    end
  end
  [processed, marked]
end

.restore_entities(text) ⇒ Object

Restore entity markers back to named entity references.



166
167
168
169
170
171
172
173
174
175
176
177
178
 
# File 'lib/moxml/entity.rb', line 166

def restore_entities(text)
  return text unless text.is_a?(String)

  # Force UTF-8 encoding since markers are UTF-8 characters
  str = text.encoding == Encoding::UTF_8 ? text : text.dup.force_encoding("UTF-8")
  # Fast path: the vast majority of documents carry no entity
  # markers — two C-level scans beat two regex passes.
  return str unless str.include?(MARKER) ||
    str.include?("&#xFFFC;")

  result = str.gsub(MARKER_RE, '&\1;')
  result.gsub(SERIALIZED_MARKER_RE, '&\1;')
end