Module: Html2rss::AutoSource::Scraper::JsonState::DocumentScanner

Defined in:
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb

Overview

Scans DOM nodes for JSON payloads containing article data.

Constant Summary collapse

GLOBAL_ASSIGNMENT_PATTERNS =

Regex patterns for known global JavaScript state assignments.

[
  /(?:window|self|globalThis)\.__NEXT_DATA__\s*=\s*/m,
  /(?:window|self|globalThis)\.__NUXT__\s*=\s*/m,
  /(?:window|self|globalThis)\.STATE\s*=\s*/m,
  /(?:window|self|globalThis)\.__REDUX_STATE__\s*=\s*/m,
  /(?:window|self|globalThis)\.__PRELOADED_STATE__\s*=\s*/m,
  /(?:window|self|globalThis)\.__APOLLO_STATE__\s*=\s*/m,
  /(?:window|self|globalThis)\.__remixContext\s*=\s*/m,
  /(?:window|self|globalThis)\.__sveltekit_data\s*=\s*/m,
  /(?:window|self|globalThis)\.GATSBY_STATE\s*=\s*/m,
  /(?:window|self|globalThis)\.__ember_meta\s*=\s*/m,
  /(?:window|self|globalThis)\.angular\s*=\s*/m
].freeze
GLOBAL_ASSIGNMENT_REGEXP =

Combined regex for faster matching of global assignments.

Regexp.union(GLOBAL_ASSIGNMENT_PATTERNS).freeze

Class Method Summary collapse

Class Method Details

.assignment_documents(parsed_body) ⇒ Array<Hash, Array>

Returns JSON documents extracted from global assignments.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    JSON documents extracted from global assignments



50
51
52
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 50

def assignment_documents(parsed_body)
  parsed_body.css('script').filter_map { parse_assignment(_1.text) }
end

.assignment_payload(text) ⇒ String?

Returns extracted JSON-like assignment payload.

Parameters:

  • text (String) —

    script text to inspect for known assignment patterns

Returns:

  • (String, nil) —

    extracted JSON-like assignment payload



63
64
65
66
67
68
69
70
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 63

def assignment_payload(text)
  trimmed = text.to_s.strip
  return if trimmed.empty?
  return unless trimmed.match?(GLOBAL_ASSIGNMENT_REGEXP)

  payload = trimmed.sub(GLOBAL_ASSIGNMENT_REGEXP, '')
  extract_assignment_payload(payload)
end

.coerce_javascript_object(payload) ⇒ String

Returns JSON-compatible payload string.

Parameters:

  • payload (String) —

    JavaScript object-literal payload

Returns:

  • (String) —

    JSON-compatible payload string



152
153
154
155
156
157
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 152

def coerce_javascript_object(payload)
  string = payload.dup

  # KISS approach: mutate common JS literal quirks instead of a full parser.
  strip_trailing_commas(quote_unquoted_keys(string))
end

.extract_assignment_payload(text) ⇒ String?

Returns normalized assignment payload.

Parameters:

  • text (String) —

    text potentially containing JSON-like payloads

Returns:

  • (String, nil) —

    normalized assignment payload



74
75
76
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 74

def extract_assignment_payload(text)
  extract_json_block(text) || text
end

.extract_json_block(text) ⇒ String?

Returns extracted JSON block spanning balanced brackets.

Parameters:

  • text (String) —

    text potentially containing JSON blocks

Returns:

  • (String, nil) —

    extracted JSON block spanning balanced brackets



80
81
82
83
84
85
86
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 80

def extract_json_block(text)
  start_index = text.index(/[\[{]/)
  return unless start_index

  stop_index = scan_for_json_end(text, start_index)
  text[start_index..stop_index] if stop_index
end

.json_documents(parsed_body) ⇒ Array<Hash, Array>

Returns parsed JSON documents discovered in scripts.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    parsed JSON documents discovered in scripts



33
34
35
36
37
38
39
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 33

def json_documents(parsed_body)
  # Use identity-based cache to avoid double-parsing of the same document.
  # WeakMap allows the Nokogiri Document (key) to be garbage collected.
  # rubocop:disable-next ThreadSafety/ClassInstanceVariable
  (@cache ||= ObjectSpace::WeakMap.new)[parsed_body] ||=
    script_documents(parsed_body) + assignment_documents(parsed_body)
end

.parse_assignment(text) ⇒ Hash, ...

Returns parsed assignment payload when available.

Parameters:

  • text (String) —

    script text that may contain a global assignment

Returns:

  • (Hash, Array, nil) —

    parsed assignment payload when available



56
57
58
59
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 56

def parse_assignment(text)
  payload = assignment_payload(text)
  parse_json(payload) if payload
end

.parse_js_object(payload, _original_error) ⇒ Hash, ...

Returns parsed payload after JavaScript coercion.

Parameters:

  • payload (String) —

    JavaScript object-literal payload

  • _original_error (JSON::ParserError) —

    original JSON parse error

Returns:

  • (Hash, Array, nil) —

    parsed payload after JavaScript coercion



138
139
140
141
142
143
144
145
146
147
148
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 138

def parse_js_object(payload, _original_error)
  coerced = coerce_javascript_object(payload)
  return unless coerced

  # Some sites emit JavaScript object literals (unquoted keys, trailing commas).
  # Coerce those payloads into valid JSON so we keep the same parsing pipeline.
  JSON.parse(coerced, symbolize_names: true)
rescue JSON::ParserError => error
  Html2rss::Log.debug("JsonState: failed to parse coerced JavaScript object (#{error.message})")
  nil
end

.parse_json(payload) ⇒ Hash, ...

Returns parsed payload or nil when parsing fails.

Parameters:

  • payload (String, nil) —

    JSON payload to parse

Returns:

  • (Hash, Array, nil) —

    parsed payload or nil when parsing fails



127
128
129
130
131
132
133
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 127

def parse_json(payload)
  return unless payload

  JSON.parse(payload, symbolize_names: true)
rescue JSON::ParserError => error
  parse_js_object(payload, error)
end

.quote_unquoted_keys(jsonish) ⇒ String

Returns payload with unquoted object keys quoted.

Parameters:

  • jsonish (String) —

    JSON-like string with potentially unquoted keys

Returns:

  • (String) —

    payload with unquoted object keys quoted



161
162
163
164
165
166
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 161

def quote_unquoted_keys(jsonish)
  jsonish.gsub(/(?<prefix>\A\s*|[{,\[]\s*)(?<key>[A-Za-z_]\w*)(?<suffix>\s*:)/) do
    captures = Regexp.last_match.named_captures(symbolize_names: true)
    "#{captures[:prefix]}\"#{captures[:key]}\"#{captures[:suffix]}"
  end
end

.scan_for_json_end(text, start_index) ⇒ Integer?

Returns index where the balanced JSON payload ends.

Parameters:

  • text (String) —

    text starting with a JSON object/array opening token

  • start_index (Integer) —

    index where JSON-like content starts

Returns:

  • (Integer, nil) —

    index where the balanced JSON payload ends



91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 91

def scan_for_json_end(text, start_index) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
  stack = []
  in_string = false
  escape = false

  i = start_index
  len = text.length
  while i < len
    char = text[i]

    if in_string
      if escape
        escape = false
      elsif char == '\\'
        escape = true
      elsif char == '"'
        in_string = false
      end
    else
      case char
      when '"' then in_string = true
      when '{' then stack << '}'
      when '[' then stack << ']'
      when '}', ']'
        expected = stack.pop
        return i if expected == char && stack.empty?
      end
    end
    i += 1
  end

  nil
end

.script_documents(parsed_body) ⇒ Array<Hash, Array>

Returns JSON documents extracted from JSON script tags.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    JSON documents extracted from JSON script tags



43
44
45
46
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 43

def script_documents(parsed_body)
  ::Html2rss::Html::Probe.scripts(parsed_body, ::Html2rss::Html::Probe::APPLICATION_JSON)
                         .filter_map { parse_json(_1.text) }
end

.strip_trailing_commas(jsonish) ⇒ String

Returns payload without trailing commas before closing tokens.

Parameters:

  • jsonish (String) —

    JSON-like string with potential trailing commas

Returns:

  • (String) —

    payload without trailing commas before closing tokens



170
171
172
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 170

def strip_trailing_commas(jsonish)
  jsonish.gsub(/,(\s*[\]}])/, '\1')
end