Module: Html2rss::PageRecon::Diagnostics

Defined in:
lib/html2rss/page_recon/diagnostics.rb

Overview

Diagnostic inspect path (not Capture or Recon ownership). Fetches via probe, then adds scraper/XHR diagnostics for curation inspect surfaces.

Defined Under Namespace

Classes: Report

Class Method Summary collapse

Class Method Details

.batch(urls:, strategy: :auto, concurrency: Batch::DEFAULT_CONCURRENCY) ⇒ Array<Report>

Runs diagnostic inspect across URLs with per-URL error isolation.

Parameters:

  • urls (Enumerable<String>)
  • strategy (Symbol, String) (defaults to: :auto)
  • concurrency (Integer) (defaults to: Batch::DEFAULT_CONCURRENCY)

Returns:



57
58
59
60
61
62
63
 
# File 'lib/html2rss/page_recon/diagnostics.rb', line 57

def batch(urls:, strategy: :auto, concurrency: Batch::DEFAULT_CONCURRENCY)
  Batch.map(Array(urls), concurrency:) do |url|
    call(url:, strategy:)
  rescue StandardError => error
    error_report(url, error)
  end
end

.call(url:, strategy: :auto, deep: false) ⇒ Report

Parameters:

  • url (String)
  • strategy (String, Symbol) (defaults to: :auto)
  • deep (Boolean) (defaults to: false) —

    when true and strategy is auto, one Botasaurus hop if configured

Returns:



42
43
44
45
46
47
48
 
# File 'lib/html2rss/page_recon/diagnostics.rb', line 42

def call(url:, strategy: :auto, deep: false)
  probe = PageRecon.probe(url, strategy: resolve_inspect_strategy(strategy, deep:))
  recon = probe.result
  response = probe.response

  Report.new(data: build_data(probe, recon, response))
end

.redacted_endpoint(entry) ⇒ String?

Returns scheme+host+path only.

Parameters:

  • entry (Hash) —

    captured response hash

Returns:

  • (String, nil) —

    scheme+host+path only



80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/html2rss/page_recon/diagnostics.rb', line 80

def redacted_endpoint(entry)
  raw = entry['url'] || entry[:url]
  return unless raw

  uri = URI.parse(raw.to_s)
  return unless uri.scheme && uri.host

  "#{uri.scheme}://#{uri.host}#{uri.path}"
rescue URI::InvalidURIError
  nil
end

.scraper_info(parsed) ⇒ Array<String>, Hash

Parameters:

  • parsed (Object) —

    parsed response body

Returns:

  • (Array<String>, Hash) 


108
109
110
111
112
113
114
115
116
 
# File 'lib/html2rss/page_recon/diagnostics.rb', line 108

def scraper_info(parsed)
  return { error: 'Response is not HTML' } unless parsed.is_a?(Nokogiri::HTML::Document)

  begin
    Html2rss::AutoSource::Scraper.from(parsed).map(&:name)
  rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
    { none_found: error.category.to_s }
  end
end

.xhr_candidate_articles?(entry) ⇒ Boolean

Parameters:

  • entry (Hash) —

    captured response hash

Returns:

  • (Boolean) 


95
96
97
98
99
100
101
102
103
 
# File 'lib/html2rss/page_recon/diagnostics.rb', line 95

def xhr_candidate_articles?(entry)
  body = entry['body'] || entry[:body]
  return false unless body.is_a?(String)

  document = JSON.parse(body, symbolize_names: true)
  AutoSource::Scraper::JsonState::CandidateDetector.candidate_array?(document)
rescue JSON::ParserError
  false
end

.xhr_capture_info(response) ⇒ Hash

Returns redacted XHR capture diagnostics (no query strings).

Parameters:

Returns:

  • (Hash) —

    redacted XHR capture diagnostics (no query strings)



68
69
70
71
72
73
74
75
 
# File 'lib/html2rss/page_recon/diagnostics.rb', line 68

def xhr_capture_info(response)
  captured = response.captured_responses
  {
    count: captured.size,
    sample_endpoints: captured.first(5).filter_map { |entry| redacted_endpoint(entry) },
    candidate_articles: captured.any? { |entry| xhr_candidate_articles?(entry) }
  }
end