Module: Html2rss::Recon

Defined in:
lib/html2rss/recon.rb

Overview

Service that runs reconnaissance on a URL to discover redirect chains, native RSS/Atom feeds, surface categorization, and emit an actionable verdict.

Defined Under Namespace

Classes: Result, Verdict

Class Method Summary collapse

Class Method Details

.batch(urls, strategy: :auto, cache_dir: nil, max_threads: 5, **options) ⇒ Array<Html2rss::Recon::Result>

Runs batch reconnaissance across an Enumerable of URLs. Results are returned in input order. Concurrency uses a Thread pool (not Ractors) for I/O overlap with one loaded gem image.

Parameters:

  • urls (Enumerable<String>) —

    list of URLs

  • strategy (Symbol) (defaults to: :auto) —

    request strategy

  • cache_dir (String, nil) (defaults to: nil) —

    optional directory to cache raw HTML bodies

  • max_threads (Integer) (defaults to: 5) —

    concurrent worker count

  • options (Hash) —

    a customizable set of options

Options Hash (**options):

  • :max_redirects (Integer, nil) —

    optional maximum redirects

  • :max_requests (Integer, nil) —

    optional request budget

Returns:



163
164
165
166
167
168
169
170
171
172
173
 
# File 'lib/html2rss/recon.rb', line 163

def batch(urls, strategy: :auto, cache_dir: nil, max_threads: 5, **options)
  url_list = urls.map { |u| u.to_s.strip }.reject(&:empty?)
  return [] if url_list.empty?

  FileUtils.mkdir_p(cache_dir) if cache_dir
  cache_mutex = Mutex.new if cache_dir

  Batch.map(url_list, concurrency: max_threads) do |target_url|
    call(target_url, strategy:, cache_dir:, cache_mutex:, **options)
  end
end

.call(url, strategy: :auto, cache_dir: nil, cache_mutex: nil) ⇒ Html2rss::Recon::Result

Runs reconnaissance on a single URL.

Parameters:

  • url (String, Html2rss::Url) —

    source page URL

  • strategy (Symbol) (defaults to: :auto) —

    request strategy (:auto, :default, :botasaurus)

  • cache_dir (String, nil) (defaults to: nil) —

    optional directory to cache raw HTML bodies

  • cache_mutex (Mutex, nil) (defaults to: nil) —

    optional mutex serializing cache writes

  • options (Hash) —

    a customizable set of options

Returns:



120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
 
# File 'lib/html2rss/recon.rb', line 120

def call(url, strategy: :auto, cache_dir: nil, cache_mutex: nil, **) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize
  url_obj = Url.from_absolute(url)

  begin
    probe = PageRecon.probe(url_obj, strategy:, **)
  rescue StandardError => error
    return error_result(url_obj, error)
  end

  cache_html_body(probe.response.body, url_obj, cache_dir, cache_mutex) if cache_dir && probe.response.body

  recon = probe.result
  native_feed, discovery_note = find_native_feed(url_obj, probe.session, probe.response)
  notes = build_notes(recon, native_feed, probe.response)
  notes << discovery_note if discovery_note
  verdict = determine_verdict(recon, native_feed)

  Result.new(
    requested_url: url_obj,
    final_url: probe.response.url,
    status: probe.response.status,
    verdict:,
    native_feed:,
    surface_category: SurfaceCategory.coerce(recon.surface_category),
    articles_count: recon.articles_count,
    scheme_downgrade: recon.scheme_downgrade,
    notes:,
    html_bytesize: probe.response.body&.bytesize
  )
end