Module: Html2rss::Batch

Defined in:
lib/html2rss/batch.rb

Overview

Concurrency-bounded batch execution over enumerables with input order preservation and per-item error isolation.

Defined Under Namespace

Classes: BatchResult

Constant Summary collapse

MAX_CONCURRENCY =

Maximum permitted worker threads for batch operations.

10
DEFAULT_CONCURRENCY =

Default worker threads for batch operations.

5

Class Method Summary collapse

Class Method Details

.batch_inspect(urls:, strategy: :auto, concurrency: DEFAULT_CONCURRENCY) ⇒ BatchResult

Inspects multiple URLs in parallel with per-URL error isolation.

Parameters:

  • list of URLs to inspect

  • (defaults to: :auto)

    request strategy (+:auto+, :default, :botasaurus)

  • (defaults to: DEFAULT_CONCURRENCY)

    number of worker threads (default 5, max 10)

Returns:



95
96
97
98
99
100
101
 
# File 'lib/html2rss/batch.rb', line 95

def batch_inspect(urls:, strategy: :auto, concurrency: DEFAULT_CONCURRENCY)
  reports = PageRecon::Diagnostics.batch(urls:, strategy:, concurrency:)
  results = reports.map(&:to_wire_h)
  successful = reports.count { |report| diagnostic_success?(report) }

  BatchResult.new(total: results.size, successful:, results:)
end

.batch_recon(urls:, strategy: :auto, concurrency: DEFAULT_CONCURRENCY) ⇒ BatchResult

Runs recon across multiple URLs in parallel with per-URL error isolation.

Parameters:

  • list of URLs to recon

  • (defaults to: :auto)

    request strategy (+:auto+, :default, :botasaurus)

  • (defaults to: DEFAULT_CONCURRENCY)

    number of worker threads (default 5, max 10)

  • a customizable set of options

Returns:



111
112
113
114
115
 
# File 'lib/html2rss/batch.rb', line 111

def batch_recon(urls:, strategy: :auto, concurrency: DEFAULT_CONCURRENCY, **)
  results = Recon.batch(urls, strategy: (strategy || :auto).to_sym, max_threads: concurrency, **)
  successful = results.count { |r| r.status ? r.status < 400 : false }
  BatchResult.new(total: results.size, successful:, results: results.map(&:to_h))
end

.batch_scrape(urls:, strategy: :auto, limit: 10, concurrency: DEFAULT_CONCURRENCY) ⇒ BatchResult

Scrapes multiple URLs in parallel with per-URL error isolation.

Parameters:

  • list of URLs to scrape

  • (defaults to: :auto)

    request strategy (+:auto+, :default, :botasaurus)

  • (defaults to: 10)

    max articles to extract per URL

  • (defaults to: DEFAULT_CONCURRENCY)

    number of worker threads (default 5, max 10)

Returns:



84
85
86
 
# File 'lib/html2rss/batch.rb', line 84

def batch_scrape(urls:, strategy: :auto, limit: 10, concurrency: DEFAULT_CONCURRENCY)
  run(urls, concurrency:) { |url| scrape_single_url(url:, strategy:, limit:) }
end

.map(items, concurrency: DEFAULT_CONCURRENCY) {|item| ... } ⇒ Array<Object>

Executes a block concurrently across items using a bounded worker pool. Results are returned in input order.

Parameters:

  • (defaults to: DEFAULT_CONCURRENCY)

    max parallel worker threads (1..10, default: 5)

Yield Parameters:

  • item (Object)

Returns:



31
32
33
34
35
36
37
 
# File 'lib/html2rss/batch.rb', line 31

def map(items, concurrency: DEFAULT_CONCURRENCY, &)
  list = Array(items)
  return [] if list.empty?

  worker_count = worker_pool_size(list.size, concurrency)
  process_queue(list, worker_count, &)
end

.run(items, concurrency: DEFAULT_CONCURRENCY) {|item| ... } ⇒ BatchResult

Runs a batch operation that yields items and wraps the results in a BatchResult.

Parameters:

  • (defaults to: DEFAULT_CONCURRENCY)

    max parallel worker threads (1..10, default: 5)

Yield Parameters:

  • item (Object)

Returns:



46
47
48
49
50
51
52
 
# File 'lib/html2rss/batch.rb', line 46

def run(items, concurrency: DEFAULT_CONCURRENCY, &)
  list = Array(items)
  results = map(list, concurrency:, &)
  successful = results.count { |entry| entry.is_a?(Hash) ? entry[:ok] : !entry.nil? }

  BatchResult.new(total: list.size, successful:, results:)
end

.scrape_wire(url:, strategy:, limit:, items_selector: nil) ⇒ Hash{Symbol => Object}

Single-URL scrape wire extraction: auto_feed_result → JSON Feed items.

Parameters:

  • (defaults to: nil)

Returns:

  • :strategy, :items, :channel_title, :admission_drops



62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/html2rss/batch.rb', line 62

def scrape_wire(url:, strategy:, limit:, items_selector: nil) # rubocop:disable Metrics/MethodLength
  plan = (strategy || :auto).to_sym
  feed_args = { strategy: plan, limit: }
  feed_args[:items_selector] = items_selector unless items_selector.nil?
  feed_result = Html2rss.auto_feed_result(url, **feed_args)
  feed = feed_result.to_json_feed
  {
    strategy: plan,
    items: feed[:items] || [],
    channel_title: feed[:title],
    admission_drops: feed_result.status.admission_drops
  }
end