Class: Html2rss::Capture

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/capture.rb

Overview

Analyzes a URL and produces a durable feed config: items selector + metadata.

Fetches via FeedPipeline (including AutoFallback for :auto), extracts articles, derives reusable items CSS selectors from SST segments, and infers directory catalog metadata.

Defined Under Namespace

Classes: CaptureResult

Constant Summary collapse

SCHEMA_MODELINE =

Packaged YAML language server modeline.

'# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json'
SEGMENT_STRATEGIES =

Ordered Segmenter strategies tried until the items selector quality gate passes.

%i[list cluster semantic].freeze
MIN_SELECTOR_MATCHES =

Minimum matched segment/article pairs required to emit an items selector.

2

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(url, strategy: :auto, **options) ⇒ Capture

Returns a new instance of Capture.

Parameters:

  • url (String) —

    source page URL

  • strategy (Symbol) (defaults to: :auto) —

    request strategy

  • options (Hash) —

    a customizable set of options

Options Hash (**options):

  • :items_selector (String, nil) —

    optional selector hint

  • :topics (Array<String>, nil) —

    optional directory topics override

  • :title (String, nil) —

    optional title override

  • :summary (String, nil) —

    optional summary override

  • :force (Boolean) —

    whether to ignore native feed detection

  • :enhance (Boolean, nil) —

    whether to force enhance: true

  • :max_redirects (Integer, nil) —

    optional redirect limit override

  • :max_requests (Integer, nil) —

    optional request budget override

  • :limit (Integer, nil) —

    max articles to keep

  • :local_file_path (String, nil) —

    optional local HTML file path



108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/html2rss/capture.rb', line 108

def initialize(url, strategy: :auto, **options) # rubocop:disable Metrics/MethodLength
  @url = url
  @strategy = strategy
  @items_selector_hint = options.delete(:items_selector)
  @topics = options.delete(:topics)
  @title = options.delete(:title)
  @summary = options.delete(:summary)
  @force = options.delete(:force) || false
  @enhance_option = options.delete(:enhance)
  @max_redirects = options.delete(:max_redirects)
  @max_requests = options.delete(:max_requests)
  @limit = options.delete(:limit)
  @local_file_path = options.delete(:local_file_path)
end

Class Method Details

.build(url, strategy: :auto) ⇒ CaptureResult

Analyzes a URL and builds a reusable feed config.

Parameters:

  • url (String) —

    source page URL

  • strategy (Symbol) (defaults to: :auto) —

    request strategy (+:auto+, :default, :botasaurus)

  • options (Hash) —

    a customizable set of options

Returns:



90
91
92
 
# File 'lib/html2rss/capture.rb', line 90

def build(url, strategy: :auto, **)
  new(url, strategy:, **).build
end

Instance Method Details

#build ⇒ CaptureResult

Runs the capture pipeline.

Returns:



127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/html2rss/capture.rb', line 127

def build # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
  outcome = FeedPipeline.new(raw_config).to_outcome
  @admission_drops = outcome.admission_drops
  selectors, segment_strategy = derive_selectors(outcome.response, outcome.articles)
  ch_title = @title || channel_title_from(outcome.response)
  topics = @topics || infer_topics("#{@url} #{ch_title}")
  native_feed = probe_native_feed(outcome.response) unless @force

  config = {
    directory: build_directory(ch_title, topics),
    channel: build_channel(outcome.response, ch_title),
    selectors: selectors.empty? ? nil : selectors,
    **strategy_stamp(outcome),
    **local_file_request_overlay
  }.compact

  CaptureResult.new(
    config:,
    yaml: "#{SCHEMA_MODELINE}\n#{Config.to_yaml(config)}",
    articles_count: outcome.articles.size,
    channel_title: ch_title,
    has_selectors: !selectors.empty?,
    segment_strategy:,
    admission_drops: outcome.admission_drops,
    selected_strategy: outcome.selected_strategy,
    inferred_topics: topics,
    native_feed:,
    suggested_channel_url: suggested_channel_url(outcome)
  )
end