Class: Html2rss::FeedPipeline

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/feed_pipeline.rb,
lib/html2rss/feed_pipeline/auto_fallback.rb,
lib/html2rss/feed_pipeline/strategy_plan.rb,
lib/html2rss/feed_pipeline/runtime_policy.rb

Overview

Builds feeds from validated config through request, extraction, and rendering stages.

Defined Under Namespace

Classes: AutoFallback, PipelineOutcome, RuntimePolicy, StrategyPlan

Instance Method Summary collapse

Constructor Details

#initialize(raw_config) ⇒ FeedPipeline

Returns a new instance of FeedPipeline.

Parameters:

  • raw_config (Hash{Symbol => Object}) —

    user-provided feed config



20
21
22
 
# File 'lib/html2rss/feed_pipeline.rb', line 20

def initialize(raw_config)
  @raw_config = raw_config
end

Instance Method Details

#deduplicated_articles(config:, response:, request_session:) ⇒ Array(Array<Html2rss::Article>, Integer, Hash)

Returns unique articles, dedup drops, admission drops.

Parameters:

Returns:

  • (Array(Array<Html2rss::Article>, Integer, Hash)) —

    unique articles, dedup drops, admission drops



75
76
77
78
79
 
# File 'lib/html2rss/feed_pipeline.rb', line 75

def deduplicated_articles(config:, response:, request_session:)
  collected, admission_drops = collect_articles(config:, response:, request_session:)
  unique = Article::Deduplicator.new(collected).call
  [unique, collected.size - unique.size, admission_drops]
end

#request_session_for(config, strategy:, resources:, scrape_url: nil) ⇒ Html2rss::RequestSession

Parameters:

Returns:



60
61
62
63
64
65
66
67
68
 
# File 'lib/html2rss/feed_pipeline.rb', line 60

def request_session_for(config, strategy:, resources:, scrape_url: nil)
  RequestSession.build(
    config:,
    strategy:,
    budget: resources.budget,
    policy: resources.policy,
    scrape_url:
  )
end

#to_outcome ⇒ PipelineOutcome

Runs the pipeline once and returns scrape-finished outcome (before Channel/Status). Used by Capture which needs the response body for SST selector derivation.

Returns:



29
30
31
32
 
# File 'lib/html2rss/feed_pipeline.rb', line 29

def to_outcome
  config = Config.from_hash(raw_config, params: raw_config[:params])
  pipeline_outcome_for(config)
end

#to_outcome_and_result ⇒ Array(PipelineOutcome, FeedResult)

Runs the pipeline once and returns scrape outcome plus Html2rss::FeedResult without a second fetch.

Returns:



38
39
40
41
42
 
# File 'lib/html2rss/feed_pipeline.rb', line 38

def to_outcome_and_result
  config = Config.from_hash(raw_config, params: raw_config[:params])
  outcome = pipeline_outcome_for(config)
  [outcome, feed_result_for(config, outcome)]
end

#to_result ⇒ Html2rss::FeedResult

Runs the pipeline once and returns an opaque, Marshal-cacheable result.

Returns:



48
49
50
51
52
 
# File 'lib/html2rss/feed_pipeline.rb', line 48

def to_result
  config = Config.from_hash(raw_config, params: raw_config[:params])
  outcome = pipeline_outcome_for(config)
  feed_result_for(config, outcome)
end