Class: Html2rss::AutoSource

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source.rb,
lib/html2rss/auto_source/cleanup.rb,
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/segment.rb,
lib/html2rss/auto_source/segmenter.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/segmenter/list.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/segmenter/cluster.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/segmenter/semantic.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/native_feed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/xhr_articles.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/segmenter/primary_link.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The AutoSource class automatically extracts articles from a given URL by utilizing a collection of Scrapers. These scrapers analyze and parse popular structured data formats—such as schema, microdata, and open graph—to identify and compile article elements into unified articles.

Scrapers supporting plain HTML are also available for sites without structured data, though results may vary based on page markup.

See Also:

Defined Under Namespace

Modules: Scraper Classes: Cleanup, Segment, Segmenter

Constant Summary collapse

DEFAULT_LIMIT =

Default max articles to keep (also the short-circuit floor across scraper tiers).

25
OPERATIONAL_ERRORS =

Operational scraper failures quarantined within a tier; programmer errors propagate.

[
  Html2rss::Error,
  ArgumentError,
  ::JSON::ParserError,
  ::Nokogiri::SyntaxError
].freeze
DEFAULT_CONFIG =

Default auto-source configuration shipped for scraper and cleanup behavior.

{
  limit: DEFAULT_LIMIT,
  entry_resolution: {
    enabled: true,
    max_probes: 5
  }.freeze,
  scraper: {
    native_feed: {
      enabled: true
    },
    wordpress_api: {
      enabled: true
    },
    sitemap: {
      enabled: true,
      min_priority: Scraper::Sitemap::Parser::DEFAULT_MIN_PRIORITY,
      max_age_days: Scraper::Sitemap::Parser::DEFAULT_MAX_AGE_DAYS
    },
    schema: {
      enabled: true
    },
    microdata: {
      enabled: true
    },
    microformats2: {
      enabled: true
    },
    json_state: {
      enabled: true
    },
    xhr_articles: {
      enabled: true
    },
    meta_oembed: {
      enabled: true
    },
    semantic_html: {
      enabled: true,
      fallback_anchorless: true
    },
    html: {
      enabled: true,
      minimum_selector_frequency: Scraper::Html::DEFAULT_MINIMUM_SELECTOR_FREQUENCY,
      use_top_selectors: Scraper::Html::DEFAULT_USE_TOP_SELECTORS,
      fallback_anchorless: true
    }
  },
  cleanup: Cleanup::DEFAULT_CONFIG
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(response, opts = DEFAULT_CONFIG, request_session: nil) ⇒ void

Parameters:

Options Hash (opts):

  • :limit (Integer) —

    max articles to keep; later tiers stop once this many survive Cleanup

  • :scraper (Hash) —

    scraper configuration map

  • :cleanup (Hash) —

    cleanup configuration map



112
113
114
115
116
117
118
119
 
# File 'lib/html2rss/auto_source.rb', line 112

def initialize(response, opts = DEFAULT_CONFIG, request_session: nil)
  @parsed_body = response.parsed_body
  @body = response.body
  @url = response.url
  @captured_responses = response.captured_responses
  @opts = opts
  @request_session = request_session
end

Class Method Details

.request_slots_for(config) ⇒ Integer

Returns the sum of required request slots for all enabled scrapers in the config.

Parameters:

  • config (Hash, nil) —

    auto_source configuration hash

Returns:

  • (Integer) —

    total request slots required by scrapers



90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/html2rss/auto_source.rb', line 90

def request_slots_for(config)
  return 0 unless config

  Scraper::SCRAPERS.sum do |scraper|
    if config.dig(:scraper, scraper.options_key, :enabled)
      opts = config.dig(:scraper, scraper.options_key)
      scraper.respond_to?(:request_slots) ? scraper.request_slots(opts) : 0
    else
      0
    end
  end
end

Instance Method Details

#admission_drops ⇒ Hash{String => Integer}

Reason → count tallies from the final Cleanup pass (empty until #articles runs).

Returns:

  • (Hash{String => Integer}) 


143
144
145
 
# File 'lib/html2rss/auto_source.rb', line 143

def admission_drops
  @admission_drops || {}
end

#articles ⇒ Array<Html2rss::Article>

Extracts articles by running scraper tiers until a sufficient set is found.

Tiers: in-page structured → follow-up IO → SemanticHtml → Html. SST is built only when a heuristic tier runs. Later tiers are skipped once limit articles with url+title remain after Cleanup; the result is capped to limit. Html is skipped when earlier tiers already admitted at least one clean article below limit — quality over padding with weaker heuristic junk.

Returns:



131
132
133
134
135
136
137
 
# File 'lib/html2rss/auto_source.rb', line 131

def articles
  @articles ||= extract_articles
rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
  Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
  @admission_drops = {}.freeze
  []
end