Class: Html2rss::AutoSource::Cleanup

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source/cleanup.rb

Overview

Cleanup is responsible for cleaning up the extracted articles. :reek:MissingSafeMethod { enabled: false } It applies various strategies to filter and refine the article list. Sole producer of admission drop tallies for Status.

Defined Under Namespace

Classes: AuditResult, Result

Constant Summary collapse

DEFAULT_CONFIG =

Default cleanup behavior for auto-sourced article lists.

{
  keep_different_domain: false
}.freeze
MIN_WORDS =

Minimum alphanumeric word count for present titles.

3
VALID_SCHEMES =

Allowed URL schemes for article filtering.

%w[http https].to_set.freeze
MIN_TITLE_LENGTH =

Minimum present-title length before short-title warnings.

4

Class Method Summary collapse

Class Method Details

.audit_feed_items(items) ⇒ AuditResult

Parameters:

  • items (Array) —

    RSS or article-like objects with title and link/url

Returns:



102
103
104
105
106
107
108
109
110
 
# File 'lib/html2rss/auto_source/cleanup.rb', line 102

def audit_feed_items(items)
  metrics = initial_audit_metrics(items.size)
  violations = Hash.new(0)
  warnings = audit_url_diversity(items, metrics, violations)
  audit_item_titles(items, metrics, violations)
  warnings = finalize_audit_warnings(warnings, metrics)
  log_audit(metrics, warnings, violations)
  AuditResult.new(warnings:, metrics: metrics.freeze, violations: violations.freeze)
end

.call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) ⇒ Result

Returns cleaned articles and frozen drop tallies.

Parameters:

  • articles (Array<Article>) —

    extracted article candidates

  • url (Html2rss::Url) —

    feed source URL used for same-host filtering

  • keep_different_domain (Boolean) (defaults to: DEFAULT_CONFIG.fetch(:keep_different_domain)) —

    whether to keep off-domain entries

Returns:

  • (Result) —

    cleaned articles and frozen drop tallies



71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/html2rss/auto_source/cleanup.rb', line 71

def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) # rubocop:disable Metrics/MethodLength -- ordered reject pipeline
  Log.debug "Cleanup: start with #{articles.size} articles"
  tallies = Hash.new(0)

  reject_invalid!(articles, tallies)
  deduplicate_by_url!(articles, tallies)
  keep_only_http_urls!(articles, tallies)
  reject_self_links!(articles, url, tallies)
  reject_different_domain!(articles, url, tallies) unless keep_different_domain
  reject_excluded_destinations!(articles, tallies)
  reject_low_quality_titles!(articles, tallies)

  Log.debug "Cleanup: end with #{articles.size} articles"
  Result.new(articles:, drop_tallies: tallies.freeze)
end

.junk_reason(title) ⇒ Symbol?

First matching junk reason for a title, or nil when the title is acceptable.

Parameters:

  • title (String, nil) —

    candidate title text

Returns:

  • (Symbol, nil) 


91
92
93
94
95
96
97
98
 
# File 'lib/html2rss/auto_source/cleanup.rb', line 91

def junk_reason(title)
  return if title.nil?

  normalized = normalize_title(title)
  return if normalized.empty?

  JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
end