Class: Html2rss::AutoSource::Scraper::Schema

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb

Overview

Scrapes articles from Schema.org objects, by looking for the objects in:

Returns a new instance of Schema.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

  • url (String, Html2rss::Url) —

    base page URL

  • opts (Hash) —

    scraper-specific options

Options Hash (**opts):

  • :_reserved (Object) —

    reserved for future scraper-specific options



163
164
165
166
167
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 163

def initialize(parsed_body, url:, **opts)
  @parsed_body = parsed_body
  @url = url
  @opts = opts
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Returns whether the page includes supported schema types.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Boolean) —

    whether the page includes supported schema types



52
53
54
55
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 52

def articles?(parsed_body)
  ::Html2rss::Html::Probe.scripts(parsed_body, ::Html2rss::Html::Probe::APPLICATION_LD_JSON)
                         .any? { |script| supported_schema_type?(script) }
end

.canonicalize_type(object) ⇒ String?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Canonical short Schema.org type name for a scalar wire form.

Parameters:

  • object (String, Symbol, nil) —

    raw @type / itemtype token

Returns:

  • (String, nil) 


126
127
128
129
130
131
132
133
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 126

def canonicalize_type(object)
  return unless object.is_a?(String) || object.is_a?(Symbol)

  short = object.to_s.sub(SCHEMA_ORG_PREFIX_RE, '')
  return if short.empty?

  CANONICAL_BY_DOWNCASE.fetch(::Html2rss::Html::Probe.fold(short), short)
end

.from(object) ⇒ Array<Hash>

Returns a flat array of all supported schema objects by recursively traversing the given object.

Prefers collection keys (COLLECTION_KEYS) when walking containers.

:reek:DuplicateMethodCall

Parameters:

  • object (Hash, Array, Nokogiri::XML::Element)

Returns:

  • (Array<Hash>) —

    the schema_objects, or an empty array



73
74
75
76
77
78
79
80
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 73

def from(object)
  case object
  when Nokogiri::XML::Element then from(parse_script_tag(object))
  when Hash then from_hash(object)
  when Array then object.flat_map { |item| from(item) }
  else []
  end
end

.normalize_types(object) ⇒ Set<String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Normalizes Schema.org @type wire forms to short type names.

Handles String, Symbol, Array, and strips https://schema.org/ / http://schema.org/ prefixes.

Parameters:

  • object (String, Symbol, Array, nil) —

    raw @type value

Returns:

  • (Set<String>) —

    short type names (e.g. "NewsArticle")



111
112
113
114
115
116
117
118
119
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 111

def normalize_types(object)
  case object
  when Array
    object.each_with_object(Set.new) { |item, set| set.merge(normalize_types(item)) }
  else
    name = canonicalize_type(object)
    name ? Set[name] : EMPTY_TYPES
  end
end

.options_key ⇒ Symbol

Returns scraper config key.

Returns:

  • (Symbol) —

    scraper config key



47
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 47

def self.options_key = :schema

.scraper_for_schema_object(schema_object) ⇒ Scraper::Schema::Thing, ...

Returns a class responding to #call.

Parameters:

  • schema_object (Hash{Symbol => Object}) —

    schema object with an @type key

Returns:



91
92
93
94
95
96
97
98
99
100
101
102
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 91

def scraper_for_schema_object(schema_object)
  types = normalize_types(schema_object[:@type])

  # Prefer article/list extractors over denied containers for multi-type @type arrays
  # (e.g. ["NewsArticle","WebPage"]).
  return ItemList if types.intersect?(ItemList::SUPPORTED_TYPES)
  return Thing if types.intersect?(Thing::SUPPORTED_TYPES)
  return nil if types.intersect?(DENIED_CONTAINER_TYPES)

  Log.debug("#{name}: unsupported schema object @type=#{schema_object[:@type].inspect}")
  nil
end

.supported_schema_object?(object) ⇒ Boolean

Returns whether an extractor exists for the candidate object.

Parameters:

  • object (Hash{Symbol => Object}) —

    schema candidate object

Returns:

  • (Boolean) —

    whether an extractor exists for the candidate object



84
85
86
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 84

def supported_schema_object?(object)
  scraper_for_schema_object(object) ? true : false
end

.supported_schema_type?(script) ⇒ Boolean

Returns whether the tag references a supported schema type.

Parameters:

  • script (Nokogiri::XML::Element) —

    schema JSON-LD script tag

Returns:

  • (Boolean) —

    whether the tag references a supported schema type



59
60
61
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 59

def supported_schema_type?(script)
  script.text.match?(SUPPORTED_TYPES_RE)
end

Instance Method Details

#each {|Hash| ... } ⇒ Array<Hash>

Returns the scraped article_hashes.

Yields:

  • (Hash) —

    Each scraped article_hash

Returns:

  • (Array<Hash>) —

    the scraped article_hashes



172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/html2rss/auto_source/scraper/schema.rb', line 172

def each
  return enum_for(:each) unless block_given?

  schema_objects.each do |schema_object|
    next unless (klass = self.class.scraper_for_schema_object(schema_object))
    next unless (results = klass.new(schema_object, url:).call)

    if results.is_a?(Array)
      results.each { yield(_1) }
    else
      yield(results)
    end
  end
end