Class: Crawlr::Context

Inherits:
Object
  • Object
show all
Defined in:
lib/crawlr/context.rb

Overview

The Context class holds metadata and shared data during a scraping session, such as URLs and crawl depth.

It acts like a small key-value store (@data) and provides helper methods to manage depth and resolve relative URLs.

Examples:

Creating a new context

ctx = Crawlr::Context.new(base_url: "https://example.com")
ctx[:title] = "Home"
ctx.increment_depth
ctx.to_h
# => { base_url: "https://example.com", page_url: nil, current_depth: 1, title: "Home" }

Since:

  • 0.1.0

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(base_url: nil, page_url: nil, current_depth: 0) ⇒ Context

Create a new scraping context.

Parameters:

  • base_url (String, nil) (defaults to: nil) —

    The root URL of the crawl

  • page_url (String, nil) (defaults to: nil) —

    The current page URL

  • current_depth (Integer) (defaults to: 0) —

    The crawl depth (default: 0)

Since:

  • 0.1.0



28
29
30
31
32
33
 
# File 'lib/crawlr/context.rb', line 28

def initialize(base_url: nil, page_url: nil, current_depth: 0)
  @base_url = base_url
  @page_url = page_url
  @current_depth = current_depth
  @data = {}
end

Instance Attribute Details

#base_url ⇒ String, ...

Returns:

  • (String, nil) —

    The base URL used for resolving relative links

  • (String, nil) —

    The current page URL

  • (Integer) —

    The current depth in the crawl hierarchy

Since:

  • 0.1.0



21
22
23
 
# File 'lib/crawlr/context.rb', line 21

def base_url
  @base_url
end

#current_depth ⇒ String, ...

Returns:

  • (String, nil) —

    The base URL used for resolving relative links

  • (String, nil) —

    The current page URL

  • (Integer) —

    The current depth in the crawl hierarchy

Since:

  • 0.1.0



21
22
23
 
# File 'lib/crawlr/context.rb', line 21

def current_depth
  @current_depth
end

#page_url ⇒ String, ...

Returns:

  • (String, nil) —

    The base URL used for resolving relative links

  • (String, nil) —

    The current page URL

  • (Integer) —

    The current depth in the crawl hierarchy

Since:

  • 0.1.0



21
22
23
 
# File 'lib/crawlr/context.rb', line 21

def page_url
  @page_url
end

Instance Method Details

#[](key) ⇒ Object?

Retrieve a stored value by key.

Parameters:

  • key (Symbol, String) —

    The key to fetch

Returns:

  • (Object, nil) —

    The stored value, or nil if not found

Since:

  • 0.1.0



39
40
41
 
# File 'lib/crawlr/context.rb', line 39

def [](key)
  @data[key]
end

#[]=(key, value) ⇒ Object

Assign a value to a key.

Parameters:

  • key (Symbol, String) —

    The key to set

  • value (Object) —

    The value to store

Returns:

  • (Object) —

    The stored value

Since:

  • 0.1.0



48
49
50
 
# File 'lib/crawlr/context.rb', line 48

def []=(key, value)
  @data[key] = value
end

#increment_depth ⇒ Integer

Increment the crawl depth by 1.

Returns:

  • (Integer) —

    The updated depth value

Since:

  • 0.1.0



68
69
70
 
# File 'lib/crawlr/context.rb', line 68

def increment_depth
  @current_depth += 1
end

#resolve_url(url) ⇒ String

Resolve a relative URL using the base_url.

Parameters:

  • url (String) —

    The relative or absolute URL

Returns:

  • (String) —

    The resolved absolute URL

Since:

  • 0.1.0



76
77
78
 
# File 'lib/crawlr/context.rb', line 76

def resolve_url(url)
  URI.join(@base_url, url).to_s
end

#to_h ⇒ Hash

Convert the context to a Hash.

Includes base_url, page_url, current_depth, and all stored data.

Returns:

  • (Hash) —

    The full context data as a Hash

Since:

  • 0.1.0



57
58
59
60
61
62
63
 
# File 'lib/crawlr/context.rb', line 57

def to_h
  {
    base_url: @base_url,
    page_url: @page_url,
    current_depth: @current_depth
  }.merge(@data)
end