Class: Crawlr::HTTPInterface

Inherits:
Object
  • Object
show all
Defined in:
lib/crawlr/http_interface.rb

Overview

Handles fetching documents via async HTTP with proxy and cookie support.

The HTTPInterface class provides a high-level async HTTP client specifically designed for web scraping. It supports proxy rotation, cookie management, configurable timeouts, and transforms raw HTTP responses into a simplified response structure suitable for content processing.

Examples:

Basic HTTP fetching

config = Crawlr::Config.new(timeout: 10)
http = Crawlr::HTTPInterface.new(config)

response = http.get('https://example.com')
puts response.status  #=> 200
puts response.body    #=> HTML content

With cookie support

config = Crawlr::Config.new(allow_cookies: true)
http = Crawlr::HTTPInterface.new(config)

# Cookies are automatically managed across requests
 = http.get('https://site.com/login')
profile_response = http.get('https://site.com/profile')  # Uses login cookies

With proxy rotation

config = Crawlr::Config.new(
  proxies: ['http://proxy1:8080', 'socks5://proxy2:1080'],
  proxy_strategy: :round_robin
)
http = Crawlr::HTTPInterface.new(config)

response = http.get('https://example.com')  # Uses proxy1
response = http.get('https://example.com')  # Uses proxy2

With request hooks

response = http.get('https://api.example.com') do |url, headers|
  headers['Authorization'] = "Bearer #{get_token()}"
  headers['X-Request-ID'] = SecureRandom.uuid
end

Since:

  • 0.1.0

Defined Under Namespace

Classes: Response

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(config) ⇒ HTTPInterface

Initializes a new HTTPInterface with the given configuration

Sets up cookie management (if enabled) and proxy rotation state. The cookie jar persists across all requests made by this interface instance.

Examples:

config = Crawlr::Config.new(
  allow_cookies: true,
  timeout: 15,
  proxies: ['http://proxy.example.com:8080']
)
http = Crawlr::HTTPInterface.new(config)

Parameters:

Options Hash (config):

  • :allow_cookies (Boolean) —

    Enable cookie jar management

  • :proxies (Array<String>) —

    List of proxy URLs

  • :proxy_strategy (Symbol) —

    Proxy selection strategy (:round_robin, :random)

  • :timeout (Integer) —

    Request timeout in seconds

  • :headers (Hash) —

    Default headers for all requests

Since:

  • 0.1.0



86
87
88
89
90
 
# File 'lib/crawlr/http_interface.rb', line 86

def initialize(config)
  @config = config
  @cookie_jars = Concurrent::Map.new if @config.allow_cookies
  @proxy_index = 0
end

Instance Attribute Details

#config ⇒ Crawlr::Config (readonly)

Returns Configuration object containing HTTP settings.

Returns:

Since:

  • 0.1.0



65
66
67
 
# File 'lib/crawlr/http_interface.rb', line 65

def config
  @config
end

Instance Method Details

#get(url) {|url, headers| ... } ⇒ HTTPInterface::Response

Performs an HTTP GET request with full async support and cookie management

This method handles the complete HTTP request lifecycle including:

  • Proxy selection and connection setup
  • Cookie retrieval and attachment
  • Request header customization via block
  • Async execution with timeout handling
  • Response cookie parsing and storage
  • Resource cleanup and connection closing

Examples:

Basic GET request

response = http.get('https://example.com/api/data')
if response.status == 200
  data = JSON.parse(response.body)
end

With custom headers

response = http.get('https://api.service.com/endpoint') do |url, headers|
  headers['Accept'] = 'application/json'
  headers['X-API-Key'] = ENV['API_KEY']
  headers['User-Agent'] = 'MyBot/1.0'
end

With authentication

response = http.get('https://secure.site.com/data') do |url, headers|
  token = authenticate_user(url)
  headers['Authorization'] = "Bearer #{token}"
end

Error handling

begin
  response = http.get('https://unreliable.com/data')
rescue Async::TimeoutError
  puts "Request timed out"
rescue StandardError => e
  puts "Request failed: #{e.message}"
end

Parameters:

  • url (String) —

    The URL to fetch

  • block (Proc) —

    Optional block for request customization

Yield Parameters:

  • url (String) —

    The URL being requested

  • headers (Hash) —

    Mutable headers hash for customization

Returns:

Raises:

  • (Async::TimeoutError) —

    When request exceeds configured timeout

  • (URI::InvalidURIError) —

    When URL is malformed

  • (StandardError) —

    For other HTTP-related errors

Since:

  • 0.1.0



138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/crawlr/http_interface.rb', line 138

def get(url) # rubocop:disable Metrics/MethodLength
  Crawlr.logger.debug "Fetching #{url}"

  uri = URI.parse(url)
  proxy_url = next_proxy
  internet = build_internet_connection(proxy_url)

  request_headers = @config.headers.dup
  handle_cookies(uri, request_headers)

  yield(url, request_headers) if block_given? # Used for request customization hook

  raw_response = nil
  begin
    Sync do |task|
      raw_response = task.with_timeout(@config.timeout) do
        internet.get(url, request_headers)
      end
    end

    parse_and_set_cookies(uri, raw_response) if @config.allow_cookies && raw_response
    make_response_struct(url, raw_response)
  rescue Async::TimeoutError
    Crawlr.logger.warn "Timeout fetching #{url} after #{@config.timeout}sec"
    raise
  ensure
    raw_response&.close
    internet&.close
    Crawlr.logger.debug "Done fetching #{url}"
  end
end