Class: ProxyFetcher::Providers::ProxyscrapeHTTP

Inherits:
Base
  • Object
show all
Defined in:
lib/proxy_fetcher/providers/proxyscrape_http.rb

Overview

FreeProxyList provider class.

Instance Method Summary collapse

Methods inherited from Base

#fetch_proxies, fetch_proxies!, #provider_headers, #provider_method, #provider_params, #xpath

Instance Method Details

#load_document(url, filters = {}) ⇒ Array

Loads provider HTML and parses it with internal document object.

Parameters:

  • url (String) —

    URL to fetch

  • filters (Hash) (defaults to: {}) —

    filters for proxy provider

Returns:

  • (Array) —

    Collection of extracted proxies with ports



25
26
27
28
29
 
# File 'lib/proxy_fetcher/providers/proxyscrape_http.rb', line 25

def load_document(url, filters = {})
  html = load_html(url, filters)

  CSV.parse(html, col_sep: "\t").map(&:first)
end

#load_proxy_list(filters = {}) ⇒ Array

Fetches HTML content by sending HTTP request to the provider URL and parses the txt document to return all the proxy entries (ip addresses and ports).

Returns:

  • (Array) —

    Collection of extracted proxies with ports



38
39
40
 
# File 'lib/proxy_fetcher/providers/proxyscrape_http.rb', line 38

def load_proxy_list(filters = {})
  load_document(provider_url, filters)
end

#provider_url ⇒ Object

Provider URL to fetch proxy list



10
11
12
 
# File 'lib/proxy_fetcher/providers/proxyscrape_http.rb', line 10

def provider_url
  "https://api.proxyscrape.com/v2/?request=getproxies&protocol=http"
end

#to_proxy(node) ⇒ ProxyFetcher::Proxy

Converts String to ProxyFetcher::Proxy object.

Parameters:

  • node (String) —

    String

Returns:



50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/proxy_fetcher/providers/proxyscrape_http.rb', line 50

def to_proxy(node)
  addr, port = node.split(":")

  ProxyFetcher::Proxy.new.tap do |proxy|
    proxy.addr = addr
    proxy.port = Integer(port)
    proxy.country = "Unknown"
    proxy.anonymity = "Unknown"
    proxy.type = ProxyFetcher::Proxy::HTTP
  end
end