Class: Firecrawl::ScrapeRequest

Inherits:
Request
  • Object
show all
Defined in:
lib/firecrawl/scrape_request.rb

Overview

The ScrapeRequest class encapsulates a ‘/scrape’ POST request to the Firecrawl API. After creating a new ScrapeRequest instance you can initiate the request by calling the submit method to perform synchronous scraping.

examples

require ‘firecrawl’

request = Firecrawl::ScrapeRequest.new( api_key: ENV[ ‘FIRECRAWL_API_KEY’ )

options = Firecrawl::ScrapeOptions.build do

format                [ :markdown, 'screenshot@full_page' ]
only_main_content     true

end

response = request.submit( ‘example.com’, options ) if response.success?

result = response.result 
puts response.metadata[ 'title ] 
puts '---'
puts response.markdown

else

puts response.result.error_description

end

Constant Summary

Constants inherited from Request

Request::BASE_URI

Instance Method Summary collapse

Methods inherited from Request

#initialize

Constructor Details

This class inherits a constructor from Firecrawl::Request

Instance Method Details

#submit(url, options = nil, &block) ⇒ Object

The submit method makes a Firecrawl ‘/scrape’ POST request which will scrape the given url.

The response is always an instance of Faraday::Response. If response.success? is true, then response.result will be an instance ScrapeResult. If the request is not successful then response.result will be an instance of ErrorResult.



37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/firecrawl/scrape_request.rb', line 37

def submit( url, options = nil, &block )    
  if options
    options = options.is_a?( ScrapeOptions ) ? options : ScrapeOptions.build!( options.to_h ) 
    options = ScrapeOptions.normalize_options( options.to_h )
  else 
    options = {}
  end
  options[ :url ] = url.to_s

  response = post( "#{BASE_URI}/scrape", options, &block )
  result = nil 
  if response.success?  
    attributes = ( JSON.parse( response.body, symbolize_names: true ) rescue nil )
    result = ScrapeResult.new( attributes )
  else 
    result = ErrorResult.new( response.status, attributes )
  end
  
  ResponseMethods.install( response, result )     
end