Rendering Javascript

If you are crawling a page that requires you to render the Javascript on the page to scrape the data you need, then we can fetch these pages using a headless browser.

To render Javascript, simply set render=true and we will use a headless Google Chrome instance to fetch the page. This feature is available on all plans.

Pass the JavaScript rendering parameter within the URL:

• API REQUEST

require 'net/http'
require 'uri'
require 'json'

# Define parameters including headers
params = {
  api_key: "<YOUR_API_KEY>",
  url: "http://httpbin.org/ip",
  render: true
}

# URI for the API endpoint (note: use HTTPS)
uri = URI('https://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)

# Create a GET request
req = Net::HTTP::Get.new(uri)
req['Accept'] = 'application/json'
req['X-MyHeader'] = '123'

# Create an HTTPS connection
http = Net::HTTP.new(uri.hostname, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

# Perform the HTTPS request
website_content = http.request(req)

# Output the response body
puts website_content.body

• PROXY MODE

#remember to install the 'httparty' gem: gem install httparty
require 'httparty'
HTTParty::Basement.default_options.update(verify: false)
response = HTTParty.get('http://httpbin.org/ip', {
  http_proxyaddr: "proxy-server.scraperapi.com",
  http_proxyport: "8001",
  http_proxyuser: "scraperapi.render=true",
  http_proxypass: "APIKEY"
})
results = response.body
puts results

• SDK Method

# remember to install the library: gem install scraperapi (compatible with Ruby 2x)
require "scraper_api"
client = ScraperAPI::Client.new("APIKEY")
result = client.get("http://httpbin.org/ip", render: true).raw_body
puts result

Pass the parameter in the headers:

• API REQUEST

require 'net/http'
require 'json'

# Define parameters including the target URL
params = {
  url: "https://httpbin.org/ip"  # Specify the URL you want to scrape
}

# Set the API endpoint URI (HTTPS)
uri = URI('https://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)  # Encode parameters into the query string

# Create a new GET request
req = Net::HTTP::Get.new(uri)

# Set Scraper API headers
req['x-sapi-render'] = 'true'  # Enable rendering
req['x-sapi-api_key'] = '<YOUR_API_KEY>'  # Replace with your actual Scraper API key

# Create an HTTPS connection
http = Net::HTTP.new(uri.hostname, uri.port)
http.use_ssl = true  # Enable SSL/TLS encryption
http.verify_mode = OpenSSL::SSL::VERIFY_PEER  # Verify the server's certificate

# Perform the HTTPS request and store the response
website_content = http.request(req)

# Output the response body (the scraped content)
puts website_content.body

• PROXY MODE

require 'httparty'

# Define headers with the required parameters
headers = {
  "x-sapi-render" => "true"
}

# Set default options for HTTParty, disabling SSL verification
HTTParty::Basement.default_options.update(verify: false)

# Make the HTTP request with HTTParty
response = HTTParty.get('http://httpbin.org/ip', {
  http_proxyaddr: "proxy-server.scraperapi.com",
  http_proxyport: "8001",
  http_proxyuser: "scraperapi",
  http_proxypass: "<YOUR_API_KEY>",
  headers: headers
})

# Capture the response body
results = response.body

# Output the response body
puts results

• SDK Method

require "scraper_api"
client = ScraperAPI::Client.new("<YOUR_API_KEY>")
headers = {
  "x-sapi-render" => "true"
}
result = client.get("http://httpbin.org/ip", headers: headers).raw_body
puts result