Parameters as Headers

Learn how to scrape websites using header-based parameters with ScraperAPI in Ruby. Pass api_key, render, and instruction_set via headers to simplify requests.

Along with the “traditional” means of passing parameters, we also support passing parameters as headers. Passing parameters such as api_key, render, ultra_premium and instruction_set is very straightforward.

• API REQUEST

Instead of including the parameters in the URL

require 'net/http'
require 'uri'
require 'json'

# Define parameters including headers
params = {
  api_key: "<YOUR_API_KEY>",
  url: "http://httpbin.org/ip",
  render: true
}

# URI for the API endpoint (note: use HTTPS)
uri = URI('https://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)

# Create a GET request
req = Net::HTTP::Get.new(uri)
req['Accept'] = 'application/json'
req['X-MyHeader'] = '123'

# Create an HTTPS connection
http = Net::HTTP.new(uri.hostname, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

# Perform the HTTPS request
website_content = http.request(req)

# Output the response body
puts website_content.body

you can just pass them as headers

require 'net/http'
require 'json'

# Define parameters including the target URL
params = {
  url: "https://httpbin.org/ip"  # Specify the URL you want to scrape
}

# Set the API endpoint URI (HTTPS)
uri = URI('https://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)  # Encode parameters into the query string

# Create a new GET request
req = Net::HTTP::Get.new(uri)

# Set Scraper API headers
req['x-sapi-render'] = 'true'  # Enable rendering
req['x-sapi-api_key'] = '<YOUR_API_KEY>'  # Replace with your actual Scraper API key

# Create an HTTPS connection
http = Net::HTTP.new(uri.hostname, uri.port)
http.use_ssl = true  # Enable SSL/TLS encryption
http.verify_mode = OpenSSL::SSL::VERIFY_PEER  # Verify the server's certificate

# Perform the HTTPS request and store the response
website_content = http.request(req)

# Output the response body (the scraped content)
puts website_content.body

Please note that the 'x-sapi-' prefix is used on each header to avoid collisions with headers used by target sites. We support all standard parameters available with the API. The instruction_set parameter is specifically supported only through headers.

• PROXY MODE

require 'httparty'

# Define headers with the required parameters
headers = {
  "x-sapi-render" => "true"
}

# Set default options for HTTParty, disabling SSL verification
HTTParty::Basement.default_options.update(verify: false)

# Make the HTTP request with HTTParty
response = HTTParty.get('http://httpbin.org/ip', {
  http_proxyaddr: "proxy-server.scraperapi.com",
  http_proxyport: "8001",
  http_proxyuser: "scraperapi",
  http_proxypass: "<YOUR_API_KEY>",
  headers: headers
})

# Capture the response body
results = response.body

# Output the response body
puts results

Note that credentials must still be passed to the proxy in the manner shown above, not as headers.