Customize Header Parameter with ScraperAPI in Python

Learn how to scrape websites using header-based parameters with ScraperAPI in Python. Pass api_key, render, and instruction_set via headers to simplify requests.

Along with the “traditional” means of passing parameters, we also support passing parameters as headers. Passing parameters such as api_key, render, ultra_premium and instruction_set are very straightforward.

• API REQUEST

Instead of including them in the URL

import requests
payload = {'api_key': 'APIKEY', 'url':'https://httpbin.org/ip', 'render': 'true'}
r = requests.get('https://api.scraperapi.com', params=payload)
print(r.text)

you can just pass the parameters as headers

import requests
payload = {'url': 'https://httpbin.org/ip'}
headers = {
    'x-sapi-render': 'true',
    'x-sapi-api_key': '<YOUR_API_KEY>'
}
r = requests.get('https://api.scraperapi.com', params=payload, headers=headers)
print(r.text)

Please note that the 'x-sapi-' prefix is used on each header to avoid collisions with headers used by target sites. We support all standard parameters available with the API. The instruction_set parameter is specifically supported only through headers.

• PROXY MODE

import requests
proxy_url = "http://scraperapi.render=true:<YOUR_API_KEY>@proxy-server.scraperapi.com:8001"
proxies = {
    "http": proxy_url,
    "https": proxy_url
}
r = requests.get('https://httpbin.org/ip', proxies=proxies, verify=False)
print(r.text)

Note that credentials must still be passed to the proxy in the manner shown above, not as headers.