Making POST/PUT Requests with ScraperAPI in Python

Learn how to send POST and PUT requests using ScraperAPI via sync, proxy, and async modes. Scrape forms, APIs, and handle raw or form data with full control.

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API ENDPOINT REQUEST

import requests
payload = {'api_key': 'APIKEY', 'url': 'https://httpbin.org/anything'}

r = requests.post('http://api.scraperapi.com', params=payload, data={'foo': 'bar'})
print(r.text)

• PROXY MODE

import requests
proxies = {
  "http": "http://scraperapi:[e-mail]:8001"
}

r = requests.post('http://httpbin.org/anything', proxies=proxies, data={'foo': 'bar'}, verify=False)
print(r.text)

• SDK Method

# remember to install the library: pip install scraperapi-sdk
  from scraperapi_sdk import ScraperAPIClient
  client = ScraperAPIClient('APIKEY')
  
  postResult = client.post(url = 'http://httpbin.org/anything', body = {'foo': 'bar'}}).text
  putResult = client.put(url = 'http://httpbin.org/anything', body = {'foo': 'bar'}}).text
  
  print(postResult)
  print(putResult)

• ASYNC Method

import requests

body = {'apiKey': 'APIKEY', 'url': 'https://httpbin.org/anything', 'method': 'POST', 'headers': {"content-type": "application/x-www-form-urlencoded"},  'body': 'foo=bar'}

r = requests.post('https://async.scraperapi.com/jobs', headers={"Content-Type": "application/json"},  json=body)
print(r.text)