Making POST/PUT Requests with ScraperAPI in Ruby

Learn how to send POST and PUT requests using ScraperAPI via sync, proxy, and async modes. Scrape forms, APIs, and handle raw or form data with full control.

Some advanced users may want to send POST/PUT requests in order to scrape forms and API endpoints directly. You can do this by sending a POST/PUT request through ScraperAPI. The return value will be stringified. So if you want to use it as JSON, you will need to parse it into a JSON object.

• API ENDPOINT REQUEST

require 'net/http'
require 'json'

## Replace POST with PUT to send a PUT request instead
params = {
:api_key => "APIKEY",
:url => "http://httpbin.org/anything"
}

uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)

website_content = Net::HTTP.post(uri, { "foo" => "bar"}.to_json, "Content-Type" => "application/json")
print(website_content.body)

## For form data
params = {
:api_key => "APIKEY",
:url => "http://httpbin.org/anything"
}

uri = URI('http://api.scraperapi.com/')
uri.query = URI.encode_www_form(params)

req = Net::HTTP::Post.new(uri)
req.set_form_data('foo' => 'bar')

website_content = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}

print(website_content.body)

• PROXY MODE

require 'httparty'
HTTParty::Basement.default_options.update(verify: false)
query = {'foo': 'bar'}

## POST/PUT Request
headers = {'Accept':'application/json', 'Content-Type':'application/json'}

response = HTTParty.post('http://httpbin.org/anything', {
  http_proxyaddr: "proxy-server.scraperapi.com",
  http_proxyport: "8001",
  http_proxyuser: "scraperapi",
  http_proxypass: "APIKEY",
  body: query.to_json,
  headers: headers
}, )
results = response.body
puts results 

## Form POST Request
headers = {'Accept':'application/json'}

response = HTTParty.post('http://httpbin.org/anything', {
  http_proxyaddr: "proxy-server.scraperapi.com",
  http_proxyport: "8001",
  http_proxyuser: "scraperapi",
  http_proxypass: "APIKEY",
  body: query.to_json,
  headers: headers
}, )
results = response.body
puts results 

• SDK Method

# remember to install the library: gem install scraperapi
require "scraper_api"
client = ScraperAPI::Client.new("APIKEY")

#POST/PUT Requests
postResult = client.post("http://httpbin.org/ip", "foo": "bar").raw_body
putResult = client.put("http://httpbin.org/ip", "foo": "bar").raw_body

puts postResult
puts putResult

#Form Post Requests
formResult = client.post("http://httpbin.org/anything", body: {"foo": "bar"}, headers: {"Accept":"application/json"}).raw_body
puts formResult

• ASYNC Method

require 'net/http'
require 'json'


uri = URI('https://async.scraperapi.com/jobs')
body = {
 "apiKey" => "APIKEY",
 "url" => "https://httpbin.org/anything",
 "method" =>  "POST",
 "headers" => {"content-type" => "application/x-www-form-urlencoded"} ,
 "body" => "foo=bar"
}

response = Net::HTTP.post(uri, body.to_json, "Content-Type" => "application/json")
print(response.body)