LogoLogo
OverviewRelease NotesDataPipelineFAQs
NodeJS
NodeJS
  • Make Requests with ScraperAPI in NodeJS
    • Use ScraperAPI Endpoint in NodeJS
    • Use ScraperAPI Proxy Port in NodeJS
    • Use ScraperAPI SDK in NodeJS
    • Make Async Requests with ScraperAPI in NodeJS
      • How to Use ScraperAPI Async Web Scraping in NodeJS
      • Use Async ScraperAPI Callbacks in NodeJS
      • Configure ScraperAPI Parameters in NodeJS
      • Request Async Batch Scraping with ScraperAPI in NodeJS
      • Decode Base64 Async Responses in NodeJS
    • ScraperAPI Structured Data Collection in NodeJS
      • Amazon Product Page API: Structured Data in NodeJS
      • Amazon Search API: Structured Data in NodeJS
      • Amazon Offers API: Structured Data in NodeJS
      • Amazon Reviews API: Structured Data in NodeJS
      • Ebay Product Page API: Structured Data in NodeJS
      • Ebay Search API: Structured Data in NodeJS
      • Google SERP API: Structured Data in NodeJS
      • Google News API: Structured Data in NodeJS
      • Google Jobs API: Structured Data in NodeJS
      • Google Shopping API: Structured Data in NodeJS
      • Google Maps Search API: Structured Data in NodeJS
      • Redfin Agent Details API: Structured Data in NodeJS
      • Redfin 'For Rent' Listings API: Structured Data in NodeJS
      • Redfin 'For Sale' Listings API: Structured Data in NodeJS
      • Redfin Listing Search API: Structured Data in NodeJS
      • Walmart Search API: Structured Data in NodeJS
      • Walmart Category API: Structured Data in NodeJS
      • Walmart Product API: Structured Data in NodeJS
      • Walmart Reviews API: Structured Data in NodeJS
    • Async Structured Data Collection Method
      • Amazon Product Page API: Async Structured Data in NodeJS
      • Amazon Search API: Async Structured Data in NodeJS
      • Amazon Offers API: Async Structured Data in NodeJS
      • Amazon Reviews API: Async Structured Data in NodeJS
      • Ebay Product Page API: Async Structured Data in NodeJS
      • Ebay Search API: Async Structured Data in NodeJS
      • Google SERP API: Async Structured Data in NodeJS
      • Google News API: Async Structured Data in NodeJS
      • Google Jobs API: Async Structured Data in NodeJS
      • Google Shopping API: Async Structured Data in NodeJS
      • Google Maps Search API: Async Structured Data in NodeJS
      • Redfin Agent Details API: Async Structured Data in NodeJS
      • Redfin 'For Rent' Listings API: Async Structured Data in NodeJS
      • Redfin 'For Sale' Listings API: Async Structured Data in NodeJS
      • Redfin Listing Search API: Async Structured Data in NodeJS
      • Walmart Search API: Async Structured Data in NodeJS
      • Walmart Category API: Async Structured Data in NodeJS
      • Walmart Product API: Async Structured Data in NodeJS
      • Walmart Reviews API: Async Structured Data in NodeJS
    • Making POST/PUT Requests with ScraperAPI in NodeJS
    • Customizing ScraperAPI Requests in NodeJS
      • Customize Amazon Requests by ZIP Code via ScraperAPI in NodeJS
      • Customize Cached Results via ScraperAPI in NodeJS
      • Customize Control Costs with ScraperAPI Parameter in NodeJS
      • Send Custom Headers with ScraperAPI in NodeJS
      • Customize Device Type with ScraperAPI in NodeJS
      • Customize Geotargeted Content Scrape via ScraperAPI in NodeJS
      • Customize Premium Geotargeted Scrape via ScraperAPI in NodeJS
      • Customize Header Parameter with ScraperAPI in NodeJS
      • Customize Premium Residential/Mobile Proxies in NodeJS
      • Customize JavaScript-Rendered Pages via ScraperAPI in NodeJS
        • Use Render Instruction Set to Scrape Dynamic Pages in NodeJS
        • Customize Taking a Website Screenshots via ScraperAPI in NodeJS
      • Customize Scrape Session-Based Proxies via ScraperAPI in NodeJS
  • Handle and Process Responses via ScraperAPI in NodeJS
    • Use API Status Codes to Retry Failed Requests in NodeJS
    • Customize Output Formats via ScraperAPI Parameters in NodeJS
      • Request JSON Response via Autoparse Parameter in NodeJS
      • Request LLM Output Formats with ScraperAPI in NodeJS
    • Request Response Encoding and Content-Type via ScraperAPI in NodeJS
  • Dashboard & Billing
    • API Key
    • Credit Usage
    • Delete Account
    • Invoice History
    • Billing Email
    • Billing Address
    • VAT Number
    • Payment Method
    • Cancel Subscription
  • Credits and Requests
  • Monitor Your ScraperAPI Account Information in NodeJS
  • Documentation Overview
Powered by GitBook

Quick links

  • Homepage
  • Dashboard
  • Pricing
  • Contact Sales

Resources

  • Developer Guides
  • Blog
  • Contact Support
  • Learning Hub
On this page

Was this helpful?

  1. Make Requests with ScraperAPI in NodeJS
  2. Make Async Requests with ScraperAPI in NodeJS

How to Use ScraperAPI Async Web Scraping in NodeJS

Learn Async scraping with ScraperAPI in NodeJS: job submission, status polling, and result retrieval. Includes POST requests, metadata, and response handling.

A simple example showing how to submit a job for scraping and receive a status endpoint URL through which you can poll for the status (and later the result) of your scraping job:

const axios = require('axios');

(async() => {
const { data } = await axios({
data: {
apiKey: 'xxxxxx',
url: 'https://example.com'
},
headers: { 'Content-Type': 'application/json' },
method: 'POST',
url: 'https://async.scraperapi.com/jobs'
});

console.log(data);
})();

You can also send POST requests to the Async scraper by using the parameter “method”: “POST”. Here is an example on how to make a POST request to the Async scraper:

const axios = require('axios');

(async() => {
const { data } = await axios({
data: {
apiKey: 'xxxxxx',
url: 'https://example.com',
method: 'POST'
body: 'var1=value1&var2=value2'
},
headers: { 'Content-Type': 'application/json' },
method: 'POST',
url: 'https://async.scraperapi.com/jobs'
});

console.log(data);
})();

Response:

{
"id":"0962a8e0-5f1a-4e14-bf8c-5efcc18f0953",
"status":"running",
"statusUrl":"https://async.scraperapi.com/jobs/0962a8e0-5f1a-4e14-bf8c-5efcc18f0953",
"url":"https://example.com"
}

Note the statusUrl field in the response. That is a personal URL to retrieve the status and results of your scraping job. Invoking that endpoint provides you with the status first:

curl "https://async.scraperapi.com/jobs/0962a8e0-5f1a-4e14-bf8c-5efcc18f0953"

Response:

const axios = require('axios');

(async() => {
const { data } = await axios.get('https://async.scraperapi.com/jobs/0962a8e0-5f1a-4e14-bf8c-5efcc18f0953');

console.log(data);
})();

You can include a meta object in your request to store custom data (your own request ID for example), which will be returned in the response as well.

Once your job is finished, the response will change and will contain the results of your scraping job:

{
"id":"0962a8e0-5f1a-4e14-bf8c-5efcc18f0953",
"status":"finished",
"statusUrl":"https://async.scraperapi.com/jobs/0962a8e0-5f1a-4e14-bf8c-5efcc18f0953",
"url":"https://example.com",
"response":{
"headers":{
"date":"Thu, 14 Apr 2022 11:10:44 GMT",
"content-type":"text/html; charset=utf-8",
"content-length":"1256",
"connection":"close",
"x-powered-by":"Express",
"access-control-allow-origin":"undefined","access-control-allow-headers":"Origin, X-Requested-With, Content-Type, Accept",
"access-control-allow-methods":"HEAD,GET,POST,DELETE,OPTIONS,PUT",
"access-control-allow-credentials":"true",
"x-robots-tag":"none",
"sa-final-url":"https://example.com/",
"sa-statuscode":"200",
"etag":"W/\"4e8-Sjzo7hHgkd15I/TYxuW15B7HwEc\"",
"vary":"Accept-Encoding"
},
"body":"<!doctype html>\n<html>\n<head>\n	<title>Example Domain</title>\n\n	<meta charset=\"utf-8\" />\n	<meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n	<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n	<style type=\"text/css\">\n	body {\n		background-color: #f0f0f2;\n		margin: 0;\n		padding: 0;\n		font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n		\n	}\n	div {\n		width: 600px;\n		margin: 5em auto;\n		padding: 2em;\n		background-color: #fdfdff;\n		border-radius: 0.5em;\n		box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n	}\n	a:link, a:visited {\n		color: #38488f;\n		text-decoration: none;\n	}\n	@media (max-width: 700px) {\n		div {\n			margin: 0 auto;\n			width: auto;\n		}\n	}\n	</style>	\n</head>\n\n<body>\n<div>\n	<h1>Example Domain</h1>\n	<p>This domain is for use in illustrative examples in documents. You may use this\n	domain in literature without prior coordination or asking for permission.</p>\n	<p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
"statusCode":200
}
}

Please note that the response for an Async job is stored for up to 72 hours (24hrs guaranteed) or until you retrieve the results, whichever comes first. If you do not get the response in due time, it will be deleted from our side and you will have to send another request for the same job.

If callbacks are used and the results are successfully delivered, we automatically delete the results.

PreviousMake Async Requests with ScraperAPI in NodeJSNextUse Async ScraperAPI Callbacks in NodeJS

Last updated 2 months ago

Was this helpful?