Callbacks

Using a status URL is a great way to test the API or get started quickly, but some customer environments may require some more robust solutions, so we implemented callbacks. Currently only webhook callbacks are supported but we are planning to introduce more over time (e.g. direct database callbacks, AWS S3, etc).

An example of using a webhook callback:

const axios = require('axios');

(async() => {
const { data } = await axios({
data: {
apiKey: 'API_KEY',
callback: {
type: 'webhook',
url: 'https://yourcompany.com/scraperapi'
},
url: 'https://example.com'
},
headers: { 'Content-Type': 'application/json' },
method: 'POST',
url: 'https://async.scraperapi.com/jobs'
});

console.log(data);
})();

Using a callback you don’t need to use the status URL (although you still can) to fetch the status and results of the job. Once the job is finished the provided webhook URL will be invoked by our system with the same content as the status URL provides.

Just replace the https://yourcompany.com/scraperapi URL with your preferred endpoint. You can even add basic auth to the URL in the following format: https://user:pass@yourcompany.com/scraperapi

By default, we'll call the webhook URL you provide for successful requests. If you'd like to receive data on failed attempts too, you will have to include the expectsUnsuccessReport: true parameter in your request structure.

An example of using callbacks that report on the failed attempts as well:

const axios = require('axios');

(async () => {
const { data } = await axios({
data: {
apiKey: 'API_KEY',
url: 'https://httpbin.org/status/500',
callback: {
type: 'webhook',
url: 'YYYYY'
},
expectsUnsuccessReport: true
},
headers: { 'Content-Type': 'application/json' },
method: 'POST',
url: 'https://async.scraperapi.com/jobs'
});

console.log(data);
})();

Response:

{
  id: 'db726d84-2609-4709-93cb-88c4a9fa4288',
  attempts: 50,
  status: 'failed',
  statusUrl: 'https://async.scraperapi.com/jobs/db726d84-2609-4709-93cb-88c4a9fa4288',
  failReason: 'failed_due_to_timeout',
  url: 'http://httpbin.org/status/500'
}

Note: The system will try to invoke the webhook URL 3 times, then it cancels the job. So please make sure that the webhook URL is available through the public internet and will be capable of handling the traffic that you need.

PreviousHow to use NextAPI Parameters

Last updated 1 month ago

Was this helpful?

const axios = require('axios'); (async() => { const { data } = await axios({ data: { apiKey: 'API_KEY', callback: { type: 'webhook', url: 'https://yourcompany.com/scraperapi' }, url: 'https://example.com' }, headers: { 'Content-Type': 'application/json' }, method: 'POST', url: 'https://async.scraperapi.com/jobs' }); console.log(data); })();

const axios = require('axios'); (async () => { const { data } = await axios({ data: { apiKey: 'API_KEY', url: 'https://httpbin.org/status/500', callback: { type: 'webhook', url: 'YYYYY' }, expectsUnsuccessReport: true }, headers: { 'Content-Type': 'application/json' }, method: 'POST', url: 'https://async.scraperapi.com/jobs' }); console.log(data); })();

{ id: 'db726d84-2609-4709-93cb-88c4a9fa4288', attempts: 50, status: 'failed', statusUrl: 'https://async.scraperapi.com/jobs/db726d84-2609-4709-93cb-88c4a9fa4288', failReason: 'failed_due_to_timeout', url: 'http://httpbin.org/status/500' }