Callbacks

Using a status URL is a great way to test the API or get started quickly, but some customer environments may require some more robust solutions, so we implemented callbacks. Currently only webhook callbacks are supported but we are planning to introduce more over time (e.g. direct database callbacks, AWS S3, etc).

An example of using a webhook callback:

try {
URL asyncApiUrl = new URL("https://async.scraperapi.com/jobs");
HttpURLConnection connection = (HttpURLConnection) asyncApiUrl.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setDoOutput(true);
try (OutputStream outputStream = connection.getOutputStream()) {
outputStream.write(("{\"apiKey\": \"xxxxxx\", " +
	"\"callback\": {" +
	"\"type\": \"webhook\"," +
	"\"url\": \"https://yourcompany.com/scraperapi\"" +
	"}," +
	"\"url\": \"https://example.com\"" +
	"}").getBytes(StandardCharsets.UTF_8));
}

int responseCode = connection.getResponseCode();

if (responseCode == HttpURLConnection.HTTP_OK) {
try (BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()))) {
String readLine = null;
StringBuffer response = new StringBuffer();

while ((readLine = in.readLine()) != null) {
	response.append(readLine);
}

System.out.println(response);
}
} else {
throw new Exception("Error in API Call");
}
} catch (Exception ex) {
ex.printStackTrace();
}

Using a callback you don’t need to use the status URL (although you still can) to fetch the status and results of the job. Once the job is finished the provided webhook URL will be invoked by our system with the same content as the status URL provides.

Just replace the https://yourcompany.com/scraperapi URL with your preferred endpoint. You can even add basic auth to the URL in the following format: https://user:[e-mail]/scraperapi

Note: The system will try to invoke the webhook URL 3 times, then it cancels the job. So please make sure that the webhook URL is available through the public internet and will be capable of handling the traffic that you need.

Hint: Webhook.site is a free online service to test webhooks without requiring you to build a complex infrastructure.