The Render Instruction Set is a set of instructions that can be used to instruct the browser on specific actions to execute during page rendering. By combining these instructions, you can execute complex operations such as completing a search form or scrolling through an endlessly scrolling page. This capability enables efficient automation of dynamic web content interactions.
How to use
To send an instruction set to the browser, you send a JSON object to the API as a header, along with any other necessary parameters, including the "render=true" parameter.
In the following example, we enter a search term into a form, click the search icon, and then wait for the search results to load.
To send the above instruction set to our API endpoint, it must be formatted as a single string and passed as a header.
Please note that the "x-sapi-" prefix should be used on each header to prevent collisions with headers used by target sites.
Reducing the number of actions you instruct the API to perform gives you a higher chance of getting a successful response. Clicking through multiple pages adds to the total latency of the request, which can lead to timeouts. For best performance, limit the instruction set down to 3-4 actions.
API REQUEST
import java.net.*;
import java.io.*;
public class Scrape {
public static void main(String[] args) {
try {
String url = "https://api.scraperapi.com?url=https://www.wikipedia.org";
URL urlForGetRequest = new URL(url);
String readLine = null;
HttpURLConnection connection = (HttpURLConnection) urlForGetRequest.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("x-sapi-render", "true");
connection.setRequestProperty("x-sapi-api_key", "<YOUR_API_KEY>");
String instructionSet = "[{\"type\": \"input\", \"selector\": {\"type\": \"css\", \"value\": \"#searchInput\"}, \"value\": \"cowboy boots\"}, {\"type\": \"click\", \"selector\": {\"type\": \"css\", \"value\": \"#search-form button[type=\\\"submit\\\"]\"}}, {\"type\": \"wait_for_selector\", \"selector\": {\"type\": \"css\", \"value\": \"#content\"}}]";
connection.setRequestProperty("x-sapi-instruction_set", instructionSet);
int responseCode = connection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuffer response = new StringBuffer();
while ((readLine = in.readLine()) != null) {
response.append(readLine);
}
in.close();
System.out.println(response.toString());
} else {
throw new Exception("Error in API Call");
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
PROXY MODE
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.HttpURLConnection;
import java.net.URL;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Properties;
public class Scrape {
public static void main(String[] args) {
try {
String apiKey = "<YOUR_API_KEY>";
String proxyHost = "proxy-server.scraperapi.com";
int proxyPort = 8001;
String proxyUser = "scraperapi";
String proxyPassword = apiKey;
// Set proxy properties
Properties systemProperties = System.getProperties();
systemProperties.setProperty("http.proxyHost", proxyHost);
systemProperties.setProperty("http.proxyPort", String.valueOf(proxyPort));
// Set authenticator for proxy authentication
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType().equals(RequestorType.PROXY)) {
return new PasswordAuthentication(proxyUser, proxyPassword.toCharArray());
}
return null;
}
});
String apiUrl = "https://www.wikipedia.org";
URL url = new URL(apiUrl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("x-sapi-render", "true");
String instructionSet = "[{\"type\": \"input\", \"selector\": {\"type\": \"css\", \"value\": \"#searchInput\"}, \"value\": \"cowboy boots\"}, {\"type\": \"click\", \"selector\": {\"type\": \"css\", \"value\": \"#search-form button[type=\\\"submit\\\"]\"}}, {\"type\": \"wait_for_selector\", \"selector\": {\"type\": \"css\", \"value\": \"#content\"}}]";
connection.setRequestProperty("x-sapi-instruction_set", instructionSet);
int responseCode = connection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuffer response = new StringBuffer();
String inputLine;
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
System.out.println(response.toString());
} else {
throw new Exception("Error in API Call, Response Code: " + responseCode);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
Supported Instructions
Browser instructions are organized as an array of objects within the instruction set, each with a specific structure. Below are the various instructions and the corresponding data they require:
Execute a set of instructions in a loop a specified number of times by using the loop instruction with a sequence of standard instructions in the “instructions” argument.
Note that nesting loops isn't supported, so you can't create a “loop” instruction inside another “loop” instruction. This method is effective for automating actions on web pages with infinitely scrolling content, such as loading multiple pages of results by scrolling to the bottom of the page and waiting for additional content to load.
Waits for an element to appear on the page. Takes a 'value' argument that instructs the rendering engine to wait for a specified number of seconds for the element to appear.
