Render Instruction Set 🆕

The Render Instruction Set is a set of instructions that can be used to instruct the browser on specific actions to execute during page rendering. By combining these instructions, you can execute complex operations such as completing a search form or scrolling through an endlessly scrolling page. This capability enables efficient automation of dynamic web content interactions.

How to use

To send an instruction set to the browser, you send a JSON object to the API as a header, along with any other necessary parameters, including the "render=true" parameter.

In the following example, we enter a search term into a form, click the search icon, and then wait for the search results to load.

[
  {
    "type": "input",
    "selector": { "type": "css", "value": "#searchInput" },
    "value": "cowboy boots"
  },
  {
    "type": "click",
    "selector": {
      "type": "css",
      "value": "#search-form button[type=\"submit\"]"
    }
  },
  {
    "type": "wait_for_selector",
    "selector": { "type": "css", "value": "#content" }
  }
]

To send the above instruction set to our API endpoint, it must be formatted as a single string and passed as a header.

Please note that the "x-sapi-" prefix should be used on each header to prevent collisions with headers used by target sites.

• API REQUEST

import java.net.*;
import java.io.*;

public class Scrape {
  public static void main(String[] args) {
    try {
      String url = "https://api.scraperapi.com?url=https://www.wikipedia.org";

      URL urlForGetRequest = new URL(url);
      String readLine = null;
      HttpURLConnection connection = (HttpURLConnection) urlForGetRequest.openConnection();
      connection.setRequestMethod("GET");
      connection.setRequestProperty("x-sapi-render", "true");
      connection.setRequestProperty("x-sapi-api_key", "<YOUR_API_KEY>");

      String instructionSet = "[{\"type\": \"input\", \"selector\": {\"type\": \"css\", \"value\": \"#searchInput\"}, \"value\": \"cowboy boots\"}, {\"type\": \"click\", \"selector\": {\"type\": \"css\", \"value\": \"#search-form button[type=\\\"submit\\\"]\"}}, {\"type\": \"wait_for_selector\", \"selector\": {\"type\": \"css\", \"value\": \"#content\"}}]";
      connection.setRequestProperty("x-sapi-instruction_set", instructionSet);

      int responseCode = connection.getResponseCode();
      if (responseCode == HttpURLConnection.HTTP_OK) {
        BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        StringBuffer response = new StringBuffer();
        while ((readLine = in.readLine()) != null) {
          response.append(readLine);
        }
        in.close();
        System.out.println(response.toString());
      } else {
        throw new Exception("Error in API Call");
      }
    } catch (Exception ex) {
      ex.printStackTrace();
    }
  }
}

• PROXY MODE

import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.HttpURLConnection;
import java.net.URL;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.Properties;

public class Scrape {
    public static void main(String[] args) {
        try {
            String apiKey = "<YOUR_API_KEY>";
            String proxyHost = "proxy-server.scraperapi.com";
            int proxyPort = 8001;
            String proxyUser = "scraperapi";
            String proxyPassword = apiKey;

            // Set proxy properties
            Properties systemProperties = System.getProperties();
            systemProperties.setProperty("http.proxyHost", proxyHost);
            systemProperties.setProperty("http.proxyPort", String.valueOf(proxyPort));

            // Set authenticator for proxy authentication
            Authenticator.setDefault(new Authenticator() {
                protected PasswordAuthentication getPasswordAuthentication() {
                    if (getRequestorType().equals(RequestorType.PROXY)) {
                        return new PasswordAuthentication(proxyUser, proxyPassword.toCharArray());
                    }
                    return null;
                }
            });

            String apiUrl = "https://www.wikipedia.org";
            URL url = new URL(apiUrl);
            HttpURLConnection connection = (HttpURLConnection) url.openConnection();
            connection.setRequestProperty("x-sapi-render", "true");

            String instructionSet = "[{\"type\": \"input\", \"selector\": {\"type\": \"css\", \"value\": \"#searchInput\"}, \"value\": \"cowboy boots\"}, {\"type\": \"click\", \"selector\": {\"type\": \"css\", \"value\": \"#search-form button[type=\\\"submit\\\"]\"}}, {\"type\": \"wait_for_selector\", \"selector\": {\"type\": \"css\", \"value\": \"#content\"}}]";
            connection.setRequestProperty("x-sapi-instruction_set", instructionSet);

            int responseCode = connection.getResponseCode();

            if (responseCode == HttpURLConnection.HTTP_OK) {
                BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
                StringBuffer response = new StringBuffer();
                String inputLine;

                while ((inputLine = in.readLine()) != null) {
                    response.append(inputLine);
                }
                in.close();

                System.out.println(response.toString());
            } else {
                throw new Exception("Error in API Call, Response Code: " + responseCode);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

Supported Instructions

Browser instructions are organized as an array of objects within the instruction set, each with a specific structure. Below are the various instructions and the corresponding data they require:

Click

Input

Loop

Scroll

Wait

Wait_for_event

Wait_for_selector

Click

Click on an element on the page.

Args	type: str = "click" selector: dict type: Enum["xpath", "css", "text"] value: str timeout: int (optional)
Example	[{ "type": "click", "selector": { "type": "css", "value": "#search-form button[type="submit\"]" } }]

Input

Enter a value into an input field on the page.

Args	type: str = "input" selector: dict type: Enum["xpath", "css", "text"] value: str value: str timeout: int (optional)
Example	[{ "type": "input", "selector": { "type": "css", "value": "#searchInput" }, "value": "cowboy boots" }]

Loop

Execute a set of instructions in a loop a specified number of times by using the loop instruction with a sequence of standard instructions in the “instructions” argument.

Note that nesting loops isn't supported, so you can't create a “loop” instruction inside another “loop” instruction. This method is effective for automating actions on web pages with infinitely scrolling content, such as loading multiple pages of results by scrolling to the bottom of the page and waiting for additional content to load.

Args	type: str="loop" for: int instructions: array
Example	[{ "type": "loop", "for": 3, "instructions": [ { "type": "scroll", "direction": "y", "value": "bottom" }, { "type": "wait", "value": 5 } ] }]

Scroll

Scroll the page in the X or Y direction, by a given number of pixels or to the top or bottom of the page.

Args	type: str = "scroll" direction: Enum["x", "y"] value: int or Enum["bottom", "top"]
Example	[{ "type": "scroll", "direction": "y", "value": "bottom" }]

Wait

Waits for a given number of seconds to elapse.

Description
Args	type: str = "wait" value: int
Example	[{ "type": "wait", "value": 10 }]

Wait_for_event

Waits for an event to occur within the browser.

Args	type: str = "wait_for_event" event: Enum["domcontentloaded", "load", "navigation", "networkidle"] timeout: int (optional)
Example	[{ "type": "wait_for_event", "event": "networkidle", "timeout": 10 }]

Wait_for_selector

Waits for an element to appear on the page. Takes a 'value' argument that instructs the rendering engine to wait for a specified number of seconds for the element to appear.

Args	type: str = "wait_for_selector" selector: dict type: Enum["xpath", "css", "text"] value: str timeout: int (optional)
Example	[{ "type": "wait_for_selector", "selector": { "type": "css", "value": "#content" }, "timeout": 5 }]