The Render Instruction Set is a set of instructions that can be used to instruct the browser on specific actions to execute during page rendering. By combining these instructions, you can execute complex operations such as completing a search form or scrolling through an endlessly scrolling page . This capability enables efficient automation of dynamic web content interactions.
How to use
To send an instruction set to the browser, you send a JSON object to the API as a header , along with any other necessary parameters, including the "render=true" parameter.
In the following example, we enter a search term into a form, click the search icon, and then wait for the search results to load.
Copy [
{
"type" : "input" ,
"selector" : { "type" : "css" , "value" : "#searchInput" } ,
"value" : "cowboy boots"
} ,
{
"type" : "click" ,
"selector" : {
"type" : "css" ,
"value" : "#search-form button[type=\"submit\"]"
}
} ,
{
"type" : "wait_for_selector" ,
"selector" : { "type" : "css" , "value" : "#content" }
}
]
To send the above instruction set to our API endpoint, it must be formatted as a single string and passed as a header .
Please note that the "x-sapi- " prefix should be used on each header to prevent collisions with headers used by target sites.
Copy import java . net . * ;
import java . io . * ;
public class Scrape {
public static void main ( String [] args) {
try {
String url = "https://api.scraperapi.com?url=https://www.wikipedia.org" ;
URL urlForGetRequest = new URL(url) ;
String readLine = null ;
HttpURLConnection connection = (HttpURLConnection) urlForGetRequest . openConnection ();
connection . setRequestMethod ( "GET" );
connection . setRequestProperty ( "x-sapi-render" , "true" );
connection . setRequestProperty ( "x-sapi-api_key" , "<YOUR_API_KEY>" );
String instructionSet = "[{\"type\": \"input\", \"selector\": {\"type\": \"css\", \"value\": \"#searchInput\"}, \"value\": \"cowboy boots\"}, {\"type\": \"click\", \"selector\": {\"type\": \"css\", \"value\": \"#search-form button[type=\\\"submit\\\"]\"}}, {\"type\": \"wait_for_selector\", \"selector\": {\"type\": \"css\", \"value\": \"#content\"}}]";
connection . setRequestProperty ( "x-sapi-instruction_set" , instructionSet);
int responseCode = connection . getResponseCode ();
if (responseCode == HttpURLConnection . HTTP_OK ) {
BufferedReader in = new BufferedReader( new InputStreamReader( connection . getInputStream())) ;
StringBuffer response = new StringBuffer() ;
while ((readLine = in . readLine ()) != null ) {
response . append (readLine);
}
in . close ();
System . out . println ( response . toString ());
} else {
throw new Exception( "Error in API Call" ) ;
}
} catch ( Exception ex) {
ex . printStackTrace ();
}
}
}
Copy import java . net . Authenticator ;
import java . net . PasswordAuthentication ;
import java . net . HttpURLConnection ;
import java . net . URL ;
import java . io . BufferedReader ;
import java . io . InputStreamReader ;
import java . util . Properties ;
public class Scrape {
public static void main ( String [] args) {
try {
String apiKey = "<YOUR_API_KEY>" ;
String proxyHost = "proxy-server.scraperapi.com" ;
int proxyPort = 8001 ;
String proxyUser = "scraperapi" ;
String proxyPassword = apiKey;
// Set proxy properties
Properties systemProperties = System . getProperties ();
systemProperties . setProperty ( "http.proxyHost" , proxyHost);
systemProperties . setProperty ( "http.proxyPort" , String . valueOf (proxyPort));
// Set authenticator for proxy authentication
Authenticator . setDefault ( new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication () {
if ( getRequestorType() . equals ( RequestorType . PROXY )) {
return new PasswordAuthentication(proxyUser , proxyPassword . toCharArray()) ;
}
return null ;
}
});
String apiUrl = "https://www.wikipedia.org" ;
URL url = new URL(apiUrl) ;
HttpURLConnection connection = (HttpURLConnection) url . openConnection ();
connection . setRequestProperty ( "x-sapi-render" , "true" );
String instructionSet = "[{\"type\": \"input\", \"selector\": {\"type\": \"css\", \"value\": \"#searchInput\"}, \"value\": \"cowboy boots\"}, {\"type\": \"click\", \"selector\": {\"type\": \"css\", \"value\": \"#search-form button[type=\\\"submit\\\"]\"}}, {\"type\": \"wait_for_selector\", \"selector\": {\"type\": \"css\", \"value\": \"#content\"}}]";
connection . setRequestProperty ( "x-sapi-instruction_set" , instructionSet);
int responseCode = connection . getResponseCode ();
if (responseCode == HttpURLConnection . HTTP_OK ) {
BufferedReader in = new BufferedReader( new InputStreamReader( connection . getInputStream())) ;
StringBuffer response = new StringBuffer() ;
String inputLine;
while ((inputLine = in . readLine ()) != null ) {
response . append (inputLine);
}
in . close ();
System . out . println ( response . toString ());
} else {
throw new Exception( "Error in API Call, Response Code: " + responseCode) ;
}
} catch ( Exception ex) {
ex . printStackTrace ();
}
}
}
Supported Instructions
Browser instructions are organized as an array of objects within the instruction set, each with a specific structure. Below are the various instructions and the corresponding data they require:
Click
Input
Loop
Scroll
Wait
Wait_for_event
Wait_for_selector
Click
Click on an element on the page.
type: str = "click"
selector: dict
type: Enum["xpath", "css", "text"]
value: str
timeout: int (optional)
[{
"type": "click",
"selector": {
"type": "css",
"value": "#search-form button[type="submit\"]"
}
}]
Input
Enter a value into an input field on the page.
type: str = "input"
selector: dict
type: Enum["xpath", "css", "text"]
value: str
value: str
timeout: int (optional)
[{
"type": "input",
"selector": {
"type": "css",
"value": "#searchInput"
},
"value": "cowboy boots"
}]
Loop
Execute a set of instructions in a loop a specified number of times by using the loop instruction with a sequence of standard instructions in the “instructions” argument.
Note that nesting loops isn't supported, so you can't create a “loop” instruction inside another “loop” instruction. This method is effective for automating actions on web pages with infinitely scrolling content, such as loading multiple pages of results by scrolling to the bottom of the page and waiting for additional content to load.
type: str="loop"
for: int
instructions: array
[{
"type": "loop",
"for": 3,
"instructions": [
{
"type": "scroll",
"direction": "y",
"value": "bottom"
},
{
"type": "wait",
"value": 5
}
]
}]
Scroll
Scroll the page in the X or Y direction, by a given number of pixels or to the top or bottom of the page.
type: str = "scroll"
direction: Enum["x", "y"]
value: int or Enum["bottom", "top"]
[{
"type": "scroll",
"direction": "y",
"value": "bottom"
}]
Wait
Waits for a given number of seconds to elapse.
Description type: str = "wait"
value: int
[{
"type": "wait",
"value": 10
}]
Wait_for_event
Waits for an event to occur within the browser.
type: str = "wait_for_event"
event: Enum["domcontentloaded", "load", "navigation", "networkidle"]
timeout: int (optional)
[{
"type": "wait_for_event",
"event": "networkidle",
"timeout": 10
}]
Wait_for_selector
Waits for an element to appear on the page. Takes a 'value' argument that instructs the rendering engine to wait for a specified number of seconds for the element to appear.
type: str = "wait_for_selector"
selector: dict
type: Enum["xpath", "css", "text"]
value: str
timeout: int (optional)
[{
"type": "wait_for_selector",
"selector": {
"type": "css",
"value": "#content"
},
"timeout": 5
}]
