Send Custom Headers with ScraperAPI in Java

Learn how to use keep_headers=true to send custom headers with ScraperAPI in Java. Recommended only when necessary. Not compatible with ultra_premium=true.

Important note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.

If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.

If you need to get results for mobile devices, use the device_type parameter to set the user-agent header for your request, instead of setting your own.

• API REQUEST

try {
String apiKey = "APIKEY";
String url = "http://api.scraperapi.com?api_key=" + apiKey + "&url=http://httpbin.org/anything&keep_headers=true";
URL urlForGetRequest = new URL(url);
String readLine = null;
HttpURLConnection httpURLConnection = (HttpURLConnection) urlForGetRequest.openConnection();
httpURLConnection.setRequestProperty("Content-Type", "application/json");
httpURLConnection.setRequestProperty("X-MyHeader", "123");
httpURLConnection.setRequestMethod("GET");
int responseCode = httpURLConnection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(httpURLConnection.getInputStream()));
StringBuffer response = new StringBuffer();
while ((readLine = in.readLine()) != null) {
response.append(readLine);
}
in.close();
System.out.println(response.toString());
} else {
throw new Exception("Error in API Call");
}
} catch (Exception ex) {
ex.printStackTrace();
}

• PROXY MODE

try {
  String apiKey = "APIKEY";
  String proxy = "http://scraperapi.keep_headers=true:" + apiKey + "@proxy-server.scraperapi.com";
  URL server = new URL("http://httpbin.org/anything");
  Properties systemProperties = System.getProperties();
  systemProperties.setProperty("http.proxyHost", proxy);
  systemProperties.setProperty("http.proxyPort", "8001");
  HttpURLConnection httpURLConnection = (HttpURLConnection) server.openConnection();
  httpURLConnection.setRequestProperty("Content-Type", "application/json");
  httpURLConnection.setRequestProperty("X-MyHeader", "123");
  httpURLConnection.connect();
  String readLine = null;
  int responseCode = httpURLConnection.getResponseCode();
  if (responseCode == HttpURLConnection.HTTP_OK) {
      BufferedReader in = new BufferedReader(new InputStreamReader(httpURLConnection.getInputStream()));
      StringBuffer response = new StringBuffer();
      while ((readLine = in.readLine()) != null) {
          response.append(readLine);
      }
      in.close();
      System.out.println(response.toString());
  } else {
      throw new Exception("Error in API Call");
  }
} catch (Exception ex) {
  ex.printStackTrace();
}

Please note that you cannot use Cutom Headers together with "ultra_premium=true". If you set "ultra_premium=true", we discard all of your custom headers even if you use "keep_headers=true".