Google News API (Async)

This endpoint will retrieve news data from a Google news result page and transform it into usable JSON.

Single Query Request:

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import javax.net.ssl.HttpsURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
public class Main {
    public static void main(String[] args) {
        try {
            String apiKey = "APIKEY";
            String jsonInputString = "{"
                + "\"apiKey\": \"" + apiKey + "\", "
                + "\"query\": \"QUERY\", "
                + "\"callback\": {"
                + "    \"type\": \"webhook\", "
                + "    \"url\": \"CALLBACK\""
                + "}}";

            URL url = new URL("https://async.scraperapi.com/structured/google/news");
            HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
            connection.setRequestMethod("POST");
            connection.setRequestProperty("Content-Type", "application/json");
            connection.setRequestProperty("Accept", "*/*");
            connection.setDoOutput(true);
            try (OutputStream os = connection.getOutputStream()) {
                byte[] input = jsonInputString.getBytes(StandardCharsets.UTF_8);
                os.write(input, 0, input.length);
            }
            int responseCode = connection.getResponseCode();
            StringBuilder response = new StringBuilder();
            BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
            String readLine;
            while ((readLine = in.readLine()) != null) {
                response.append(readLine);
            }
            in.close();
            if (responseCode == HttpURLConnection.HTTP_OK) {
                System.out.println("Response: " + response.toString());
            } else {
                throw new Exception("Error in API Call: Response code " + responseCode + "\nbody: " + response.toString());
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import javax.net.ssl.HttpsURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
public class Main {
    public static void main(String[] args) {
        try {
            String apiKey = "APIKEY";
            String jsonInputString = "{"
                + "\"apiKey\": \"" + apiKey + "\", "
                + "\"queries\": [\"QUERY1\", \"QUERY2\", \"QUERY3\"], "
                + "\"callback\": {"
                + "    \"type\": \"webhook\", "
                + "    \"url\": \"CALLBACK\""
                + "}}";

            URL url = new URL("https://async.scraperapi.com/structured/google/news");
            HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
            connection.setRequestMethod("POST");
            connection.setRequestProperty("Content-Type", "application/json");
            connection.setRequestProperty("Accept", "*/*");
            connection.setDoOutput(true);
            try (OutputStream os = connection.getOutputStream()) {
                byte[] input = jsonInputString.getBytes(StandardCharsets.UTF_8);
                os.write(input, 0, input.length);
            }
            int responseCode = connection.getResponseCode();
            StringBuilder response = new StringBuilder();
            BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
            String readLine;
            while ((readLine = in.readLine()) != null) {
                response.append(readLine);
            }
            in.close();
            if (responseCode == HttpURLConnection.HTTP_OK) {
                System.out.println("Response: " + response.toString());
            } else {
                throw new Exception("Error in API Call: Response code " + responseCode + "\nbody: " + response.toString());
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

Multiple Parameters can be used with this method:

REQUIRED

REQUIRED
`APIKEY`	User account’s normal API key.
`QUERY`	Google Search Query.

APIKEY

User account’s normal API key.

QUERY

Google Search Query.

OPTIONAL

OPTIONAL
`TLD`	Set this value to scrape the respective Google domain. Valid values include tlds for those countries or regions where Google has a search engine: ae: () ca: () cn: () co.jp: () co.uk: () com: () com.au: () com.be: () com.br: () com.mx: () com.tr: () de: () eg: () es: () fr: () in: () it: () nl: () pl: () sa: () se: () sg: ()
`COUNTRY`	Valid values are two letter country codes for which we offer Geo Targeting (e.g. “au”, “es”, “it”, etc.). Where a Google domain needs to be scraped from another country (e.g. scraping Google.com from Canada, both TLD and COUNTRY parameters must be specified.
We also support Google Search parameters for this endpoint.	`UULE`: Set a region for a search. For example: `w+CAIQICINUGFyaXMsIEZyYW5jZQ`. You can find an online `UULE` generator here: `NUM`: Number of results `HL`: Host Language. For example: `DE` `GL`: Boosts matches whose country of origin matches the parameter value. For example: `DE` `IE`: Character encoding how the engine interpret the query string. For example: `UTF8` `OE`: Character encoding used for the results. For example: `UTF8` `START`: Set the starting offset in the result list. When `start=10` set the first element in the result list will be the 10th search result (meaning it starts with page 2 of results if the "num" is 10)

TLD

Set this value to scrape the respective Google domain. Valid values include tlds for those countries or regions where Google has a search engine:

ae: ()

ca: ()

cn: ()

co.jp: ()

co.uk: ()

com: ()

com.au: ()

com.be: ()

com.br: ()

com.mx: ()

com.tr: ()

de: ()

eg: ()

es: ()

fr: ()

in: ()

it: ()

nl: ()

pl: ()

sa: ()

se: ()

sg: ()

COUNTRY

Valid values are two letter country codes for which we offer Geo Targeting (e.g. “au”, “es”, “it”, etc.).

Where a Google domain needs to be scraped from another country (e.g. scraping Google.com from Canada, both TLD and COUNTRY parameters must be specified.

We also support Google Search parameters for this endpoint.

UULE: Set a region for a search. For example: w+CAIQICINUGFyaXMsIEZyYW5jZQ.

You can find an online UULE generator here: NUM: Number of results HL: Host Language. For example: DE GL: Boosts matches whose country of origin matches the parameter value.

For example: DE IE: Character encoding how the engine interpret the query string. For example: UTF8 OE: Character encoding used for the results. For example: UTF8 START: Set the starting offset in the result list. When start=10 set the first element in the result list will be the 10th search result (meaning it starts with page 2 of results if the "num" is 10)

Sample Response

Single Query Request:

{
	"id": "89b388bf-0cea-49c1-8db6-9e00042c8c3a",
	"status": "running",
	"statusUrl": "http://async.scraperapi.com/structured/google/news/89b388bf-0cea-49c1-8db6-9e00042c8c3a",
	"query": "US economy"
}

Multiple Query Request:

[
	{
		"id": "2955ad23-c812-475c-bc52-572576815d78",
		"status": "running",
		"statusUrl": "http://async.scraperapi.com/structured/google/news/2955ad23-c812-475c-bc52-572576815d78",
		"query": "US Economy"
	},
	{
		"id": "120a7344-7832-4fd0-a64f-ce9cd39726f3",
		"status": "running",
		"statusUrl": "http://async.scraperapi.com/structured/google/news/120a7344-7832-4fd0-a64f-ce9cd39726f3",
		"query": "Crowdfunding"
	}
]

After the job(s) finish, you will find the result under the response key in the response JSON object. The structure is the same as in the corresponding SYNC data endpoint.

Last updated 6 months ago