Release NotesDataPipelineFAQs

Input and Parameters

Input options

It is possible to scrape up to 10,000 URL, ASINs or keywords within one project. Depending on the project type, the type of Input can be different. E.g. Walmart category IDs for Category search or Amazon AINS for Amazon product data.

Input options are:

  • Input field: Copy paste the URLs or search terms into the input field. ‼️ Each term or URL must be in a separate line. ‼️

  • CSV file: Simply drag and drop the file into the field. The file should not contain a header row or any commas.

  • Webhook: Add a webhook URL where we poll the entries like URLs or search terms from. The file will be validated on the fly. This gives you a more flexible and dynamic way to update the pages and data you need.

Each term should be in one line without commas.

Input for use case specific project types:

Domain
Specific Use Case
Input

Google

Search Results

Query string / keywords that a user wants to search for.

Maps

Query string / keywords that a user wants to search for.

Jobs

Query string / keywords that a user wants to search for.

News

Query string / keywords that a user wants to search for.

Shopping

Query string / keywords that a user wants to search for.

Amazon

Product page data

ASIN = Amazon Standard Identification Number. Please not that ASIN's are market specific (TLD). You can usually find the ASINs in the URL of an Amazon product (example: B07FTKQ97Q)

Search page results

Amazon Search query string

Offers

ASIN = Amazon Standard Identification

Walmart

Product page data

Walmart Product ID = You can find the product ID in the URL. Example: 5253396052 (/ip/5253396052)

Category Pages

Walmart category ID = You can find the walmart category ID in the URL. Example: 3944_1089430_37807 (/browse/3944_1089430_37807)

Search page results

Query string / keywords that a user wants to search for.

Ebay

Product page data

Ebay product ID = You can find the 12 digit ID in the URL. Example: 116506192782 (/itm/116506192782?)

Search page results

Query string / keywords that a user wants to search for.

Redfin

Single for Sale listings

URLs of the on sale listings

Single for Rent listings

URLs of the for rent listings

Listing search page

URLs of search page

Parameters and filters

We have multiple parameters available that will improve the success of a job. Some parameters will increase the price for your job. Be aware that not all parameters might be in your plan! Read more about the credit costs here.

  • Activate premium residential and mobile IPs: Activate premium proxies. A simple request costs 10 API credits and 25 in combination with javascript rendering.

  • Activate advanced bypass mechanism: Activate advanced bypass for very difficult domains. A request costs 30 API credits.

  • Disable follow redirects

  • Activate javascript rendering: Activate if the targeted URLs require rendering javascript. We will fetch these pages using a headless browser.

  • Activate to parse results: Activate to receive your results as JSON for certain domains instead of plain HTML. The domains are Google, Amazon and Walmart.

  • Activate retrying 404 responses: Activate to retry URLs if they responded with 404 error.

  • Country code: Activate country geo-targeting to use country specific proxies, for example US proxies. This parameter does not increase the cost.

  • Device type: Default is device_type=desktop. The device type will be overridden if you use keep_headers=true and send your own user agent in the request header.

Last updated

Was this helpful?