Customizing Requests
Customize API Functionality
ScraperAPI enables you to customize the API’s functionality by adding additional parameters to your requests. The API will accept the following parameters:
render
Activate javascript rendering by setting render=true in your request. The API will automatically render the javascript on the page and return the HTML response after the javascript has been rendered.
Requests using this parameter cost 10 API credits, or 75 if used in combination with ultra-premium ultra_premium=true.
screenshot
Gves you the ability to take a screenshot of the target page through the use of the parameter screenshot=true. This parameter automatically enables JS rendering to get the full page content, before taking a screenshot.
country_code
Activate country geotargeting by setting country_code=us to use US proxies for example.
This parameter does not increase the cost of the API request.
premium
Activate premium residential and mobile IPs by setting premium=true. Using premium proxies costs 10 API credits, or 25 API credits if used in combination with Javascript rendering render=true. (Can not be combined withUltra_Premium)
session_number
Reuse the same proxy by setting session_number=123 for example.
This parameter does not increase the cost of the API request. (Can not be combined with Premium/Ultra_Premium)
binary_target
Helpful when trying to scrape files or images. This tells our API that the target is a file.
keep_headers
Use your own custom headers by setting keep_headers=true along with sending your own headers to the API.
This parameter does not increase the cost of the API request.
device_type
Set your requests to use mobile or desktop user agents by setting device_type=desktop or device_type=mobile.
This parameter does not increase the cost of the API request.
autoparse
Activate auto parsing for select websites by setting autoparse=true. The API will parse the data on the page and return it in JSON format.
This parameter does not increase the cost of the API request.
ultra_premium
Activate our advanced bypass mechanisms by setting ultra_premium=true.
Requests using this parameter cost 30 API credits, or 75 if used in combination with javascript rendering. (Can not be combined with Premium)
output_format
The output_format parameter allows you to instruct the API on what the response file type should be. Valid options:
markdown
text
SDEs valid options:
json
csv
markdown
text
Amazon ZIP Code Targeting
Please be aware that our current support is limited to US ZIP Codes exclusively
Target specific ZIP codes to scrape product listings, offers, search queries, and more from regions of interest, providing valuable insights for market research, competitor analysis, and pricing optimization. When you need to focus solely on a particular ZIP code, incorporate the zip parameter into your requests.
Here are some example ZIP codes that we support:
33837
62864
92223
92392
Davenport, FL
Mount Vernon, VA
Beaumont, TX
Victorville, CA
For example, set zip=92223to retrieve information tailored to the Beaumont, TX area within the response:
API REQUEST
PROXY MODE
SDE METHOD
ASYNC SDE METHOD
Coming soon
We encourage you to explore the ZIP code targeting feature to see how it can enhance your data collection. Try various ZIP codes to get a feel for how the targeted information changes and tailor your requests to suit your needs. If certain ZIP codes are not working or accepted during your testing, please contact our support team to enable them for you.
Cached Results
ScraperAPI's caching system unlocks a new level of efficiency. By leveraging advanced decision-making mechanisms, we maintain an extensive amount of cached data, that is ready to be served if called upon. When you request a page with a cached response available, you'll receive the cached data, ensuring much faster access to the information you need with a 100% success rate.
Why is this good?
Difficult Pages: Perfect for pages that are challenging to scrape.
10-Minute Updates: No older than 10 minutes cached value.
Guaranteed Success: 100% success rate for cached results.
Faster Response Times: Retrieve data quicker from cached results.
Fewer Retries: Reduced number of retries needed to serve back the response.
What's in it for you?
🚀
Increased Efficiency:Save time and resources by reducing the need to scrape the same page multiple times.✅
Improved Reliability:Enhance the reliability of your scraping tasks with consistent and timely data retrieval.
Advanced Use-Case Scenario
For cases where real-time data is required, you can ensure the API serves uncached data by adding the parameter cache_control=no-cache to the URL, as shown below:
We tag cached responses with the sa-from-cache: 1 response header, making it easy to distinguish between cached and non-cached responses.
Future Enhancements
We're always looking to improve. Stay tuned for the upcoming max_age option, which will give you even more control over your caching preferences.
Cost control
ScraperAPI helps you control and manage your costs efficiently. By using the max_cost parameter with your requests, you instruct the API to set a limit on the maximum API credits you'd like to spend per each individual scrape. This helps prevent overspending, ensuring you stay within your individual project's budget.
API REQUEST
ASYNC REQUEST
PROXY MODE
If the scrape cost exceeds your limit, a 403 status code will be returned for the request, with the following error message:
"This request exceeds your max_cost. You can view the cost per request in your response header or in the API Playground on the dashboard."
Custom Headers
Important note: Only use this feature if you need to send custom headers to retrieve specific results from the website. Within the API we have a sophisticated header management system designed to increase success rates and performance on difficult sites. When you send your own custom headers you override our header system, which oftentimes lowers your success rates. Unless you absolutely need to send custom headers to get the data you need, we advise that you don’t use this functionality.
If you would like to use your own custom headers (user agents, cookies, etc.) when making a request to the website, simply set keep_headers=true and send the API the headers you want to use. The API will then use these headers when sending requests to the website.
If you need to get results for mobile devices, use the device_type parameter to set the user-agent header for your request, instead of setting your own.
API REQUEST
PROXY MODE
Device Type
If your use case requires you to exclusively use either desktop or mobile user agents in the headers it sends to the website then you can use the device_type parameter.
Set
device_type=desktopto have the API set a desktop (e.g. iOS, Windows, or Linux) user agent. Note: This is the default behavior. Not setting the parameter will have the same effect.Set
device_type=mobileto have the API set a mobile (e.g. iPhone or Android) user agent.
Note: The device type you set will be overridden if you use keep_headers=true and send your own user agent in the requests header.
API REQUEST
PROXY MODE
Geotargeting
Certain websites (typically e-commerce stores and search engines) display different data to different users based on the geolocation of the IP used to make the request to the website. In cases like these, you can use the API’s geotargeting functionality to easily use proxies from the required country to retrieve the correct data from the website.
To control the geolocation of the IP used to make the request, simply set the country_code parameter to the country you want the proxy to be from and the API will automatically use the correct IP for that request.
For example: to ensure your requests come from the United States, set the country_code parameter to country_code=us.
Business and Enterprise Plan users can geotarget their requests to the following 69 countries (Hobby and Startup Plans can only use US and EU geotargeting) by using the country_code in their request:
us
United States
Hobby Plan and higher.
eu
Europe (general)
Hobby Plan and higher.
au
Australia
Business Plan and higher.
ae
United Arab Emirates
Business Plan and higher.
ar
Argentina
Business Plan and higher.
at
Austria
Business Plan and higher.
bd
Bangladesh
Business Plan and higher.
be
Belgium
Business Plan and higher.
br
Brazil
Business Plan and higher.
ca
Canada
Business Plan and higher.
ch
Switzerland
Business Plan and higher.
cl
Chile
Business Plan and higher.
cn
China
Business Plan and higher.
de
Germany
Business Plan and higher.
dk
Denmark
Business Plan and higher.
eg
Egypt
Business Plan and higher.
es
Spain
Business Plan and higher.
fi
Finland
Business Plan and higher.
fr
France
Business Plan and higher.
gr
Greece
Business Plan and higher.
hk
Hong Kong
Business Plan and higher.
hu
Hungary
Business Plan and higher.
id
Indonesia
Business Plan and higher.
ie
Ireland
Business Plan and higher.
il
Israel
Business Plan and higher.
in
India
Business Plan and higher.
it
Italy
Business Plan and higher.
jo
Jordan
Business Plan and higher.
jp
Japan
Business Plan and higher.
ke
Kenya
Business Plan and higher.
kr
South Korea
Business Plan and higher.
mx
Mexico
Business Plan and higher.
my
Malaysia
Business Plan and higher.
ng
Nigeria
Business Plan and higher.
nl
Netherlands
Business Plan and higher.
no
Norway
Business Plan and higher.
nz
New Zealand
Business Plan and higher.
pe
Peru
Business Plan and higher.
ph
Phillipines
Business Plan and higher.
pk
Pakistan
Business Plan and higher.
pl
Poland
Business Plan and higher.
pt
Portugal
Business Plan and higher.
ru
Russia
Business Plan and higher.
sa
Saudi Arabia
Business Plan and higher.
se
Sweden
Business Plan and higher.
sg
Singapore
Business Plan and higher.
th
Thailand
Business Plan and higher.
tr
Turkey
Business Plan and higher.
tw
Taiwan
Business Plan and higher.
ua
Ukraine
Business Plan and higher.
uk
United Kingdom
Business Plan and higher.
ve
Venezuela
Business Plan and higher.
vn
Vietnam
Business Plan and higher.
za
South Africa
Business Plan and higher.
bg
Bulgaria
Business Plan and higher.
hr
Croatia
Business Plan and higher.
cy
Cyprus
Business Plan and higher.
cz
Czechia
Business Plan and higher.
ee
Estonia
Business Plan and higher.
is
Iceland
Business Plan and higher.
lv
Latvia
Business Plan and higher.
li
Liechtenstein
Business Plan and higher.
lt
Lithuania
Business Plan and higher.
mt
Malta
Business Plan and higher.
ro
Romania
Business Plan and higher.
sk
Slovakia
Business Plan and higher.
si
Slovenia
Business Plan and higher.
ec
Ecuador
Business Plan and higher.
pa
Panama
Business Plan and higher.
Other countries are available to Enterprise customers upon request.
ZIP Code Geo targeting is currently supported for the Amazon domain. To find out more, please visit this link.
API REQUEST
PROXY MODE
Geotargeting (Premium)
In addition to the list of Country Codes listed on the Geotargeting page, our premium geotargeting service offers advanced capabilities for accessing and targeting exclusive geographic locations.
To get access to these geographic locations, please use premium=true together with the country_code parameter:
API REQUEST
PROXY MODE
ad
Andorra
Business Plan and higher.
af
Afghanistan
Business Plan and higher.
ag
Antigua and Barbuda
Business Plan and higher.
ai
Anguilla
Business Plan and higher.
al
Albania
Business Plan and higher.
am
Armenia
Business Plan and higher.
ao
Angola
Business Plan and higher.
as
American Samoa
Business Plan and higher.
aw
Aruba
Business Plan and higher.
ax
Åland Islands
Business Plan and higher.
az
Azerbaijan
Business Plan and higher.
ba
Bosnia and Herzegovina
Business Plan and higher.
bb
Barbados
Business Plan and higher.
bf
Burkina Faso
Business Plan and higher.
bh
Bahrain
Business Plan and higher.
bi
Burundi
Business Plan and higher.
bj
Benin
Business Plan and higher.
bl
Saint Barthélemy
Business Plan and higher.
bm
Bermuda
Business Plan and higher.
bn
Brunei
Business Plan and higher.
bo
Bolivia
Business Plan and higher.
bq
Bonaire, Sint Eustatius and Saba
Business Plan and higher.
bs
Bahamas
Business Plan and higher.
bt
Bhutan
Business Plan and higher.
bw
Botswana
Business Plan and higher.
by
Belarus
Business Plan and higher.
bz
Belize
Business Plan and higher.
cd
Democratic Republic of the Congo
Business Plan and higher.
cf
Central African Republic
Business Plan and higher.
cg
Republic of the Congo
Business Plan and higher.
ci
Côte d'Ivoire (Ivory Coast)
Business Plan and higher.
ck
Cook Islands
Business Plan and higher.
cm
Cameroon
Business Plan and higher.
cr
Costa Rica
Business Plan and higher.
cu
Cuba
Business Plan and higher.
cv
Cape Verde
Business Plan and higher.
cw
Curaçao
Business Plan and higher.
dj
Djibouti
Business Plan and higher.
dm
Dominica
Business Plan and higher.
do
Dominican Republic
Business Plan and higher.
dz
Algeria
Business Plan and higher.
er
Eritrea
Business Plan and higher.
et
Ethiopia
Business Plan and higher.
fj
Fiji
Business Plan and higher.
fk
Falkland Islands
Business Plan and higher.
fm
Micronesia
Business Plan and higher.
fo
Faroe Islands
Business Plan and higher.
ga
Gabon
Business Plan and higher.
gd
Grenada
Business Plan and higher.
ge
Georgia
Business Plan and higher.
gf
French Guiana
Business Plan and higher.
gg
Guernsey
Business Plan and higher.
gh
Ghana
Business Plan and higher.
gi
Gibraltar
Business Plan and higher.
gl
Greenland
Business Plan and higher.
gm
The Gambia
Business Plan and higher.
gn
Guinea
Business Plan and higher.
gp
Guadeloupe
Business Plan and higher.
gq
Equatorial Guinea
Business Plan and higher.
gt
Guatemala
Business Plan and higher.
gu
Guam
Business Plan and higher.
gw
Guinea-Bissau
Business Plan and higher.
gy
Guyana
Business Plan and higher.
hn
Honduras
Business Plan and higher.
ht
Haiti
Business Plan and higher.
im
Isle of Man
Business Plan and higher.
io
British Indian Ocean Territory
Business Plan and higher.
je
Jersey
Business Plan and higher.
jm
Jamaica
Business Plan and higher.
kg
Kyrgyzstan
Business Plan and higher.
kh
Cambodia
Business Plan and higher.
km
Comoros
Business Plan and higher.
kn
Saint Kitts and Nevis
Business Plan and higher.
kw
Kuwait
Business Plan and higher.
ky
Cayman Islands
Business Plan and higher.
kz
Kazakhstan
Business Plan and higher.
la
Laos
Business Plan and higher.
lc
Saint Lucia
Business Plan and higher.
lk
Sri Lanka
Business Plan and higher.
lr
Liberia
Business Plan and higher.
ls
Lesotho
Business Plan and higher.
lu
Luxembourg
Business Plan and higher.
ly
Libya
Business Plan and higher.
ma
Morocco
Business Plan and higher.
mc
Monaco
Business Plan and higher.
md
Moldova
Business Plan and higher.
me
Montenegro
Business Plan and higher.
mf
Saint Martin
Business Plan and higher.
mg
Madagascar
Business Plan and higher.
mh
Marshall Islands
Business Plan and higher.
mk
North Macedonia
Business Plan and higher.
ml
Mali
Business Plan and higher.
mm
Myanmar
Business Plan and higher.
mn
Mongolia
Business Plan and higher.
mo
Macau
Business Plan and higher.
mp
Northern Mariana Islands
Business Plan and higher.
mq
Martinique
Business Plan and higher.
mr
Mauritania
Business Plan and higher.
ms
Montserrat
Business Plan and higher.
mu
Mauritius
Business Plan and higher.
mv
Maldives
Business Plan and higher.
mw
Malawi
Business Plan and higher.
mz
Mozambique
Business Plan and higher.
na
Namibia
Business Plan and higher.
nc
New Caledonia
Business Plan and higher.
ne
Niger
Business Plan and higher.
ni
Nicaragua
Business Plan and higher.
np
Nepal
Business Plan and higher.
pf
French Polynesia
Business Plan and higher.
pg
Papua New Guinea
Business Plan and higher.
pm
Saint Pierre and Miquelon
Business Plan and higher.
pr
Puerto Rico
Business Plan and higher.
ps
Palestine
Business Plan and higher.
pw
Palau
Business Plan and higher.
py
Paraguay
Business Plan and higher.
qa
Qatar
Business Plan and higher.
re
Réunion
Business Plan and higher.
rw
Rwanda
Business Plan and higher.
sb
Solomon Islands
Business Plan and higher.
sc
Seychelles
Business Plan and higher.
sd
Sudan
Business Plan and higher.
sh
Saint Helena
Business Plan and higher.
sj
Svalbard and Jan Mayen
Business Plan and higher.
sl
Sierra Leone
Business Plan and higher.
sm
San Marino
Business Plan and higher.
sn
Senegal
Business Plan and higher.
so
Somalia
Business Plan and higher.
sr
Suriname
Business Plan and higher.
ss
South Sudan
Business Plan and higher.
st
São Tomé and Príncipe
Business Plan and higher.
sv
El Salvador
Business Plan and higher.
sx
Sint Maarten
Business Plan and higher.
sz
Eswatini
Business Plan and higher.
tc
Turks and Caicos Islands
Business Plan and higher.
td
Chad
Business Plan and higher.
tj
Tajikistan
Business Plan and higher.
tl
Timor-Leste
Business Plan and higher.
tm
Turkmenistan
Business Plan and higher.
tn
Tunisia
Business Plan and higher.
tv
Tuvalu
Business Plan and higher.
tz
Tanzania
Business Plan and higher.
ug
Uganda
Business Plan and higher.
uy
Uruguay
Business Plan and higher.
uz
Uzbekistan
Business Plan and higher.
va
Vatican City
Business Plan and higher.
vc
Saint Vincent and the Grenadines
Business Plan and higher.
vg
British Virgin Islands
Business Plan and higher.
vi
U.S. Virgin Islands
Business Plan and higher.
vu
Vanuatu
Business Plan and higher.
wf
Wallis and Futuna
Business Plan and higher.
ws
Samoa
Business Plan and higher.
ye
Yemen
Business Plan and higher.
yt
Mayotte
Business Plan and higher.
zm
Zambia
Business Plan and higher.
zw
Zimbabwe
Business Plan and higher.
ZIP Code Geo targeting is currently supported for the Amazon domain. To find out more, please visit this link.
Parameters as Headers
Along with the “traditional” means of passing parameters, we also support passing parameters as headers. Passing parameters such as api_key, render, ultra_premium and instruction_set is very straightforward.
API REQUEST
Instead of including the parameters in the URL
you can just pass them as headers
PROXY MODE
Note that credentials must still be passed to the proxy in the manner shown above, not as headers.
Premium Residential/Mobile Proxy Pools
Our standard proxy pools include millions of proxies from over a dozen ISPs and should be sufficient for the vast majority of scraping jobs. However, for a few particularly difficult to scrape sites, we also maintain a private internal pool of residential and mobile IPs. This pool is available to all paid users.
Requests through our premium residential and mobile pool are charged at 10 times the normal rate (every successful request will count as 10 API credits against your monthly limit). Each request that uses both javascript rendering and our premium proxy pools will be charged at 25 times the normal rate (every successful request will count as 25 API credits against your monthly limit). To send a request through our premium proxy pool, please set the premium query parameter to premium=true.
We also have a higher premium level that you can use for really tough targets, such as LinkedIn. You can access these pools by adding the ultra_premium=true query parameter. These requests will use 30 API credits against your monthly limit, or 75 if used together with rendering. Please note, this is only available on our paid plans. Requests with the ultra_premium=true parameter are cached (by default) to enhance performance and efficiency. For detailed information about how caching works and its benefits, please refer to our Cached Results page.
API REQUEST
PROXY MODE
Learn more about our rotating proxy and residential proxy solution.
Rendering Javascript
If you are crawling a page that requires you to render the Javascript on the page to scrape the data you need, then we can fetch these pages using a headless browser.
To render Javascript, simply set render=true and we will use a headless Google Chrome instance to fetch the page. This feature is available on all plans.
Pass the JavaScript rendering parameter within the URL:
API REQUEST
PROXY MODE
Pass the parameter in the headers:
API REQUEST
PROXY MODE
Render Instruction Set
The Render Instruction Set is a set of instructions that can be used to instruct the browser on specific actions to execute during page rendering. By combining these instructions, you can execute complex operations such as completing a search form or scrolling through an endlessly scrolling page. This capability enables efficient automation of dynamic web content interactions.
How to use
To send an instruction set to the browser, you send a JSON object to the API as a header, along with any other necessary parameters, including the "render=true" parameter.
In the following example, we enter a search term into a form, click the search icon, and then wait for the search results to load.
To send the above instruction set to our API endpoint, it must be formatted as a single string and passed as a header.
API REQUEST
PROXY MODE
Supported Instructions
Browser instructions are organized as an array of objects within the instruction set, each with a specific structure. Below are the various instructions and the corresponding data they require:
Click
Click on an element on the page.
Args
type: str = "click" selector: dict type: Enum["xpath", "css", "text"] value: str timeout: int (in seconds, optional)
Example
[{
"type": "click", "selector": { "type": "css", "value": "#search-form button[type="submit\"]" } }]
Input
Enter a value into an input field on the page.
Args
type: str = "input" selector: dict type: Enum["xpath", "css", "text"] value: str value: str timeout: int (in seconds, optional)
Example
[{
"type": "input", "selector": { "type": "css", "value": "#searchInput" }, "value": "cowboy boots" }]
Loop
Execute a set of instructions in a loop a specified number of times by using the loop instruction with a sequence of standard instructions in the “instructions” argument.
Note that nesting loops isn't supported, so you can't create a “loop” instruction inside another “loop” instruction. This method is effective for automating actions on web pages with infinitely scrolling content, such as loading multiple pages of results by scrolling to the bottom of the page and waiting for additional content to load.
Args
type: str="loop" for: int instructions: array
Example
[{
"type": "loop", "for": 3, "instructions": [ { "type": "scroll", "direction": "y", "value": "bottom" }, { "type": "wait", "value": 5 } ] }]
Scroll
Scroll the page in the X (horizontal) or Y (vertical) direction, by a given number of pixels or to the top or bottom of the page.
Args
type: str = "scroll" direction: Enum["x", "y"] value: int or Enum["bottom", "top"]
Example
[{ "type": "scroll", "direction": "y", "value": "bottom" }]
Wait
Waits for a given number of seconds to elapse.
Args
type: str = "wait" value: int
Example
[{ "type": "wait", "value": 10 }]
Wait_for_event
Waits for an event to occur within the browser.
Args
type: str = "wait_for_event" event: Enum["domcontentloaded", "load", "navigation", "networkidle"] timeout: int (in seconds, optional)
Example
[{ "type": "wait_for_event", "event": "networkidle", "timeout": 10 }]
- domcontentloaded = initial HTML loaded - load = full page load - navigation = page navigation - networkidle = network requests stopped - stabilize = page reaches steady state
Wait_for_selector
Waits for an element to appear on the page. Takes a 'value' argument that instructs the rendering engine to wait for a specified number of seconds for the element to appear.
Args
type: str = "wait_for_selector" selector: dict type: Enum["xpath", "css", "text"] value: str timeout: int (in seconds, optional)
Example
[{ "type": "wait_for_selector", "selector": { "type": "css", "value": "#content"
}, "timeout": 5 }]
Screenshot Capture🆕
Our Java Script solution now gives you the ability to take a screenshot of the target page through the use of the parameter screenshot=true. This parameter automatically enables JS rendering to get the full page content, before taking a screenshot.
After we serve the response, you can find the screenshot URL in the sa-screenshot response header. The screenshot format is PNG:
API REQUEST
ASYNC REQUEST
PROXY MODE - COMING SOON!
Sessions
To reuse the same proxy for multiple requests, simply use the session_number parameter by setting it equal to a unique integer for every session you want to maintain (e.g. session_number=123). This will allow you to continue using the same proxy for each request with that session number. To create a new session simply set the session_number parameter with a new integer to the API. The session value can be any integer. Sessions expire 15 minutes after the last usage.
API REQUEST
PROXY MODE