Claude Desktop
Remote Integration
Prerequisites
ScraperAPI account.
Claude (used in this guide).
Setup
If you don't have an account with us yet, head over to scraperapi.com to create one and grab your API key from the Dashboard area. You will need it to authenticate the requests that your LLM client will be making.
Configuration for Claude Desktop App:
Open Claude Desktop Application.
Access the Settings Menu.
Click on the settings icon (typically a gear or three dots in the upper right corner).
Select the "Developer" tab.
Click on "Edit Config" and paste the JSON block in the configuration file.
{
"mcpServers": {
"ScraperAPIRemote": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.scraperapi.com/mcp",
"--header",
"Authorization: Bearer {YOUR_API_KEY}"
]
}
}
}Parameters
scrape (required)
Tells the LLM to scrape a URL from the internet using ScraperAPI.
url (requierd)
URL you wish to scrape.
render (optional)
Defaults to False. Set to True if the page requires JavaScript rendering to display its contents.
country_code (optional)
Activate country geotargeting (e.g. “us”, “es”, “uk”, etc.).
premium (optional)
Set to True to use residential IPs with your scrapes.
ultra_premium (optional)
Activates advanced bypass mechanisms when set to True. Can not be combined with premium.
device_type (optional)
Defaults to desktop. Set to mobile to use mobile user agents with the scrapes.
Prompt Examples
Scrape Zillow to get Real Estate insights
- "Scrape this Zillow search results URL https://www.zillow.com/queens-new-york-ny/under-500000/ and return a list of homes with: address, neighborhood, price, beds, baths, square feet (if available), listing URL, and a 1-2 sentence summary."
Response
Claude successfully scraped the search results page, but the response exceeded Claude’s 1MB MCP output limit, so the data could not be returned in full at first. To work around this limit, the request was retried using a different output format.
A list of properties, including pricing details and property-related information, was successfully extracted and returned once the response was reformatted to fit within Claude's MCP output size limit.
Crawl Walmart Seller Profile pages to extract product data
- "I want to crawl this Walmart seller page https://www.walmart.com/brand/bose/10026932 to get and scrape the product URLs for that seller (max depth should be 1), then have the data streamed to this webhook: https://webhook.site/203557d4-0c71-437d-9093-86d42f5d2b79. Starting URL is https://www.walmart.com/brand/bose/10026932. The budget for this is 500 API Credits. Use the following as regexp: urlRegexpInclude: .*/ip/.* The name of the crawling job should be 'Walmart Seller Profile page products' and it should be ran only once."
Response
Claude automatically configured and created a new Crawler job via the ScraperAPI Crawler, using the provided information. It also offers to check the crawl job status, making it easy to monitor progress and confirm when results are being delivered.
Google Shopping product search
- "I want to get an All-Clad kitchen utensils set. Check what's available on Google in the US and return the top 10 results. My budget is $2000."
Response
Claude executed a Google Shopping search using ScraperAPI, then extracted and returned the top 10 results, presenting them in a structured table for easy comparison.
ScraperAPI plays a key role in making these workflows possible by handling the heavy lifting behind the scenes. It ensures that Claude can retrieve accurate results efficiently, even when dealing with difficult-to-scrape websites.
Local (Self-Hosted) Integration
Prerequisites
ScraperAPI Account
Python 3.11 +
Claude (used in this guide)
Docker (optional)
Install the scraperapi-mcp-server using pip:
Setup
If you don't have an account with us yet, head over to scraperapi.com to create one and grab your API key from the Dashboard area. You will need it to authenticate the requests that your LLM client will be making.
Now that you have your API Key, add this JSON block to your client configuration file:
Using Python:
Using Docker:
Configuration for Claude Desktop App:
Open Claude Desktop Application.
Access the Settings Menu.
Click on the settings icon (typically a gear or three dots in the upper right corner).
Select the "Developer" tab.
Click on "Edit Config" and paste the JSON block in the configuration file.
That's it, the MCP Server is fully configured! Include the keyword scrape in a prompt, and the LLM will automatically use ScraperAPI to retrieve the data you need.
If you are interested in digging deeper, you can set up the MCP server locally on your machine. Here you will find instructions on how to setup, debug and customize the server for advanced development.
Parameters
scrape (required)
Tells the LLM to scrape a URL from the internet using ScraperAPI.
url (requierd)
URL you wish to scrape.
render (optional)
Defaults to False. Set to True if the page requires JavaScript rendering to display its contents.
country_code (optional)
Activate country geotargeting (e.g. “us”, “es”, “uk”, etc.).
premium (optional)
Set to True to use residential IPs with your scrapes.
ultra_premium (optional)
Activates advanced bypass mechanisms when set to True. Can not be combined with premium.
device_type (optional)
Defaults to desktop. Set to mobile to use mobile user agents with the scrapes.
Prompt Template
Please scrape this URL
<URL>. If you receive a 500 server error, identify the website's geo-targeting and add the corresponding country_code to overcome geo-restrictions. If errors continues, upgrade the request to use premium proxies by adding premium=true. For persistent failures, activate ultra_premium=true to use enhanced anti-blocking measures.Can you scrape URL
<URL>to extract<SPECIFIC_DATA>? If the request returns missing/incomplete<SPECIFIC_DATA>, set render=true to enable JS Rendering.
Example Prompt
"Please scrape this URL https://www.lowes.com/pd/Kozyard-12-ft-x-16-ft-Gazebo-Dark-Brown-Metal-Square-Screened-Gazebo-with-Steel-Roof/5014900669. If you receive a 500 server error identify the website's geo-targeting and add the corresponding country_code to overcome geo-restrictions. If errors continues, upgrade the request to use premium proxies by adding premium=true. For persistent failures, activate ultra_premium=true to use enhanced anti-blocking measures. Get me the price of the product."
Response
The LLM first attempted to scrape the product page with a flat request (no extra parameters), but it was unable to find the price information requested in the prompt. It then identified that the price was likely being loaded dynamically via JavaScript and determined that enabling JS Rendering (render=true) would be the next step.
The LLM then retried the request, this time with render=true and successfully retrieved the price information.
The full set of price options, product details, and availability were successfully extraced once JavaScript Rendering was applied.
This illustrates why ScraperAPI is essential to the process. Without the smart proxy, user agent rotation, and JavaScript Rendering, the LLM would not have been able to access the webpage undisrupted and capture the price details successfully.
Last updated