LlamaIndex Integration

Overview

The LlamaIndex Tools integration plugs ScraperAPI directly to LlamaIndex agents, so that LLM pipelines can execute scraping tasks. This integration lets your LlamaIndex agent scrape web pages and pull structured data from Amazon, Google, eBay, Walmart and Redfin.

Useful links:

• PyPI package: llama-index-tools-scraperapi

• GitHub repo: https://github.com/scraperapi/llama-index-tools-scraperapi

Installation

pip install llama-index-tools-scraperapi

Using the LlamaIndex ToolSpec

Prerequisites:

• ScraperAPI Account

• OpenAI Account (used in the example)

• Python 3.10+ installed

import asyncio
import os
from llama_index.tools.scraperapi import ScraperAPIToolSpec
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

async def main():
    scraper_tool = ScraperAPIToolSpec(
        api_key=os.environ["SCRAPERAPI_API_KEY"],
    )
    agent = FunctionAgent(
        tools=scraper_tool.to_tool_list(),
        llm=OpenAI(model="gpt-4.1"),
    )
    response = await agent.run(
        "Scrape https://example.com and summarize the content"
    )
    print(response)

asyncio.run(main())

Example Response

The content of the page at "https://example.com" is a placeholder text indicating that
the domain is intended for use in documentation examples without requiring permission. 
It advises against using it in operational contexts. 
More information can be found at [iana.org/domains/example](https://iana.org/domains/example).

Available Tools

Tool	Description	Example usage
scrape	Return the raw HTML content of any URL.	"Scrape https://example.com"
amazon	Extract structured data from an Amazon product page (title, price, images, etc.).	"Look up the Amazon product page for ASIN B0FFMRH228"
google	Perform a Google Search and return organic results (title, link, snippet).	"Search Google for 'best web scraping tools 2026'"
ebay	Search eBay listings and extract product information.	"Find the top three eBay listings for 'vintage headphones'"
walmart	Extract product data from Walmart pages.	"Search Walmart for 'wireless headphones' and list the top 3 results"
redfin	Extract real‑estate data from Redfin listings.	"Get the details of 123 Main St from Redfin"

Each of these tools uses ScraperAPI on the backend. Requests count toward your ScraperAPI credit pool, and domains with anti-bot protection may consume more API credits.

MCP Server Integration

If you prefer not to install the tool package, you can access the same tools through our MCP server instead. The MCP server exposes these tools over a streamable HTTP API, allowing you to use them without installing an additional Python package.

Checkout the MCP Server section for more info on the MCP server itself.

Prerequisites:

pip install llama-index llama-index-tools-mcp llama-index-llms-openai

Example

"""
Example: Using ScraperAPI MCP Server with LlamaIndex

Prerequisites:
    pip install llama-index llama-index-tools-mcp llama-index-llms-openai

Usage:
    export SCRAPERAPI_KEY="your-scraperapi-key"
    export OPENAI_API_KEY="your-openai-api-key"
    python examples/llamaindex_example.py
"""

import asyncio
import os

from llama_index.core.agent import FunctionAgent, AgentWorkflow
from llama_index.llms.openai import OpenAI
from llama_index.tools.mcp import BasicMCPClient, McpToolSpec


async def main():
    scraperapi_key = os.environ["SCRAPERAPI_KEY"]
    mcp_url = os.environ.get("MCP_URL", "https://mcp.scraperapi.com/mcp")

    # Connect to the ScraperAPI MCP server over Streamable HTTP
    mcp_client = BasicMCPClient(
        mcp_url,
        headers={"Authorization": f"Bearer {scraperapi_key}"},
    )

    # Load all MCP tools as LlamaIndex tools
    mcp_tool_spec = McpToolSpec(client=mcp_client)
    tools = await mcp_tool_spec.to_tool_list_async()

    print(f"Loaded {len(tools)} tools:")
    for tool in tools:
        print(f"  - {tool.metadata.name}: {tool.metadata.description[:80]}...")

    # Create an agent with the MCP tools
    llm = OpenAI(model="gpt-4o")
    agent = FunctionAgent(
        name="scraper_agent",
        description="Agent that uses ScraperAPI tools to search and scrape the web",
        tools=tools,
        llm=llm,
    )
    workflow = AgentWorkflow(agents=[agent])

    # Example queries
    queries = [
        "Search Google for 'best web scraping tools 2026' and summarize the top results",
        "Look up the Amazon product page for ASIN B0FFMRH228",
        "Search Walmart for 'wireless headphones' and list the top 3 results with prices",
    ]

    query = queries[1]
    print(f"\n{'='*60}")
    print(f"Query: {query}")
    print(f"{'='*60}\n")

    response = await workflow.run(user_msg=query)
    print(f"\nAgent response:\n{response}")


if __name__ == "__main__":
    asyncio.run(main())

Example Respose

Loaded 22 tools:
  - scrape: Scrape the full content of any web page via ScraperAPI. Returns the page body in...
  - crawler_job_start: Start an asynchronous crawl that discovers and scrapes pages by following links ...
  - crawler_job_status: Check the progress of a running or completed crawler job started with crawler_jo...
  - crawler_job_delete: Cancel a running crawler job and permanently delete its data. This action is irr...
  - google_search: Search Google and get structured results (organic links, snippets, knowledge pan...
  - google_jobs: Search Google Jobs and get structured job listing results (title, company, locat...
  - google_news: Search Google News and get structured news article results (title, source, date,...
  - google_shopping: Search Google Shopping and get structured product results (title, price, seller,...
  - google_maps_search: Search Google Maps and get structured place results (name, address, rating, phon...
  - amazon_product: Get structured details for a single Amazon product by its ASIN (title, price, ra...
  - amazon_search: Search Amazon by keyword and get structured product results (title, ASIN, price,...
  - amazon_offers: Get all seller offers for an Amazon product by ASIN (seller name, price, conditi...
  - walmart_search: Search Walmart by keyword and get structured product results (title, price, rati...
  - walmart_product: Get structured details for a single Walmart product by its product ID (title, pr...
  - walmart_category: Browse products in a Walmart category by its category ID. Returns parsed JSON wi...
  - walmart_review: Get structured customer reviews for a Walmart product by its product ID (rating,...
  - ebay_search: Search eBay by keyword and get structured listing results (title, price, conditi...
  - ebay_product: Get structured details for a single eBay listing by its item ID (title, price, c...
  - redfin_for_sale: Get structured data for a Redfin property listing that is for sale (price, beds,...
  - redfin_for_rent: Get structured data for a Redfin rental property listing (rent price, beds, bath...
  - redfin_search: Search Redfin property listings and get structured results (address, price, beds...
  - redfin_agent: Get structured data for a Redfin real estate agent profile (name, brokerage, rat...

============================================================
Query: Look up the Amazon product page for ASIN B0FFMRH228
============================================================


Agent response:
Here is the information for the Amazon product with ASIN B0FFMRH228:

### Product Details
- **Name:** Vivimeng USA Sweatshirt Hoodie for Men American Flag Sweater Women Casual Long Sleeve Pullover Unisex USA Sweater Hooded
- **Brand:** [Visit the Vivimeng Store](https://www.amazon.com/stores/DailyNewLook/page/E3CC6E5E-0DB5-4E66-B8B9-9E585AB6B159?lp_asin=B0FS1J95KY&ref_=ast_bln&store_ref=bl_ast_dp_brandlogo_sto)
- **Price:** $24.99
- **Department:** Womens
- **Date First Available:** September 22, 2025
- **Best Sellers Rank:**
  - #478,166 in Clothing, Shoes & Jewelry
  - #540 in Women's Novelty Hoodies
  - #7,094 in Men's Novelty Hoodies
  - #304,756 in Women's Fashion
- **Customer Reviews:** 3.8 out of 5 stars (33 ratings)
- **Availability:** Only 2 left in stock - order soon.
- **Ships From:** Amazon
- **Sold By:** [Pinrong-](https://www.amazon.com/gp/help/seller/at-a-glance.html/ref=dp_merchant_link?ie=UTF8&seller=AEPY0S5X9FKQY&asin=B0FFMRH228&ref_=dp_merchant_link&isAmazonFulfilled=1)

### Description
- **Fabric Type:** 100% Polyester
- **Care Instructions:** Hand Wash Only
- **Origin:** Imported
- **Closure Type:** Pull On
- **Features:**
  - Show your patriotic spirit with the American Flag Sweater Hoodie, featuring a striking USA print.
  - Crafted from high-quality materials for exceptional comfort and durability.
  - Relaxed fit with a kangaroo pocket, easy to pair with jeans, shorts, or skirts.
  - Versatile for any occasion, ideal for layering or wearing on its own.

### Images
![Product Image](https://m.media-amazon.com/images/I/41Aj3isz70L._AC_.jpg)

### Reviews
- **5 Stars:** "Patriotic Sweater" - My dad loves the color RED and is very patriotic. He needs some more sweaters without holes. This is a nice bright red, soft, and big enough for him to wear.
- **2 Stars:** "Not impressed. Will not buy again." - These sweatshirts do not run true to size. Run very small.
- **4 Stars:** "Just found by searching" - It does shrink if you put it in the dryer.

For more details, you can visit the [Amazon product page](https://www.amazon.com/dp/B0FFMRH228).