Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Scrape Any URL

API Playground is ScraperAPI's request builder for testing and configuring scraping requests. In this tutorial, you will use the API Playground to scrape any URL. By the end, you will have working request code, with proxies and anti-bot challenges handled, so you can integrate web data into your application without building scraping infrastructure.

Step 1: Build the request

Open the API Playground from the left sidebar.

  1. Select scraping method: API.

  2. In the URL field, enter https://quotes.toscrape.com/.

  3. Review the Summary panel on the right to understand the cost of the job before you run it.

  4. Scroll down to the generated code and click Try it.

The Playground runs the request and shows the raw HTML response inline.

Use the language dropdown to switch to Python, Node.js, PHP, Ruby, or Java. Click Copy to clipboard to run the request in your own terminal instead.

Step 2: Additional options and filters

Use Additional options & filters to fine-tune your request. The Playground adds the relevant parameters to the generated code automatically.

  • Premium residential and mobile IPs adds premium=true. Higher success rate on difficult sites, at a higher credit cost.

  • Advanced bypass mechanism adds ultra_premium=true. For the most aggressively protected sites.

  • Javascript rendering adds render=true. Use for pages that load content dynamically.

  • Country code adds country_code. Targets a specific country's proxies so the response reflects what a local user would see.

  • Device type adds device_type=mobile or device_type=desktop. Use mobile to receive the mobile version of a page.

  • Max Cost sets a credit ceiling per request. The request fails rather than exceeding the limit you set.

Some parameters increase the cost per request. Check the Summary panel before committing to a configuration at scale.

Step 3: Submit a batch of URLs

For large URL lists, switch the scraping method to Async in the Playground. Instead of a single URL field, you get a text area that accepts multiple URLs, one per line. The generated request changes from a synchronous GET to an async batch job submission. Each URL in the batch gets its own job, and the response returns a statusUrl per job that you use to check progress.

Paste these URLs to try it:

Step 4: Poll for results

Async jobs run in the background. Use the statusUrl from the batch response to check progress. When status is finished, the response.body field contains your scraped data.

Prefer a no-code setup? Scrape any URL with DataPipeline instead.

More API tutorials

PreviousAPI TutorialsNextTrack Prices

Last updated