Scrape Any URL

DataPipeline is ScraperAPI's no-code dashboard for scheduled scraping. In this tutorial, you will use DataPipeline to scrape any URL list on a recurring schedule. By the end, you will have a working scraper for any URL list, with proxies and anti-bot challenges handled automatically, so you can ship data features instead of writing maintenance code.

Step 1: Create the project

Click DataPipeline in the left sidebar.

Select New project.
Select Extract data from arbitrary websites at the top of the tile grid.

Step 2: Configure input settings

Under Input settings, choose List and enter your URLs, one per line:

https://quotes.toscrape.com/page/1/
https://quotes.toscrape.com/page/2/
https://quotes.toscrape.com/page/3/

For larger URL lists, switch to Upload file and upload a .txt or .csv file with one URL per row (up to 100,000).

Step 3: Configure output settings

Under Output settings, choose Download. Results will be available in the dashboard when the job completes. For larger setups, you can enter a Webhook URL to have results posted to your server automatically.

Step 4: Set scraping frequency

Under Scraping frequency, select Just scrape once. To run on a recurring schedule, switch this to Daily, Weekly, or another interval. Recurring schedules require a paid plan.

Step 5: Review and start

Click Review & start scraping. The Job summary panel shows the estimated credit cost before you confirm.

Step 6: Review results

Click into the project after the run completes and click Download results. Results are retained for 30 days.

Additional options

Under Additional options & filters you can fine-tune your requests:

Activate premium residential and mobile IPs: routes requests through residential or mobile proxies to reduce blocks on harder targets.
Activate advanced bypass mechanism: enables additional anti-bot evasion for sites with aggressive detection.
Disable follow redirects: stops the scraper from following HTTP redirects, returning the redirect response directly.
Activate javascript rendering: runs the page in a full browser to capture content loaded dynamically by JavaScript.
Generate screenshot: captures a screenshot of the rendered page alongside the HTML response.
Activate retrying 404 responses: retries requests that return a 404 instead of treating them as final.
Country code: targets a specific country's proxies so the response reflects what a local user would see.
Device type: returns the desktop or mobile version of the page.

Some parameters increase the cost per request. Check the Cost per Job panel before committing to a configuration at scale.

Prefer to set this up in your own codebase? Try the API playground instead.