Anti-Bots & CAPTCHAs

Getting bans from Cloudflare, PerimeterX, Datadome, etc.

Along with continuously fine-tuning our proxy and header pools, we've built numerous anti-bot bypasses, that enable the API to bypass most challenges thrown by popular bot-protection systems. The success rates can be a slightly lower on sites that make heavy use of anti-bot protection, but you should still be able to scrape those sites reliably at scale with ScraperAPI.

If you are fully blocked by a site or your success rate drops below 70%, please notify our support team. This usually happens when a site uses multiple anti-bot systems at once or is using a higher-security custom implementation that blocks our bypasses.

How CAPTCHA solving works

After we receive a HTML response back from your target website, we automatically run it through our ban and CAPTCHA detection algorithms. If the API detects a CAPTCHA or a bot block page, it will retry the request with a different IP and User Agent. This internal retry mechanism ensures that you don't get an invalid 200 response and you don’t wait until the CAPTCHA is solved, before you can retry the request.

Getting a CAPTCHA as successful response

We use a large database of known CAPTCHAs and block-page signatures to detect when a request has been challenged or blocked. That said, its not uncommon for bot protection systems to introduce new CAPTCHAs and modify old ones. If you receive a CAPTCHA or bot-block page as a successful status 200 response, contact our support team and they'll add it to our detection system, so its automatically retried internally in the future.

Solving embedded CAPTCHAs

Currently, the API doesn’t solve CAPTCHAs that are permanently embedded on the page, like those often found on forms or buttons to reveal personal information. You will need to use a dedicated CAPTCHA solver service to solve these CAPTCHAs. For Enterprise plans, we can look into handling the CAPTCHA-solving for a given domain or use-case, subject to technical feasibility and scope.