Some websites may block requests from known scraper user agents. Using your own custom headers can help disguise your requests as normal web traffic and avoid getting blocked. By setting keep_headers=true and sending the API the headers you want to use, you can customize your requests and potentially increase your success rates.
For more information on how to use custom headers and our other features, please see our documentation.
If you're new to custom headers, we don't recommend this approach. Custom headers are typically employed in particular situations, such as making subsequent requests along with a session_number or when you need to send custom headers to obtain specific results from a website. Please note that when you use your own custom headers, our header system will be overridden, which may occasionally lead to lower success rates. Nonetheless, if you need help setting this up let us know, we're always here to help and support you through the process!