SDK Method

Learn to implement ScraperAPI's Python SDK for efficient web scraping. Includes setup for Requests and Scrapy, with advanced features like JS rendering.

To make it easier to get up and running, you can also use our Python SDK. First, you will need to install the SDK:

pip install scraperapi-sdk

Then integrate it into your code:

from scraperapi_sdk import ScraperAPIClient
client = ScraperAPIClient('APIKEY')
result = client.get(url = 'https://httpbin.org/ip')
print(result)
# Scrapy users can simply replace the urls in their start_urls and parse function
# Note for Scrapy, you should not use DOWNLOAD_DELAY and
# RANDOMIZE_DOWNLOAD_DELAY, these will lower your concurrency and are not
# needed with our API
# ...other scrapy setup code
start_urls =[client.scrapyGet(url = 'https://httpbin.org/ip')]
def parse(self, response):
# ...your parsing logic here
yield scrapy.Request(client.scrapyGet(url = 'https://httpbin.org/ip'), self.parse)

To enable extra functionality while using our SDK, add an additional parameter to the GET request:

from scraperapi_sdk import ScraperAPIClient
client = ScraperAPIClient('APIKEY')
#Requests Users
result = client.get(url = 'https://httpbin.org/ip', params={'render': True})
# Scrapy users
yield scrapy.Request(client.scrapyGet(url = 'https://httpbin.org/ip', params={'render': True}), self.parse)

To use two or more parameters, simply add them to the GET request:

from scraperapi_sdk import ScraperAPIClient
client = ScraperAPIClient('APIKEY')
#Requests Users
result = client.get(url = 'https://httpbin.org/ip', params={'render': True, 'premium': True})
# Scrapy users
yield scrapy.Request(client.scrapyGet(url = 'https://httpbin.org/ip', params={'render': True, 'premium': True}), self.parse)