Endpoints and Parameters 🧩
The following section provides a list of the current API endpoints and parameters. Explore these options to tailor your experience and optimize your integration with our DataPipeline.
List of Endpoints
Fields / Parameters
name
The project name is purely for representation purposes.
schedulingEnabled
true / false
This specifies the automatic rescheduling of a project. If it's set to false, the project won't be rescheduled after the next run (if applicable)
scrapingInterval
- hourly - daily - weekly - monthly - cron expression like "10 * * * *"
scheduledAt
Next project runtime. Should be specified in ISO 8601 format. Simply use "now" as your reference point to initiate at the earliest possible time.
createdAt
The ISO 8601 representation of the date when the project was created
projectType
This can be URLs for simple URL scraping, or a number of other values if you need to get structured data (JSON,CSV)
Valid projectType parameters:
- urls - amazon_product - amazon_offer - amazon_review - amazon_search - google_jobs - google_news - google_search - google_shopping - walmart_category - walmart_product - walmart_search
projectInput
The source of the list of URLs, search terms etc. for the project.
There are 2 valid variants
- A simple list (this is the default):
{"type": "list", "list": ["<https://example.com>", "<https://httpbin.org>"] }
- Webhook input. The result of the address of the webhook has to be a new line delimited list of search terms, asins, etc.
{"type": "webhook_input", "url": "<https://the.url.where.your.list.is>" }
webhookOutput
The results are always saved and you can download them through the results link or through the UI.
If you want to get a webhook callback from your jobs you can add an optional parameter:
...
"webhookOutput": { "url": "<url>"}
...
If you want to encode the results as multipart/form-data (because for example you use webhook.site) you can use the extra
webhookEncoding parameter like that:
"webhookOutput": {
"url": "<url>",
"webhookEncoding": "multipart_form_data_encoding"}
To unset a webhookOutput set it's value ot null
"webhookOutput": null
notificationConfig
Conifugre when to send email notifications about the finished jobs
•notifyOnSuccess - valid values never, with_every_run, daily, weekly
•notifyOnFailure - valid values with_every_run, daily, weekly
apiParams
Please refer to the Async API params —> https://docs.scraperapi.com/making-requests/async-requests-method/api-parameters
output_format
csv / JSON Set output format you want to get for SDE
Last updated