Endpoints and Parameters 🧩

The following section provides a list of the current API endpoints and parameters. Explore these options to tailor your experience and optimize your integration with our DataPipeline.

List of Endpoints

Create a project
POST   <https://datapipeline.scraperapi.com/api/projects>

Get a single project
GET    <https://datapipeline.scraperapi.com/api/projects/:id>

Update a project
PATCH  <https://datapipeline.scraperapi.com/api/projects/:id>

Archive a project
DELETE <https://datapipeline.scraperapi.com/api/projects/:id>   

List projects
GET    <https://datapipeline.scraperapi.com/api/projects>

List the jobs of a project
GET    <https://datapipeline.scraperapi.com/api/projects/:id/jobs>

Cancel a job
DELETE <https://datapipeline.scraperapi.com/api/projects/:id/jobs/:jobId>

Fields / Parameters


The project name is purely for representation purposes.


true / false

This specifies the automatic rescheduling of a project. If it's set to false, the project won't be rescheduled after the next run (if applicable)


- hourly - daily - weekly - monthly - cron expression like "10 * * * *"


Next project runtime. Should be specified in ISO 8601 format. Simply use "now" as your reference point to initiate at the earliest possible time.


The ISO 8601 representation of the date when the project was created


This can be URLs for simple URL scraping, or a number of other values if you need to get structured data (JSON,CSV)

Valid projectType parameters:

- urls - amazon_product - amazon_offer - amazon_review - amazon_search - google_jobs - google_news - google_search - google_shopping - walmart_category - walmart_product - walmart_search


The source of the list of URLs, search terms etc. for the project.

There are 2 valid variants - A simple list (this is the default): {"type": "list", "list": ["<https://example.com>", "<https://httpbin.org>"] } - Webhook input. The result of the address of the webhook has to be a new line delimited list of search terms, asins, etc. {"type": "webhook_input", "url": "<https://the.url.where.your.list.is>" }


The results are always saved and you can download them through the results link or through the UI. If you want to get a webhook callback from your jobs you can add an optional parameter: ... "webhookOutput": { "url": "<url>"} ... If you want to encode the results as multipart/form-data (because for example you use webhook.site) you can use the extra webhookEncoding parameter like that: "webhookOutput": { "url": "<url>", "webhookEncoding": "multipart_form_data_encoding"} To unset a webhookOutput set it's value ot null "webhookOutput": null


Conifugre when to send email notifications about the finished jobs

notifyOnSuccess - valid values never, with_every_run, daily, weekly

notifyOnFailure - valid values with_every_run, daily, weekly



csv / JSON Set output format you want to get for SDE

Last updated