search

How to use

To help you get started with the DataPipeline API endpoints, please review the following examples. These will guide you through the process of creating, managing and integrating your projects and jobs programmatically, ensuring a seamless access to this powerful feature.

hashtag
Table of contents

hashtag
Create a New Project

triangle-exclamation

Scheduling is enabled by default and set to a weekly interval. You can disable it by adding schedulingEnabled: false

Example:

curl -X POST --data '{ "projectInput": {"type": "list", "list": 
["https://www.amazon.com/AmazonBasics-3-Button-Wired-Computer-1-Pack/dp/B005EJH6RW/"] } }' 
-H 'content-type: application/json' 
'https://datapipeline.scraperapi.com/api/projects?api_key=xxxxxx'

Output:

{
    "id": 522,
    "name": "Project created at 2024-05-10T16:04:28.263Z",
    "schedulingEnabled": true,
    "scrapingInterval": "weekly",
    "createdAt": "2024-05-10T16:04:28.306Z",
    "projectType": "urls",
    "projectInput": {
        "type": "list",
        "list": [
            "https://www.amazon.com/AmazonBasics-3-Button-Wired-Computer-1-Pack/dp/B005EJH6RW/"
        ]
    },
    "projectOutput": {
        "type": "save"
    },
    "notificationConfig": {
        "notifyOnSuccess": "never",
        "notifyOnFailure": "with_every_run"
    }
}

Except for id and the createdAt fields, everything can be specified during project creation.

hashtag
Specifying Parameters

This is what project creation looks like when additional parameters are specified

Example:

Output:

hashtag
Updating Parameters Of An Existing Project

Example:

Output:

hashtag
Specifying Project Type

Similarly to the DataPipeline GUI, you can specify a project type (e.g. Google Search) here as well. If not specified, projectType will be defaulted to URLs (for simple URL scraping).

Example:

Output:

hashtag
Specifying Project Input

Choose the source for providing lists of URLs, search terms, and other relevant data for your project:

-A simple list (this is the default)

-Webhook Input - With every run the system downloads the contents from the webhook, allowing you to update the URLs list, ASINs, etc. without interacting with ScraperAPI.

Example:

Output:

hashtag
Changing Output Format For SDE Project Types

The default output format is JSON, but you can opt for CSV if it better suits your needs for the SDE project types.

Example:

Output:

hashtag
Changing Scheduling Interval

Example:

Output:

hashtag
Creating a Project With a Custom Cron Scheduling

Cron is a time-based job scheduler. It allows users to schedule jobs to run periodically at fixed times, dates, or intervals. A cron expression consists of five fields that define when a job should execute:

  • Minute (0-59)

  • Hour (0-23)

  • Day of the month (1-31)

  • Month (1-12 or JAN-DEC)

  • Day of the week (0-7 or SUN-SAT, where 0 and 7 represent Sunday)

For example, a cron expression "0 0 * * 1" means the job will run at midnight (00:00) every Monday (1 represents Monday in cron):

Output:

hashtag
Rescheduling The Project's Next Run Date

This works even with schedulingEnabled set to false, ensuring the project runs once.

circle-info

If you want to schedule the project to run as soon as possibele you can use the now interval.

hashtag
Configuring Webhook Output

circle-info

If you are using webhook.site, it needs the content to be encoded as multipart/form-data so an extra parameter is used here.

hashtag
Set the webhook output

Example:

Output:

hashtag
Unset the webhook output

hashtag
Configuring Notifications

Define when you would like to receive email notifications for completed jobs. This can be done at project level creation or eddited later on. Both notifyOnSuccess and notifyOnFailure have to be specified.

Example (on project creation):

Output:

Example (updating existing project):

Output:

hashtag
Check The List of Jobs For a Project

You can list project's jobs with this endpoint:

hashtag
Cancel a Job Within a Project

Canceling a job within a project allows you to manage projects effectively by halting specific jobs that are no longer needed.

Example:

Once you delete the job, the system will respond back with ok if the command was successful

If we now lookup that same project, we'll see the job status is cancelled:

hashtag
Archiving a Project

To archive an existing project, just send the following request:

Example:

When you try to look up that project now, the system will return Project not found

These changes are also visible on the Dashboard, under the DataPipeline Projects list.

Before (7 Projects in total):

After (6 Projects in total):

hashtag
Error Messages

PreviousEndpoints and ParametersNextJobs & Costs Management

Last updated