Scraperapi Datapipeline

ScraperAPI DataPipeline

DataPipeline is a managed scraping product. You define a project (what to scrape, how often, where to send results), and ScraperAPI runs it on your schedule without you managing proxies, retries, or infrastructure.

When NOT to use DataPipeline

• One-off scrapes of a known URL list → use the Async API — faster, cheaper, no project setup.
• Exploring a site without known URLs → use the Crawler.
• Need results in real-time within your code → Async API is programmable; DataPipeline is scheduled.
• Free plan, need recurring execution → recurring schedules require a paid plan.

Use DataPipeline when: scraping runs on a fixed schedule, the input list is large (up to 100,000 items), results should flow to a webhook automatically, or you want email notifications on job completion.

Base URL and Auth

Base URL: https://datapipeline.scraperapi.com/api
Auth:     ?api_key=YOUR_KEY  (query parameter on every request)

Project Types

Set projectType in the create request to choose what to scrape:

Creating a Project

import os, requests

API_KEY = os.environ["SCRAPERAPI_API_KEY"]
BASE    = "https://datapipeline.scraperapi.com/api"

project = requests.post(
    f"{BASE}/projects",
    params={"api_key": API_KEY},
    json={
        "name":               "Weekly Amazon price monitor",
        "projectType":        "amazon_product",
        "schedulingEnabled":  True,
        "scrapingInterval":   "weekly",
        "scheduledAt":        "now",
        "projectInput": {
            "type": "list",
            "list": ["B09V3KXJPB", "B08N5WRWNW"]   # ASINs
        },
        "apiParams": {
            "country_code": "us"
        },
        "webhookOutput": {
            "url":             "https://yourapp.com/pipeline-results",
            "webhookEncoding": "multipart_form_data_encoding"
        },
        "notificationConfig": {
            "notifyOnSuccess": "with_every_run",
            "notifyOnFailure": "with_every_run"
        }
    }
).json()

print(f"Project created: id={project['id']}")

Create request fields

Input Methods

Direct list (up to 500 items)

{
  "projectInput": {
    "type": "list",
    "list": ["query one", "query two", "B09V3KXJPB"]
  }
}

CSV file (up to 100,000 items)

Upload a CSV with one URL/query/ASIN per line — no header rows, no commas. Do this through the dashboard when creating a project; the API accepts list inputs only.

Webhook input (dynamic polling)

{
  "projectInput": {
    "type": "webhook",
    "webhookUrl": "https://yourapp.com/input-items"
  }
}

ScraperAPI polls your webhook URL for the item list when the job starts. One item per line; no commas. Useful for dynamically generated lists (e.g., new ASINs added since the last run).

Scheduling Options

Recurring schedules (hourly, daily, weekly, monthly, cron) require a paid plan.

Set "scheduledAt": "now" to trigger the first run immediately when the project is created.

Output / Delivery

Webhook delivery

Results are POSTed to your webhook URL as they complete. The webhookEncoding field controls the format:

{
  "webhookOutput": {
    "url":             "https://yourapp.com/results",
    "webhookEncoding": "multipart_form_data_encoding"
  }
}

Dashboard download

Omit webhookOutput and results are saved for download in the DataPipeline dashboard. Results are retained for 30 days then automatically deleted.

Output formats by project type:

• urls / urls_with_js → HTML wrapped in JSONL
• Structured types (Amazon, Google, Walmart, eBay, Redfin) → JSON or CSV

Managing Projects

# List all projects
projects = requests.get(f"{BASE}/projects", params={"api_key": API_KEY}).json()

# Get a single project
project = requests.get(f"{BASE}/projects/525", params={"api_key": API_KEY}).json()

# Update (partial update — only include fields to change)
requests.patch(
    f"{BASE}/projects/525",
    params={"api_key": API_KEY},
    json={
        "scrapingInterval": "daily",
        "apiParams":        {"premium": True},
        "notificationConfig": {"notifyOnSuccess": "never"}
    }
)

# Delete / archive (irreversible without support)
requests.delete(f"{BASE}/projects/525", params={"api_key": API_KEY})

Updatable fields: scrapingInterval, scheduledAt, outputFormat, apiParams, notificationConfig.

Managing Jobs

# List jobs for a project
jobs = requests.get(
    f"{BASE}/projects/525/jobs",
    params={"api_key": API_KEY}
).json()

# Cancel a running job
requests.delete(
    f"{BASE}/projects/525/jobs/{job_id}",
    params={"api_key": API_KEY}
)
# Running requests within the job finish first; final status becomes "Cancelled"

A new job can only start if no other job for that project is currently running.

Notification Config

{
  "notificationConfig": {
    "notifyOnSuccess": "with_every_run",
    "notifyOnFailure": "with_every_run"
  }
}

Options for both fields: "never", "with_every_run", "daily", "weekly".

apiParams Reference

All standard ScraperAPI parameters are supported inside apiParams:

Credit Costs

DataPipeline uses the same underlying credit rates as the Standard API. Cost is the sum of all requests in a job run. Preview the estimated cost before launching a project from the dashboard.

Only successful 200 and 404 responses are charged; failed requests are not.

Limits

Documentation

• DataPipeline overview
• Dashboard — manage projects
• API reference