# Scrape

> End-to-end web scraping workflow — from URL to working spider with web-poet page objects. Use this for full-site or multiple-page crawls that create a new scraping workflow. Do not use for fixing, debugging, or modifying an existing spider.

- Skill: `zytedata/scrape` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add zytedata/scrape`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zytedata/scrape/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: zytedata (https://skillmd.com/u/zytedata)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zytedata/scrape

---


You are orchestrating the full web scraping workflow, from a user's prompt to a
working Scrapy spider with web-poet page objects.

## Prerequisites

Requires `uv`. Install if missing.

## Input

The raw argument string is `$ARGUMENTS`. Split it into 2 positional arguments:

1. **url**: target website URL (first whitespace-separated token)
2. **what**: what the user wants to extract (the rest after the URL, e.g. "product", "job listing", "recipe" — free text, may contain spaces)

## Track progress

Before Stage 1, create exactly these tasks with `TaskCreate`, in order:
  1. "Decide which fields to extract" — `/scrape-define`
  2. "Analyze the website" — `/scrape-spec`
  3. "Create the Scrapy project" — `/scrape-ensure-project`
  4. "Generate the extraction code" — `/scrape-codegen` (one per data type)
  5. "Generate the spider" — `/scrape-create-spider`

As you launch each skill, `TaskUpdate` the task to `in_progress`. Mark it `completed`
only after the skill (all instances of the skill in case of `/scrape-codegen`)
returns successfully. Do not batch updates — flip status at the boundary so the user
sees live progress.

Do NOT create tasks inside the sub-skills; they share this session's task list
and would duplicate entries.

## Stage 1: Define schema

Invoke `/scrape-define` with the user's arguments. This downloads 1 detail page,
discovers fields, and runs a fast terminal approval loop for the schema.

Output: `.scrape/{site_name}/` with approved schema (including `examples` from user-verified values). No stored pages or value files — those come from Stage 2.

## Stage 2: Explore and validate

Invoke `/scrape-spec .scrape/{site_name}`. This downloads more detail and listing
pages, compares HTML variants, extracts values, and optionally presents a browser
review.

## Stage 3: Generate working project

### Ensure the Scrapy project

Derive a project name from the domain (e.g., `books_toscrape_com`).

```
/scrape-ensure-project ./{project_name} {project_name}
```

### Generate page objects (per data type)

After `scrape-spec` returns, re-read `{site_path}/spec.json` to get the final
`data_types` list — `scrape-spec` may have replaced `{data_type}` with
`{data_type}-list` when all requested fields were found on list pages.

Identify the primary data type: the first entry in `data_types` that is not
`"navigation"`. If it ends with `-list` (e.g., `"products-list"`), the site
will use list-page extraction. Otherwise it uses detail-page extraction (the
default). Navigation is always present.

Call codegen once per data type in `data_types`:

```
/scrape-codegen .scrape/{site_name}/{primary_type} ./{project_name}
/scrape-codegen .scrape/{site_name}/navigation ./{project_name}
```

Where `{primary_type}` is whatever `scrape-spec` wrote — `products-list` for
the list path, `products` (or the site's data type name) for the detail path.
Each call adds its own item class, page object, and fixtures to the project.

## Spider generation

After codegen, determine the PO class import paths from the generated files, then:

```
/scrape-create-spider ./{project_name} {item_po_import_path} {nav_po_import_path}
```

Also provide start URLs in the prompt (the site URL from spec.json).

When the primary data type ends with `-list`, the generated item PO class name
ends with `ListPage` (e.g., `ProductListPage`). `scrape-create-spider` detects
this suffix and automatically generates a list-extraction spider (items yielded
directly from `parse()`, no `parse_item` callback).

When the primary data type is a plain detail type, the spider uses the standard
detail-extraction pattern (navigation in `parse()`, items in `parse_item()`).

Tests with a limited crawl.

## Report

```
Created scraping solution for {domain}:
  Project: ./{project_name}/
  Spider: uv run scrapy crawl {spider_name}
  Tests: uv run pytest fixtures/
```

Offer to help the user deploy to [Scrapy
Cloud](https://docs.zyte.com/scrapy-cloud/get-started.md) if they wish. It's
useful for scheduled or long-running crawls, to keep a job history with results
and logs, for job monitoring (with an API that an LLM can use), and more. There
is also a [free tier](https://docs.zyte.com/scrapy-cloud/pricing.md).

If they deploy and run it there (via `/scrape-scrapy-cloud`), the run will by
default wait for the job to finish and validate its results (errors, field
coverage, and whether items match the request) before reporting back — so
expect it to take as long as the crawl does unless they opt out.