# Scrape Define

> Create a new extraction spec from a URL — explore a detail page, discover fields, quick schema approval

- Skill: `zytedata/scrape-define` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zytedata/scrape-define`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zytedata/scrape-define/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: zytedata (https://skillmd.com/u/zytedata)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zytedata/scrape-define

---


You are helping the user quickly define what to extract from a website. Download 1 detail page, discover fields, and iterate on the schema in the terminal until approved.

Read `${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md`.

The output is a draft spec folder with an approved schema and values (no stored pages — those are Stage 2's job). It can be expanded with `/scrape-spec` (more pages, variant comparison, navigation).

**Hard constraints — never violate these:**
- You MUST NOT fetch, read, grep, or parse any HTML file yourself. Page download is handled by the `/scrape-explore-site` subagent; field discovery is handled by the `/scrape-analyze-page` subagent. The main agent only orchestrates and consumes their outputs.
- This includes `raw.html`, `rendered.html`, `*.cleaned.html`, and any other saved page HTML. You may check whether a file exists, but you MUST NOT open it, print it, grep it, parse it, or run extraction scripts against it from the main agent.
- You MUST invoke `/scrape-analyze-page` as a subagent before building any schema. Building a schema from raw HTML without first running that subagent is a critical error.
- While waiting for `/scrape-analyze-page`, do NOT read HTML files, run `clean_html.py`, `extract_metadata.py`, or any inline parser code, even if the agent response suggests continuing unrelated work. Wait for the subagent to complete, then read only its saved JSON output.


## Parse intent

From $ARGUMENTS, determine:
- **target_url**: the starting URL
- **data_type**: what is being extracted, **always singular** — whether the user names it or you infer it from the URL/page. If the user says "books" use "book", "products" → "product", "articles" → "article". If inferring, use the singular item noun (e.g. "product" not "products", "book" not "books"). Singularize before using anywhere.
- **field_hints**: any specific fields the user mentioned (may be empty)
- **site_name**: a short identifier (e.g. "books-toscrape", "realestate-listings") — derive from the site and data type

## Step 1: Set up folders

Ensure site_name is unique:
```bash
BASE="books-toscrape"  # your derived site_name
NAME="$BASE"
N=2
while [ -d ".scrape/$NAME" ]; do NAME="${BASE}-${N}"; N=$((N+1)); done
echo "$NAME"
```

Create the folder structure (`{data_type}` must be singular, e.g. `book` not `books`):
```
.scrape/{site_name}/
  {data_type}/
.scrape/.work/{site_name}/
  explore/
  analyze-page/
```

## Step 2: Get a detail page

Ask the user how to obtain a detail page via `AskUserQuestion`:

- **question**: "How should I get a detail page to analyze?"
- **header**: "Detail page"
- **options**:
  - `Provide a URL` — "I'll paste a detail page URL (e.g. a product page)."
  - `Explore the site` — "Find one automatically from the target URL."

**If the user picks "Provide a URL"**: follow up with a plain-text prompt — "Paste the detail page URL." — and wait for their next message. Then download that page using `download.py`:
```bash
uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/download.py --skill scrape-define --log-file .scrape/.work/{site_name}/explore/download.log <<'EOF'
[{"url": "USER_URL", "output_dir": ".scrape/.work/{site_name}/explore/pages/detail-1", "page_type": "detail"}]
EOF
```

**If the user picks "Explore the site"**: Use a subagent to run the `/scrape-explore-site` skill with minimal counts. The subagent prompt should be:

> Run /scrape-explore-site {target_url} .scrape/.work/{site_name}/explore 1 0

This downloads the homepage + 1 detail page into `.scrape/.work/{site_name}/explore/pages/`.

If the site is blocked, suggest Zyte. Only invoke `/scrape-zyte-login` if the user agrees. After it returns, re-run the `scrape-explore-site` subagent above.

## Step 3: Analyze the detail page

Use `rendered.html` for analysis. If it doesn't exist, fall back to `raw.html`. Choose between them with a file-existence check only; do not read, grep, print, clean, or parse either HTML file in the main agent.

Run a subagent with the analysis skill. The subagent prompt MUST start with `/scrape-analyze-page`, and the subagent must invoke that skill; it is not acceptable for the main agent to run `clean_html.py`, `extract_metadata.py`, `sed`, `grep`, `cat`, or inline Python/lxml parsing as a substitute.

```
Agent(description="analyze detail-1 rendered", prompt="/scrape-analyze-page Extract data from .scrape/.work/{site_name}/explore/pages/detail-1/rendered.html and save it into .scrape/.work/{site_name}/analyze-page/detail-1.rendered.json")
```

After starting the analysis subagent, wait for it to finish. If the expected JSON file is not present, retry the `/scrape-analyze-page` subagent or report a blocker. Do NOT create the analysis JSON yourself from HTML.

Read the analysis result from `.scrape/.work/{site_name}/analyze-page/detail-1.rendered.json`. Do not inspect the downloaded page HTML or any cleaned HTML after the analysis subagent returns; the JSON result is the only allowed field-discovery input for the main agent.

## Step 4: Build schema

From the analysis result, build a JSON Schema:
- Collect all field names and their types
- Map types: `str` → `string`, `float` → `number`, `int` → `integer`, `list` → `array`, `dict` → `object`
- Add `description` for each field (infer from context)
- Mark fields matching field_hints as `"source": "requested"`, others as `"source": "discovered"`

## Step 5: Quick schema check

Present the schema with values in the terminal. Group by requested/discovered:

```
Found {N} fields on {detail_page_url}:

Requested:
  title (string): "A Light in the Attic"
  price (string): "£51.77"

Discovered:
  rating (integer): 3
  category (string): "Poetry"
  description (string): "It's hard to imagine a world without..." (2340 chars)
  upc (string): "a897fe39b1053632"
  image_url (string): "https://books.toscrape.com/media/cache/fe/72/..."
```

Wait for user response. The user can:
- **Approve**: "looks good", "ok", "approve", etc. → go to step 6
- **Drop fields**: "drop description, image_url" → remove from schema, show updated
- **Keep discovered fields**: "keep rating" → change source to "requested"
- **Rename**: "rename upc to product_code" → rename in schema
- **Edit descriptions**: "price should be without currency symbol" → update description and value
- **Other instructions**: apply and show updated schema

Loop until the user approves. Re-display the schema after each change.

## Step 6: Save spec

### Data type spec

Write `.scrape/{site_name}/{data_type}/spec.json`. The output MUST be valid JSON Schema (draft/2020-12) — use exactly the structure shown below. Do NOT invent a custom format (e.g. a plain array of field objects). Include `examples` in each schema field — the user-approved value (with any corrections). This tells Stage 2 the expected format. Include `examples` for all fields. Truncate values longer than 200 chars with "..." — keep enough to show the format.

```json
{
  "url": "https://books.toscrape.com",
  "data_type": "product",
  "html_variant": "rendered",
  "schema": {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
      "title": {
        "type": "string",
        "description": "Book title as displayed on the detail page",
        "source": "requested",
        "examples": ["A Light in the Attic"]
      },
      "price": {
        "type": "string",
        "description": "Book price without currency symbol",
        "source": "requested",
        "examples": ["51.77"]
      }
    }
  }
}
```

### Site-level spec

Write `.scrape/{site_name}/spec.json`:
```json
{
  "url": "https://books.toscrape.com",
  "data_types": ["{data_type}"]
}
```

Note: `data_types` only lists the primary data type. `/scrape-spec` adds "navigation" later.

## Done

```
Spec draft saved to .scrape/{site_name}/:
  {data_type}: {N} fields

Run /scrape-spec .scrape/{site_name} to explore more pages and validate.
```