# Scrape Spec

> Expand a spec created by /scrape-define — download more pages, compare HTML variants, extract values, optional browser review

- Skill: `zytedata/scrape-spec` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zytedata/scrape-spec`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zytedata/scrape-spec/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: zytedata (https://skillmd.com/u/zytedata)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zytedata/scrape-spec

---


You are expanding and validating an extraction spec that was drafted by `/scrape-define`. Download diverse detail and listing pages, compare HTML variants, extract values, and optionally present for browser-based review.

Read `${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md`.

The input is a spec folder with at least one data type containing a schema and 1 page (from `/scrape-define`). The output is the same folder with more pages, validated values, and a navigation data type.

## Input

The raw argument string is `$ARGUMENTS` — a single value, used as-is:

1. **site_path**: path to the site spec folder, e.g. `.scrape/books-toscrape`

## Step 1: Read existing spec

Read `{site_path}/spec.json` to get `url` and `data_types`.

For the primary data type (first in `data_types` that isn't "navigation"), read:
- `{site_path}/{data_type}/spec.json` → `schema`, `html_variant`

Derive `site_name` from `site_path` (last component, e.g. `books-toscrape`).

## Step 2: Explore the site

Stage 1 doesn't store pages — only schema and values. Stage 2 downloads a small fresh sample using `/scrape-explore-site` in a subagent.

```
Agent(description="explore-site", prompt="Run /scrape-explore-site {url} .scrape/.work/{site_name}/explore 2 2")
```

This downloads the start page + up to 2 detail pages + up to 2 list pages, classifies links, and generates navigation values. All output goes to `.scrape/.work/{site_name}/explore/`.

If the subagent reports that the site is blocked, invoke `/scrape-zyte-login`. After it returns, **re-run** the `scrape-explore-site` subagent above. Only proceed once exploration succeeds.

Then distribute pages to the right data-type subfolders:
```bash
mkdir -p {site_path}/{data_type}/pages {site_path}/navigation/pages {site_path}/navigation/values

# Detail pages → data type
for d in .scrape/.work/{site_name}/explore/pages/detail-*; do
  [ -d "$d" ] && cp -r "$d" {site_path}/{data_type}/pages/
done

# Start + list pages → navigation
for d in .scrape/.work/{site_name}/explore/pages/start-* .scrape/.work/{site_name}/explore/pages/list-*; do
  [ -d "$d" ] && cp -r "$d" {site_path}/navigation/pages/
done

# Navigation values (generated by explore-site)
cp .scrape/.work/{site_name}/explore/values/*.json {site_path}/navigation/values/ 2>/dev/null || true
```

## Step 3: Determine navigation HTML variant

Before analyzing detail pages, independently determine the HTML variant navigation pages need. Run `extract_links.py` on all navigation pages using both variants:

```bash
mkdir -p .scrape/.work/{site_name}/analyze-nav

uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/extract_links.py \
  {site_path}/navigation/pages/*/raw.html \
  --group --base-url-from-meta \
  > .scrape/.work/{site_name}/analyze-nav/nav.raw.json 2>/dev/null || true

uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/extract_links.py \
  {site_path}/navigation/pages/*/rendered.html \
  --group --base-url-from-meta \
  > .scrape/.work/{site_name}/analyze-nav/nav.rendered.json 2>/dev/null || true
```

Read both output files and analyze the link groups. Determine which HTML variant provides better navigation coverage — consider the group structure, whether meaningful navigation links appear in each variant, and whether one variant reveals links the other misses. **Prefer raw unless there is clear evidence that rendered provides materially better navigation coverage.**

Store the result as `nav_html_variant` for use in Step 6.

## Step 4: Analyze detail sample and list pages in parallel

First analyze a bounded sample of detail pages with both HTML variants. Use the first detail page that has both `raw.html` and `rendered.html` when list pages are present; otherwise use the first 1-2 detail pages. This means the detail-page variant comparison uses at most 1 page when list pages are also being analyzed, and at most 2 pages when no list pages are available. Do not launch more than 4 detail-page analysis agents for the variant comparison. This keeps the workflow responsive while still testing whether raw or rendered can extract the approved schema fields.

Also analyze all list pages from `{site_path}/navigation/pages/` with the raw variant using the data type schema. Launch the detail sample agents **plus** one Agent per list page in a **single message** for parallel execution. Keep the combined launch at 5 Agents or fewer to avoid provider thread limits; if there are list pages, prefer `detail-1` raw/rendered plus the list-page agents over analyzing a second detail page. Issue all the Agent calls in one response with no tool calls in between — do not wait for any result before launching the rest.

Before launching the agents, create the output directories:
```bash
mkdir -p .scrape/.work/{site_name}/analyze-page
mkdir -p {site_path}/{data_type}-list/values
```

```
Agent(description="analyze detail-1 raw", prompt="/scrape-analyze-page Extract data from {site_path}/{data_type}/pages/detail-1/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-1.raw.json ")
Agent(description="analyze detail-1 rendered", prompt="Run /scrape-analyze-page {site_path}/{data_type}/pages/detail-1/rendered.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-1.rendered.json")
Agent(description="analyze list-1", prompt="/scrape-analyze-page --list-mode Extract all {data_type} items from {site_path}/navigation/pages/list-1/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into {site_path}/{data_type}-list/values/list-1.json")
Agent(description="analyze list-2", prompt="/scrape-analyze-page --list-mode Extract all {data_type} items from {site_path}/navigation/pages/list-2/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into {site_path}/{data_type}-list/values/list-2.json")
... (all list pages in the same message)

If there are no list pages, add the second detail-page pair instead:

Agent(description="analyze detail-2 raw", prompt="Run /scrape-analyze-page {site_path}/{data_type}/pages/detail-2/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-2.raw.json")
Agent(description="analyze detail-2 rendered", prompt="Run /scrape-analyze-page {site_path}/{data_type}/pages/detail-2/rendered.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-2.rendered.json")
```

Skip variants whose HTML files don't exist. If there are no list pages in `{site_path}/navigation/pages/`, skip the list-page agents and the `mkdir` for `{data_type}-list/values`. The schema_path gives analyze-page the approved field names, descriptions, and examples — so it extracts with the correct names and value formats. After Step 5 chooses the variant, do not launch extra detail-page analyses. Do not wait for every list-page agent before continuing: once the bounded detail sample is complete and at least one list-page values file is available, proceed to Step 4.5 using the completed list outputs so slow optional list analyses do not block finalization.

## Step 4.5: Decide whether to create a list-page data type

This step runs only if list pages were found and analyzed in Step 4.

Read the completed `{site_path}/{data_type}-list/values/list-*.json` files. For each completed file, check whether every `source: "requested"` field from the schema appears as a key in at least one item in the `values` array. If no list values file has completed yet, set `list_spec_created = false` and proceed with detail-page extraction rather than waiting.

**If ALL requested fields are found in EVERY completed list-page values file:**

Set `list_spec_created = true`.

1. Write `{site_path}/{data_type}-list/spec.json`:
   ```json
   {
     "url": "{url}",
     "data_type": "{data_type}-list",
     "html_variant": "raw",
     "schema": {<same schema object as in {site_path}/{data_type}/spec.json>}
   }
   ```

2. Copy list pages into the new data type:
   ```bash
   mkdir -p {site_path}/{data_type}-list/pages
   for d in {site_path}/navigation/pages/list-*; do
     [ -d "$d" ] && cp -r "$d" {site_path}/{data_type}-list/pages/
   done
   ```

**Otherwise:** set `list_spec_created = false`. Proceed with detail-page extraction as normal — the `{data_type}-list/values/` directory may remain but will be ignored.

## Step 5: Choose HTML variant

Compare raw vs rendered results across the bounded sample from `.scrape/.work/{site_name}/analyze-page/`.

For each page, compare `{page_id}.raw.json` and `{page_id}.rendered.json`:
- Which variant found more of the schema fields?
- Which fields are only in one variant?
- Do values differ between variants for the same field?

Use a single small Python script for this comparison. The script should load `{site_path}/{data_type}/spec.json`, enumerate `.scrape/.work/{site_name}/analyze-page/*.json`, compare only schema field coverage and values, print a concise summary, and recommend `raw` unless rendered consistently finds requested schema fields that raw misses.

**Raw is preferred by default** — it's faster, cheaper, and more reliable. Only use rendered if it finds schema fields that raw consistently misses, or if raw HTML is essentially empty (SPA site).

Present the comparison in the terminal:
```
HTML variant comparison:
  Both variants found: name, price, brand, description, image_url
  Only in rendered: reviews_count (3/3 pages), sale_price (2/3 pages)
  Only in raw: (none)
  Value differences:
    price: raw="$29.99", rendered="$24.99" (detail-1) — rendered may show sale price

  Recommendation: raw (all schema fields found)
  Note: rendered also has reviews_count, sale_price — say "use rendered" if you need these.
```

If it's clear which variant to use, use it. If it's not, for example when the raw view contains
most of the fields but not all of them, ask the user via `AskUserQuestion` which variant to use,
providing details:

- **question**: "Which HTML variant should I use for extraction?"
- **header**: "HTML variant"
- **options**: "Use raw" and "Use rendered", mark the recommended one as such.

In this case use the variant that the user selects.

If both variants find all approved schema fields in the bounded sample, choose `raw` immediately. If the variant changes from what Stage 1 used, update `{site_path}/{data_type}/spec.json` with the new `html_variant`. Once the variant is chosen, continue directly to Step 6; do not start more analysis agents.

## Step 6: Extract values (skip if list_spec_created)

If `list_spec_created` is true, skip this step — values were already written to `{data_type}-list/values/` in Step 4.

Otherwise, use `extract_values.py` to build values from analysis files, filtered by the schema:

```bash
uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/extract_values.py \
  .scrape/.work/{site_name}/analyze-page/ \
  {site_path}/{data_type}/spec.json \
  --variant {html_variant} \
  -O {site_path}/{data_type}/values/
```

This overwrites any existing values files (including the one from Stage 1) with fresh extractions from the analyzed detail-page sample. No `--renames` needed — Stage 2's analyze-page receives the full schema (names + descriptions + examples), so it extracts with the correct field names directly.

### Navigation

Write `{site_path}/navigation/spec.json` with the fixed navigation schema from `${CLAUDE_SKILL_DIR}/../scrape/references/extraction-spec.md`, using the site URL and `nav_html_variant` (determined in Step 3).

Navigation values were already copied from explore-site output in Step 2.

## Step 7: Optional browser review

Tell the user the extraction stats first ("Extracted values for {N} detail pages and {M} navigation pages."), then ask via `AskUserQuestion` whether they want a browser review.

If the environment cannot ask follow-up questions, skip browser review and continue.

- **question**: "Open a browser review of the extracted values?"
- **header**: "Browser review"
- **options**:
  - `Skip browser review` — "Continue without opening the browser."
  - `Open browser review` — "Review the values in the browser."

If the user picks `Skip review`, go to step 9.

If the user picks `Open browser review`, invoke `/scrape-review-schema` with the data type spec:
```
/scrape-review-schema {site_path}/{data_type} .scrape/.work/{site_name} {schema_json} {html_variant}
```

Report the review dir path to the user before opening.

## Step 8: Apply feedback

If the user reviews and provides feedback:

If feedback starts with `APPROVED`: apply any included schema changes (drops, renames, description edits, kept fields → change source to "requested") and skip to step 9. The user has signed off.

If feedback does NOT start with `APPROVED` (user clicked "Request changes"):
- Apply schema changes (drops, renames, description edits) directly
- For value corrections: check the other variant's analysis files first. If the other variant has the correct value, suggest switching variants:
  ```
  You corrected price to "$24.99" — the rendered variant already extracts this correctly (across 3/3 pages).
  Switch to rendered? [Y/n]
  ```
  If the user agrees, switch variant, update spec, and regenerate values.
- If the other variant doesn't help: re-run analysis for corrected fields across ALL detail pages (using the chosen variant). Launch parallel Agents, one per page.

After re-analysis or variant switch, re-extract values (step 6) and offer review again. Pass a changes summary as the 5th argument to `/scrape-review-schema`:
```
/scrape-review-schema {site_path}/{data_type} .scrape/.work/{site_name} {schema_json} {html_variant} '["Re-analyzed price across all pages","Dropped field isbn"]'
```

Loop steps 7-8 until the user approves or skips review.

## Step 9: Finalize

Update `{site_path}/{data_type}/spec.json` with any schema changes from the review (only relevant when `list_spec_created` is false, since detail pages were used).

Update `{site_path}/spec.json` — set `data_types` as follows:
- If `list_spec_created` is true: `["{data_type}-list", "navigation"]` — omit `"{data_type}"` since detail-page extraction is not needed.
- Otherwise: `["{data_type}", "navigation"]`

Report:

If `list_spec_created` is true:
```
Spec finalized at {site_path}/:
  {data_type}-list: {N} list pages, {F} fields, {K} items extracted
  navigation: {M} pages

All requested fields found on list pages — skipping detail-page extraction.
Ready for codegen: /scrape-codegen {site_path}/{data_type}-list ./{project_dir}
```

Otherwise:
```
Spec finalized at {site_path}/:
  {data_type}: {N} detail pages, {F} fields
  navigation: {M} pages

Ready for codegen: /scrape-codegen {site_path}/{data_type} ./{project_dir}
```