# Scrape Codegen Generate

> Generate web-poet page object code from per-page extraction analyses

- Skill: `zytedata/scrape-codegen-generate` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zytedata/scrape-codegen-generate`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zytedata/scrape-codegen-generate/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: zytedata (https://skillmd.com/u/zytedata)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zytedata/scrape-codegen-generate

---


You are generating web-poet page object code. You receive per-page extraction analyses (from Stage 1) that describe WHERE and HOW each field can be extracted from pages on a given domain. Your job is to synthesize these analyses into a single page object class that works across the entire domain.

## Input

The raw argument string is `$ARGUMENTS`. Split it into 3 whitespace-separated positional arguments:

1. **work_path**: directory containing Stage 1 analysis files, e.g. `.scrape/.work/spec`
2. **output_path**: where to save the generated page object, e.g. `.scrape/spec/page_object.py`
3. **spec_path**: path to spec.json file, e.g. `.scrape/spec/spec.json`

Plus, taken from the surrounding prompt text (not from the argument string):

- **fields**: optional, specific fields to generate (provided in the prompt as "Only generate these fields: ..."). When set, only generate `@field` methods for those fields. When not set, generate all fields found in the analyses.

## Process

### 1. Read inputs

Read `web-poet.md` and `docs-access.md` from `${CLAUDE_SKILL_DIR}/../scrape/references/`.

Read the schema from `{spec_path}` — use the `properties` object inside `schema`.
Also read `data_type` from `{spec_path}`.

If `data_type` ends with `-list`, set `is_list_type = true`.

Read all Stage 1 analysis files from `{work_path}/codegen-analyze/`:
```
{work_path}/codegen-analyze/list-1.json
{work_path}/codegen-analyze/list-2.json
...
```
(or `detail-*.json` for non-list specs)

### 2. Build consensus across pages

For each field in the schema, review all per-page analyses together:

- **Identify common patterns**: Do all pages use the same CSS selector / JSON-LD path? If so, that's a strong signal.
- **Resolve disagreements**: If analyses recommend different approaches for different pages, determine:
  - Is one approach more general? (e.g., a selector that works on all pages vs one that only works on some)
  - Should you combine approaches? (e.g., try JSON-LD first, fall back to CSS)
  - Are there structural differences between pages that require conditional logic?
- **Validate target values**: Check that the recommended extraction method would produce the target values across all pages.
- **Decide on data source**: For each field, decide whether to use HTML selectors, JSON-LD, microdata, URL, or a combination.

### 3. Generate page object code

**No content filtering — ever.** Even if the user's prompt asks to filter,
exclude, or limit results by value, do NOT implement that logic in the page
object. Page objects extract; spiders filter. Mention in your summary that
filtering belongs at the spider level.

Generate a complete, self-contained Python module following the web-poet reference. The code must:

- **Work across the domain**: not just for the analyzed pages. Avoid overfitting — no hardcoded product names, specific if/else for individual pages, etc.
- **Be simple**: prefer the simplest approach that works. Only add fallbacks when analyses show the data genuinely comes from different sources on different pages.
- Use BrowserPage as base class if a browser response is needed.
- **Handle missing data**: return `None` when a field is not present (never empty string or `[]`).
- **Match schema types**: return values matching the JSON schema types.
- **Use recommended libraries**: `extruct` for JSON-LD/microdata, `price_parser` for prices, `jmespath` for JSON queries.
- **Include all imports**: the module must be self-contained and runnable.

**For non-list types** (`is_list_type = false`), standard structure:
```python
from web_poet import WebPage, field
# ... other imports as needed

class PageObject(WebPage[dict]):
    # shared helpers as @cached_property if multiple fields need them

    @field
    def field_name(self) -> type | None:
        # extraction logic
        ...
```

**For list types** (`is_list_type = true`), use web-poet's `SelectorExtractor`
(see the "Processors for nested fields" section of the
[web-poet fields reference](https://web-poet.readthedocs.io/en/stable/page-objects/fields.md)).
Define a private `SelectorExtractor` subclass for per-item extraction, then
iterate over the container selector in the `items` field. Each analysis file has
`container_selector` and per-field relative selectors.

```python
import logging
from web_poet import WebPage, Returns, SelectorExtractor, field, handle_urls
from {project}.items import Product, ProductListItems

logger = logging.getLogger(__name__)


class _ProductExtractor(SelectorExtractor[Product]):
    @field
    def title(self) -> str | None:
        return self.css("h3 a::attr(title)").get()

    @field
    def price(self) -> str | None:
        return self.css("p.price_color::text").get()


@handle_urls("example.com")
class ProductListPage(WebPage, Returns[ProductListItems]):

    @field
    async def items(self) -> list[Product] | None:
        result = []
        for el in self.css("article.product_pod"):
            try:
                result.append(await _ProductExtractor(el).to_item())
            except Exception:
                logger.error("Failed to extract item from %s", self.url, exc_info=True)
        return result or None
```

Key rules for list-type code:
- Define a `_<ItemClass>Extractor` class (e.g. `_ProductExtractor`) that extends
  `SelectorExtractor[ItemClass]` and has one `@field` method per item field
- The `items` field must be `async def` because `to_item()` is a coroutine
- Wrap each `await extractor.to_item()` in `try/except Exception` with a
  `logger.error(..., exc_info=True)` call — this isolates extraction failures to
  individual items and logs a full traceback without aborting the whole list
- Use `return result or None` (never return `[]`)
- The container selector must be consistent across all analyzed pages; if analyses
  show different selectors, choose the most general one
- Field selectors in the extractor are relative to each container element
  (`self.css(...)` on the extractor operates on `el`, not the full page)
- Import both `ItemClass` and `WrapperClass` from the project's items module

### 4. Save and report

Save the generated code to `{output_path}`.

Return a summary of what was generated:
```
Generated page object with N fields:
  name: CSS h1.product-title::text
  price: JSON-LD offers.price, fallback to CSS span.price
  description: CSS div.product-description (text join)
  rating: JSON-LD aggregateRating.ratingValue
  image_url: CSS img.product-image::attr(src) + urljoin
```

Include notes on any fields where consensus was difficult or where extraction may be fragile.