# Scraper Contract > Spec a scraper that fails loudly instead of returning quiet garbage. Defines a closed failure taxonomy (unreachable / blocked / timeout / rate-limited / invalid / empty / unknown), makes empty output a hard failure not a success, sets a per-field stability tier and the minimum identifier a record must have to be accepted, and decides what counts as a broken run. Use this before pointing Claude, Apify, or Playwright at a site. Produces a scrape contract and stops. - Skill: `robdasi/scraper-contract` (Agent Skill) - Install (CLI): `npx skillmds@latest add robdasi/scraper-contract` - Raw SKILL.md: https://api.skillmd.com/api/skills/robdasi/scraper-contract/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: robdasi (https://skillmd.com/u/robdasi) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/robdasi/scraper-contract --- # Scraper Contract A scraper rarely fails by crashing. It fails by returning nothing, or half a record, while the pipeline downstream carries on as if the data were real. A week later you're emailing "Hi {first_name}" because one field came back null and nobody decided that was a failure. This skill writes the contract that stops that. You describe what you're scraping and from where, it produces a contract that classifies every failure, treats empty as broken, and says which fields you're allowed to trust. It's the wrapper I put around every scraper before it goes near a pipeline. Build the contract, then stop. ## Inputs (ask for whatever is missing) - **The target** (required): the site/source and the fields you need off it. - **How you're fetching**: Apify / Playwright / a fetch-and-parse / an API, and whether the source is one you control. - *Optional:* a real example of a good record, and the worst page you've hit (paywalled, JS-rendered, captcha'd). ## The method 1. **Define the failure taxonomy.** A closed enum of failure codes, each mapped from what you can observe. The set that has covered every real case for me: - **unreachable** — DNS/connection failure, 404/502/503/504, ENOTFOUND, ECONNRESET. - **blocked** — 403, captcha, "access denied", a bot wall. - **timeout** — the request or render exceeded its budget. - **rate-limited** — 429, or the source's throttle response. - **invalid-input** — the URL/target was malformed before you even left. - **empty-content** — the fetch succeeded but the thing you came for isn't there. - **unknown** — everything else, logged with the raw cause so the taxonomy can grow. Write the classify rule: status code and message substrings in, one code out. Every failure gets a code; "it just didn't work" is not allowed. 2. **Make empty a hard failure.** This is the one everyone skips. A response with no markdown, an empty list, a missing required field is **empty-content**, and it throws or returns a failure — never a success with blank data. "No content" being a normal outcome is how garbage enters the pipeline silently. 3. **Tier every field by stability.** For each field you extract, mark how fragile it is: - **stable** — a reliable selector / API field unlikely to move. - **extracted** — pulled by the model from prose; resilient to layout change but needs a sanity check. - **volatile** — a brittle selector or a layout-dependent grab that will break on the next redesign. The volatile fields are your decay risk. They get the canary in step 5. 4. **Set the minimum identifier.** State the smallest set of fields a record must have to be accepted at all — for a person, at least a profile URL or an email; for a company, at least a name or domain. A record missing the minimum is dropped, not patched with guesses. Validate this before the record is allowed into the pipeline. 5. **Add a decay canary per volatile field.** A cheap assertion that fires when the shape changes: a field that's suddenly null across the whole batch means the site changed, not that everyone lacks it. Pair it with the escalation rule — on **blocked**/**rate-limited**, back off (and escalate to a proxy/stealth fetch only if it's worth it); don't hammer. 6. **Keep the batch alive.** Wrap each record in its own try/catch so one malformed row never kills the run. Collect the failures with their codes instead of throwing at the first one. 7. **Define a broken run.** The threshold that flips the whole run from "fine" to "stop and alert": e.g. ">30% empty-content or any blocked = the source changed or shut us out, halt and tell me." Without this line, a scraper that returns 90% nulls looks like a successful run. 8. **Request only what you need.** Pull the expensive formats (rendered HTML, screenshots) only when a field actually requires them, and let an optional extra (a screenshot) fail soft — warn and proceed — without failing the core record. Cost and fragility both drop. ## Output Produce the scrape contract as: a **field table** (field, stability tier, required-for-minimum?, canary check), the **failure enum** with its classify rules, the **broken-run threshold**, and the **fetch economy** notes (which formats, what degrades soft). Flag every volatile field with no canary as a decay risk to fix. Then stop. The contract is the deliverable. The scraper you build against it will tell you when it breaks, instead of quietly feeding you nothing.