# Add Scraper > Add support for scraping property listings from a new website. Use when the user wants to add a new scraper, create a new mapping, or support a new property listing site. Use when this capability is needed. - Skill: `tomevault-io/add-scraper` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/add-scraper` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/add-scraper/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/add-scraper --- # Add Scraper for a New Website You are helping the user add support for scraping property listings from a new website. This is a multi-step process that requires creating a mapping file, capturing a test fixture, and wiring everything up. ## Inputs The user will provide `$ARGUMENTS` which should be a URL of a sample listing page from the target website, or just the website name. If no URL is provided, ask for one. ## Step-by-step workflow ### Phase 1: Gather information 1. **Get a sample URL** if the user didn't provide one 2. **Determine the scraper name** — derive from the hostname (e.g. `www.example-realty.com` → `example_realty`). Use lowercase, underscores for separators, keep it short. Ask the user to confirm the name. 3. **Get the HTML** — Ask the user how they want to provide the HTML: - If the site is static (no JS rendering needed), we can fetch it directly - If the site is JS-heavy (React/Vue/Angular SPA), the user should save the page from their browser ("Save As" → "Web Page, HTML Only") and provide the file path - They can also pipe it: `curl ... | npm run capture-fixture -- --stdin --url ` ### Phase 2: Analyze the HTML and create the mapping 4. **Capture the fixture** using the capture-fixture utility: ```bash cd astro-app # For URL fetch: npm run capture-fixture -- --name # For local file: npm run capture-fixture -- --file --url --name --no-extract ``` 5. **Analyze the saved HTML fixture** at `astro-app/test/fixtures/.html`. Look for: - **Title**: Usually in `

`, ``, or `og:title` meta tag - Price: Look for price elements, currency symbols, `itemprop="price"` microdata - Address/Location: Address strings, postal codes, `og:` meta tags, or structured data - Coordinates: Often in `<script>` tags as JSON (lat/lng), or in meta tags - Bedrooms/Bathrooms: Usually near icons or labels like "bed", "bath", "BR" - Images: `<img>` tags in gallery sections, or `og:image` meta tags - Reference/ID: Property ID in URL path, hidden inputs, or `data-` attributes - For sale/rent: Status indicators, URL patterns (`/to-rent/`, `/for-sale/`) - Description: `meta[name=description]` or content sections - Area/Size: Square footage/meters, usually near bed/bath counts 6. Create the mapping JSON at `config/scraper_mappings/<name>.json`. Follow the format described in the reference file at `.claude/skills/add-scraper/reference.md`. Key rules: - Each field must appear in exactly ONE section. If the same field appears in multiple sections, the last one processed wins (processing order: defaultValues → images → features → intFields → floatFields → textFields → booleanFields) - Always add `cssCountId: "0"` when a CSS selector might match multiple elements - Use `cssAttr` to read attribute values (e.g. `content` from meta tags) - Use `scriptRegEx` for data embedded in `<script>` tags - Use `urlPathPart` to extract segments from the URL path - Prefer class-based or ID-based selectors over `nth-child` chains ### Phase 3: Validate and register 7. Run the capture-fixture utility again with extraction enabled to preview results: ```bash cd astro-app npm run capture-fixture -- --file test/fixtures/<name>.html --url <source_url> --name <name> --force ``` 8. Review the extraction preview. If fields are wrong: - Check selectors against the actual HTML - Fix the mapping JSON - Re-run capture-fixture to verify 9. Add to the hostname map — Add entries for the new hostname to: - `astro-app/scripts/capture-fixture.ts` in the `HOSTNAME_MAP` constant - `astro-app/src/lib/services/url-validator.ts` in the `LOCAL_HOST_MAP` constant 10. Add to the test manifest — Copy the manifest stub from capture-fixture output into `astro-app/test/fixtures/manifest.ts`. Review and adjust expected values — remove fields that are zero/empty and not meaningful, ensure types are correct (integers vs strings vs booleans). 11. Run the test suite to verify everything passes: ```bash cd astro-app && npx vitest run ``` ### Phase 4: Summary 12. Present a summary of all files created/modified and offer to commit the changes. ## Important notes - The mapping file format is JSON but parsed with JSON5 (comments are allowed) - The `name` field in the mapping must match the scraper name used everywhere else - `defaultValues` always produce strings (e.g. `"true"` not `true`) - `stripPunct` removes `.` and `,` only — not currency symbols - `stripString` removes the first occurrence of an exact substring, runs after split - Cheerio converts `<br>` to `\n`, never `\r` - `parseFloat`/`parseInt` return `0` on failure, not an error ## Existing scrapers for reference These scrapers are already configured: uk_rightmove, uk_zoopla, uk_onthemarket, uk_jitty, es_idealista, es_fotocasa, es_pisos, pt_idealista, ie_daft, us_realtor, us_forsalebyowner, us_mlslistings, us_wyomingmls, in_realestateindia, de_immoscout, au_domain, au_realestate. See `config/scraper_mappings/` for their mapping files and `astro-app/test/fixtures/manifest.ts` for expected values. --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/realestatewebtools) — claim your Tome and manage your conversions.