# Scrape And Load

> Run the Wookieepedia scraper, convert data, load into SQLite, and verify with migration and full tests.

- Skill: `sap-samples/scrape-and-load` (Agent Skill)
- Install (CLI): `npx skillmds@latest add sap-samples/scrape-and-load`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sap-samples/scrape-and-load/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: sap-samples (https://skillmd.com/u/sap-samples)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/sap-samples/scrape-and-load

---


Run the full data pipeline **sequentially** from the `cap/` directory.

## Steps

1. **Scrape Star Wars data**
   ```bash
   cd cap && npm run scrape
   ```
   Uses committed cache by default (fast). If the user requests `--bypass-cache`, ask for explicit confirmation first, then run `npm run scrape:bypass-cache` instead.

2. **Build CDS artifacts**
   ```bash
   cd cap && npm run build
   ```

3. **Load fixture data into SQLite**
   ```bash
   cd cap && npm run load_sqlite
   ```

4. **Run migration tests**
   ```bash
   cd cap && npm run test:migration
   ```
   Verifies data conversion logic and report generation.

5. **Run full test suite**
   ```bash
   cd cap && npm test
   ```

## Rules

- **Stop on failure.** If any step fails, report the error and do not continue.
- **Cache-first by default.** Never bypass the scraper cache without user confirmation — fresh fetches hit Wookieepedia and take significantly longer.
- **Report results.** Summarize each step's outcome: scrape stats, build status, load counts, and test pass/fail/skip.