# Scraper

> Add, debug, or modify price scrapers for e-commerce sites (Shopee, Lazada, Tiki, etc.). Use when working with scraper configs or pullData modules. Use when this capability is needed.

- Skill: `tomevault-io/scraper` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/scraper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/scraper/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/scraper

---


# Scraper Skill

Work with PriceTrack's price scrapers — add new sites, debug existing ones, or modify scraping logic.

## Architecture

Scrapers are split into two parts:

1. **Config** (`functions/config/<site>.js`) — Site-specific selectors and URL patterns. Exports an object with:
   - `urlPattern` — regex to match product URLs
   - `selectors` — CSS selectors for price, title, image, etc.
   - `transform` — optional data transformation functions

2. **Module** (`functions/modules/pullData.js`) — Generic pull logic that reads configs and fetches/scrapes product pages using:
   - Puppeteer-core + @sparticuz/chromium for JS-rendered pages (serverless Chromium)
   - jsdom for static HTML parsing
   - axios for simple HTTP fetches

## Adding a New Scraper

1. Create `functions/config/<site>.js` with the site's URL pattern and selectors
2. Register it in the config index (check how existing sites are imported)
3. Test with: `firebase functions:shell → pullData({url: '<product-url>'})`
4. Verify the scraped data structure matches what `modules/pullData.js` expects

## Debugging

- Check Firebase Functions logs: `firebase functions:log`
- Puppeteer scrapers run headless on serverless Chromium — no GPU, limited memory
- Some sites require specific headers or cookie consent handling
- `@sparticuz/chromium` has size limits; check if the binary is up to date

## Key Files

- `functions/config/` — per-site scraper configurations
- `functions/modules/pullData.js` — main scraping orchestration
- `functions/modules/updateInfo.js` — product info updates
- `functions/utils/fetch.js` — HTTP fetch utilities

---
> Source: [duyet/pricetrack](https://github.com/duyet/pricetrack) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-06-17 -->