# Web Scraping

> Web Scraping Capability

- Skill: `lensetek/web-scraping` (Agent Skill)
- Install (CLI): `npx skillmds@latest add lensetek/web-scraping`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lensetek/web-scraping/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: lensetek (https://skillmd.com/u/lensetek)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lensetek/web-scraping

---

# Web Scraping Capability

## When to Use

Use this capability to extract public structured data from web pages, such as product listings, headings, metadata, FAQ content, pricing tables, schema markup, or competitor landing page elements.

## Preferred Implementation

- Use lightweight HTTP clients and HTML parsers for static pages.
- **Dynamic Pages:** For pages that require JavaScript rendering or interaction, use Chrome DevTools MCP (`chrome-devtools-mcp`) tools (such as `chrome-devtools/evaluate_script` or `chrome-devtools/take_snapshot`) or Puppeteer/Playwright scripts first.
- Prefer small, respectful, rate-limited, and task-specific extraction.
- Use Computer Use only if a page cannot be accessed with scripts and the user approves UI-based extraction.

## Inputs

- URLs.
- Data fields to extract.
- Rate and scope limits.
- Allowed pages.
- Output format.

## Outputs

- Extracted data table or JSON.
- Source URL per row.
- Extraction notes.
- Errors or missing fields.
- Data quality warnings.

## Security and Ethics

- Scrape only public data or user-authorized data.
- Do not bypass access controls, CAPTCHAs, login gates, or paywalls.
- Do not collect credentials, private customer data, or sensitive personal data.
- Respect site terms and rate limits.