# Web Scraper

> Scrape and extract content from web pages. Supports extracting text, links, headings, and structured data. Use when the user asks to read a web page, extract information from a URL, or scrape website content.

- Skill: `ericwang915/web-scraper` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add ericwang915/web-scraper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ericwang915/web-scraper/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ericwang915 (https://skillmd.com/u/ericwang915)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ericwang915/web-scraper

---


## Instructions

Scrape and extract readable content from any web page.

### Prerequisites

Install dependencies: `pip install requests beautifulsoup4`

### Usage

```bash
python {skill_path}/scrape.py URL [--format text|json|links|headings]
```

Formats:
- `text` (default) — cleaned readable text
- `json` — structured JSON with title, text, links, headings
- `links` — all links on the page
- `headings` — all headings (h1–h6)

### Examples

- "Read the content of https://example.com"
- "Extract all links from https://news.ycombinator.com"
- "What does this page say? https://some-article.com/post"

## Resources

| File | Description |
|------|-------------|
| `scrape.py` | Generic web page scraper |