# Web Scraper

> Scrape web pages, search the internet, and extract structured content using Python. Includes anti-bot/Cloudflare bypass via Scrapling. Use when the user wants to fetch a webpage, search for information online, extract links, crawl JS-rendered pages, or bypass bot protection.

- Skill: `the-hugin/web-scraper` (Agent Skill)
- Install (CLI): `npx skillmds@latest add the-hugin/web-scraper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/the-hugin/web-scraper/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: the-hugin (https://skillmd.com/u/the-hugin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/the-hugin/web-scraper

---


# Web Scraper

Fetch, search, and extract content from websites.

## When to use this skill

- User asks to fetch or read a webpage / URL
- User wants to search the internet for information
- User needs to extract links, tables, or structured data from a website
- User asks to crawl a JavaScript-rendered (dynamic) page
- User wants web content converted to clean Markdown for analysis

## Scripts overview

| Script | Purpose | Dependencies |
|---|---|---|
| `fetch_page.py` | Fetch a URL and extract readable content as Markdown | `requests`, `beautifulsoup4`, `readability-lxml`, `html2text` |
| `search_web.py` | Search the web via DuckDuckGo | `ddgs` |
| `crawl_dynamic.py` | Crawl JS-rendered pages with a headless browser | `crawl4ai` |
| `extract_links.py` | Extract and categorize all links from a page | `requests`, `beautifulsoup4` |
| `scrapling_fetch.py` | Anti-bot / Cloudflare bypass scraping | `scrapling[all]>=0.4.2` (Python 3.10+) |

## Steps

### 1. Install dependencies (first time only)

For lightweight scraping (static pages, search, link extraction):
```bash
pip install requests beautifulsoup4 readability-lxml html2text ddgs
```

For dynamic / JavaScript-rendered pages (heavier, installs Playwright + Chromium):
```bash
pip install crawl4ai
crawl4ai-setup
```

> **Note**: `crawl4ai-setup` downloads a Chromium browser (~150 MB). Only install if you actually need dynamic page support.

> **CRITICAL — Dependency Error Recovery**: If ANY script below fails with an `ImportError` or "module not found" error, install the missing dependencies using the command above, then **re-run the EXACT SAME script command that failed**. Do NOT write inline Python code (`python -c "..."`) or your own ad-hoc scripts as a substitute. These scripts handle encoding, error handling, and output formatting that inline code will miss.

### 2. Fetch a web page (static — recommended first choice)

Use this for most websites. It's fast, lightweight, and works for articles, docs, blogs, etc.

```bash
python scripts/fetch_page.py "URL"
```

Options:
- `--raw` — Output full page Markdown instead of extracted article content
- `--selector "CSS_SELECTOR"` — Extract only elements matching the CSS selector (e.g. `".article-body"`, `"table"`, `"#content"`)
- `--save OUTPUT_PATH` — Also save output to a file
- `--max-length N` — Truncate output to N characters (default: no limit)

Examples:
```bash
# Fetch an article
python fetch_page.py "https://example.com/article"

# Extract only tables
python fetch_page.py "https://example.com/data" --selector "table"

# Fetch raw full-page markdown, limit to 5000 chars
python fetch_page.py "https://example.com" --raw --max-length 5000
```

### 3. Search the web

Search using DuckDuckGo (no API key required).

```bash
python scripts/search_web.py "search query"
```

Options:
- `--max-results N` — Number of results to return (default: 10)
- `--region REGION` — Region code, e.g. `cn-zh`, `us-en`, `jp-jp` (default: `wt-wt` for worldwide)
- `--news` — Search news instead of general web

Examples:
```bash
# General search
python search_web.py "Python web scraping best practices 2025"

# News search, Chinese region, 5 results
python search_web.py "AI 最新进展" --news --region cn-zh --max-results 5
```

### 4. Crawl a dynamic / JavaScript-rendered page

Use this only when `fetch_page.py` returns empty or incomplete content (SPA, React/Vue apps, pages that load content via JS).

```bash
python scripts/crawl_dynamic.py "URL"
```

Options:
- `--wait N` — Wait N seconds after page load for JS to finish (default: 3)
- `--selector "CSS_SELECTOR"` — Wait for a specific element to appear before extracting
- `--scroll` — Scroll to bottom of page to trigger lazy loading
- `--save OUTPUT_PATH` — Also save output to a file
- `--max-length N` — Truncate output to N characters

### 5. Extract links from a page

Extract all links with their text labels, categorized by type (internal, external, resource).

```bash
python scripts/extract_links.py "URL"
```

Options:
- `--filter PATTERN` — Only show links matching a regex pattern (applied to URL)
- `--external-only` — Only show external links
- `--json` — Output as JSON instead of Markdown

### 5. Anti-bot / Cloudflare protected sites

Use when `crawl_dynamic.py` returns 403, CAPTCHA, or Cloudflare challenge page.

Install (Python 3.10+ required):
```bash
pip install "scrapling[all]>=0.4.2"
```

```bash
# Browser automation (JS-heavy sites, not Cloudflare)
python scripts/scrapling_fetch.py "URL"

# Stealth mode (Cloudflare bypass, TLS fingerprinting)
python scripts/scrapling_fetch.py "URL" --stealth

# With CSS selector
python scripts/scrapling_fetch.py "URL" --stealth --selector ".article-body"

# Save output
python scripts/scrapling_fetch.py "URL" --stealth --save output.txt
```

Options:
- `--stealth` — Enable Cloudflare bypass (slower, more effective)
- `--selector CSS` — Extract only elements matching the CSS selector
- `--save PATH` — Save output to file
- `--max-length N` — Truncate output to N characters (default: 10000)

---

## Decision guide: which script to use

1. **Start with `fetch_page.py`** — handles 90% of websites (articles, docs, blogs, wikis).
2. If `fetch_page.py` returns empty/garbled content → try **`crawl_dynamic.py`** (the page needs JavaScript).
3. If `crawl_dynamic.py` returns 403 / Cloudflare block → try **`scrapling_fetch.py --stealth`** (anti-bot bypass).
4. Need to find URLs first? → Use **`search_web.py`** to discover relevant pages.
5. Need to navigate a site structure? → Use **`extract_links.py`** to map out links, then fetch individual pages.

## Common workflows

### Research a topic
1. `search_web.py "topic"` → get relevant URLs
2. `fetch_page.py "best_url"` → read the content
3. Repeat for multiple sources, then synthesize

### Scrape structured data from a page
1. `fetch_page.py "url" --selector "table"` → extract tables
2. Or `fetch_page.py "url" --selector ".product-card"` → extract specific elements

### Crawl a modern web app (SPA)
1. `crawl_dynamic.py "url" --wait 5 --scroll` → full JS-rendered content

## Edge cases

- **Paywalled sites**: May return partial content or login pages. Inform the user.
- **Rate limiting / CAPTCHAs**: If requests fail with 403/429, wait and retry or inform the user.
- **Very large pages**: Use `--max-length` to truncate output and avoid overwhelming the context window.
- **Encoding issues**: Scripts handle UTF-8 by default. Exotic encodings may need manual adjustment.
- **Robots.txt**: These scripts do not check robots.txt. Use responsibly and respect website terms of service.

## Scripts

- [fetch_page.py](scripts/fetch_page.py) — Fetch and extract readable content as Markdown
- [search_web.py](scripts/search_web.py) — Search the web via DuckDuckGo
- [crawl_dynamic.py](scripts/crawl_dynamic.py) — Crawl JavaScript-rendered pages
- [extract_links.py](scripts/extract_links.py) — Extract and categorize page links