# Scraping

> Web scraping skill with a 4-tier strategy: WebFetch (fast, static), searxng (search-driven), BrightData scrape_as_markdown (anti-bot, JS-rendered), Playwright (interactive, login-required). Picks the right tool per target URL. [WHAT] Scraping for OSINT, intelligence monitoring, source verification. Default: start cheapest (WebFetch), escalate as needed. BrightData for Cloudflare / anti-bot-protected sites. Playwright for login or heavy SPAs. [WHEN] Use when: scrape, fetch URL, crawl, extract content, read site, capture article, mirror page, extract markdown, anti-bot. NOT for: web search (use searxng or exa), archiving (use Wayback). [LANGUAGE] English and other languages. Output content matches source language.

- Skill: `carlheath/scraping` (Agent Skill)
- Install (CLI): `npx skillmds@latest add carlheath/scraping`
- Raw SKILL.md: https://api.skillmd.com/api/skills/carlheath/scraping/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: carlheath (https://skillmd.com/u/carlheath)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/carlheath/scraping

---


# Scraping

**Role:** 4-tier scraping strategy.

## 4-tier escalation

### Tier 1: WebFetch (default)
Fast, free, works for 70% of sites. Static HTML, no JS rendering.

```
WebFetch URL prompt="Extract main article text"
```

### Tier 2: searxng (search-driven)
When you know the approximate target but not the exact URL. Searches first, fetches the best hit.

```
mcp__searxng__searxng_web_search query
mcp__searxng__web_url_read url
```

### Tier 3: BrightData scrape_as_markdown
For Cloudflare, anti-bot, JS rendering, sites that block simple bots.

```
mcp__brightdata__scrape_as_markdown URL
mcp__brightdata__discover URL  # for link discovery
```

### Tier 4: Playwright (interactive)
Login-required, multi-step (click first, then extract), heavy SPAs.

```
mcp__playwright__browser_navigate URL
mcp__playwright__browser_snapshot
mcp__playwright__browser_evaluate "..." # for dynamic content
```

## Decision rules

| Site type | Tier |
|-----------|------|
| Regular blog / Wikipedia / news article without paywall | 1 |
| Unknown URL but known topic | 2 |
| Cloudflare-protected / Reddit / X (if public) / site returning 403 to bots | 3 |
| Requires login / needs clicks / heavy SPA | 4 |

## Domain-specific defaults

| Domain | Tier strategy |
|--------|---------------|
| Government sites (parliament, ministries) | Tier 1 |
| State media with anti-bot defenses | Tier 3 |
| Telegram channels | Tier 4 (Playwright or Telegram API) |
| LinkedIn profile/post (public) | LinkedIn MCP > Tier 4 |
| Bluesky | Bluesky MCP, not scraping |
| Academic papers (arXiv, PubMed) | arxiv MCP > Tier 1 |
| Paywalled press | Tier 1 (open lede), Tier 3 if bypass via gift link |

## Output format

Markdown always. Preserve source language. Add a metadata block at the top:

```markdown
---
url: [original]
fetched: 2026-05-02T10:30:00Z
tier: 3
title: [extracted title]
---

[content]
```

## Archiving

For relevant content, offer to save to the user's vault via the `archive-to-vault` skill (typically routes to a sources MOC or an unsorted inbox).

## Version history

- v3.0 (2026-05-02): initial public release.