URL to Markdown
Fetch any web URL and get clean, readable Markdown — main content only, no
navigation/footer/ads. Local + free by default; smart fallback to Exa MCP
when the page can't be extracted locally.
Workflow (the only thing the agent needs to remember)
Try trafilatura first:
python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"
If exit code is 1 or 2 → fall back to Exa MCP with the same URL:
mcp__exa__web_search_advanced_exa(
query="<URL>",
includeDomains=["<host of URL>"],
numResults=1,
textMaxCharacters=50000,
type="auto"
)
(mcp__exa__crawling works too if the server exposes it; the web_search_advanced_exa
call above is the always-available variant — pin the host with includeDomains and
use the URL itself as the query.)
Exit code 3 means trafilatura is not installed — install once:
python3 -m pip install --break-system-packages trafilatura
Exit codes (what they mean for the fallback decision)
| Code |
Meaning |
Action |
| 0 |
Markdown printed to stdout |
done |
| 1 |
DownloadError — network/HTTP/timeout/anti-bot block at fetch |
fall back to Exa |
| 2 |
ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) |
fall back to Exa |
| 3 |
trafilatura missing |
install (see above), then retry |
| 4 |
UnsupportedContentTypeError — URL is binary (PDF, image, archive) |
don't fall back to Exa; use the right specialized skill (e.g. pdf for PDFs) |
Defaults baked into the script
output_format="markdown", include_formatting=True — keeps headings/lists/code structure where the source HTML uses real <h1..h6> etc.
include_links=True, include_tables=True
with_metadata=True → emits a YAML frontmatter (title, author, date, url, hostname)
favor_recall=True, deduplicate=True — readable but trims duplicates
- Real-browser User-Agent + 30s timeout configured in
scripts/settings.cfg
- Anti-stub guards (built into the script):
- rejects
Content-Type other than text/html|application/xhtml+xml|text/plain|application/xml|text/xml → exit 4
- sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit
2
- rejects extracted bodies under 50 chars (configurable via
--min-body N, 0 to disable) → exit 2
Useful flags
... fetch_url.py "<URL>" --no-links # strip hyperlinks
... fetch_url.py "<URL>" --no-tables # strip tables
... fetch_url.py "<URL>" --no-metadata # omit YAML header
... fetch_url.py "<URL>" --comments # include user comments (off by default — usually noise)
... fetch_url.py "<URL>" --images # include image refs (experimental)
... fetch_url.py "<URL>" --precision # terser output, drops borderline content
When to choose what
| Situation |
Tool |
| Article, blog post, docs, README, wiki |
trafilatura (default) — local, free |
| JS-heavy SPA, login-walled, Cloudflare |
Exa fallback (the script will signal exit 2) |
| Bulk / many URLs |
trafilatura — no quota, no API key |
| Already failed twice on a domain |
Exa directly |
1---2name: fetch-url-as-markdown3description: Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints.4metadata:5 source: https://skillmd.com/skills/codealive-ai-agents-reflection-skills/fetch-url-as-markdown6---78# URL to Markdown910Fetch any web URL and get clean, readable Markdown — main content only, no11navigation/footer/ads. Local + free by default; smart fallback to Exa MCP12when the page can't be extracted locally.1314## Workflow (the only thing the agent needs to remember)15161. **Try trafilatura first**:1718 ```bash19 python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"20 ```21222. **If exit code is 1 or 2 → fall back to Exa MCP** with the same URL:2324 ```25 mcp__exa__web_search_advanced_exa(26 query="<URL>",27 includeDomains=["<host of URL>"],28 numResults=1,29 textMaxCharacters=50000,30 type="auto"31 )32 ```3334 (`mcp__exa__crawling` works too if the server exposes it; the `web_search_advanced_exa`35 call above is the always-available variant — pin the host with `includeDomains` and36 use the URL itself as the query.)37383. Exit code `3` means trafilatura is not installed — install once:3940 ```bash41 python3 -m pip install --break-system-packages trafilatura42 ```4344## Exit codes (what they mean for the fallback decision)4546| Code | Meaning | Action |47|---|---|---|48| 0 | Markdown printed to stdout | done |49| 1 | DownloadError — network/HTTP/timeout/anti-bot block at fetch | fall back to Exa |50| 2 | ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) | fall back to Exa |51| 3 | trafilatura missing | install (see above), then retry |52| 4 | UnsupportedContentTypeError — URL is binary (PDF, image, archive) | **don't** fall back to Exa; use the right specialized skill (e.g. `pdf` for PDFs) |5354## Defaults baked into the script5556- `output_format="markdown"`, `include_formatting=True` — keeps headings/lists/code structure where the source HTML uses real `<h1..h6>` etc.57- `include_links=True`, `include_tables=True`58- `with_metadata=True` → emits a YAML frontmatter (`title`, `author`, `date`, `url`, `hostname`)59- `favor_recall=True`, `deduplicate=True` — readable but trims duplicates60- Real-browser User-Agent + 30s timeout configured in `scripts/settings.cfg`61- Anti-stub guards (built into the script):62 - rejects `Content-Type` other than `text/html|application/xhtml+xml|text/plain|application/xml|text/xml` → exit `4`63 - sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit `2`64 - rejects extracted bodies under 50 chars (configurable via `--min-body N`, `0` to disable) → exit `2`6566## Useful flags6768```bash69... fetch_url.py "<URL>" --no-links # strip hyperlinks70... fetch_url.py "<URL>" --no-tables # strip tables71... fetch_url.py "<URL>" --no-metadata # omit YAML header72... fetch_url.py "<URL>" --comments # include user comments (off by default — usually noise)73... fetch_url.py "<URL>" --images # include image refs (experimental)74... fetch_url.py "<URL>" --precision # terser output, drops borderline content75```7677## When to choose what7879| Situation | Tool |80|---|---|81| Article, blog post, docs, README, wiki | trafilatura (default) — local, free |82| JS-heavy SPA, login-walled, Cloudflare | Exa fallback (the script will signal exit 2) |83| Bulk / many URLs | trafilatura — no quota, no API key |84| Already failed twice on a domain | Exa directly |