How do I install the Fetch Url As Markdown skill?

Run npx skillmds@latest add codealive-ai-agents-reflection-skills/fetch-url-as-markdown in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Fetch Url As Markdown safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: executes scripts, makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Fetch Url As Markdown?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Fetch Url As Markdown free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Fetch Url As Markdown?

CodeAlive-AI (@codealive-ai-agents-reflection-skills) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Fetch Url As Markdown by codealive-ai-agents-reflection-skills

Name: Fetch Url As Markdown
Rating: 4.2 (5 reviews)
Author: CodeAlive-AI

Fetch Url As Markdown

Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints.

CodeAlive-AI Updated Sep 10, 2026

File contents

URL to Markdown

Fetch any web URL and get clean, readable Markdown — main content only, no navigation/footer/ads. Local + free by default; smart fallback to Exa MCP when the page can't be extracted locally.

Workflow (the only thing the agent needs to remember)

Try trafilatura first:

python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"

If exit code is 1 or 2 → fall back to Exa MCP with the same URL:
```
mcp__exa__web_search_advanced_exa(
    query="<URL>",
    includeDomains=["<host of URL>"],
    numResults=1,
    textMaxCharacters=50000,
    type="auto"
)
```
(mcp__exa__crawling works too if the server exposes it; the web_search_advanced_exa call above is the always-available variant — pin the host with includeDomains and use the URL itself as the query.)
Exit code 3 means trafilatura is not installed — install once:
```
python3 -m pip install --break-system-packages trafilatura
```

Exit codes (what they mean for the fallback decision)

Code	Meaning	Action
0	Markdown printed to stdout	done
1	DownloadError — network/HTTP/timeout/anti-bot block at fetch	fall back to Exa
2	ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars)	fall back to Exa
3	trafilatura missing	install (see above), then retry
4	UnsupportedContentTypeError — URL is binary (PDF, image, archive)	don't fall back to Exa; use the right specialized skill (e.g. `pdf` for PDFs)

Defaults baked into the script

output_format="markdown", include_formatting=True — keeps headings/lists/code structure where the source HTML uses real <h1..h6> etc.
include_links=True, include_tables=True
with_metadata=True → emits a YAML frontmatter (title, author, date, url, hostname)
favor_recall=True, deduplicate=True — readable but trims duplicates
Real-browser User-Agent + 30s timeout configured in scripts/settings.cfg
Anti-stub guards (built into the script):
- rejects Content-Type other than text/html|application/xhtml+xml|text/plain|application/xml|text/xml → exit 4
- sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit 2
- rejects extracted bodies under 50 chars (configurable via --min-body N, 0 to disable) → exit 2

Useful flags

... fetch_url.py "<URL>" --no-links     # strip hyperlinks
... fetch_url.py "<URL>" --no-tables    # strip tables
... fetch_url.py "<URL>" --no-metadata  # omit YAML header
... fetch_url.py "<URL>" --comments     # include user comments (off by default — usually noise)
... fetch_url.py "<URL>" --images       # include image refs (experimental)
... fetch_url.py "<URL>" --precision    # terser output, drops borderline content

When to choose what

Situation	Tool
Article, blog post, docs, README, wiki	trafilatura (default) — local, free
JS-heavy SPA, login-walled, Cloudflare	Exa fallback (the script will signal exit 2)
Bulk / many URLs	trafilatura — no quota, no API key
Already failed twice on a domain	Exa directly

---name: fetch-url-as-markdowndescription: Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints.---# URL to MarkdownFetch any web URL and get clean, readable Markdown — main content only, nonavigation/footer/ads. Local + free by default; smart fallback to Exa MCPwhen the page can't be extracted locally.## Workflow (the only thing the agent needs to remember)1. **Try trafilatura first**: ```bash python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>" ```2. **If exit code is 1 or 2 → fall back to Exa MCP** with the same URL: ``` mcp__exa__web_search_advanced_exa( query="<URL>", includeDomains=["<host of URL>"], numResults=1, textMaxCharacters=50000, type="auto" ) ``` (`mcp__exa__crawling` works too if the server exposes it; the `web_search_advanced_exa` call above is the always-available variant — pin the host with `includeDomains` and use the URL itself as the query.)3. Exit code `3` means trafilatura is not installed — install once: ```bash python3 -m pip install --break-system-packages trafilatura ```## Exit codes (what they mean for the fallback decision)| Code | Meaning | Action ||---|---|---|| 0 | Markdown printed to stdout | done || 1 | DownloadError — network/HTTP/timeout/anti-bot block at fetch | fall back to Exa || 2 | ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) | fall back to Exa || 3 | trafilatura missing | install (see above), then retry || 4 | UnsupportedContentTypeError — URL is binary (PDF, image, archive) | **don't** fall back to Exa; use the right specialized skill (e.g. `pdf` for PDFs) |## Defaults baked into the script- `output_format="markdown"`, `include_formatting=True` — keeps headings/lists/code structure where the source HTML uses real `<h1..h6>` etc.- `include_links=True`, `include_tables=True`- `with_metadata=True` → emits a YAML frontmatter (`title`, `author`, `date`, `url`, `hostname`)- `favor_recall=True`, `deduplicate=True` — readable but trims duplicates- Real-browser User-Agent + 30s timeout configured in `scripts/settings.cfg`- Anti-stub guards (built into the script): - rejects `Content-Type` other than `text/html|application/xhtml+xml|text/plain|application/xml|text/xml` → exit `4` - sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit `2` - rejects extracted bodies under 50 chars (configurable via `--min-body N`, `0` to disable) → exit `2`## Useful flags```bash... fetch_url.py "<URL>" --no-links # strip hyperlinks... fetch_url.py "<URL>" --no-tables # strip tables... fetch_url.py "<URL>" --no-metadata # omit YAML header... fetch_url.py "<URL>" --comments # include user comments (off by default — usually noise)... fetch_url.py "<URL>" --images # include image refs (experimental)... fetch_url.py "<URL>" --precision # terser output, drops borderline content```## When to choose what| Situation | Tool ||---|---|| Article, blog post, docs, README, wiki | trafilatura (default) — local, free || JS-heavy SPA, login-walled, Cloudflare | Exa fallback (the script will signal exit 2) || Bulk / many URLs | trafilatura — no quota, no API key || Already failed twice on a domain | Exa directly |

Frequently asked questions