Fetch URL
fetch-url "https://example.com" # That's it.
Extracts text from any webpage. Works globally after install. No credentials needed — the free tools (w3m, lynx, jina, markdown) cover most sites.
Keep it simple
The default auto-selects the best tool per site. Don't add flags you don't need.
| ✗ Don't |
✓ Do |
export PATH="$HOME/.local/bin:$PATH"; fetch-url "url" --tool jina -v 2>&1 | head -100 |
fetch-url "url" |
fetch-url "url" --tool auto --clean -v |
fetch-url "url" |
- Output is already cleaned (empty lines collapsed). No
head, no 2>&1.
fetch-url: command not found? Install once (below) or use the full path ~/.local/bin/fetch-url "url". Don't prefix every call with export PATH=.- Tool auto-selects per site (Reddit→w3m, GitHub→jina, StackOverflow→markdown). Reach for
--tool only when the default output is poor.
- JS/Cloudflare-protected site?
fetch-url "url" --tool chrome (or --tool markdown --md-method browser).
Install
This skill ships in the skale-skills pi package — install the repo once:
pi install git:github.com/devskale/skale-skills
That loads the skill into pi. To also get a global fetch-url shell command, run the installer from this skill's own directory (next to SKILL.md):
./install.sh # → creates ~/.local/bin/fetch-url (uv auto-installed)
install.bat # Windows, same directory
Update
fetch-url --update # Manual update (git pull + uv sync)
fetch-url --selfcheck # Show version + last update date
Auto-updates in background every 7 days.
Usage
fetch-url "https://example.com" # Auto-selects best tool
fetch-url "https://reddit.com/r/python" # Redirects to old.reddit.com, uses w3m
fetch-url "https://news.ycombinator.com" # Uses w3m (free, local)
fetch-url "URL" --tool jina # Force specific tool
fetch-url "URL" -v # Verbose (shows tool + redirects)
Options
| Option |
Description |
--tool NAME |
w3m, lynx, jina, markdown, chrome, chawan, api |
-v, --verbose |
Show tool selection and redirects |
--no-clean |
Keep empty lines |
--update |
Update the skill now |
--selfcheck |
Show version and last update |
Tools (Auto-Selected)
Priority: free local tools first (w3m, lynx), then free APIs (jina, markdown), then chrome.
| Tool |
Best For |
Cost |
| w3m |
Reddit, HN, simple HTML sites |
Free, local |
| lynx |
Wikipedia, text-heavy sites |
Free, local |
| jina |
Docs, blogs, GitHub, Medium |
Free API |
| markdown |
StackOverflow, fallback |
Free API (50/day) |
| api |
Skale fetch endpoint (amd.skale.dev) |
Free, needs credgoo key |
| chrome |
Cloudflare/JS-protected sites |
Free, needs Chrome |
Site-Specific Behavior
| Site |
Strategy |
| Reddit |
Auto-redirects to old.reddit.com, uses w3m |
| HN |
w3m (clean output) |
| Wikipedia |
jina (cleanest), lynx/w3m fallback |
| GitHub |
jina (clean markdown) |
| StackOverflow |
markdown (bypasses blocks) |
| Medium |
jina |
| Cloudflare sites |
Chrome headless |
Configure (optional)
# Credgoo for the 'api' tool (retrieve shared team token)
credgoo FETCH_URL_BEARER
# No 'credgoo' command? Install it once:
uv tool install "credgoo @ git+https://github.com/devskale/python-openutils.git#subdirectory=packages/credgoo"
# Optional browsers for better fallback
brew install w3m lynx chawan
Troubleshooting
| Problem |
Fix |
fetch-url: command not found |
Add ~/.local/bin to PATH |
| gunzip error on GitHub |
Use --tool jina |
| Empty result |
Try --tool jina or --tool chrome |
| Dependency error |
fetch-url --update |
References
- references/sites.md — per-site tool rankings. Read when a site returns poor output and you need to pick a better
--tool.
- references/github.md — GitHub raw-URL patterns. Read when fetching GitHub file/README/directory content.
- references/troubleshooting.md — detailed fixes. Read when a fetch fails or returns empty/garbled output.
1---2name: fetch-url3description: Fetch and extract readable text from any web page — auto-selects the best backend (w3m, lynx, jina, markdown, chrome) with smart fallback. Use when the user wants to read an article, docs, or scrape text from a page. Triggers on: fetch this URL, read this page, extract the text, scrape this site, get the article content. Works on Reddit, StackOverflow, GitHub, docs sites, and more.4---56# Fetch URL78```bash9fetch-url "https://example.com" # That's it.10```1112Extracts text from any webpage. Works globally after install. **No credentials needed** — the free tools (w3m, lynx, jina, markdown) cover most sites.1314## Keep it simple1516The default auto-selects the best tool per site. Don't add flags you don't need.1718| ✗ Don't | ✓ Do |19|---------|------|20| `export PATH="$HOME/.local/bin:$PATH"; fetch-url "url" --tool jina -v 2>&1 \| head -100` | `fetch-url "url"` |21| `fetch-url "url" --tool auto --clean -v` | `fetch-url "url"` |2223- **Output is already cleaned** (empty lines collapsed). No `head`, no `2>&1`.24- **`fetch-url: command not found`?** Install once (below) or use the full path `~/.local/bin/fetch-url "url"`. Don't prefix every call with `export PATH=`.25- **Tool auto-selects per site** (Reddit→w3m, GitHub→jina, StackOverflow→markdown). Reach for `--tool` only when the default output is poor.26- **JS/Cloudflare-protected site?** `fetch-url "url" --tool chrome` (or `--tool markdown --md-method browser`).2728## Install2930This skill ships in the **skale-skills** pi package — install the repo once:3132```bash33pi install git:github.com/devskale/skale-skills34```3536That loads the skill into pi. To also get a global `fetch-url` shell command, run the installer from **this skill's own directory** (next to `SKILL.md`):3738```bash39./install.sh # → creates ~/.local/bin/fetch-url (uv auto-installed)40install.bat # Windows, same directory41```4243## Update4445```bash46fetch-url --update # Manual update (git pull + uv sync)47fetch-url --selfcheck # Show version + last update date48```4950Auto-updates in background every 7 days.5152## Usage5354```bash55fetch-url "https://example.com" # Auto-selects best tool56fetch-url "https://reddit.com/r/python" # Redirects to old.reddit.com, uses w3m57fetch-url "https://news.ycombinator.com" # Uses w3m (free, local)58fetch-url "URL" --tool jina # Force specific tool59fetch-url "URL" -v # Verbose (shows tool + redirects)60```6162## Options6364| Option | Description |65|--------|-------------|66| `--tool NAME` | w3m, lynx, jina, markdown, chrome, chawan, api |67| `-v, --verbose` | Show tool selection and redirects |68| `--no-clean` | Keep empty lines |69| `--update` | Update the skill now |70| `--selfcheck` | Show version and last update |7172## Tools (Auto-Selected)7374Priority: free local tools first (w3m, lynx), then free APIs (jina, markdown), then chrome.7576| Tool | Best For | Cost |77|------|----------|------|78| w3m | Reddit, HN, simple HTML sites | Free, local |79| lynx | Wikipedia, text-heavy sites | Free, local |80| jina | Docs, blogs, GitHub, Medium | Free API |81| markdown | StackOverflow, fallback | Free API (50/day) |82| api | Skale fetch endpoint (`amd.skale.dev`) | Free, needs credgoo key |83| chrome | Cloudflare/JS-protected sites | Free, needs Chrome |8485## Site-Specific Behavior8687| Site | Strategy |88|------|----------|89| Reddit | Auto-redirects to old.reddit.com, uses w3m |90| HN | w3m (clean output) |91| Wikipedia | jina (cleanest), lynx/w3m fallback |92| GitHub | jina (clean markdown) |93| StackOverflow | markdown (bypasses blocks) |94| Medium | jina |95| Cloudflare sites | Chrome headless |9697## Configure (optional)9899```bash100# Credgoo for the 'api' tool (retrieve shared team token)101credgoo FETCH_URL_BEARER102103# No 'credgoo' command? Install it once:104uv tool install "credgoo @ git+https://github.com/devskale/python-openutils.git#subdirectory=packages/credgoo"105106# Optional browsers for better fallback107brew install w3m lynx chawan108```109110## Troubleshooting111112| Problem | Fix |113|---------|-----|114| `fetch-url: command not found` | Add `~/.local/bin` to PATH |115| gunzip error on GitHub | Use `--tool jina` |116| Empty result | Try `--tool jina` or `--tool chrome` |117| Dependency error | `fetch-url --update` |118119## References120121- [references/sites.md](references/sites.md) — per-site tool rankings. **Read when** a site returns poor output and you need to pick a better `--tool`.122- [references/github.md](references/github.md) — GitHub raw-URL patterns. **Read when** fetching GitHub file/README/directory content.123- [references/troubleshooting.md](references/troubleshooting.md) — detailed fixes. **Read when** a fetch fails or returns empty/garbled output.
devskale