Trafilatura

Trafilatura

CLI skill for trafilatura: extract main article text and metadata from URLs, piped HTML, or local files. Follow the stdout-first protocol before any batch writes.

Scope: Single-page extraction, metadata JSON, piped/local HTML, bounded URL discovery (--list), and user-approved batch processing. NOT for: multi-source research (research), PDF/tables (docling), raw page fetch (fetch/fetcher), video hosts (yt-dlp), or paywall bypass.

Canonical Vocabulary

Dispatch

Auto-Detection Heuristic

1. URL + "sitemap", "feed", "crawl", "list URLs" → Discover (--list only).
2. URL + "metadata", "title", "author", "date" → Metadata.
3. Path to .html or "pipe", "stdin", "local file" → Pipe.
4. Path to URL list file or "batch" → Batch (approval gate).
5. Directory of saved HTML → Local (approval gate).
6. "Archive", "wayback", "link rot" after failed fetch → Archived.
7. Bare URL or "extract", "clean", "article text" → Extract.
8. Ambiguous → ask: extract, metadata, discover list, or batch?

Stdout-First Protocol

Run stages in order. Do not skip doctor after CLI errors. Do not batch-write until single-URL needs are met or the user explicitly approves bulk scope.

Stage 1 — Doctor (once per session or after errors)

uv run python scripts/doctor.py --format json

Stop on ok: false or any check with status: fail (missing trafilatura binary).

Stage 2 — Extract (preferred for single URLs)

uv run python scripts/extract_url.py --url 'https://…' --format json

Markdown (default readable):

uv run python scripts/extract_url.py --url 'https://…' --output-format markdown

From probe JSON, report: url, title, date (if metadata), text_length, and format. Load references/output-formats.md when choosing formats.

Stage 3 — Discover (list only by default)

uv run python scripts/list_urls.py --mode sitemap --url 'https://…' --format json

Present url_count and a sample of URLs. Do not chain into batch download without explicit user approval.

Stage 4 — Batch / local (explicit approval required)

Before running:

1. Confirm URL count or directory scope.
2. Confirm output directory (default: $HOME/Downloads/trafilatura/output/).
3. Confirm politeness acknowledgment — see references/politeness-and-ethics.md.

mkdir -p "$HOME/Downloads/trafilatura/output"
trafilatura -i list.txt -o "$HOME/Downloads/trafilatura/output/" --markdown --backup-dir "$HOME/Downloads/trafilatura/html-backup/"

Stage 5 — Escalation (missing or blocked content)

1. Retry with --recall (CLI) or --recall flag on extract_url.py.
2. If download blocked: wget -qO- 'URL' | trafilatura or curl -sL 'URL' | trafilatura.
3. If unavailable: extract_url.py --archived or trafilatura --archived -u 'URL'.
4. If still empty: note JS-rendered page limitation — no in-tool browser automation. See references/troubleshooting.md.

MCP fallback (no Bash)

When doctor fails because the harness has no shell, use MCPHub trafilatura → fetch_and_extract. Document degraded path. See references/mcp-fallback.md.

Mode Details

Extract

• Use bundled extract_url.py (wraps trafilatura -u).
• Read-only regarding repo tree; stdout or JSON envelope only.
• Default output: markdown for human summary; json when structured metadata is needed.

Metadata

uv run python scripts/extract_url.py --url 'https://…' --output-format json --with-metadata

Pipe

trafilatura --markdown < page.html
cat page.html | trafilatura --json --no-tables

Discover

Modes: feed, sitemap, crawl, probe. Always --list via list_urls.py unless user explicitly requests harvest.

Optional filters: --url-filter, --target-language (requires trafilatura[all]). See references/discovery.md.

Batch

• Parse list file (one URL per line).
• Require approval before -i runs.
• Recommend --backup-dir for HTML archival.
• Never write into the agents repo.

Local

• --input-dir for previously downloaded HTML.
• Mirror structure with --keep-dirs when needed.
• Approval gate same as batch.

Archived

• Use when fetch_url equivalent returns nothing.
• Slow; best for small URL sets.

Doctor

uv run python scripts/doctor.py --format json

Help

Show dispatch table, stdout-first protocol, default paths, and reference index.

Gallery (Empty Arguments)

Pick a number, a mode from the dispatch table, or paste a URL and say whether you need extract, metadata, or URL discovery.

Critical Rules

1. Doctor before first extract in a session (or after CLI errors).
2. Stdout-first for single URLs — use extract_url.py before filesystem batch writes.
3. Discover list-only by default — present URL count; no auto bulk download.
4. Explicit approval for batch/local — confirm scope, output dir, politeness.
5. No paywall bypass coaching — see references/politeness-and-ethics.md.
6. Not Fetch MCP primary — article main-text extraction belongs here when trafilatura is appropriate.
7. Not research — multi-source synthesis uses /research.
8. Not docling — PDFs and table-heavy documents use docling MCP.
9. Not yt-dlp — video/audio hosts use /yt-dlp.
10. Report provenance — URL, format, title/date if present, text length in summary.

Troubleshooting

See references/troubleshooting.md for precision/recall, archived fallback, wget/curl pipe, and JS-rendered pages.

References

Scripts

Examples

# Preflight
uv run python scripts/doctor.py --format json

# Extract (stdout-first)
uv run python scripts/extract_url.py --url 'https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/' --output-format markdown

# Metadata JSON
uv run python scripts/extract_url.py --url 'https://example.org' --output-format json --with-metadata

# Discover sitemap URLs (list only)
uv run python scripts/list_urls.py --mode sitemap --url 'https://www.sitemaps.org/' --format json

# Pipe local HTML
trafilatura --markdown < saved.html

# Batch (after user approval)
trafilatura -i urls.txt -o "$HOME/Downloads/trafilatura/output/" --markdown --backup-dir "$HOME/Downloads/trafilatura/html-backup/"