Crawl Web Text
Overview
Extract readable text from public webpages with conservative crawling behavior and clear source records. Use the bundled script for repeatable extraction, and read references/extraction-guidelines.md when access boundaries, cleanup rules, or failure reporting need closer judgment.
Workflow
- Confirm the requested source is public and does not require login, payment, CAPTCHA, or bypassing access controls.
- Prefer official APIs, feeds, exports, or provided files when available.
- For live URLs, check robots.txt and use polite fetch settings.
- Extract text with
scripts/extract_web_text.py when a deterministic artifact is useful.
- Preserve source metadata with the extracted text.
- Report failures plainly when access is blocked, restricted, or technically unavailable.
Quick Start
Extract a public webpage to Markdown:
python3 skills/crawl-web-text/scripts/extract_web_text.py --url "https://example.com/" --output page.md
Extract a saved HTML file without network access:
python3 skills/crawl-web-text/scripts/extract_web_text.py --input-file saved-page.html --format json
Script Behavior
scripts/extract_web_text.py:
- accepts one
--url or one --input-file;
- supports
--format markdown, --format json, and --format text;
- refuses non-http(s) URLs and URLs with embedded credentials;
- checks robots.txt before live fetches and fails closed if the policy cannot be fetched;
- removes script, style, hidden, and non-text elements;
- records source, final URL, fetch time, HTTP status, content type, robots decision, title, character count, and word count.
Access Rules
- Do not bypass login walls, paywalls, robots restrictions, rate limits, CAPTCHA, IP blocks, or anti-bot controls.
- Do not scrape pages that are clearly private, credentialed, or contractually restricted.
- Do not retry aggressively after 401, 403, 407, 429, or CAPTCHA-like responses.
- If the user provides saved HTML from a page they can access, extract from the file and record it as a file source.
Reference
Read references/extraction-guidelines.md for detailed boundaries, polite crawling defaults, extraction quality guidance, and failure modes.
1---2name: crawl-web-text3description: Extract clean, source-recorded text from public webpages or saved HTML files. Use when Codex needs to crawl or fetch public web pages for textual content, convert HTML into readable text, preserve source metadata, or prepare webpage text for summarization, analysis, citation, or downstream processing while respecting robots.txt, access limits, paywalls, logins, CAPTCHA, and anti-bot restrictions.4---56# Crawl Web Text78## Overview910Extract readable text from public webpages with conservative crawling behavior and clear source records. Use the bundled script for repeatable extraction, and read `references/extraction-guidelines.md` when access boundaries, cleanup rules, or failure reporting need closer judgment.1112## Workflow13141. Confirm the requested source is public and does not require login, payment, CAPTCHA, or bypassing access controls.152. Prefer official APIs, feeds, exports, or provided files when available.163. For live URLs, check robots.txt and use polite fetch settings.174. Extract text with `scripts/extract_web_text.py` when a deterministic artifact is useful.185. Preserve source metadata with the extracted text.196. Report failures plainly when access is blocked, restricted, or technically unavailable.2021## Quick Start2223Extract a public webpage to Markdown:2425```bash26python3 skills/crawl-web-text/scripts/extract_web_text.py --url "https://example.com/" --output page.md27```2829Extract a saved HTML file without network access:3031```bash32python3 skills/crawl-web-text/scripts/extract_web_text.py --input-file saved-page.html --format json33```3435## Script Behavior3637`scripts/extract_web_text.py`:3839- accepts one `--url` or one `--input-file`;40- supports `--format markdown`, `--format json`, and `--format text`;41- refuses non-http(s) URLs and URLs with embedded credentials;42- checks robots.txt before live fetches and fails closed if the policy cannot be fetched;43- removes script, style, hidden, and non-text elements;44- records source, final URL, fetch time, HTTP status, content type, robots decision, title, character count, and word count.4546## Access Rules4748- Do not bypass login walls, paywalls, robots restrictions, rate limits, CAPTCHA, IP blocks, or anti-bot controls.49- Do not scrape pages that are clearly private, credentialed, or contractually restricted.50- Do not retry aggressively after 401, 403, 407, 429, or CAPTCHA-like responses.51- If the user provides saved HTML from a page they can access, extract from the file and record it as a file source.5253## Reference5455Read `references/extraction-guidelines.md` for detailed boundaries, polite crawling defaults, extraction quality guidance, and failure modes.
CoUse-123