# Doc Harvester > Mirror an external platform's documentation site into the current project as local markdown files, so the coding agent can cross-check the official docs offline while building. Use this whenever a documentation or wiki URL appears together with a request to study, learn, fetch, download, mirror, save, scrape, or "vendor" the docs — including phrasings like "изучи документацию", "скачай документацию по ссылке", "собери вики", "study these docs", "pull the API docs into the repo", "I'm integrating with X, get its docs". Use it even if the user just pastes a docs link and says "learn this" without naming a tool, and even if they only give a deep link to one page. Prefer this skill over ad-hoc WebFetch whenever the goal is to capture a whole documentation set rather than read a single page. - Skill: `arturayupov/doc-harvester` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add arturayupov/doc-harvester` - Raw SKILL.md: https://api.skillmd.com/api/skills/arturayupov/doc-harvester/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: arturayupov (https://skillmd.com/u/arturayupov) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/arturayupov/doc-harvester --- # doc-harvester Turn any documentation site into a folder of local markdown inside the user's project, so that during development the coding agent can grep and read the official docs instead of guessing or fetching pages one at a time. A plain single-URL fetch fails on real doc sites for three reasons, and this skill is built to defeat all three: (1) one fetch only sees one page; (2) most modern docs are client-rendered single-page apps, so a raw fetch returns an empty HTML shell with no content; (3) content hidden behind tabs and nested sections is never discovered. The engine works around this with a ladder of strategies, cheapest and most reliable first. ## Quickstart The engine is the bundled script `scripts/harvest.py` (Python 3, standard library only — nothing to install for the common cases). Run it from the project root: ```bash python3 /scripts/harvest.py "" --out docs/vendor ``` `` is the directory containing this SKILL.md. `` can be any page of the target docs — the engine figures out the root itself. Output lands in `docs/vendor//`. The engine auto-selects a method and prints a final `RESULT:` line. Read it: it reports which method succeeded, how many pages were saved, and the output directory. Then read `_index.md` and skim `llms-full.md` in the output folder to confirm the content is real documentation and not an error page. ## How the engine chooses a method The engine tries each step below and stops at the first that yields content. Understanding the ladder helps when you need to intervene. 1. **`llms-full.txt`** — many modern doc platforms (Mintlify, parts of GitBook, Fern, some Docusaurus setups) publish the entire documentation as one LLM-ready file at `/llms-full.txt`. One request, the whole corpus. Always the best outcome — if you see a "Copy page" button or a Markdown/ChatGPT/Claude option on the docs site, this file very likely exists. 2. **`llms.txt`** — a curated index of links. The engine fetches every linked page as markdown and also assembles a merged `llms-full.md`. 3. **`sitemap.xml`** — every documentation URL is enumerated from the sitemap (and `robots.txt`), then each page is fetched. For each URL it first tries the `URL + ".md"` source endpoint (Mintlify/Fern serve clean markdown there), and falls back to Jina Reader. 4. **Jina Reader BFS** — when there is no sitemap, `r.jina.ai` renders the entry page (executing JavaScript, so SPAs work) and the engine crawls same-site documentation links breadth-first. Free, no API key. 5. **`sitefetch`** — a last-resort whole-site crawler run via `npx sitefetch` (needs Node). Best effort; output parsing is defensive. You normally just run the script once and let it pick. Use `--method ` only to force a specific step for debugging. ## When the engine cannot collect the docs If the `RESULT:` line says `status=failed`, the site is fully client-rendered with no `llms.txt` and no sitemap. Escalate in this order: - **Retry with Jina explicitly**: `--method jina`. If Jina was rate-limited (HTTP 429), wait a minute and retry, or raise `--delay 1.0`. A Jina API key removes the limit — if the user has one they can request a free key at jina.ai; the engine does not require it. - **crawl4ai — final resort.** `crawl4ai` is a free, open-source crawler with a real headless browser. It is **not** bundled because installing it downloads a browser (~150 MB) and its Python API changes between versions. Only use it if the steps above failed, and **ask the user first** before installing. Then: ```bash pip install -U crawl4ai crawl4ai-setup # installs the headless browser, one time ``` Check the installed version's deep-crawl API (`python3 -c "import crawl4ai, inspect"` or the project README), then write a short script that deep-crawls the docs root, collects each result's `.markdown`, and saves the pages exactly the way `harvest.py` does — one `.md` per page under `pages/`, a merged `llms-full.md`, and an `_index.md`. Keep the same output layout so the rest of this skill's conventions still hold. Report honestly what was and was not captured. A partial harvest (e.g. sitemap found but some pages 404'd) is still useful — say so rather than implying it is complete. ## Output layout Everything goes under `docs/vendor//`: ``` docs/vendor// llms-full.md the entire documentation merged into one file (best for grep / full reads) pages/... one markdown file per page, mirroring the URL path (when pages were fetched) _index.md table of contents — the site's own llms.txt when available _meta.json source URL, method used, timestamp, per-page checksums ``` `--out` changes the base directory and `--name` overrides the auto-derived platform folder name. Re-running overwrites the page files (a fresh snapshot) without deleting unrelated files. ## After harvesting — make the docs actually get used Collecting the docs is only half the job; the point is that the in-repo agent consults them. After a successful harvest, do this so future development cross-checks the official docs: 1. Tell the user the path and that they can now reference it. For a quick targeted lookup, `grep` inside `docs/vendor//` instead of re-fetching the web. 2. Offer to register the docs in the project's agent guide. If a `CLAUDE.md` (or `AGENTS.md` / `.cursorrules`) exists at the project root, add a short line such as: > Official documentation is mirrored in `docs/vendor//`. Consult > `llms-full.md` there before implementing anything that touches the API. This is what makes the coding agent reach for the local docs automatically. 3. If the integration is non-trivial, skim `llms-full.md` now and confirm the key facts the current task depends on (auth, endpoints, rate limits) before writing code. ## Options reference | Flag | Purpose | |------|---------| | `--out DIR` | base output directory (default `docs/vendor`) | | `--name NAME` | output sub-folder name (default: derived from the host) | | `--method auto\|llms\|sitemap\|jina\|sitefetch` | force one strategy (default `auto`) | | `--max-pages N` | cap pages fetched (default 1500) | | `--include-prefix /path` | restrict the sitemap/jina crawl to one section | | `--delay SECONDS` | politeness delay between requests (default 0.15; raise if throttled) | | `--quiet` | reduce logging | ## Notes - The engine sniffs response bodies, not just status codes, because single-page-app docs answer `200 OK` with an HTML shell for *any* path — including a missing `llms.txt`. A candidate counts only if the body is non-trivial and not HTML. - Be polite: the default delay is small but real; raise `--delay` for small or rate-limited sites rather than hammering them. - The harvested docs are a point-in-time snapshot. For long-lived projects, re-run the skill periodically so the local copy stays current with the upstream documentation.