Doc Harvester

doc-harvester

Turn any documentation site into a folder of local markdown inside the user's project, so that during development the coding agent can grep and read the official docs instead of guessing or fetching pages one at a time.

A plain single-URL fetch fails on real doc sites for three reasons, and this skill is built to defeat all three: (1) one fetch only sees one page; (2) most modern docs are client-rendered single-page apps, so a raw fetch returns an empty HTML shell with no content; (3) content hidden behind tabs and nested sections is never discovered. The engine works around this with a ladder of strategies, cheapest and most reliable first.

Quickstart

The engine is the bundled script scripts/harvest.py (Python 3, standard library only — nothing to install for the common cases). Run it from the project root:

python3 <skill_dir>/scripts/harvest.py "<DOC_URL>" --out docs/vendor

<skill_dir> is the directory containing this SKILL.md. <DOC_URL> can be any page of the target docs — the engine figures out the root itself. Output lands in docs/vendor/<platform>/.

The engine auto-selects a method and prints a final RESULT: line. Read it: it reports which method succeeded, how many pages were saved, and the output directory. Then read _index.md and skim llms-full.md in the output folder to confirm the content is real documentation and not an error page.

How the engine chooses a method

The engine tries each step below and stops at the first that yields content. Understanding the ladder helps when you need to intervene.

1. llms-full.txt — many modern doc platforms (Mintlify, parts of GitBook, Fern, some Docusaurus setups) publish the entire documentation as one LLM-ready file at /llms-full.txt. One request, the whole corpus. Always the best outcome — if you see a "Copy page" button or a Markdown/ChatGPT/Claude option on the docs site, this file very likely exists.
2. llms.txt — a curated index of links. The engine fetches every linked page as markdown and also assembles a merged llms-full.md.
3. sitemap.xml — every documentation URL is enumerated from the sitemap (and robots.txt), then each page is fetched. For each URL it first tries the URL + ".md" source endpoint (Mintlify/Fern serve clean markdown there), and falls back to Jina Reader.
4. Jina Reader BFS — when there is no sitemap, r.jina.ai renders the entry page (executing JavaScript, so SPAs work) and the engine crawls same-site documentation links breadth-first. Free, no API key.
5. sitefetch — a last-resort whole-site crawler run via npx sitefetch (needs Node). Best effort; output parsing is defensive.

You normally just run the script once and let it pick. Use --method <name> only to force a specific step for debugging.

When the engine cannot collect the docs

If the RESULT: line says status=failed, the site is fully client-rendered with no llms.txt and no sitemap. Escalate in this order:

• Retry with Jina explicitly: --method jina. If Jina was rate-limited (HTTP 429), wait a minute and retry, or raise --delay 1.0. A Jina API key removes the limit — if the user has one they can request a free key at jina.ai; the engine does not require it.

• crawl4ai — final resort. crawl4ai is a free, open-source crawler with a real headless browser. It is not bundled because installing it downloads a browser (~150 MB) and its Python API changes between versions. Only use it if the steps above failed, and ask the user first before installing. Then:

  pip install -U crawl4ai
  crawl4ai-setup            # installs the headless browser, one time
  

  Check the installed version's deep-crawl API (python3 -c "import crawl4ai, inspect" or the project README), then write a short script that deep-crawls the docs root, collects each result's .markdown, and saves the pages exactly the way harvest.py does — one .md per page under pages/, a merged llms-full.md, and an _index.md. Keep the same output layout so the rest of this skill's conventions still hold.

Report honestly what was and was not captured. A partial harvest (e.g. sitemap found but some pages 404'd) is still useful — say so rather than implying it is complete.

Output layout

Everything goes under docs/vendor/<platform>/:

docs/vendor/<platform>/
  llms-full.md   the entire documentation merged into one file (best for grep / full reads)
  pages/...      one markdown file per page, mirroring the URL path (when pages were fetched)
  _index.md      table of contents — the site's own llms.txt when available
  _meta.json     source URL, method used, timestamp, per-page checksums

--out changes the base directory and --name overrides the auto-derived platform folder name. Re-running overwrites the page files (a fresh snapshot) without deleting unrelated files.

After harvesting — make the docs actually get used

Collecting the docs is only half the job; the point is that the in-repo agent consults them. After a successful harvest, do this so future development cross-checks the official docs:

1. Tell the user the path and that they can now reference it. For a quick targeted lookup, grep inside docs/vendor/<platform>/ instead of re-fetching the web.

2. Offer to register the docs in the project's agent guide. If a CLAUDE.md (or AGENTS.md / .cursorrules) exists at the project root, add a short line such as:

   Official documentation is mirrored in docs/vendor/<platform>/. Consult llms-full.md there before implementing anything that touches the API.

   This is what makes the coding agent reach for the local docs automatically.

3. If the integration is non-trivial, skim llms-full.md now and confirm the key facts the current task depends on (auth, endpoints, rate limits) before writing code.

Options reference

Notes

• The engine sniffs response bodies, not just status codes, because single-page-app docs answer 200 OK with an HTML shell for any path — including a missing llms.txt. A candidate counts only if the body is non-trivial and not HTML.
• Be polite: the default delay is small but real; raise --delay for small or rate-limited sites rather than hammering them.
• The harvested docs are a point-in-time snapshot. For long-lived projects, re-run the skill periodically so the local copy stays current with the upstream documentation.