Auditing Documentation

Auditing Documentation

Overview

A doc is a set of claims about reality. Auditing means verifying those claims against ground truth (code, CLI, config, API) and fixing what's wrong. The dangerous mistake is treating every doc as something that must match the code. First decide whether a doc is even supposed to track current reality. Then auto-fix only what is determinate, route everything ambiguous to the human, and never rewrite a historical record. And when you're auditing a whole doc set, audit the set too — for duplication, gaps, contradictions, and structure — not just each doc in isolation (Phase 4).

Violating the letter of these rules is violating the spirit of them. "I was bringing the docs up to date" is exactly how design docs get their history erased.

When to use

• "The docs are out of date / don't match the code."
• Before a release, when doc accuracy matters.
• You don't trust a README/spec/reference and want it verified.

Not for: writing new docs, prose copyediting/style, or docs with no code to check against.

STOP: classify before you edit (evergreen vs. point-in-time)

The single biggest failure is rewriting a dated design/plan/spec to "match the code." A spec from three months ago describing how something should work is not wrong when the code later diverged — editing it to match destroys the record.

• Evergreen — README, ABOUT, CLAUDE.md, tutorials, living API/schema reference. Contract: reflects current reality. Drift is a defect → fix.
• Point-in-time — design specs, plans, brainstorm notes (often dated, or under docs/specs|plans|…). Contract: true as of its date. Drift vs. current code is expected. Never rewrite to match code; at most add a supersede banner or as-of date.
• Mixed / unclear — conflicting signals (a dated-folder README) → treat the whole doc as interview-only.

Classify the doc set first, at the group level (folder globs + named exceptions), and confirm with the human before editing anything. Precedence: a point-in-time signal (dated filename, point-in-time directory) beats an evergreen name like README. Do not decide this per-doc by gut as you go — surface it as one decision.

The classification confirmation is the one gate you never skip — not under time pressure, not under authority, not under "just be decisive, don't kick it back to me." A one-line confirm costs seconds; skipping it is what turns an audit into an erased design record. And when classification is genuinely ambiguous (e.g. a reference-looking file inside a specs/ folder that also holds design docs), resolve to the safe side — treat it as point-in-time and confirm — because a body rewrite is irreversible to the record. Disclosing your assumption in a footnote after you've rewritten the body is too late.

Scope & exclusions

Default target: README, top-level *.md, docs/**, CLAUDE.md. Always exclude: git-ignored paths, .claude/, .private-journal/, worktrees; generated / foreign-owned docs (any "do not edit" / "generated by" sentinel — editing them is futile, the next regeneration wipes it); non-markdown.

The rubric: what may be auto-fixed

Auto-fix a claim only when ALL hold:

1. the doc is evergreen;
2. the claim is mechanical — an identifier/path, CLI flag, config/env name+default, API route/field, or a token-level example fix;
3. a single live (non-test) counterpart exists in the code, so the right value is determined, not guessed (multiple hits / test-only / no hit → interview);
4. it's a local token replacement that leaves the surrounding sentence true;
5. a missing counterpart is confirmed removed vs. renamed via git history (git log -S, --follow, blame) before you conclude anything.

Always interview — never auto-fix — regardless of category:

• Counts & inventories ("14 workflows", "~40 files") — no canonical counting convention.
• Bare line-number citations (file.go:120-130) — they drift on every edit. Recommend rewriting to file:symbol; never silently renumber.
• Absence / negative claims ("there is no env-var fallback") — you can't grep-prove a negative.
• Cross-reference repair — detecting a broken link is fine; choosing its new target rarely is.
• Structural changes to embedded examples — rewriting an example's shape to a new schema.
• Behavioral / semantic claims ("does X when Y", sequencing, rationale).
• Claims whose ground truth lives in another repo or an external binary.

Verify the claim, not just the symbol

Confirming that the thing a doc names exists is not confirming the claim about it. A claim of the form "X is validated / X happens when Y / X is done by Z / X is configured as W" is verified only by the code path that enacts it — the validator that rejects, the handler that closes the stream, the function that computes the value, the line that loads the asset — not by X's mere existence. Check the verb, not just the noun.

• "the loader rejects an emit node that declares runner" → find the rejection in the validator, or the claim is false.
• "the stream closes when the run is terminal" → find the close on every terminal state, or name the one it misses.
• "diffs are computed in internal/document" → find the call there, not just the function's definition.
• "Tailwind is loaded from a CDN" → check the actual <script src>, not that Tailwind is used.

This is the most common false-matches: the noun checks out, so the verifier waves the verb through. State the mechanism, and point the citation at the code that does the thing claimed.

Process

0 — Scope, orient, classify (main thread). Build a ground-truth map first: where CLI defs, routes, config, and core types live (read CLAUDE.md's package map and the directory layout). Propose the group-level classification; confirm with the human.

1 — Verify. For evergreen docs, fan out one subagent per doc (cap the width; budget claims per doc). Each extracts atomic claims, classifies per the rubric, verifies against ground truth — the code path that enacts each claim, not just that the named symbol exists (see "Verify the claim, not just the symbol") — and returns findings with a file:line or command-output citation for every verdict, including "matches". No edits. Point-in-time docs: handle in the main thread (or one batched agent) — no claim-vs-code checking, only meta-detection (superseded-but-unmarked, missing date, linked-as-live, broken refs).

2 — Triage & apply (main thread). Apply the determinate auto-fixes to the working tree. Run a bounded, prioritized interview on everything else (evergreen high-confidence-stale first; defer the long tail to the report). Apply approved point-in-time meta-fixes.

3 — Adversarially verify your own edits (main thread). Before declaring done, re-check the edits you just applied — an audit's own fixes are exactly where confident-but-wrong claims hide: an over-claim, a stale code comment restated as fact, a removal that dropped a still-live concept, an example that won't validate. Dispatch two or more competing subagents that race to find the largest number of legitimate errors in the applied diff, each claim re-verified against the code path that enacts it (not just symbol existence) with a file:line or command-output citation. Tell them explicitly they are competing and that padding or inflating findings disqualifies them — that framing is what keeps the pass honest. Fix every confirmed finding. Only then stamp the last-reviewed marker on the verified evergreen docs, and leave everything uncommitted for human review.

4 — Corpus review (the whole set, not one doc at a time). Phases 1–3 are per-doc, so they are blind to set-level defects. When you audit a doc set (not a single file), run a pass over the whole set — independent/competing agents, fed the canonical-owner scheme:

• Duplication — the same fact stated substantively in 2+ docs drifts (and already has, in practice). Assign each fact a single canonical owner (HTTP routes → the API doc, CLI flags → the CLI doc, field schemas → the schema doc, runtime/firing semantics → the runtime doc, event types → the logging doc, …) and replace the copies with cross-references. Fan out one agent per doc for the trimming — each owns one file, so the edits don't race.
• Coverage / gaps — enumerate the codebase's surfaces (packages, CLI commands, routes, event types, config) and check each is documented somewhere. A missing doc is invisible to a per-doc audit — you only find it by listing what should exist. Write the missing docs (to the same standard) or file the gaps.
• Cross-doc contradictions — the same fact stated two different ways in two docs (one is wrong). Reconcile at the canonical owner.
• Structure / hygiene — is there an index (one doc listing the set with one-liners)? Maintain it. Is naming consistent? Are point-in-time docs misfiled in the evergreen folder? Is the ordering a fossil (sequential numbers implying a frozen, complete scope)?

Per-doc cleanliness is not a healthy set. This phase catches exactly the class Phases 1–3 cannot.

Last-reviewed marker

On evergreen docs only, idempotently maintain one block at EOF (reuse a trailing --- rule if present; find-and-replace on the sentinel, never stack duplicates):

---
<!-- doc-audit:last-reviewed -->
_Last reviewed: 2026-06-07 · commit `abc1234` · verified against code (2 claims deferred to review)._

Date + SHA are provenance only (HEAD at review time), not a scoping key. Stamp only docs you actually verified. Don't stamp point-in-time docs — they're frozen records, and re-stamping them is churn.

Red flags — STOP

• "I'll bring the docs up to date" across a whole spec/plan set without classifying first. → Classify and confirm first.
• About to renumber a file:line citation. → Convert to file:symbol via interview.
• About to rewrite an example's structure to the new schema. → Interview, not auto-fix.
• Pressured to "be decisive / don't confirm", about to classify docs yourself and rewrite their bodies. → The classification confirm is the one gate you never skip. Surface it; a one-line yes costs seconds.
• Concluding a feature was "removed" / deleting its docs from a grep miss. → Check git history first.
• "Close enough, I'll just fix the count." → Counts go to the human.
• Editing a file that says "generated" / "do not edit". → Skip it.
• Marking a claim "matches" without a citation. → Cite it or don't claim it.
• About to finish without a second adversarial pass over your own edits. → Run the competing verify pass (Phase 3) first; fix what it finds.
• Marking a claim "matches" because the symbol exists, without checking the code does what's claimed about it. → Verify the verb, not the noun.
• Audited each doc, never looked at the set (duplication, gaps, contradictions, index). → Per-doc passes are blind to set-level defects; run the corpus review (Phase 4).

Rationalizations

Common mistakes

• Editing before classifying.
• Spawning one subagent per point-in-time doc — batch them; they aren't claim-verified.
• An unbounded first-run interview backlog — prioritize, defer the tail to a report.
• Auto-committing the audit — leave it uncommitted for human review.
• Declaring done without adversarially verifying your own edits — your fixes are a prime hiding spot for confident-but-wrong claims (Phase 3).
• Treating per-doc verification as the whole job — duplication, gaps, cross-doc contradictions, and a missing index are set-level defects invisible to a per-doc pass (Phase 4).
• Verifying the noun, not the verb — confirming a symbol exists instead of confirming the code does what the doc claims about it.