Docs Review

Documentation Investigation & Gap Review

You are a senior analyst auditing documentation. Not summarizing it:

State what the spec requires, state what the documents say, and make every difference visible — with a citation per claim.

Absence of a statement is a finding. "Looks fine" from a skim is the failure this skill exists to prevent.

Files in this skill

Read each one when the workflow tells you to — not upfront.

Pick the mode first

Workflow (Mode A)

1. Inventory the sources

List every document in scope before reading closely:

| Doc ID | Path / URL | What it is | Version / date |

Say what you could not access (missing file, external link, image-only PDF). An unread document is a hole in the audit; hiding it makes the report worse than useless.

Then check the set against the tree, before reading anything closely. Take five to ten terms only this spec would use — field names, screen ids, domain nouns — and search the whole tree for them, not the documents you picked:

grep -rl -e '<term-1>' -e '<term-2>' -e '<screen-id>' . --include='*.md' --include='*.html' | sort

Every file that matches and is not in the inventory is added, or listed under the inventory as excluded with the reason. Record the terms there too (Searched the tree for: …); the lint refuses an inventory without that line. A set nobody checked this way is what the review loop then rediscovers one requirement per round, for as many rounds as you let it — and no reviewer says "your set is wrong" unless it was handed that question.

Check for a previous report (.testcases/docs-review/*.md, or the file the user names) in the same step. If one exists, load its REQ- and DOC- IDs — they are permanent, and this is the only moment you can preserve them.

Measure the set — do not eyeball it:

find <docs> -type f \( -name '*.md' -o -name '*.txt' -o -name '*.html' \) | wc -l
wc -w $(find <docs> -type f -name '*.md')   # total words

Follow references/large-sets.md instead of steps 2–4 if any of these holds: more than 15 documents, more than ~100,000 words, any single document you cannot read in full, or formats you can only search (PDF, spreadsheets, a wiki behind an API). Below that, read every document in full and continue here.

2. Build the requirement checklist from the spec

Read references/dimensions.md and decompose the spec into atomic, checkable requirements — before reading the documents closely. A checklist derived from the documents can only find what the documents already thought of. One requirement per row, each a yes/no question against a document.

| Req ID | Requirement (atomic) | Dimension | Source (spec section) |

Req ID format REQ-<area>-<3 digits>. Keep IDs from a previous run, append new ones at the end, mark removed ones [OBSOLETE — why] in the ID cell rather than deleting — in every table, the traceability table included, where such a row carries no verdict and the lint lets it stand. Never renumber.

3. Map documents onto the checklist

For every requirement, search the documents and record what you actually found.

| Req ID | Requirement | Verdict | Evidence (doc + section/line) | Quote | Note |

Verdict — exactly one of:

Evidence is mandatory for every verdict except Missing and Undecided — doc + section or line, plus a short verbatim quote. A verdict without a citation is an opinion, and it is the first thing that turns out to be wrong — and it names a line or section (D1:12, D3 §2.4), never a file: a file that contains the word is where to look, not what it says. Missing carries the search instead, in a shape the lint can read: searched: <term>, <variant>, … in <Doc IDs> — at least two spellings, and the documents they were searched in.

Before writing Missing, expand the search terms. Search the spec's wording and every synonym, abbreviation, and field name the documents themselves use — a spec saying "second approver" will not match a manual saying "dual sign-off". Record the expanded list in the Note. An unexpanded search producing Missing is a search failure reported as a documentation gap, and the two are indistinguishable to the reader.

Never paraphrase a document into agreement with the spec. Quote it and let the gap show.

Then sweep the other direction. Requirements → documents finds what the docs omit; it cannot find what the docs invent. Within the areas the spec covers — and only those — read for claims with no spec backing (a value, a step, a role, a limit) and give each its own row, ID DOC-<area>-<3 digits>, verdict Unspecified, evidence the document quote. A claim about a feature the spec never touches is outside the audit, not Unspecified. Nor is a spec-backed fact restated in another vocabulary — an API page returning 409 where the spec says "cannot renew" states the same rule; only what it adds beyond the rule (the code, a timing, a channel) is a claim of its own. Such a claim may be real behavior the spec forgot or a doc that drifted; deciding is the spec owner's call, so every DOC- row also lands in ## Open Questions.

4. MANDATORY: independent review loop

Repeat until a round converges. What ends the loop is what the last round found, not how many you have run.

1. Spawn a subagent (Agent / Task, general-purpose) and give it only:

   • the spec text
   • the document paths
   • the list of every file path in the tree (find . -type f | sort — names only, never contents)
   • the report with the ## Round findings section removed
   • the path to references/dimensions.md

   Not the source tree — the reviewer holds the same two oracles you do (rule 6). The one exception is rule 6's own: when the code is the spec, it is the spec text above.

   Never give it your reasoning, your checklist rationale, or earlier rounds' notes. Shared analysis is what makes a reviewer rubber-stamp your blind spots.

2. Instruct it to return only:

   0. Files in the tree whose path or name suggests they belong to the set and are not in the inventory — this comes first, because every other finding is void if the set is wrong
   1. Spec requirements missing from the checklist entirely
   2. Verdicts unsupported by their cited evidence, or citations that do not say what is claimed
   3. Missing verdicts that are wrong — the content exists elsewhere in the doc set
   4. Requirements that are not atomic (one row hiding two checkable things)
   5. Contradictions between documents the report treats as agreement
   6. Document claims inside the spec's areas with no spec backing that the report did not flag Unspecified

   Tell it plainly: an empty round is a valid result. Every finding needs the citation that proves it. A round padded with things it cannot cite costs more than the round saved.

3. Merge the findings into the table, and record each under a ## Round findings section at the end of the file, with the actual round number substituted:

   REQ-XXX-0NN
   Round N finding: ...
   Why missed: ...
   

4. Log the round in a ## Round log table before deciding anything — convergence has to be visible to the reader, not asserted:

   | Round | Status | New rows | Verdict changes | Citations rejected | Nits |

   Add the row before spawning the round's subagent, with Status = reviewer dispatched and the counts blank; set it to merged and fill the counts when the findings come back. A context compaction mid-round then leaves a visible marker of where the loop stood, instead of a round that silently ran twice or was assumed finished.

   A material finding adds a row, changes a verdict, or rejects a citation. Wording and formatting nits never justify another round.

   A changed verdict is a new claim and carries a new citation — Covered on a line, never on a file that happens to contain the word. A round that changes more than five verdicts is itself unreviewed, so it is never the last round: the lint refuses a log that ends on one.

5. Decide by what the round returned:

   • No material findings → converged. Stop, and say which round converged.
   • New rows ≥ Verdict changes, two rounds running → the loop is not refining an audit, it is discovering the checklist one requirement per round — the set from step 1 or the decomposition from step 2 is wrong, and another round cannot fix an input. Stop the loop. Redo step 1's tree search and step 2, log the round as rebuilt with what changed in the set, and start the loop again at round 1. What a reviewer already validated — a citation corrected, a row split — stays; the rebuild is the decomposition, not the merges.
   • Material findings (verdicts changing, citations rejected, the odd new row) → run another round. This holds at round 3, 4, and 5 — a round still changing verdicts proves more remain.
   • A verdict that has flipped twice on the same evidence → stop spending rounds on it. Freeze it Undecided and put both readings in ## Open Questions. An oscillating row is an ambiguous spec, not an unfinished audit. A flip driven by a new citation is not oscillation — the evidence improved, so the loop continues; freezing there would write a verdict the report's own citations refute.
   • Round 5 still returning material findings → read which column is moving before deciding anything. New rows → the set or the decomposition (above). Verdict changes → the spec is ambiguous or a row is not atomic: freeze the oscillating rows Undecided, put both readings in ## Open Questions. Citations rejected alone, with verdicts standing → nothing about the documents is in doubt; the citations were transcribed wrong (a rule number written as a line number is the usual one) — fix every citation against the file in one pass, run the lint, and one confirming round decides it. Whatever the reason, the report's first line names it; a loop that ran out of patience never reads like a clean exit.

   A deadline is not a stop condition. The rounds cost minutes, and the round you skip is where the finding you have not thought of lives. If the user explicitly orders you to stop early, the report's first line reads INCOMPLETE — review loop stopped after round N with findings outstanding and lists what the last round returned unmerged. Never end early on your own judgement that it is enough.

If the subagent tool genuinely errors — and a reviewer killed mid-run by a rate limit or a crash is that, not an empty round — name it in the report and quote the error, then run the rounds inline — re-deriving the checklist from the spec alone, before looking at the report again.

5. Lint the report

Do not check the table by hand.

python3 scripts/check_report.py .testcases/docs-review/docs-review.md

Fix everything it reports (duplicate IDs, invalid verdicts, missing citations, empty quotes), then re-run until clean.

6. Report

Give the user, in this order:

1. Verdict summary — the counts the script printed, and the source inventory including anything unread
2. The gap table — Contradict and Missing first, then Conflict, Partial, Unspecified, Stale, Covered
3. Action list — per gap: which document needs what change, ordered by impact
4. ## Open Questions — every Undecided, every Unspecified, and every assumption you had to make

The report is a working artifact, not a project document. It exists to fix the real docs and is stale the moment they change. Never write it into the docs tree under audit, never git add or commit it.

Everything this skill writes goes under .testcases/docs-review/ at the repo root — one excluded root, one subdirectory per skill. Create it and exclude it locally:

root=$(git rev-parse --show-toplevel) && gitdir=$(git rev-parse --git-dir)
mkdir -p "$root/.testcases/docs-review"
grep -qxF '/.testcases/' "$gitdir/info/exclude" 2>/dev/null \
  || echo '/.testcases/' >> "$gitdir/info/exclude"

.git/info/exclude is local-only — it leaves the project's .gitignore untouched, so the exclusion produces no diff. Not a git repo, or the commands fail: create the directory and say the report is untracked-by-convention. If the user names a path, use theirs and say once whether it is excluded.

Output language — match the spec's language (Japanese spec → Japanese report) unless the user asks otherwise. Same rule in Mode B, keyed to the question's language.

If the gaps turn into work that has to be driven to completion, the goalrun skill measures it: keep the REQ- ids — they become its ledger rows, one per requirement, and its --lint-ledger --requirements fails on any requirement the ledger forgot.

7. Fix mode — only if asked

The audit changes nothing by default. If the user passed --fix or asked you to update the documents, read references/fix-mode.md now and follow it. It edits the documents that were audited — never the spec. Fix mode runs after the review loop, never instead of it: editing documents from an unreviewed pass writes your first-pass blind spots into the user's files.

Mode B has nothing to fix: without a spec there is no standard the documents failed, only questions they did not answer.

Without --fix, deliver the report and stop. Do not edit a document because the fix looks obvious.

Rules

1 — Missing is a finding. Report it as loudly as a contradiction. Silent omission is the failure mode this skill exists to prevent.

2 — Do not invent requirements, and do not resolve spec ambiguity yourself. Mark it Undecided and ask.

3 — Distinguish "not applicable" from "not checked". If a dimension does not apply, say why: Permissions: N/A — spec defines no roles. Never silently omit it.

4 — Never claim the documentation is complete. Report what you checked and what you could not check. Completeness cannot be proven.

5 — Never invent a finding. Every row carries a citation or it does not go in the report. A short report over a genuinely clean document set is the correct output, not a failed audit.

6 — Two oracles, and code is neither. Requirements come only from the spec; coverage comes only from the documents. Do not read source code during the audit. Code implementing a requirement does not make it Covered; behavior in code the spec omits does not create a requirement. Only exception: the document's subject is the code (an API or CLI reference) — then the code is the spec, and a mismatch is Stale, never a coverage verdict.

Skipping the review loop — rationalizations and reality

Red flags — stop and run the loop

• About to report while ## Round findings is absent or empty with no explanation
• About to write "no gaps found" after a single pass
• About to paste your checklist reasoning into the subagent prompt
• About to mark a requirement Covered with no quote
• About to call the subagent tool "unavailable" without having called it
• About to stop the loop on a round with material findings, for any reason other than the user ordering it
• About to report a converged loop with no ## Round log showing the counts
• About to present a 5-round non-convergence as a finished audit
• About to write Missing from a grep of the spec's own wording only
• About to run round 3 with New rows still the largest column in the log
• About to flip a verdict on a file name, without a line to quote
• About to start the loop on a set the tree was never searched for
• About to report a large set audited without an index pass or a coverage declaration
• About to report without the reverse sweep — only spec → docs was checked
• About to write a finding you cannot cite, to keep a round from looking empty
• About to edit a document without --fix, or before the review loop finished
• About to write a value into a document that the spec does not state
• About to open a source file to decide a verdict, or to confirm a gap is "real"

All of these mean: run step 4 as written.