Corpus Review
A single reader cannot hold a few hundred files in working memory, and a single
pass over them misses what only a second angle catches. The pattern that scales:
partition the corpus, review each partition blind and in parallel, verify a
finding before acting on it, fix without collisions, and re-audit until the
findings dry up.
This is a flexible skill, and it ships no engine — it orchestrates the audit
on whatever parallel-agent or workflow primitive the harness provides, and
degrades to plain sequential review when none is available. What stays firm is
the shape of the loop.
The loop
- Partition the corpus into coherent groups — by directory, subsystem, or
artifact kind — each small enough for one reviewer to read closely together
with the real source it should agree with (a doc against the
src/ it
describes, a config against what consumes it).
- Review each partition blind and in parallel. One reviewer per group, each
unaware of the others, so findings are not anchored by a shared first read.
The reviewer reads the artifact and its ground truth, never the artifact
alone.
- Adversarially verify every high-severity finding before acting. A finding
is a claim until a second, skeptical agent confirms it against the code.
Unverified findings are where a fan-out wastes the most effort — a confident
reviewer can be confidently wrong.
- Fix in disjoint file partitions. Parallel editors that never touch the
same file need no worktrees and cannot collide; partition the confirmed
findings by file before fixing.
- Re-audit with fresh eyes, and stop on convergence. Each round's
high-severity yield should fall; halt when it drops below a threshold instead
of running an open-ended next round on noise. A falling curve (for example
157 → 49 → 9 findings) is the stop signal.
Execute the artifact, don't only read it
For a corpus with executable parts — hooks, scripts, example programs, a CLI —
the highest-assurance round runs the artifact with synthetic inputs rather than
reading it. Executing every hook with a sample payload and every example script
catches what reading cannot: a fix-introduced regression, a reference to a
symbol that no longer exists, an example that no longer runs. Reading-only
rounds routinely pass these.
Concurrency and resume
- Cap peak concurrency. A large fan-out hits provider rate limits; a ceiling
on simultaneous reviewers trades a little wall-clock for not having to re-run a
rate-limited slice.
- Make slices resumable. Record which partitions converged, so a failed or
rate-limited slice re-runs on its own without redoing the rest.
Common failure modes
| Pattern |
What it costs |
| Reading the artifact without its source |
Misses doc-vs-code drift; confirms the doc against itself. |
| Acting on unverified findings |
Fixes a confidently-wrong claim; adds churn. |
| One reviewer for too large a group |
Attention thins; the tail of the group gets a shallow read. |
| Reading-only on an executable corpus |
Runtime breakage and fix-introduced regressions survive. |
| Parallel editors on shared files |
Collisions, lost edits, or a forced serialization. |
| No convergence rule |
An open-ended Nth round reviews mostly noise. |
Boundaries
- review-panel convenes a fresh-eyes panel on a single design, spec, or
artifact the user has iterated on. A corpus is the other axis — many artifacts,
one angle each, fanned out — so it is this skill's job, not a panel's.
- code-review / a diff review reads one change's diff for bugs and
regressions. A corpus review reads whole files against their ground truth,
not a diff.
- evaluate-skill measures whether a skill triggers and performs — behavioral
measurement of one tool, not a content audit of a file set.
1---2name: corpus-review3description: Audit a large file corpus — dozens to hundreds of docs, configs, or mixed code-plus-docs-plus-tests — by fanning out blind reviewers over partitions, adversarially verifying every high-severity finding before acting on it, fixing in disjoint file partitions, and re-auditing with fresh eyes until the findings converge. Use when reviewing or auditing a whole repo's documentation, a release's doc set, an entire plugin or package, or any file set too large for one reader to hold at once; on asks like "review all the docs before we push", "audit the whole repo for X", "blind review across these N files", "do a pre-push review of everything", or "check the docs still match the code across the project". It orchestrates the audit on the harness's parallel subagent and workflow primitives and ships no engine of its own. Not for a fresh-eyes panel on a single design, spec, or artifact (that is review-panel), not for reviewing one change's diff for bugs and regressions (that is a diff review / code-review), and not for 4---56# Corpus Review78A single reader cannot hold a few hundred files in working memory, and a single9pass over them misses what only a second angle catches. The pattern that scales:10partition the corpus, review each partition blind and in parallel, verify a11finding before acting on it, fix without collisions, and re-audit until the12findings dry up.1314This is a **flexible** skill, and it ships no engine — it orchestrates the audit15on whatever parallel-agent or workflow primitive the harness provides, and16degrades to plain sequential review when none is available. What stays firm is17the shape of the loop.1819## The loop20211. **Partition the corpus into coherent groups** — by directory, subsystem, or22 artifact kind — each small enough for one reviewer to read closely together23 with the real source it should agree with (a doc against the `src/` it24 describes, a config against what consumes it).252. **Review each partition blind and in parallel.** One reviewer per group, each26 unaware of the others, so findings are not anchored by a shared first read.27 The reviewer reads the artifact *and* its ground truth, never the artifact28 alone.293. **Adversarially verify every high-severity finding before acting.** A finding30 is a claim until a second, skeptical agent confirms it against the code.31 Unverified findings are where a fan-out wastes the most effort — a confident32 reviewer can be confidently wrong.334. **Fix in disjoint file partitions.** Parallel editors that never touch the34 same file need no worktrees and cannot collide; partition the confirmed35 findings by file before fixing.365. **Re-audit with fresh eyes, and stop on convergence.** Each round's37 high-severity yield should fall; halt when it drops below a threshold instead38 of running an open-ended next round on noise. A falling curve (for example39 157 → 49 → 9 findings) is the stop signal.4041## Execute the artifact, don't only read it4243For a corpus with executable parts — hooks, scripts, example programs, a CLI —44the highest-assurance round runs the artifact with synthetic inputs rather than45reading it. Executing every hook with a sample payload and every example script46catches what reading cannot: a fix-introduced regression, a reference to a47symbol that no longer exists, an example that no longer runs. Reading-only48rounds routinely pass these.4950## Concurrency and resume5152- **Cap peak concurrency.** A large fan-out hits provider rate limits; a ceiling53 on simultaneous reviewers trades a little wall-clock for not having to re-run a54 rate-limited slice.55- **Make slices resumable.** Record which partitions converged, so a failed or56 rate-limited slice re-runs on its own without redoing the rest.5758## Common failure modes5960| Pattern | What it costs |61|---------|---------------|62| Reading the artifact without its source | Misses doc-vs-code drift; confirms the doc against itself. |63| Acting on unverified findings | Fixes a confidently-wrong claim; adds churn. |64| One reviewer for too large a group | Attention thins; the tail of the group gets a shallow read. |65| Reading-only on an executable corpus | Runtime breakage and fix-introduced regressions survive. |66| Parallel editors on shared files | Collisions, lost edits, or a forced serialization. |67| No convergence rule | An open-ended Nth round reviews mostly noise. |6869## Boundaries7071- **review-panel** convenes a fresh-eyes panel on a *single* design, spec, or72 artifact the user has iterated on. A corpus is the other axis — many artifacts,73 one angle each, fanned out — so it is this skill's job, not a panel's.74- **code-review / a diff review** reads one change's diff for bugs and75 regressions. A corpus review reads whole files against their ground truth,76 not a diff.77- **evaluate-skill** measures whether a skill triggers and performs — behavioral78 measurement of one tool, not a content audit of a file set.
grimaldost