docs-corpus-audit -- whole-corpus regrounding pass
The docs corpus drifts silently between releases. docs-sync catches
drift introduced by individual PRs at PR-open time. This skill catches
the accumulated drift that slips past per-PR review -- stale flag
names, dead nav links from past IA reshuffles, deprecation banners
that outlived their version targets, factual claims whose source-side
truth has moved.
The pattern is A1 PANEL + WAVE EXECUTION + S7 DETERMINISTIC TOOL
BRIDGE + A8 ALIGNMENT LOOP + A9 SUPERVISED EXECUTION. The corpus is
split into disjoint page scopes; one verifier subagent owns each
scope; agents extract factual claims, S7-verify against source, apply
surgical fixes inline. The orchestrator then runs an alignment-loop
pass to re-verify that applied edits actually ground out true.
This skill is ADVISORY but ACTIONABLE: agents apply edits inline on a
working branch. The orchestrator is the sole writer to git -- stages,
commits, pushes. Maintainer reviews the resulting PR.
Sibling contract with docs-sync
These two skills share substrate. Be explicit:
| Shared resource |
Owner |
Both use |
.apm/docs-index.yml (corpus map) |
docs-sync |
yes |
| doc-writer persona |
shared |
yes (per-page edits) |
| python-architect persona |
shared |
yes (S7 verification) |
| editorial-owner persona |
shared |
optional (voice pass at scale) |
| cdo persona |
shared |
yes (final synthesis) |
assets/panelist-return-schema.json |
docs-sync (mirrored) |
yes |
Trigger boundary (avoid DISPATCH COLLISION):
docs-sync triggers on a PR event ("PR opened/synchronized",
source-diff-driven).
docs-corpus-audit triggers on a maintainer ask for a
WHOLE-CORPUS pass ("audit the corpus", "reground", "pre-release
sweep") -- no PR required, no diff required, the whole corpus
is the input.
If a maintainer asks "review this PR's doc impact", route to
docs-sync. If they ask "audit all our docs" or "the docs feel
stale everywhere", route here.
Architecture invariants
- Wave-batched, not flat. Pages are partitioned into 6-8 disjoint
scopes; each scope is one verifier subagent. Cost scales with
wave size, not corpus size. A wave of 6 agents on ~10 pages each
is the canonical shape.
- Disjoint page ownership. Each subagent has EDIT AUTHORITY on
its scope only. No two agents touch the same file -- guarantees
no merge conflicts during fan-in.
- S7 verification is mandatory. Every factual claim is verified
against deterministic source:
uv run apm <verb> --help for CLI,
grep -n src/apm_cli/ for symbols, python -c "import ..." for
module shape, file-existence checks for nav links. Never assert
from LLM recall.
- Surgical edits only. 1-3 line patches per drift, preserving
voice. Restructuring is deferred to the orchestrator post-pass,
never auto-applied by per-scope agents.
- Single-writer interlock for git. Subagents NEVER run
git commit, git push, or gh pr <write>. Orchestrator
commits per wave; pushes once per session.
- Alignment loop (A8). After waves return, orchestrator
re-greps the corpus for the patterns the agents claimed to fix.
Any residue triggers a targeted re-dispatch (max 2 redrafts) or
is escalated to maintainer.
Roster (composition, not invention)
Reuse docs-sync's personas. Do NOT invent a one-off "grounding-
verifier" role; that's R3 EXTRACT in reverse.
| Role |
Persona |
Always active? |
| Per-scope verifier+editor |
python-architect (S7) and doc-writer (edits), bundled into one subagent prompt per scope |
Yes -- one per page scope, parallel fan-out |
| Cross-corpus post-pass |
orchestrator (deterministic greps via scripts/scan-cross-corpus-drift.sh) |
Yes -- once after waves return |
| Alignment-loop checker |
orchestrator (deterministic re-grep + targeted re-dispatch) |
Yes -- once after post-pass |
| Voice pass (optional) |
editorial-owner |
Only when >20 edits to keep tone coherent |
| Final synthesis |
cdo |
Once, for the PR summary comment |
The per-scope subagent prompt that composes python-architect +
doc-writer is in assets/subagent-prompt-template.md -- the
orchestrator substitutes scope + working dir + branch and dispatches
via the task tool.
Process
1. PROBE (A9 SUPERVISED EXECUTION)
- Check working tree: docs/src/content/docs/ exists?
- Check working tree: packages/apm-guide/.apm/skills/apm-usage/
exists? (Rule-4 backfill target. If missing, the audit cannot
close Rule 4; ask maintainer before continuing.)
- Check `.apm/docs-index.yml` reachable.
- Verify on a working branch (not main).
2. RISK-TRIAGE (orchestrator, ~1 LLM call)
- Read .apm/docs-index.yml only (NOT the corpus body).
- Bucket pages by drift risk: HIGH (CLI ref, schemas, consumer
flows), MEDIUM (producer, enterprise policy), LOW (concepts,
contributing, troubleshooting, integrations).
- Decide wave order: HIGH first, MEDIUM next, LOW last.
3. WAVE-PLANNER (orchestrator, deterministic)
- Partition pages into 6-8 disjoint scopes per wave.
- Each agent gets ~9 pages, mixed surface types.
4. WAVE EXECUTION (parallel, one subagent per scope)
- Orchestrator dispatches one task per scope using the prompt
template in assets/subagent-prompt-template.md.
- Subagents read pages, extract claims, S7-verify, apply
surgical edits, return JSON per the docs-sync panelist
schema (mirrored at assets/panelist-return-schema.json).
- Validate every return against the schema; reject malformed
JSON.
5. CROSS-CORPUS POST-PASS (orchestrator, deterministic)
- Run scripts/scan-cross-corpus-drift.sh to grep for patterns
a per-scope agent cannot see (IA-reshuffle dead links, stale
deprecation version targets, phantom flag references).
- Patch residue inline.
6. ALIGNMENT LOOP (orchestrator, deterministic)
- Re-run scripts/scan-cross-corpus-drift.sh.
- Re-grep for claims the agents marked DRIFTED-FIXED.
- If residue: targeted re-dispatch to the owning agent
(bounded: max 2 redrafts per wave).
7. COMMIT + PUSH (orchestrator, single writer)
- One commit per wave; structured message naming closed items.
- Push to working branch.
8. PR + SUMMARY COMMENT (orchestrator)
- If no PR exists: open one with the [pr-description-skill]
(../pr-description-skill/SKILL.md).
- Post per-wave summary comment: pages audited, drift caught,
fixes applied, items deferred, alignment-loop residue.
Bundled assets
assets/subagent-prompt-template.md -- the per-scope prompt the
orchestrator substitutes and dispatches. Composes python-architect
(S7) + doc-writer (surgical edit). Loaded once per scope.
assets/panelist-return-schema.json -- subagent return schema,
mirrored from docs-sync. Loaded once at wave start; validated
against every return.
scripts/scan-cross-corpus-drift.sh -- deterministic grep sweep
for cross-corpus patterns (IA dead links, stale deprecation
targets, phantom flags). Non-interactive; emits structured
matches on stdout, diagnostics on stderr. Run --help for
pattern list. Update this script after each major IA reshuffle.
Cost model
| Wave size |
Pages |
Subagents |
LLM dispatches |
Wall time |
| Small |
~30 |
4 |
~5 |
~3 min |
| Medium (default) |
~55 |
6 |
~7 |
~5 min |
| Large |
~110 (full corpus) |
12 (two medium waves) |
~14 |
~10 min |
Compared to docs-sync (15-call flat ceiling), this skill scales as
O(waves), not O(claims), because per-agent work fits in one context
window. S7 verification dominates wall-time, not LLM cost.
Boundary (what this skill does NOT do)
- Per-PR doc-impact review -- use
docs-sync.
- Single-page typo or copy edit -- direct edit is faster.
- Writing docs for a brand-new feature -- use
docs-impact-architect
and doc-writer directly.
- Auto-merging or pushing without maintainer review.
- Reviewing code quality, security, or test coverage (out of scope).
Evals
See evals/:
evals/content-evals.json -- 3 corpus snapshots with seeded drift
(stale CLI flag, dead nav link, expired deprecation target);
expected behavior is that the skill catches all three and applies
surgical fixes that ground out true on re-verification.
evals/trigger-evals.json -- 10 should-trigger + 10 should-NOT-
trigger queries, 60/40 train/val. The val split is the ship gate
(>=0.5 should-trigger AND <0.5 should-not-trigger).
evals/README.md -- how to run.
Provenance
This skill was extracted from a real session that audited the
microsoft/apm corpus across 3 waves (PR #1511, 2026-05-27):
112/112 pages audited, 49 surgical fixes, ~25 LLM dispatches,
~30 min wall-time. The session design artifact (genesis hand-off
packet) lives in session state, not in this bundle (maintainer-
scope, not runtime-loaded).
1---2name: docs-corpus-audit3description: >-4 Use this skill to run a holistic regrounding pass on the entire5 microsoft/apm documentation corpus against current source code,6 page-by-page, and emit surgical fixes for stale claims. Activate7 when the maintainer wants a WHOLE-CORPUS audit (not per-PR review)8 -- typical triggers include "audit the docs", "reground the9 corpus", "check every page against code", "pre-release docs10 sweep", "the docs have drifted everywhere", or "we just reshaped11 the TOC, find dead links". Wave-batched and S7-verified; scales12 to the full ~112-page corpus in ~10 minutes wall-time. This is a13 SIBLING to docs-sync, not a replacement: docs-sync is per-PR14 (triggered by a diff); this skill is per-corpus (triggered by a15 maintainer ask). They share agent personas, schemas, and the16 docs index, but their triggers MUST NOT collide. Does NOT17 auto-merge, does NOT push without maintainer review, and does18 NOT replace per-PR drift detection.19metadata:20 source: https://skillmd.com/skills/microsoft/docs-corpus-audit21---2223# docs-corpus-audit -- whole-corpus regrounding pass2425The docs corpus drifts silently between releases. `docs-sync` catches26drift introduced by individual PRs at PR-open time. This skill catches27the **accumulated** drift that slips past per-PR review -- stale flag28names, dead nav links from past IA reshuffles, deprecation banners29that outlived their version targets, factual claims whose source-side30truth has moved.3132The pattern is **A1 PANEL + WAVE EXECUTION + S7 DETERMINISTIC TOOL33BRIDGE + A8 ALIGNMENT LOOP + A9 SUPERVISED EXECUTION**. The corpus is34split into disjoint page scopes; one verifier subagent owns each35scope; agents extract factual claims, S7-verify against source, apply36surgical fixes inline. The orchestrator then runs an alignment-loop37pass to re-verify that applied edits actually ground out true.3839This skill is ADVISORY but ACTIONABLE: agents apply edits inline on a40working branch. The orchestrator is the sole writer to git -- stages,41commits, pushes. Maintainer reviews the resulting PR.4243## Sibling contract with docs-sync4445These two skills share substrate. Be explicit:4647| Shared resource | Owner | Both use |48|---|---|---|49| `.apm/docs-index.yml` (corpus map) | docs-sync | yes |50| [doc-writer](../../agents/doc-writer.agent.md) persona | shared | yes (per-page edits) |51| [python-architect](../../agents/python-architect.agent.md) persona | shared | yes (S7 verification) |52| [editorial-owner](../../agents/editorial-owner.agent.md) persona | shared | optional (voice pass at scale) |53| [cdo](../../agents/cdo.agent.md) persona | shared | yes (final synthesis) |54| `assets/panelist-return-schema.json` | docs-sync (mirrored) | yes |5556**Trigger boundary (avoid DISPATCH COLLISION):**5758- `docs-sync` triggers on a PR event ("PR opened/synchronized",59 source-diff-driven).60- `docs-corpus-audit` triggers on a maintainer ask for a61 WHOLE-CORPUS pass ("audit the corpus", "reground", "pre-release62 sweep") -- no PR required, no diff required, the whole corpus63 is the input.6465If a maintainer asks "review this PR's doc impact", route to66`docs-sync`. If they ask "audit all our docs" or "the docs feel67stale everywhere", route here.6869## Architecture invariants7071- **Wave-batched, not flat.** Pages are partitioned into 6-8 disjoint72 scopes; each scope is one verifier subagent. Cost scales with73 wave size, not corpus size. A wave of 6 agents on ~10 pages each74 is the canonical shape.75- **Disjoint page ownership.** Each subagent has EDIT AUTHORITY on76 its scope only. No two agents touch the same file -- guarantees77 no merge conflicts during fan-in.78- **S7 verification is mandatory.** Every factual claim is verified79 against deterministic source: `uv run apm <verb> --help` for CLI,80 `grep -n src/apm_cli/` for symbols, `python -c "import ..."` for81 module shape, file-existence checks for nav links. Never assert82 from LLM recall.83- **Surgical edits only.** 1-3 line patches per drift, preserving84 voice. Restructuring is deferred to the orchestrator post-pass,85 never auto-applied by per-scope agents.86- **Single-writer interlock for git.** Subagents NEVER run87 `git commit`, `git push`, or `gh pr <write>`. Orchestrator88 commits per wave; pushes once per session.89- **Alignment loop (A8).** After waves return, orchestrator90 re-greps the corpus for the patterns the agents claimed to fix.91 Any residue triggers a targeted re-dispatch (max 2 redrafts) or92 is escalated to maintainer.9394## Roster (composition, not invention)9596Reuse docs-sync's personas. Do NOT invent a one-off "grounding-97verifier" role; that's R3 EXTRACT in reverse.9899| Role | Persona | Always active? |100|---|---|---|101| Per-scope verifier+editor | [python-architect](../../agents/python-architect.agent.md) (S7) and [doc-writer](../../agents/doc-writer.agent.md) (edits), bundled into one subagent prompt per scope | Yes -- one per page scope, parallel fan-out |102| Cross-corpus post-pass | orchestrator (deterministic greps via `scripts/scan-cross-corpus-drift.sh`) | Yes -- once after waves return |103| Alignment-loop checker | orchestrator (deterministic re-grep + targeted re-dispatch) | Yes -- once after post-pass |104| Voice pass (optional) | [editorial-owner](../../agents/editorial-owner.agent.md) | Only when >20 edits to keep tone coherent |105| Final synthesis | [cdo](../../agents/cdo.agent.md) | Once, for the PR summary comment |106107The per-scope subagent prompt that composes `python-architect` +108`doc-writer` is in `assets/subagent-prompt-template.md` -- the109orchestrator substitutes scope + working dir + branch and dispatches110via the task tool.111112## Process113114```1151. PROBE (A9 SUPERVISED EXECUTION)116 - Check working tree: docs/src/content/docs/ exists?117 - Check working tree: packages/apm-guide/.apm/skills/apm-usage/118 exists? (Rule-4 backfill target. If missing, the audit cannot119 close Rule 4; ask maintainer before continuing.)120 - Check `.apm/docs-index.yml` reachable.121 - Verify on a working branch (not main).1221232. RISK-TRIAGE (orchestrator, ~1 LLM call)124 - Read .apm/docs-index.yml only (NOT the corpus body).125 - Bucket pages by drift risk: HIGH (CLI ref, schemas, consumer126 flows), MEDIUM (producer, enterprise policy), LOW (concepts,127 contributing, troubleshooting, integrations).128 - Decide wave order: HIGH first, MEDIUM next, LOW last.1291303. WAVE-PLANNER (orchestrator, deterministic)131 - Partition pages into 6-8 disjoint scopes per wave.132 - Each agent gets ~9 pages, mixed surface types.1331344. WAVE EXECUTION (parallel, one subagent per scope)135 - Orchestrator dispatches one task per scope using the prompt136 template in assets/subagent-prompt-template.md.137 - Subagents read pages, extract claims, S7-verify, apply138 surgical edits, return JSON per the docs-sync panelist139 schema (mirrored at assets/panelist-return-schema.json).140 - Validate every return against the schema; reject malformed141 JSON.1421435. CROSS-CORPUS POST-PASS (orchestrator, deterministic)144 - Run scripts/scan-cross-corpus-drift.sh to grep for patterns145 a per-scope agent cannot see (IA-reshuffle dead links, stale146 deprecation version targets, phantom flag references).147 - Patch residue inline.1481496. ALIGNMENT LOOP (orchestrator, deterministic)150 - Re-run scripts/scan-cross-corpus-drift.sh.151 - Re-grep for claims the agents marked DRIFTED-FIXED.152 - If residue: targeted re-dispatch to the owning agent153 (bounded: max 2 redrafts per wave).1541557. COMMIT + PUSH (orchestrator, single writer)156 - One commit per wave; structured message naming closed items.157 - Push to working branch.1581598. PR + SUMMARY COMMENT (orchestrator)160 - If no PR exists: open one with the [pr-description-skill]161 (../pr-description-skill/SKILL.md).162 - Post per-wave summary comment: pages audited, drift caught,163 fixes applied, items deferred, alignment-loop residue.164```165166## Bundled assets167168- `assets/subagent-prompt-template.md` -- the per-scope prompt the169 orchestrator substitutes and dispatches. Composes python-architect170 (S7) + doc-writer (surgical edit). Loaded once per scope.171- `assets/panelist-return-schema.json` -- subagent return schema,172 mirrored from docs-sync. Loaded once at wave start; validated173 against every return.174- `scripts/scan-cross-corpus-drift.sh` -- deterministic grep sweep175 for cross-corpus patterns (IA dead links, stale deprecation176 targets, phantom flags). Non-interactive; emits structured177 matches on stdout, diagnostics on stderr. Run `--help` for178 pattern list. Update this script after each major IA reshuffle.179180## Cost model181182| Wave size | Pages | Subagents | LLM dispatches | Wall time |183|---:|---:|---:|---:|---:|184| Small | ~30 | 4 | ~5 | ~3 min |185| Medium (default) | ~55 | 6 | ~7 | ~5 min |186| Large | ~110 (full corpus) | 12 (two medium waves) | ~14 | ~10 min |187188Compared to docs-sync (15-call flat ceiling), this skill scales as189O(waves), not O(claims), because per-agent work fits in one context190window. S7 verification dominates wall-time, not LLM cost.191192## Boundary (what this skill does NOT do)193194- Per-PR doc-impact review -- use `docs-sync`.195- Single-page typo or copy edit -- direct edit is faster.196- Writing docs for a brand-new feature -- use `docs-impact-architect`197 and `doc-writer` directly.198- Auto-merging or pushing without maintainer review.199- Reviewing code quality, security, or test coverage (out of scope).200201## Evals202203See `evals/`:204- `evals/content-evals.json` -- 3 corpus snapshots with seeded drift205 (stale CLI flag, dead nav link, expired deprecation target);206 expected behavior is that the skill catches all three and applies207 surgical fixes that ground out true on re-verification.208- `evals/trigger-evals.json` -- 10 should-trigger + 10 should-NOT-209 trigger queries, 60/40 train/val. The val split is the ship gate210 (>=0.5 should-trigger AND <0.5 should-not-trigger).211- `evals/README.md` -- how to run.212213## Provenance214215This skill was extracted from a real session that audited the216microsoft/apm corpus across 3 waves (PR #1511, 2026-05-27):217112/112 pages audited, 49 surgical fixes, ~25 LLM dispatches,218~30 min wall-time. The session design artifact (genesis hand-off219packet) lives in session state, not in this bundle (maintainer-220scope, not runtime-loaded).