Audit With Plans

Audit-with-plans

Run an audit-fix loop for projects using the planning-with-files skill. Repeatedly launch independent, fresh-context audit workers that compare the code to .planning/<active>/{task_plan, findings, progress}.md, then fix whichever side is wrong. Exit only when a fresh audit returns zero findings.

Prerequisite

Require a planning-with-files plan at .planning/<plan-id>/{task_plan, findings, progress}.md. If none exists, explain that the audit needs this baseline and suggest creating a plan first.

Adapt to the host agent

• Use the host's fresh-context delegation capability for each audit pass. In Codex, use a new subagent or delegated agent task; in Claude Code, use a new Agent invocation. Use an equivalent isolated worker on other hosts.
• Start a new worker for every pass. Never continue a prior audit worker or include earlier audit findings or fix history.
• Use the host's normal file-reading, searching, editing, and shell-execution tools. Do not depend on product-specific tool names.
• Keep the audit worker read-only with respect to project files. Run output-producing commands in a temporary copy or temporary output directory. Apply fixes only in the coordinating agent, then launch another cold-start worker.
• Inspect entry points for external or destructive side effects before execution. Never submit jobs, deploy, publish, SSH, or modify external systems; report the check as blocked when safe local execution is impossible.
• If the host cannot create an isolated worker, do not claim independent convergence. Ask the user to run the supplied prompt in a clean session, then continue the loop from its report.

The loop

audit  →  fix  →  audit (fresh, no memory of prior pass)  →  fix  →  …  →  no findings  →  exit

Exit condition: an independent fresh-context audit worker returns 0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW.

NOT exit conditions:

• "I think I fixed everything."
• "An audit worker that I told my fix history to confirmed my fixes landed."
• "All HIGH/MEDIUM closed; LOW items remain."

Audit-worker prompt template

Read templates/audit_subagent_prompt.md before launching the first audit worker. Fill its placeholders from the active planning files and pass the resulting prompt to a new isolated worker. The required structure is:

1. Role: "You are a code reviewer auditing <PROJECT_PATH>."
2. Project goal: one paragraph stating what the project is trying to achieve (auditor cross-checks against this).
3. Primary baseline: "Start from .planning/<active-plan>/{task_plan, findings, progress}.md. Treat other Markdown files as secondary evidence. Planning files may themselves be stale, so determine the correct side from executable behavior, project intent, and current artifacts."
4. Section-level audit hooks (see below).
5. Two failure modes: (a) code does X but .planning says Y; (b) .planning describes a workflow / artifact / value that the code is missing / broken / out of sync. Either side can be wrong.
6. Execution requirement: "For every documented entry point, execute with the documented invocation, check exit code, compare stdout/stderr/file outputs to the .planning description."
7. Artifact verification: "For every on-disk artifact .planning/*.md cites, verify the file exists; if a specific number is cited, grep for it in the file; if script-generated, regenerate + diff against the on-disk version where practical."
8. Safety bounds: "Do not modify project files or external systems. Run output-producing checks in temporary locations. Do not submit SLURM/batch jobs, deploy, publish, or SSH. Report checks that cannot run safely."
9. Independence: "You have no prior context on this project. Do not assume any audit has run before. Do not look for 'things prior passes might have missed'. Start cold from the three planning files and work outward."

Never include in the audit-worker prompt

• A list of prior findings.
• A list of what was just fixed.
• "Pass-N closed X; verify each landed."
• "Don't re-flag closed items."
• Any narrative from your prior fix history.

These turn the auditor into a verifier of your fixes. The verifier confirms what you told it is fixed and misses what you introduced, rationalized away, or never noticed. Each pass must be cold-start.

Section-level audit hooks (planning-with-files structure)

Anchor the audit worker's checks to the canonical sections of the three planning files:

Bidirectional fix policy

For each discrepancy:

• If the code does X and .planning says Y, and .planning is RIGHT → fix the code.
• If the code does X and .planning says Y, and the code has legitimately evolved (algorithm changed, cutoff tightened, path renamed) → fix the .planning file.

Stale documentation is as broken as stale code. Either direction is valid; the goal is convergence between code and .md, not one-way fixing of code.

Fix every reported issue

When an audit returns findings:

• Fix all of them (CRITICAL through LOW).
• The audit worker may explicitly mark some as "no fix required" / "informational" / "open by design pending user decision" — those don't count against the severity counts but should be acknowledged in the exit report.
• Do NOT punt items to "next session" without explicit user approval.
• Do NOT batch fixes across multiple audit passes; close each pass's findings before launching the next.

Anti-patterns

After a plan switch (.planning/.active_plan changed)

Switching the active plan is a major event on par with a restructure — it silently invalidates all plan-scoped tooling. When .active_plan changes (or when you start an audit and find the project has multiple plans):

1. Verify the self-audit script (audit_planning.py or equivalent) resolves .active_plan and dispatches the matching check set. A script hard-coded to the previous plan will print OKs about the wrong artifacts. This is an automatic HIGH finding until fixed.
2. Verify every plan-scoped tool / path constant points at the active plan, not the old one.
3. Re-run the self-audit and read WHICH plan's checks fired — not just the OK/FAIL counts. If you see checks naming the inactive plan's artifacts, the dispatch is broken.
4. The exit condition (0/0/0/0) is only meaningful once the self-audit is confirmed to be auditing the active plan.

After a major restructure (paths moved, files relocated)

Path-update sweeps with grep+sed are necessary but NOT sufficient. They catch literal-string references. They miss:

1. Assembled paths — OTF / "01_qm_opt/sp.fchk" where OTF is a Path variable; the prefix is in the variable, the suffix in the assembled string.
2. Test expectations — tests usually use Path(__file__).parent.parent / "templates"; restructure breaks them silently until pytest runs.
3. Template inheritance gap — the canonical template was updated; existing on-disk copies of that template (in working dirs, T-folders, etc.) were not re-templated.
4. README claims — "Phase complete" was true before the move but now references files at stale paths or observables that haven't been measured.

Mandatory post-restructure audit pass:

• bash -n on every shell script under the new tree.
• python3 -m py_compile on every Python script.
• Run every documented entry-point script with a trivial invocation; check exit code.
• Run the project's test suite (pytest etc.).
• Regenerate every cited analysis output; diff against on-disk version.
• Diff every working-dir copy of dynamic.sh / minimization.sh / build.sh against the canonical templates.
• For every phase N complete claim, verify the artifact exists, is non-empty, and was produced by the current code.

Termination report template

When the loop converges, report:

## Loop converged.

Fresh independent audit returned 0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW.

| Check | Status |
|---|---|
| Every cited file path in planning | ✓ exists |
| Every cited numeric value | ✓ matches artifact to documented precision |
| Every documented entry point | ✓ exit 0 |
| <add another project-specific category> | ✓ <status> |

One explicitly open item (documented, not an audit failure): <if any>

The remaining work is purely <physics / business logic / etc.>.

If items remain documented-but-unresolved (e.g. an explicit Open Blocker awaiting user decision), name them in the termination report so the user can see what's deferred by intent vs what's deferred by oversight.

Related skill

• planning-with-files — prerequisite. audit-with-plans audits the artifacts that planning-with-files produces.