Lbstrydom Claude Engineering Skills Audit Code

Code Audit Loop

Multi-pass code audit with GPT + Gemini final review. Iterates until findings stabilise or max 6 rounds.

Input: $ARGUMENTS — plan file path (the spec the code is being audited against). Optional flags: --scope diff|plan|full.

Step 0 — Parse Input and Validate

Validate: plan file exists, OPENAI_API_KEY is set. Optional: GEMINI_API_KEY for Step 7 (falls back to Claude Opus when absent). SUPABASE_AUDIT_URL for cloud learning (optional).

Initialise session ID: SID=audit-code-$(date +%s).

Show kickoff card:

═══════════════════════════════════════
  /audit-code — Starting
  Plan: <path> | Max 6 rounds | SID: $SID
═══════════════════════════════════════

Step 0.5 — Architectural-memory catalogue (--scope=full only)

When --scope=full, fetch the repo's symbol catalogue from architectural memory (if populated) and inline a "Symbol catalogue" section into the prompt context. This helps the auditor catch cross-file duplication that diff-scope cannot see.

# 1. Resolve repo identity + active snapshot
node scripts/cross-skill.mjs get-active-refresh-id --repo-uuid "$(node scripts/cross-skill.mjs resolve-repo-identity | jq -r .repoUuid)"
# 2. Fetch top-N symbols (env-tunable: ARCH_AUDIT_FULL_TOPN, default 200)
node scripts/cross-skill.mjs list-symbols-for-snapshot --json '{"refreshId":"<from step 1>","limit":200}'

Format the rows as a ## Symbol catalogue (top N by domain) section with (domain alphabetical, symbol_name alphabetical) ordering. Note truncation in the section header if count == limit.

States:

• cloud:false or refreshId:null → skip section silently (audit proceeds normally).
• RPC_ERROR → skip section, log warning to stderr.

This step is advisory. Audit must work even if architectural memory is unavailable — never fail the run.

Step 1 — Choose Audit Scope

CRITICAL: GPT doesn't know what's "new" vs pre-existing — it flags everything in scope. Choose deliberately:

Default is --scope diff. Switch only when the user explicitly asks or git diff is empty.

Dirty-aware base (default) + the --base override (read this before clustered/resumed audits). With no --base, openai-audit.mjs resolves the diff base by the working tree's state: dirty → HEAD (audit only your uncommitted work) / clean → HEAD~1 (audit your last commit). This prevents the over-capture failure where an already-shipped + already-audited prior commit gets re-pulled into scope and floods the audit with out-of-scope findings (observed in ai-organiser: 33/34 findings were a previous audited cluster). The resolved base is logged as [scope] base resolved to <ref> ….

⚠ When the prior commit was already audited but you have BOTH committed and uncommitted work (e.g. resuming a clustered build where Cluster A/B is committed and Cluster C is in-flight), pass --base explicitly to scope to exactly the new work — e.g. --base <clusterStartRef> or --base HEAD — and likewise scope the Gemini gate's --diff to the same range. Never rely on the default to separate audited-from-unaudited across a commit boundary.

Step 2 — Run Code Audit

Round 1

node scripts/openai-audit.mjs code <plan-file> \
  --scope diff \
  --out /tmp/$SID-r1-result.json \
  2>/tmp/$SID-r1-stderr.log

Round 2+

R2+ mode changes the prompt rubric and enables ledger-driven suppression. Full flag contract, smart pass selection, automatic behaviour, and tool pre-pass rules: references/r2-plus-mode.md.

# Dirty-aware base — match R1's scope (which uses `git status --porcelain`, so
# UNTRACKED files count as dirty; do NOT use `git diff --quiet`, it ignores
# untracked). Dirty tree → HEAD (uncommitted work only); clean → HEAD~1 (last
# commit). The CLI's `[scope] base resolved to <ref>` log is the source of
# truth. Pass --base/clusterStartRef instead when separating audited-from-
# unaudited across a commit boundary.
BASE=$([ -n "$(git status --porcelain)" ] && echo HEAD || echo HEAD~1)
git diff "$BASE" -- . > /tmp/$SID-diff.patch
node scripts/openai-audit.mjs code <plan-file> \
  --round 2 \
  --ledger /tmp/$SID-ledger.json \
  --diff /tmp/$SID-diff.patch \
  --changed <csv> --files <csv> --passes <csv> \
  --out /tmp/$SID-r2-result.json \
  2>/tmp/$SID-r2-stderr.log

Requirements rubric (automatic)

When .requirements/ledger.json exists, every code-audit pass is given a <requirements_rubric> block — the repo's de-facto invariants (security / safety / correctness / behavioural / persistence) the diff must not violate. It is assembled by getRequirementsContext and injected through the shared prompt builder; in-scope active requirements appear in full, the rest as an index. No flag needed — ledger absent → audit is unaffected. A stale ledger (in-scope files changed since extraction) or uncovered target files surface as a [requirements] stderr line; if you see [stale], run node scripts/requirements.mjs extract --files <…> then reconcile to refresh it. See docs/completed/requirements-layer.md.

Handle results

If verdict is INCOMPLETE (passes timed out), offer: re-run with higher timeout, or continue with partial results.

Show results

═══════════════════════════════════════
  ROUND 1 AUDIT — SIGNIFICANT_ISSUES
  H:6 M:10 L:5 | Deduped: 3 | Cost: ~$0.45
  Top: [H1] Missing auth on /api/...
═══════════════════════════════════════

Step 3 — Triage (validity × scope × action)

You are a peer, not a subordinate. For each finding, record three orthogonal judgements:

Triage rules

• validity=invalid → action MUST be dismiss or rebut
• validity=uncertain → action MUST be rebut (GPT deliberation)
• validity=valid + scope=in-scope + HIGH/MEDIUM → fix-now (unless accepted-permanent debt)
• validity=valid + scope=out-of-scope + load-bearing → fix-now (treat as in-scope — see impact test below)
• validity=valid + scope=out-of-scope + independent → defer eligible (pre-existing debt)
• validity=valid + scope=in-scope + LOW → operator choice
• Only validity=valid findings can be deferred

Scope is decided by impact, not authorship (load-bearing test). "My PR didn't touch this line" is NOT a defer pass. Before any out-of-scope finding routes to defer, apply the test: does the correctness or stability of the change I'm shipping depend on the cited code path?

• Load-bearing — the new code calls into, reads state from, or otherwise rides on the cited path → it is in-scope for the fix/defer decision even if pre-existing. fix-now, or explicitly gate the feature on it and say so. Never silent-defer a load-bearing finding.
• Independent — the path fails identically with or without this change; the new code does not depend on it → genuine defer (pre-existing debt).

A pre-existing finding in a file you changed is a yellow flag, not a green one: you usually touched the file because your change now rides on its behaviour. Default such findings to "prove independence" rather than to defer. Passing tests don't clear this — a green suite only covers exercised paths, not the load-bearing path's failure modes.

Honest-deferral check (Design right-sizing, AGENTS.md — the band-aid escape hatch). defer is the place "patched the easy way" hides. A defer of a valid in-scope finding must name three things: (1) the root cause; (2) the minimal in-scope fix you considered and rejected; (3) the residual risk. A defer of an out-of-scope finding must additionally name the independence — one sentence stating the new code does not call/depend on the cited path (the load-bearing test above). If you can't write that sentence truthfully, it's load-bearing → fix-now. Invariant: never defer because the correct fix is merely larger — size is not scope, and neither is authorship. Legitimate defer = a true (impact-tested) scope boundary or explicitly accepted, documented debt. Strongest form: drop a TODO at the cited line naming the root cause, so the dodge becomes a visible artifact. (The over-engineering cliff is caught symmetrically by Gemini's over_engineering_flags in the final review.)

Mechanical vs architectural

Each finding has is_mechanical: true/false from GPT:

• Mechanical: deterministic fix. Fix immediately, no deliberation.
• Architectural: judgement call. Needs deliberation, resets stability if new.

Tiered rebuttal (when action=rebut)

Only send rebuttal if rebut HIGH or MEDIUM findings exist:

node scripts/openai-audit.mjs rebuttal <plan-file> <rebuttal-file> \
  --out /tmp/$SID-resolution.json 2>/tmp/$SID-rebuttal-stderr.log

Convergence

Quality threshold: HIGH == 0 && MEDIUM <= 2 && quickFix == 0

Stability uses _hash for exact cross-round matching:

• New hash not in prior set = genuinely new → resets stability
• Mechanical-only findings do NOT require stability rounds

Max 6 rounds for code audits.

Step 7 (Gemini final review) is MANDATORY after the last audit round, regardless of convergence — except when both GEMINI_API_KEY and ANTHROPIC_API_KEY are absent.

Step 3.5 — Update Adjudication Ledger

After each deliberation round, write ledger entries for every finding before proceeding to Step 4. The ledger drives R2+ rulings injection and post-output suppression.

Full writer invocation example + status field semantics: references/ledger-format.md.

Step 3.5b — Record Triage Outcomes (closes the adaptive-learning loop)

MANDATORY after the ledger is written each round. Persist the round's accepted/dismissed outcomes so the adaptive-learning loop has ground truth:

node scripts/write-code-outcomes.mjs \
  --result /tmp/$SID-r<N>-result.json \
  --ledger /tmp/$SID-ledger.json \
  --round <N>

This bridges the adjudication ledger → outcome-sync → finding_adjudication_events

• audit_pass_stats (accepted/dismissed) + audit_findings + audit_runs.labeled (cloud) and .audit/outcomes.jsonl (local bandit reward). Without it those columns stay 0 and the bandit / FP-learning / prompt evolution get no training signal — the audit works but never improves.

Best-effort: a cloud failure logs and falls back to local-only; it never blocks the audit. Run it once per round, after Step 3.5.

Step 3.6 — Debt Capture

Persist out-of-scope valid findings to .audit/tech-debt.json so future audits suppress them automatically. Eligible candidates: Step 3 triage findings with action = defer.

Full per-reason field requirements, capture flow, sensitivity-scan rules, and status card format: references/debt-capture.md.

Execution order — critical

Wait for rebuttal BEFORE fixing.

1. Send rebuttal (if rebut HIGH/MEDIUM findings from triage)
2. Wait for rebuttal response
3. Write adjudication ledger (Step 3.5)
4. Record triage outcomes — write-code-outcomes.mjs (Step 3.5b)
5. Capture deferrable debt (Step 3.6)
6. Fix ALL findings together (Step 4)
7. Run tests
8. Verification audit (Step 5) — debt suppression runs automatically

Step 4 — Fix Findings

ALL HIGH must be fixed. MEDIUM until ≤2 remain. LOW if mechanical.

Track which files you modify — you'll need this for --changed in Step 5.

═══════════════════════════════════════
  FIXING — 17 findings
  Auto-fixed: 3 (mechanical)
  Fixed per recommendation: 8
  Compromises: 2
  Skipped (LOW): 4
  Files modified: shared.mjs, openai-audit.mjs
═══════════════════════════════════════

List each fix: [ID] description → file:lines.

After fixing, update ledger entries to remediationState: 'fixed' for fixed items.

Step 5 — Verify and Loop (R2+ Mode)

After fixes, re-audit using R2+ mode (back to Step 2):

1. Collect files modified during Step 4 → --changed
2. Compute scope: changed + importers → --files
3. Generate diff (dirty-aware base, matching R1 — untracked counts): BASE=$([ -n "$(git status --porcelain)" ] && echo HEAD || echo HEAD~1); git diff "$BASE" -- . > /tmp/$SID-diff.patch
4. Build --passes from file types
5. Run R2+ audit with --round <N> --ledger --diff --changed --files

Track finding churn using _hash fields: resolved / recurring / new.

═══════════════════════════════════════
  ROUND 2 → ROUND 3 (R2+ mode)
  H:0 M:2 L:1 | New: 0 | Suppressed: 11
  Stable: 1/2
═══════════════════════════════════════

Step 5.1 — Debt Resolution Prompt

After verification, reopened debt topics with no matching finding this round are candidates for resolution. Full prompt + resolver invocation: references/debt-capture.md.

Step 6 — Convergence Report (Pre-Final)

═══════════════════════════════════════
  CONVERGED — Round 4
  Final: H:0 M:2 L:1
  Rounds: 4 | Time: 14m | Cost: ~$0.20
  Files changed: 6
  Remaining (accepted): [M3], [M7]
═══════════════════════════════════════

Save convergence snapshot to docs/plans/<name>-audit-summary.md.

Do not close the loop in Step 6 — completion requires Step 7.

Step 6.5 — Regenerate Telemetry Dashboard (advisory, source-repo only)

Source-repo-gated — run ONLY when package.json.name === "claude-engineering-skills". Skip silently in consumer repos (the dashboard is opt-in there — docs/plans/local-dashboard.md §7.3). Never blocks the audit.

node scripts/build-dashboard.mjs telemetry 2>&1 || true

This refreshes the gitignored dashboard/telemetry.html so the just-run audit's findings are visible in the local dashboard. Print the link:

Telemetry dashboard: file://<abs-path>/dashboard/telemetry.html

telemetry.html is gitignored — never staged, never committed.

Step 7 — Gemini Independent Review (MANDATORY)

Run Gemini 3.1 Pro as the final gate. Falls back to Claude Opus when GEMINI_API_KEY is absent.

# Pass --run-id <_cloudRunId> when the audit --out JSON carries one, so the
# final-review (and the optional shadow A/B reviewer) persist their per-finding
# results keyed to this audit_run. Read it from the audit result:
#   RUN_ID=$(node -e "process.stdout.write(require('/tmp/'+process.env.SID+'-result.json')._cloudRunId||'')")
# Omit --run-id when absent (cloud off) → gemini-review runs local-only.
node scripts/gemini-review.mjs review <plan-file> /tmp/$SID-transcript.json \
  --out /tmp/$SID-gemini-result.json \
  ${RUN_ID:+--run-id "$RUN_ID"} 2>/tmp/$SID-gemini-stderr.log

Verdict handling: APPROVE → done. CONCERNS → deliberate, fix, re-run Gemini. REJECT → present to user.

Shadow A/B reviewer (optional, observation-only): set FINAL_REVIEW_SHADOW (e.g. claude-opus) to run a second blind reviewer in parallel with the primary — it never gates the build, attributes findings per source_model, and persists the diff for final-review-stats. No-op when unset or under an Azure profile. See docs/plans/final-review-shadow-reviewer.md.

Full transcript-building, verdict routing, Step 7.1 deliberation protocol, and category-error handling: references/gemini-gate.md.

UX Rules

1. Status card after every phase (compact format above)
2. Never dump raw JSON — parse and summarise
3. Show every fix with file + line reference
4. Cost tracking: cost ≈ (input × 2.5 + output × 10) / 1M
5. Batch all user decisions into one prompt
6. Progress: show pass timings from stderr

Key Principles

1. Peer relationship — neither model blindly defers
2. Three-model system — Claude (author) + GPT (auditor) + Gemini (final arbiter)
3. Fix all HIGH, MEDIUM until ≤2, LOW optional
4. Stability over speed — 2 clean rounds required
5. No quick fixes — band-aids rejected by all models
6. Deliberation is final — no infinite debate
7. Graceful degradation — failed passes, missing keys, missing ledger all skip cleanly
8. No self-review — Step 7 final gate reviews Claude-GPT transcript
9. Adaptive learning — outcomes logged, FP patterns tracked, prompts improve

Reference files

This skill's canonical flow is above. The files below cover specialised situations — read them only when the trigger applies.

Source: Lbstrydom/claude-engineering-skills — distributed by TomeVault.