Verify Repair Loop

verify-repair-loop

A bounded generate → run → revise → re-run cycle whose pass signal is the executed verdict — the project's real test/quality output, scored against a numeric threshold — escalated to a judge for qualitative confirmation only after the numeric gate passes (staged escalation). The loop is the conversation, not a process: this loop introduces no resident process of its own and no persistent cross-session state. Use it to drive a change to green when the verdict source is runtime execution, not a judge reading a diff.

When to use

• A change must converge against the project's test runner / quality tools and you want the agent to iterate until checks pass before asking for review.
• Test/quality coverage is the authoritative signal (high coverage, actionable failures) — not subjective craft.
• You want bounded auto-repair with a hard stop, not an open-ended fix loop.

Do NOT use when:

• The verdict is subjective craft (naming, architecture, style) or coverage is incomplete → use do-and-judge in subagent-orchestration (judge reads the diff and is sovereign).
• You only need a one-shot multi-judge review of a finished diff → use /review-changes.
• The verdict must come from a live app (Playwright against running services) → deferred (see Scope below); use test/quality verdicts until that trigger fires.

The Iron Law

THE NUMERIC VERDICT IS NEVER SOVEREIGN.
TESTS DECIDE WHEN TO ESCALATE — A JUDGE DECIDES WHETHER TO APPLY.
A REVISION THAT BREAKS A PREVIOUSLY-GREEN CHECK STOPS THE LOOP,
EVEN IF THE OVERALL SCORE ROSE.
GENERATOR NEVER APPROVES ITS OWN CHANGE.

A loop that optimizes for pass_count ≥ N can game the metric — delete tests, skip assertions, weaken checks — and still hit the threshold. The numeric gate only decides when to escalate; a judge with veto confirms the change is real.

Procedure

1. Freeze the contract (spec fingerprint)

Before the first run, capture what "done" means and snapshot it:

• the task's acceptance criteria / requirement,
• the set of checks that define the verdict (test files + quality commands),
• a spec fingerprint = a hash of (requirement text + the check set).

Record the baseline green set — which checks pass before any edit — so regressions are detectable. Pick the verdict commands per the project's toolchain-resolver (PHP / JS-TS / Python / Go / Rust), not a hardcoded runner.

2. Set the gate

3. The loop (multi-turn — the conversation IS the loop)

Each iteration is agent turns, never executing control-flow code:

1. Generate / revise — make the smallest change toward the contract.

2. Run — execute the verdict commands; parse the structured result (counts green/total, which checks failed). This is one tool call, read in context.

3. Score + regression check — compute the numeric score; compare failures against the baseline green set.

   • Regression (a baseline-green check now fails) → STOP, hand back with the regression named, even if the score rose. No whack-a-mole.
   • Plateau (last plateau_window scores within tolerance) → STOP; surface suspected flakiness or a stuck point. Do not thrash.
   • attempts == max_attempts → STOP, hand back the best envelope.

4. Tamper check (deterministic, loop-local) — before the numeric gate is allowed to say anything, run the detector over the diff this attempt produced and over the recorded verdict:

   ./scripts-run src/scripts/detect_verification_tampering \
       --diff <this attempt's diff> --message <this attempt's message> \
       --verdict <the runner's recorded verdict>
   

   It is fail-closed inside the loop: outcome violations or stale_verdict STOPS the loop and hands back with the finding's file:line, whatever the score says. clean changes nothing — an ordinary passing attempt is unaffected, because the detector reports only on shapes the diff introduced.

   The check answers exactly the gap the Iron Law names: assertion-removed, assertion-weakened, verification-skipped (warn), expectation-rewritten, early-exit-injected, test-file-deleted, subject-mocked-away, and stale_verdict for a runner that finished before the last edit. A change that legitimately deletes an obsolete test or relaxes an over-tight assertion declares it in the attempt's message — tamper-allow: <detector-id> — <reason> — scoped to one id, never to all.

   It is not a CI gate and does not become one here. The tree's measured lesson is that reach, not content, is the scarce thing; a new always-on gate is the opposite of that lesson. Promotion owes a measured false-positive rate first.

5. Numeric gate — score < threshold and attempts remain → back to step 1 (attempts++). Score ≥ threshold, no regression, tamper check clean → escalate (step 4 of the stage).

4. Judge escalation (only after the numeric gate passes)

Dispatch the judge as a subagent with fresh context (the judge-* cluster via subagent-orchestration) that sees only the diff + the executed results — never the generator's reasoning. This is the real generator ≠ judge separation; single-agent persona-switching is theater.

• judge apply → DONE.
• judge revise → back to step 3.1 (attempts++; still bounded).
• judge reject → STOP, hand back; the approach must change.

5. Mid-loop invalidation (user-interrupt-priority)

Re-check the spec fingerprint each iteration. If the user changes the requirement mid-loop (a new instruction in the conversation), the fingerprint changes → abort and hand back per user-interrupt-priority. Never keep iterating against a stale contract.

Scope — what runs, what is deferred

State lives in the conversation, optionally mirrored to a re-read working file — never a daemon or persistent cross-session store (honors resident-process-governance).

Validation

Before finalizing, confirm:

1. The numeric gate escalated to a judge — it did not apply on its own.
2. No baseline-green check regressed (or the loop stopped and said so).
3. The tamper check ran on the final attempt and returned clean — or the loop stopped on its finding. A green verdict recorded over a violations or stale_verdict outcome is not a green verdict.
4. The loop stopped at max_attempts, a plateau, or a judge apply — never ran unbounded.
5. The judge saw only the diff + results, dispatched in fresh context.
6. No unsupervised resident process introduced — and none of the four governance conditions in resident-process-governance.md left unmet if a supervised one is. (Amended 2026-08-27, ADR-249: this read "No daemon / persistent runtime introduced" when a daemon was prohibited outright. The check still fires; what it tests moved from existence to governance.)

Output format

verify-repair-loop
Contract:   <one-line requirement> · threshold <X> · allow_regressions false
Attempts:   <k>/<max>  scores: [s1, s2, …]   stop: threshold|plateau|cap|regression|reject
Verdict:    DONE | DONE_WITH_CONCERNS | BLOCKED
Judge:      <judge skill> → apply|revise|reject
Evidence:   <green/total> · regressions: none | <named checks>

Required fields (ordered):

1. Contract — the frozen requirement + threshold + allow_regressions.
2. Attempts — k/max, the score history, and the exact stop reason (threshold / plateau / cap / regression / reject).
3. Verdict — DONE, DONE_WITH_CONCERNS, or BLOCKED.
4. Judge — the dispatched judge skill and its apply / revise / reject verdict (omit only when the loop stopped before escalation).
5. Evidence — final green/total and any regressed checks by name.

Examples

Good — staged, bounded, judge confirms:

iter1 84/100 (<90) → revise
iter2 93/100 (≥90, no regression) → escalate → judge apply → DONE

Bad — numeric verdict treated as sovereign (rejected by the Iron Law):

iter1 91/100 ≥ threshold → "PASS, done"   ✗ no judge escalation; metric may be gamed
iter2 fixes A, breaks B, 92% → "PASS"      ✗ regression ignored

Gotcha

• Metric gaming — the agent "passes" by deleting a failing test or weakening an assertion. The numeric gate still cannot catch this, and that is why loop step 3.4 exists: detect_verification_tampering decides the mechanical shapes deterministically, and the fresh-context judge remains the safeguard for everything shape cannot decide. If the diff removes checks, that is a reject, not a pass.
• Phantom verification — the loop records a green verdict it never re-ran, so the exit code describes a tree that no longer exists. The verdict carries the runner's own timestamp; one that predates the last edit is stale_verdict, its own outcome, never a pass.
• Flake mistaken for a plateau — a single non-deterministic test flips the score and the window-comparison reads it as "no improvement". Re-run the suspected check before declaring a plateau; surface flakiness instead of thrashing.
• Score rose, but a regression hid inside it — fixing 6 checks while breaking 2 still raises the total. Without the regression guard the loop ships a regression. Always diff failures against the baseline green set, not just the aggregate score.
• Persona-switch theater — asking the same agent to "now judge what you wrote" is not separation. Dispatch the judge as a fresh-context subagent that never saw the generator's reasoning.
• Stale contract — iterating against a requirement the user changed mid-loop. Re-check the spec fingerprint each turn; a change aborts.

Do NOT

• NEVER let the numeric score apply a change without judge confirmation.
• NEVER record a green verdict over a tamper finding or a stale runner timestamp — inside the loop the detector is fail-closed.
• NEVER widen a tamper-allow declaration past the one id it names.
• NEVER continue past a regression because the overall score improved.
• NEVER run the live-app Playwright path here — it is deferred (needs a runtime).
• NEVER judge with the generator's own context — dispatch a fresh-context judge.
• NEVER loop past max_attempts or a detected plateau.

See also

• subagent-orchestration — do-and-judge (judge-sovereign, diff-read) is the sibling; this skill is test-verdict-gated.
• /review-changes — one-shot multi-judge review; wires this loop as an opt-in step.
• playwright-testing, quality-tools — verdict sources.
• src/scripts/detect_verification_tampering.ts — the loop-local tamper and phantom-verification check bound at step 3.4, with its fixture corpus in tests/fixtures/tamper-corpus/ and its polarity proven both ways by --self-test.
• autonomous-execution — the N=3 cap.
• resident-process-governance — the runtime-free constraint this loop honors.
• Optional impact pre-step (when a code-graph is present, ADR-124): run agent-config code-graph affected --since <ref> to scope which symbols a change touches before choosing verdict sources — cited, not duplicated; the loop itself is unchanged when no graph exists.