Iterate Until Verified

Iterate Until Verified

When the job is to produce an artifact (rendered image, SVG, chart, config file, regex, generated code, rubric-graded prose) and a concrete pass/fail check exists, use the check tool to run the loop instead of eyeballing the result. The extension tracks iterations on disk + in the session branch, enforces budgets, and catches "looks right to me" claims you didn't actually verify.

You are the actor: you edit the artifact. The check (a bash command or a critic subagent) is the environment. Every check run returns a verdict; you use it to decide the next edit - the same shape as a tight test-driven-development loop, just formalized so weak models can follow it too.

When to declare a check

Reach for check when all of these hold:

• You are producing a concrete artifact: a file, a diagram, a chart, a config, a generated snippet, a rendered image.
• A pass/fail contract exists or can be stated: "the file exists and contains X", "the SVG has three pie slices", "the regex matches these examples and rejects those", "the output validates against this schema", "the rubric says…".
• You expect to iterate at least once. If the first draft is obviously right, you still benefit - one check run catches the off-by-one case cheaper than a user round-trip.

Skip the check when:

• The deliverable IS the answer to a question ("what does this function do?"). No artifact, nothing to re-render.
• A single tool call completes the task with deterministic success (git status, "read line 40"). The round trip costs more than it saves.
• The artifact is genuinely throwaway (one-off /tmp scratch). If the user will look at it once and discard it, don't spin up the loop.
• You already have a user-facing test command you'd run anyway (npm test). Run it inline - the iteration-loop's bash kind is for cases where the check is the whole contract, not a superset of an existing suite.

How to pick a check kind

Rules of thumb:

• Start with bash if any deterministic validator exists. Exit codes are cheap, zero-cost, and leave no ambiguity.
• Use critic when the rubric fits better as English than as code. "Three slices, correct labels, brand palette" is harder to express as a validator than as a rubric.
• For visual critics on cheap local models, use modelOverride: "llama-cpp/qwen3-6-35b-a3b" on the critic spec so the judge runs on a vision-capable local model instead of a strong paid one.
• SVG+vision path on this host: magick in.svg out.png (ImageMagick is available; rsvg-convert is not). Have the critic read the rendered PNG - vision models can't parse raw SVG reliably.

Authorship flow

Hybrid authorship is deliberate - you draft, the user reviews, only then does the loop start.

1. Declare. Call check with action: "declare":

{
  "action": "declare",
  "kind": "bash",
  "artifact": "out.svg",
  "cmd": "xmllint --noout out.svg",
  "maxIter": 5,
  "maxCostUsd": 0.1
}

Or for a critic:

{
  "action": "declare",
  "kind": "critic",
  "artifact": "out.svg",
  "rubric": "SVG is a pie chart with exactly 3 slices. Each slice has a visible label with its percentage. Uses the brand palette (red #e53, blue #36c, yellow #fd3).",
  "maxIter": 5
}

The tool writes .pi/checks/<task>.draft.json. task defaults to default; v1 supports one active task at a time.

2. Surface the draft. Print the draft inline so the user can review, and tell them how to start:

Proposed check (.pi/checks/default.draft.json): bash xmllint --noout out.svg, max 5 iterations, $0.10 budget. Review the draft, then run check with action: "accept" (or /check accept default) to begin iterating.

Do NOT call check run before the user accepts - the tool will refuse while a draft is pending.

3. Accept. On user confirmation, call check with action: "accept". The extension renames draft → active and seeds the iteration state. The system prompt starts showing a ## Iteration Loop block every turn from now on.

4. Iterate. See next section.

5. Close. On stopReason: "passed" or when the user accepts best-so-far, call check close with an appropriate reason (passed, budget-iter, budget-cost, wall-clock, fixpoint, user-closed). The task directory archives under .pi/checks/archive/<timestamp>-<task>/.

Iteration discipline

This is where small models lose - they edit, say "looks right", and skip the verification step. The rules below mechanize the contract.

• After every substantive edit of the artifact, run the check. The extension counts edits since the last check run; hitting the threshold (default 2) triggers a strict nudge at turn end.
• Read the verdict before editing again. Open the returned observation_path if you need the raw output, check the issues array, and address the highest-severity issue first (blocker → major → minor).
• One focused edit per iteration. Shotgun edits make verdicts hard to attribute. If multiple issues exist, fix the blocker, re-run, then move to the next tier.
• Never claim "the artifact is done" without a passing check run this turn. The claim-nudge extension catches phrases like "looks right", "matches the spec", "the chart is ready" when no successful verdict was recorded - either run the check or retract.
• Honor the injected status block. Every turn while a task is active, the system prompt ends with ## Iteration Loop (task: default) carrying iteration count, last verdict, best-so-far, remaining budget, and a Next step: line. That line is your directive - follow it.
• Budget exhaustion is not failure. If stopReason comes back as budget-iter, budget-cost, or wall-clock, the extension returns best-so-far. Report it to the user and ask whether to accept, extend the budget via a new declare, or reshape the rubric.
• Fixpoint = stop editing blindly. stopReason: "fixpoint" means two consecutive iterations produced identical artifact bytes. Edit differently or close the loop - spinning won't help.

Composition with todo

One iteration loop = one todo. Keeps the plan and the loop in sync so the user can see both in the injected blocks.

• When you check accept, park the matching todo in review with a note like "iterating - parked on check/default".
• While iterating, the todo stays in review; individual iterations are NOT separate todos (they'd churn the plan).
• Only complete the todo after check run returns stopReason: "passed" (or the user accepted best-so-far on budget exhaustion). Completion note: "check default passed on iter 3, score 1.00".
• If the loop terminates without a pass (budget-iter, fixpoint) and the user wants to retry differently, reopen the todo and declare a new check with adjusted rubric/budget.

Use scratchpad for cross-iteration context you want surviving compaction: the rendering pipeline command (magick in.svg out.png), non-obvious rubric interpretations, the draft's model override. One short note per item, not an iteration log - the extension already maintains history.

Anti-patterns

• Don't skip authorship because "the check is obvious." Weak models often draft checks that look sensible and subtly mis-spec the rubric. The user's review catches that cheaply.
• Don't check run without editing between iterations. Wasted spin; every call snapshots the artifact and (for critic) burns cost. The fixpoint detector will stop you after two identical bytes, but don't make it do your job.
• Don't rely on the claim nudge to catch edits. The strict edit-without-check nudge exists because the claim nudge only fires when the model vocalizes a completion claim - and small models often skip the claim entirely. Run the check proactively.
• Don't iterate without reading the prior verdict. The issues array is your signal. Making a guess without opening it is how loops wander.
• Don't widen the rubric mid-loop to make the check pass. If the rubric was wrong, check close with reason: "user-closed", then declare a corrected check. Never hand-edit .pi/checks/default.json mid-run to cheat the verdict.
• Don't forget to close. Active tasks keep injecting the status block every turn; a stale active check clutters every unrelated conversation until the user notices. close on every terminal stop-reason.
• Don't declare a second task while one is active. v1 enforces single-task - the second declare will error. Finish or close the first.
• Don't use modelOverride on the critic without a reason. The critic inherits by default. Override only to push vision grading to a cheap local model (llama-cpp/qwen3-6-35b-a3b) or to deliberately downshift when the judge is the bottleneck.

Quick reference