Goal Loop

Goal Loop

A loop that only checks the criteria it shows the implementer will, eventually, produce an implementer that's very good at passing exactly those criteria and nothing more — memorizing the test, not solving the problem. This skill treats "loop until it works" as gradient descent: forward → loss → gradient → update, with a held-out set the implementer never sees, so the final verdict can tell the difference between "solved" and "gamed the visible checks."

When to use this

• "Keep iterating until this passes", "loop until the goal is actually met."
• Bug fixes, feature completion, or refactors with fuzzy or compositional acceptance criteria — where a narrow visible test could pass without the real problem being solved.
• Skip it for one-shot deterministic tasks with no real risk of overfitting the checks (a config change, a straightforward CRUD endpoint with a spec that's already unambiguous) — the loop machinery is overhead you don't need there.

Instructions

1. Set up the goal contract before looping

• State the goal as something checkable, not a vibe: "the cache handles concurrent writes without corruption," not "make the cache better."
• Split evidence into two sets:
  • Visible checks — what the implementer is told to satisfy this iteration (a specific failing test, a described scenario). This is the implementer's target.
  • Held-out checks — compositional or edge-case scenarios written separately, covering what the visible checks don't directly test. Never shown to the implementer during the loop — only used at the final gate.
• Evidence must be an observable command or artifact — something that exits 0/nonzero, a diff against expected output. Self-assessment is not evidence, on either side of the split.
• See references/holdout-design.md for how to write held-out checks that actually catch overfitting instead of just duplicating the visible ones.

2. Run the loop — one iteration at a time

Each iteration ("epoch"):

1. Forward. A fresh attempt — ideally a fresh context, not the same one carrying accumulated rationalizations — gets the goal and the visible checks only, and proposes/applies a change.
2. Loss. Run the visible checks; record pass/fail. Separately, run the held-out checks and record the result somewhere the next iteration's context won't see — held-out results inform the loop's own steering, not the implementer.
3. Textual gradient. Don't just log "failed" — diagnose which behavior caused the failure and what class of fix would address it, as a reusable insight, not a one-off patch. For held-out failures specifically: turn them into directional guidance without leaking the held-out check's content or exact output — e.g. "the current approach breaks under concurrent access," not "test_concurrent_3 failed with AssertionError on line 42."
4. Update, with momentum. Keep a running log of recurring failure patterns with occurrence counts across iterations (not just the latest epoch's noise). The next attempt acts on the consolidated pattern, so it fixes the root issue instead of whack-a-moling symptoms or flip-flopping between two partial fixes.

See references/loop-mechanics.md for a worked micro-example of one full epoch.

3. Stop early — don't loop blindly

• Set a budget up front (default: 3–5 iterations).
• If the same top failure pattern recurs across ≥2 iterations with no visible-check improvement, narrow the edit scope (broad → targeted) rather than trying progressively bigger rewrites.
• After a patience budget (default: 2 iterations) with no visible-check improvement, stop with status EXHAUSTED and recommend decomposing the goal into smaller sub-goals rather than continuing to loop.

4. Gate on the held-out set — this is the anti-gaming check

• Only evaluate this once the visible checks are all green.
• Compare visible pass rate to held-out pass rate. A positive gap (visible passing, held-out failing) is the signature of gaming — the implementer optimized the checks it could see instead of the underlying goal.
• A positive gap with green visible evidence means the verdict is NOT MET, full stop, regardless of how convincing the visible evidence looks. Report the gap explicitly, don't soften it.
• Verdict is MET only when both visible and held-out checks pass.
• Watch specifically for these gaming patterns (detail in references/reward-hacking-patterns.md): hardcoding to the visible check's expected output, special-casing on how the check invokes the code rather than the general behavior, weakening or tampering with the check itself, and narrowly overfitting the literal visible scenario instead of the general case it's supposed to represent.
• If a genuinely independent judge is available (see codex-judge), use it for the final MET/NOT MET call on anything high-stakes — it's structurally better positioned to catch gaming than the loop that just ran the implementer.

5. Report

State: iterations run, visible/held-out pass rate per iteration, the final verdict, any recurring failure patterns found, and — if EXHAUSTED — the recommended decomposition into sub-goals.

Reference files

• references/loop-mechanics.md — a worked example of one full forward → loss → gradient → update epoch.
• references/holdout-design.md — how to write held-out checks that catch overfitting, not just duplicate the visible ones.
• references/reward-hacking-patterns.md — the gaming patterns to watch for at the final gate, in depth.