Outer Loop

/outer-loop — improve the process, not the output

An inner loop optimizes the work: propose → run → measure → keep or roll back. An outer loop optimizes how the inner loop searches: read its traces, find where the search itself is stuck, change the mechanism, run again.

The finding worth internalizing (Qu & Lu, verified): both levels used the same LLM. The 5x came from architecture, not a smarter model. And crucially — parameter-level adjustments without mechanism changes yielded no reliable gains. Tuning the knobs does nothing. Changing how it searches is the whole effect.

What their outer loop actually found: the inner loop kept falling back on the model's priors — retrying familiar moves even after those moves stopped working. The outer loop broke the pattern by forcing exploration in directions the model's instincts avoid. Prior-lock is the default failure of any repeated agent process. That is what you are hunting.

PRECONDITION — refuse to run without these four

An outer loop is expensive and easy to fool. It only earns its cost when the inner loop has:

1. A metric. A number that moves, in the right direction, without a human deciding. (val_bpb, tests passing, exit code, row-count parity, build time, cost per run.) No metric → stop. Say so plainly and offer to build the metric first.
2. Traces. Logs of what the inner loop actually tried and what happened — not just the final artifact. You cannot diagnose a search you cannot see.
3. Repetition. It has run enough times that a pattern exists to find. A handful of runs is anecdote, not a stuck search.
4. A budget that can absorb it. Two levels of loop burn far more than one.

If any fail: do not build the outer loop. Fix the missing piece, or run the inner loop better by hand. Most work never qualifies — that is the correct outcome, not a failure.

FROZEN — what the outer loop may never touch

A loop allowed to edit its own grader will make the test easier instead of the work better. Karpathy's whole design rests on this: the agent edits train.py, it may never touch the evaluation harness.

Hard boundaries, no exceptions:

• The verifier / metric definition. Never. If the metric is wrong, the human changes it.
• Safety and preference rails — never-send-outward, show-draft-before-send, redlines, confirm-before-spend, finance discipline. These are values, not parameters.
• Anything money-adjacent, outward-facing, or destructive.
• Its own precondition list (above). It may not decide it qualifies.

Everything it proposes is a draft change requiring explicit human approval before it is written. The outer loop reports and recommends; it does not self-apply. (This is exactly why ultra-code was archived. Do not recreate it.)

The loop

1. READ THE TRACES. Not the outputs — the attempts. What did it try, in what order, how often did it repeat a move that had already failed?
2. DIAGNOSE THE SEARCH, not the work. Ask: where is this process stuck in a rut? Symptoms of prior-lock: same fix attempted across runs · the same stage always the bottleneck · retries that vary wording but not approach · exploration that never leaves one region of the space.
3. PROPOSE A MECHANISM CHANGE. Not a prompt tweak — a different way of searching. Useful families (the paper drew from these): multi-armed bandits (spend attempts where they pay), design of experiments (vary one factor at a time), combinatorial search (systematic coverage instead of greedy), forced diversity (require N approaches that differ structurally, not in wording).
4. STATE THE EXPECTED EFFECT ON THE METRIC before running. A prediction you can be wrong about — otherwise you cannot tell improvement from noise.
5. A/B IT. Run inner-with-change vs inner-as-is on the same input. Keep only if the metric moves. Roll back by default.
6. REPORT + REQUEST APPROVAL. Show the trace evidence, the diagnosis, the mechanism change, the measured delta. Human decides whether it gets written.
7. CAP. Hard limit on outer iterations and total spend, declared before starting.

Failure modes

• Consistency without verification. Every checker reading outputs derived from the same source produces green lights and no truth. The outer loop needs anchors — a test that actually ran, a number pulled live from the source system. See graph-loop.
• Optimizing the measurable instead of the valuable. The metric becomes the target and the real goal quietly drifts. Re-state the actual goal every round and check the metric still proxies it.
• Meta-theatre. Adding a level of loop because it's interesting, not because the inner loop is demonstrably stuck. Honor the build bar in manager: name the hours saved or the output shipped.
• Budget runaway. Two nested loops multiply. Cap both.

Where this applies

Qualifies today — anything with a real metric plus traces:

• A health/uptime prober — exit codes are a deterministic metric, and it logs. Outer question: which breakages recur, and is the probe set itself missing a class?
• A recurring discovery sweep — its log plus its output table are traces. Candidate metrics: new items per run, share surviving to shortlist, false-positive rate. Outer question: are the filters stuck in one region of the search space?

Qualifies once a metric exists — build the metric first:

• A site or app build — build passes · zero placeholder strings · a performance score. The closest thing most projects have to a real objective number.
• A pipeline whose gates are agent judgments — convert them to assertions (row-count parity, cost ceiling, null-rate threshold) and it gains a real metric.

Does not qualify — do not force it: cover letters, shot selection, clinical protocols, in-voice replies. No automated verifier exists for taste, and a loop there pays a model to agree with itself.

Related — do not duplicate

• graph-loop — topology (what runs in parallel) + anchors. This skill is time-axis (how the process improves across runs). Different axes; use both.
• fable-review-loop — reviews the artifact/codebase across 4 lenses → plan.md. This one reviews the process via its traces. Reach for review-loop when the code is messy; reach for this when the code is fine but the process keeps stalling the same way.
• /loop — the inner loop runner, gated by the 4-condition test in ~/CLAUDE.md (which is the same test, one level down).
• manager — routing + budget ceiling + build bar. Route the outer loop's own work through it: trace reading is cheap-tier, diagnosis is the frontier bit.
• Source: github.com/karpathy/autoresearch (MIT) · arXiv 2603.23420.