# Outer Loop > Bilevel loop engineering — put a loop on top of a loop. When a recurring process already runs (a sweep, a pipeline, a review cycle) but keeps getting stuck the same way, this reads its traces, finds where the SEARCH is stuck rather than where the output is wrong, and proposes a change to the mechanism itself. Based on Karpathy's AutoResearch loop plus Qu & Lu's "Bilevel Autoresearch" (arXiv 2603.23420), which got a verified 5x over a single loop using the SAME model at both levels — the gain is architectural, not intelligence. Requires an inner loop with traces and a metric; refuses to run without them. Triggers - "outer loop", "/outer-loop", "bilevel", "loop on the loop", "why does this loop keep getting stuck", "improve the process not the output", "meta-loop". - Skill: `rishbjain1/outer-loop` (Agent Skill) - Install (CLI): `npx skillmds@latest add rishbjain1/outer-loop` - Raw SKILL.md: https://api.skillmd.com/api/skills/rishbjain1/outer-loop/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: rishbjain1 (https://skillmd.com/u/rishbjain1) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/rishbjain1/outer-loop --- # /outer-loop — improve the process, not the output An **inner loop** optimizes the work: propose → run → measure → keep or roll back. An **outer loop** optimizes *how the inner loop searches*: read its traces, find where the search itself is stuck, change the mechanism, run again. The finding worth internalizing (Qu & Lu, verified): **both levels used the same LLM.** The 5x came from architecture, not a smarter model. And crucially — *parameter-level adjustments without mechanism changes yielded no reliable gains.* Tuning the knobs does nothing. Changing **how it searches** is the whole effect. **What their outer loop actually found:** the inner loop kept falling back on the model's **priors** — retrying familiar moves even after those moves stopped working. The outer loop broke the pattern by forcing exploration in directions the model's instincts avoid. **Prior-lock is the default failure of any repeated agent process.** That is what you are hunting. ## PRECONDITION — refuse to run without these four An outer loop is expensive and easy to fool. It only earns its cost when the inner loop has: 1. **A metric.** A number that moves, in the right direction, without a human deciding. (`val_bpb`, tests passing, exit code, row-count parity, build time, cost per run.) *No metric → stop. Say so plainly and offer to build the metric first.* 2. **Traces.** Logs of what the inner loop actually tried and what happened — not just the final artifact. You cannot diagnose a search you cannot see. 3. **Repetition.** It has run enough times that a *pattern* exists to find. A handful of runs is anecdote, not a stuck search. 4. **A budget that can absorb it.** Two levels of loop burn far more than one. If any fail: **do not build the outer loop.** Fix the missing piece, or run the inner loop better by hand. Most work never qualifies — that is the correct outcome, not a failure. ## FROZEN — what the outer loop may never touch A loop allowed to edit its own grader will make the test easier instead of the work better. Karpathy's whole design rests on this: the agent edits `train.py`, it may **never** touch the evaluation harness. **Hard boundaries, no exceptions:** - **The verifier / metric definition.** Never. If the metric is wrong, the *human* changes it. - **Safety and preference rails** — never-send-outward, show-draft-before-send, redlines, confirm-before-spend, finance discipline. These are values, not parameters. - **Anything money-adjacent, outward-facing, or destructive.** - **Its own precondition list** (above). It may not decide it qualifies. Everything it proposes is a **draft change requiring explicit human approval before it is written.** The outer loop reports and recommends; it does not self-apply. *(This is exactly why `ultra-code` was archived. Do not recreate it.)* ## The loop 1. **READ THE TRACES.** Not the outputs — the *attempts*. What did it try, in what order, how often did it repeat a move that had already failed? 2. **DIAGNOSE THE SEARCH, not the work.** Ask: where is this process stuck in a rut? Symptoms of prior-lock: same fix attempted across runs · the same stage always the bottleneck · retries that vary wording but not approach · exploration that never leaves one region of the space. 3. **PROPOSE A MECHANISM CHANGE.** Not a prompt tweak — a different way of searching. Useful families (the paper drew from these): multi-armed bandits (spend attempts where they pay), design of experiments (vary one factor at a time), combinatorial search (systematic coverage instead of greedy), forced diversity (require N approaches that differ structurally, not in wording). 4. **STATE THE EXPECTED EFFECT ON THE METRIC** before running. A prediction you can be wrong about — otherwise you cannot tell improvement from noise. 5. **A/B IT.** Run inner-with-change vs inner-as-is on the same input. Keep only if the metric moves. **Roll back by default.** 6. **REPORT + REQUEST APPROVAL.** Show the trace evidence, the diagnosis, the mechanism change, the measured delta. Human decides whether it gets written. 7. **CAP.** Hard limit on outer iterations and total spend, declared before starting. ## Failure modes - **Consistency without verification.** Every checker reading outputs derived from the same source produces green lights and no truth. The outer loop needs **anchors** — a test that actually ran, a number pulled live from the source system. See `graph-loop`. - **Optimizing the measurable instead of the valuable.** The metric becomes the target and the real goal quietly drifts. Re-state the actual goal every round and check the metric still proxies it. - **Meta-theatre.** Adding a level of loop because it's interesting, not because the inner loop is demonstrably stuck. Honor the build bar in `manager`: name the hours saved or the output shipped. - **Budget runaway.** Two nested loops multiply. Cap both. ## Where this applies **Qualifies today — anything with a real metric plus traces:** - **A health/uptime prober** — exit codes are a deterministic metric, and it logs. Outer question: *which breakages recur, and is the probe set itself missing a class?* - **A recurring discovery sweep** — its log plus its output table are traces. Candidate metrics: new items per run, share surviving to shortlist, false-positive rate. Outer question: *are the filters stuck in one region of the search space?* **Qualifies once a metric exists — build the metric first:** - **A site or app build** — build passes · zero placeholder strings · a performance score. The closest thing most projects have to a real objective number. - **A pipeline whose gates are agent judgments** — convert them to assertions (row-count parity, cost ceiling, null-rate threshold) and it gains a real metric. **Does not qualify — do not force it:** cover letters, shot selection, clinical protocols, in-voice replies. No automated verifier exists for taste, and a loop there pays a model to agree with itself. ## Related — do not duplicate - **`graph-loop`** — topology (what runs in parallel) + anchors. This skill is time-axis (how the process improves across runs). Different axes; use both. - **`fable-review-loop`** — reviews the *artifact/codebase* across 4 lenses → `plan.md`. This one reviews the *process* via its traces. Reach for review-loop when the code is messy; reach for this when the code is fine but the process keeps stalling the same way. - **`/loop`** — the inner loop runner, gated by the 4-condition test in `~/CLAUDE.md` (which is the same test, one level down). - **`manager`** — routing + budget ceiling + build bar. Route the outer loop's own work through it: trace reading is cheap-tier, diagnosis is the frontier bit. - **Source:** github.com/karpathy/autoresearch (MIT) · arXiv 2603.23420.