# Autoresearch

> A hypothesis-generating loop that validates itself, run entirely inside Claude Code. Claude proposes a change, runs a fixed-budget experiment, measures a deterministic metric, then git-commits wins and git-reverts losses — repeating ~10–100x while you're AFK. The harness is `/loop` (single agent iterating) or `/agent-teams` (a proposer + a validator/judge running in parallel) — no external scripts, full progress visible in the session. Use whenever there's a single scalar metric to push (lower KL / RMSE / loss / latency, higher accuracy / score / win-rate / pass-rate) and the validation runs end-to-end without human judgment. Works on simulators, fitting routines, portfolio optimizers, eval pass rates, hyperparameter tuning, solver tuning, fuzzing, perf budgets, prompt scores — anything optimizable. Triggers: "autoresearch", "run autoresearch", "optimize this overnight", "AFK optimize", "autonomous experiments", "iterate until <metric> hits <target>".

- Skill: `jonmumm/autoresearch` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jonmumm/autoresearch`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jonmumm/autoresearch/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: jonmumm (https://skillmd.com/u/jonmumm)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jonmumm/autoresearch

---


# Autoresearch — Self-Validating Hypothesis Loop

Autoresearch is a **pattern**, not a package. The shape is: agent generates a
hypothesis → agent runs a bounded experiment → the experiment produces a number →
git keeps the change if the number improved (and regressions still pass), reverts
otherwise. Repeat for hours.

Everything runs **inside Claude Code**. The harness is one of two skills you
already have:

- **`/loop`** — Claude reruns the same prompt on a cadence (or self-paced),
  driving experiments one at a time. Progress streams into the session as it
  happens, so you can watch the metric move.
- **`/agent-teams`** — spin up a proposer agent and an independent
  validator/judge agent. The proposer changes code; the validator runs the
  metric script + regression tests and decides commit vs. revert. Two
  perspectives, harder to game.

No `loop.sh`, no external script, no separate framework. You write `program.md`
describing the metric and the rules, then invoke `/loop` or `/agent-teams` with
that file as the brief.

It applies to anything where:

1. A single scalar metric defines "better" (lower or higher).
2. One run of the validation script produces that metric, end-to-end, without human input.
3. The metric is reproducible (or stable enough that noise < signal per run).

> **Goodhart warning.** The agent will optimize the metric *exactly*, including
> degenerate paths (overfitting val set, exploiting timing noise, removing safety
> checks). Define the metric so the only way to move it is the way you want, and
> always pair it with a regression test suite that must also pass before commit.

## Canonical example: f1-dk race-model fitting

The f1-dk codebase **is already running autoresearch** — every recent commit on
`main` is a kept experiment:

```
9c731f1 Drop per-driver alphas (parameterless win): -22 params, -2.25 held-out KL
2ea6592 Extend Powell+bounds to Phase 1 + Phase 3 (codex Rank 6 fully applied)
1baf367 Multi-start + ensemble — best held-out KL yet (53.05)
```

The shape:

| Loop piece | f1-dk implementation |
|---|---|
| **Metric** | Held-out KL on a frozen race set (lower better, current best 53.05) |
| **Validator** | `python train.py …` printing the KL, deterministic with fixed seeds |
| **Modifiable** | `train.py` — parameterization, optimizer choice, multi-start, bounds |
| **Frozen** (per `CLAUDE.md`) | `prepare.py`, `config.py`, scoring rules in `test_scoring.py` |
| **Regression gate** | `pytest test_scoring.py -v` (DK 2026 rules must still pass) |
| **Keep/revert ledger** | `git log` — commit messages carry the metric delta |
| **Harness** | `/loop` proposing changes, or `/agent-teams` (proposer + validator) |

The loop is just "propose a change in `train.py`, run the fitter, parse KL,
compare to last commit, run pytest, commit or revert." Run that for a night and
the agent claws the metric down a slice at a time.

## When to reach for autoresearch

✅ **Good fit** — deterministic and/or optimizable:
- Numerical fitting / inference: f1-dk's Thurstone race model, any MLE/MAP loop with held-out likelihood
- Simulators with a fitness metric: portfolio expected value, betting-odds RMSE, lineup `mean(max-score-per-sim)`
- Hyperparameter search where grid/random is too coarse
- Algorithmic tuning: solver iterations, beam width, sampling temp, ensemble weights
- Performance budgets: bundle size, p99 latency, peak RSS
- Fuzzing/property-test minimization: shortest counterexample, highest crash count
- Eval pass-rate against a frozen rubric or labeled set

❌ **Wrong fit:**
- Subjective quality (UI polish, copy tone) — use `/autodesign` or a calibrated LLM judge
- Tasks needing human-in-the-loop confirmation each step
- Metrics that aren't reproducible or whose run cost is hours, not minutes
- Anywhere "kept" experiments can break unrelated behavior without the regression gate noticing

## Core loop

```
human writes program.md (metric, modifiable files, frozen files, regression gate, budget)
    ↓
/loop or /agent-teams reads program.md and current code
    ↓
proposes a change (architecture, hparam, algorithm, ensemble weight…)
    ↓
runs the validator: bounded budget (5 min / N sims / fixed seed)
    ↓
parses metric → compares vs. last committed best
    ↓
improved AND regression tests pass → git commit
not improved OR regressions → git reset --hard HEAD
    ↓
repeat (~10–100 experiments per session)
```

`git log --oneline` is the results table. Every commit is a real improvement on
a real metric.

## Setup pattern (works in any project)

Three pieces in the repo. Build once, then the harness does the rest.

### 1. `program.md` — the agent's standing orders

Your leverage point. Specify:

- **Goal** — single scalar metric, direction (lower/higher better), current best,
  target if you have one. *"Lower held-out KL on `data/2026/holdout.yaml`. Current
  best 53.05. Target < 50."*
- **What's fair game** — files the agent may modify. *"Modify only `train.py` and
  `model.py`. `prepare.py`, `config.py`, `scoring.py` are frozen."*
- **Validation command** — exact command, exact line that holds the metric.
  *`python train.py --quiet | tail -1`* and *"line is `KL=<float>`"*.
- **Time budget per experiment** — wall-clock cap so attempts are comparable.
- **Regression gates** — tests that must pass before commit.
- **Commit/revert protocol** — strict `<` or `>`, regression tests must pass,
  commit message format: `autoresearch: <metric> (was <prev>) — <hypothesis>`.
- **Heuristics** — "small changes first", "if 3 attempts in a direction fail
  switch directions", "explain the hypothesis in 2 sentences before each change",
  "after a kept commit try a *different* axis."

### 2. Validation script — deterministic, bounded, single number

- Print the metric on its own parseable line (`METRIC=0.4523`, `KL=53.05`, etc.).
- Fix all randomness (seeds, data order). Noise must be smaller than the
  improvements you care about.
- Cap runtime. If validation can blow past the budget, the loop stalls.

### 3. Regression tests — `pytest` (or equivalent) that must pass

The agent runs these *before* committing. If the metric improved but tests fail,
it discards. This is the guard against Goodhart.

## Launch

### Option A: `/loop` (single agent, simplest)

In Claude Code at the project root:

```
/loop Read program.md. Run one autoresearch iteration:
  1. Propose one change to a modifiable file, with a 2-sentence hypothesis.
  2. Run the validator command, parse the metric.
  3. If metric strictly improved AND `pytest -q` passes:
       git add -A && git commit -m "autoresearch: <metric> (was <prev>) — <hypothesis>"
     else:
       git reset --hard HEAD
  4. Append one row to results.tsv: <metric>, <kept|discarded>, <hypothesis>.
  Stop after this iteration — /loop will fire the next one.
```

Omit the interval to let Claude self-pace. Each tick is one experiment, fully
visible in the session — you see the diff, the metric, the keep/revert decision.

### Option B: `/agent-teams` (proposer + validator, harder to game)

For longer runs or when you don't trust a single agent to police itself:

```
/agent-teams Run autoresearch on program.md.
  Roles:
    Proposer — reads program.md and the current code, proposes one change with a
      hypothesis, makes the edit, hands off to Validator.
    Validator — runs the validator command, parses the metric, runs the
      regression tests, decides commit vs. revert. Writes the result to
      results.tsv. Hands back to Proposer for the next iteration.
  Stop condition: 50 attempts, OR target metric reached, OR human interrupts.
```

The validator never proposes; the proposer never decides. Splitting the roles
makes it much harder for the loop to "succeed" by gaming the metric, since the
validator has no incentive to accept a sketchy diff.

> **First-time check.** Watch the first 2–3 iterations before going AFK. Verify
> the validator is actually being run, the metric is being parsed, and
> commits/reverts are happening. The most common silent failure: the agent
> "tries" a change without ever running the validator.

## More recipes

### Recipe: Portfolio / lineup optimization (f1-dk)

- **Metric**: `python eval_portfolio.py --quiet` → `mean(max(lineup_score per sim))`. Higher better.
- **Modifiable**: `portfolio.py`, `optimize.py`.
- **Frozen**: scoring rules, race model.
- **Regression gate**: `pytest test_scoring.py` + a lineup-validity check (salary cap, roster rules).
- **Budget**: `--sims 10000` cap.

### Recipe: Hyperparameter / solver tuning

- **Metric**: solver objective on a held-out instance set.
- **Modifiable**: a single `config.yaml`.
- **Regression gate**: solver still produces a feasible solution on every instance.
- **Budget**: per-instance timeout × number of instances.

### Recipe: Performance budgets

- **Metric**: p99 latency / bundle size / peak RSS.
- **Modifiable**: hot-path code, build config.
- **Regression gate**: full functional test suite + golden output diff. *Do not* skip — this recipe is the easiest to silently break correctness.
- **Budget**: per-bench wall-time × repetitions for noise control.

### Recipe: Prompt / eval-set tuning

- **Metric**: pass-rate or F1 against a frozen labeled set, judged by a frozen LLM judge (`temperature=0`, pinned model version).
- **Modifiable**: prompt files.
- **Regression gate**: held-out canary set the agent never sees.
- **Budget**: cost-cap (max $X per experiment) since judge calls dominate.
- Watch Goodhart hard — judges are gameable. Calibrate against human labels first.

## Tips for getting real results

1. **Smoke-test the validator first.** One manual run, end-to-end. If you can't reliably get a metric out, the agent can't either.
2. **Tighten `program.md` over time.** Most "agent wandered" failures are underspecified instructions.
3. **Most experiments fail.** 10–20 keepers per 100 attempts is normal. The git log filters to wins automatically.
4. **Watch for noise > signal.** If repeated runs of the same code give different metrics by more than a typical improvement, fix the seed / increase samples / lengthen the budget *before* starting the loop.
5. **Strict comparison.** Use `<` / `>` not `≤` / `≥`. Otherwise the agent drifts on within-noise "neutral" changes.
6. **Commit message = experiment description.** Include a one-line hypothesis in each commit. `git log` becomes a research diary.
7. **Prefer `/agent-teams` for unattended overnight runs** — the proposer/validator split closes the most common Goodhart failure.

## Output artifacts after a session

- **`git log --oneline`** — chronological list of wins, each with the metric in its message.
- **`results.tsv`** — every attempt with metric, duration, kept/discarded, hypothesis. Lets you plot metric vs. experiment number.
- **Session transcript** — because everything ran in Claude Code, the proposals, runs, and decisions are all in one scrollback.

## Failure modes

| Symptom | Cause | Fix |
|---|---|---|
| Agent never invokes the validator | `program.md` underspecifies the loop | Make the validator command mandatory in the launch prompt; require its output be quoted before the keep/revert decision. |
| Metric improves but tests break | Missing regression gate | Add `pytest`/equivalent step before `git commit`. Revert if it fails. |
| Same metric forever | Validator is too noisy or too short | Fix seed; lengthen budget; aggregate over multiple seeds. |
| Agent loops on the same change | Not tracking what's been tried | Append every attempt to `results.tsv`; instruct it to read the file before proposing. |
| All experiments time out | Budget too tight or change exploded runtime | Lower problem size; raise budget; cap memory. |
| Massive metric jump that looks fake | Goodhart — agent gamed the metric (overfit, removed asserts, hardcoded answer) | Inspect the diff. Add the gamed path as a regression test. Switch to `/agent-teams` so a separate validator inspects each diff. |