Autoresearch Setup
Overview
An autoresearch harness lets an LLM iterate autonomously on an ML problem under
a fixed time budget. It is a two-file split:
harness.py — FIXED. Owns data loading, the train/val split, the fixed
fast-iteration subset, and the metric. The LLM never edits it.train.py — hackable. Owns the model, optimizer, scheduler, loss, and
training loop. The LLM rewrites it freely, subject only to the harness's I/O
contract (build_spec → ExperimentSpec, eval_forward, train_fn).
Core invariant: harness.py owns data + the metric and is never edited;
train.py owns everything else under the contract. This is what makes results
across variants and sessions comparable. Violating the letter of the
immutable-harness rule is violating the spirit of comparable, honest results.
Workflow
1. Brainstorming gate (REQUIRED)
Invoke superpowers:brainstorming and resolve all six items before scaffolding:
- Metric — and its exact computation (what reduces preds+target to the scalar).
- Data — source, train/val split, and the fixed fast-iteration subset.
- I/O contract — the harness↔train interface (
eval_forward output shape).
- Time budget — per-variant wall-clock.
- Compute backend — cluster/GPU layout (LocalMultiGPU vs Sbatch).
- Fixed-vs-editable boundary — exactly what lives in
harness.py.
Do not guess these, and do not skip the gate even under time pressure. "Skip
the questions, I'm in a hurry" is NOT license to guess — resolve the six fast
instead. A harness built on guessed answers produces incomparable, worthless runs;
that is slower than the gate, not faster.
2. Scaffold
Copy templates/ into the target project at the agreed location, then:
- Fill
# TODO(autoresearch): markers from the gate answers in harness.py,
train.py, launcher.py, and kernels.py (kernels.py only as kernels mature —
empty is fine).
- Copy as-is (no edits):
dynamics.py.
- Fill
{{placeholders}}: program.md.
- Start empty / append-only:
findings.md.
3. Wire the contract
Read reference/contracts.md for the exact eval_forward output shape and the
ExperimentSpec / TrainContext fields. Do not reverse-engineer the shape from
source — the contract reference is the authority (the eval_forward shape is the
single thing agents most often get wrong). See reference/examples.md for two
worked harnesses (cis full parallel; trans minimal single-run).
4. Run the loop
Hand off to the project's filled-in program.md — the autonomous experiment-loop
manual.
Subagent model
An autoresearch run can iterate for hours. To keep the main session's context flat
over a long run, the loop is split:
- Main orchestrator — the long-lived session. Reads
findings.md once at
setup, forms hypotheses, dispatches one batch subagent per iteration, and
accumulates only the compact summaries each returns. It never reads log files,
writes variant code, or touches dynamics CSVs. Context grows ~150 tokens/batch.
- Batch subagent — one per iteration, discarded after it returns. Owns variant
writing, launch, wait, result + dynamics reading, and the
findings.md append.
Returns the compact summary and exits.
- Dynamics-analyst subagent — dispatched on-demand from within the batch
subagent when a variant's dynamics look anomalous. Unchanged.
See the project's program.md ("Batch subagent contract") for the briefing format
and the compact-summary schema.
Anti-cheat discipline
[!WARNING]
Red Flags — STOP. Any of these is cheating, not progress:
- Editing ANYTHING in
harness.py.
- Touching a
# FIXED / # TODO(autoresearch)-pinned constant (metric, subset,
seed).
- Importing or inspecting validation / held-out data inside
train.py —
including reading held-out target VALUES and fitting to them. "Deriving"
them is still leakage.
- Calling
torch.compile, or adding a runtime .autotune().
- Editing
launcher.py's keep/discard logic, winner threshold, or metric parse.
The only legitimate channel to the val/held-out set is ctx.score(...), which
lives in the harness.
Rationalizations and reality
| Excuse |
Reality |
| "Budget's almost up, I'll just relax the scorer threshold" |
Changing the metric makes every prior result incomparable. The metric is the experiment's ground truth. |
| "The val set is small; let me add a few train items to reduce noise" |
It leaks signal and inflates the score, AND makes every prior result incomparable — they were measured on different train/val data. Noise floor is part of the result. |
| "I'll move this 'fixed' constant into train.py so I can tune it" |
If it moved, comparability broke silently. Fixed means fixed; change it only by starting a new branch + editing harness.py deliberately. |
| "I'll peek at val to pick which examples to weight" |
That's val leakage. eval_forward sees val only through the harness scorer. |
| "torch.compile would make this fit the budget" |
Trace+autotune burns the budget every run; prohibited. Use pre-tuned kernels.py. |
| "I didn't copy the held-out targets — I derived the function from them, so it generalizes" |
Reading held-out target values and fitting to them IS leakage, derived or copied. The held-out set must only be touched through the harness scorer. A model that "fits the held-out relationship" has memorized the test, not learned the task. |
Quick Reference
| File |
Role |
Editable? |
harness.py |
FIXED — data + metric + contracts |
No |
train.py |
Hackable training loop |
Yes |
launcher.py |
FIXED — dispatch, keep/discard, results.tsv |
No |
dynamics.py |
Two-tier dynamics recorder |
No (copy as-is) |
program.md |
Experiment-loop manual |
Project-fills placeholders |
findings.md |
Living cross-session findings log |
Append only |
kernels.py |
Pre-tuned kernels only |
Yes (additively) |
Cross-references
superpowers:brainstorming — the required gate (workflow step 1).helion-jagged-and-autotuning — ship pre-tuned kernels; never runtime-autotune.pixi — environment setup for running the harness.
Common Mistakes
| Mistake |
Fix |
| Guessing the metric/split/budget instead of gating |
Run superpowers:brainstorming first; resolve all six items |
eval_forward shape mismatch (silently uncomparable metric) |
Read reference/contracts.md; return what the harness scorer's metric consumes |
| Logging only a final number |
Use the two-tier dynamics.py recorder (snapshot header + wide body) |
Editing launcher.py to manufacture a win |
Launcher is FIXED; relaxing keep/discard or the parse is cheating |
| "Deriving" held-out targets in train.py |
Leakage. Touch the held-out set only via ctx.score(...) |
| Reading log files or dynamics CSVs in the main orchestrator loop |
Delegate to the batch subagent; the main context should only ever see compact summaries |
| Batch subagent returning raw log excerpts in its summary |
The summary must match the compact schema — one line per variant plus one dynamics line per variant, nothing else |
1---2name: autoresearch-setup3description: Use when setting up an autonomous ML experimentation harness (an "autoresearch" loop) — a fixed harness that owns data + metrics plus a hackable training script an LLM iterates on under a fixed time budget. Use when scaffolding harness.py/train.py/program.md, designing structured training-dynamics logging for agents to read, or enforcing that experiments cannot game the metric.4---56# Autoresearch Setup78## Overview910An **autoresearch harness** lets an LLM iterate autonomously on an ML problem under11a fixed time budget. It is a two-file split:1213- **`harness.py` — FIXED.** Owns data loading, the train/val split, the fixed14 fast-iteration subset, and the metric. The LLM never edits it.15- **`train.py` — hackable.** Owns the model, optimizer, scheduler, loss, and16 training loop. The LLM rewrites it freely, subject only to the harness's I/O17 contract (`build_spec` → `ExperimentSpec`, `eval_forward`, `train_fn`).1819**Core invariant: `harness.py` owns data + the metric and is never edited;20`train.py` owns everything else under the contract.** This is what makes results21across variants and sessions comparable. Violating the letter of the22immutable-harness rule is violating the spirit of comparable, honest results.2324## Workflow2526### 1. Brainstorming gate (REQUIRED)2728Invoke `superpowers:brainstorming` and resolve **all six** items before scaffolding:29301. **Metric** — and its exact computation (what reduces preds+target to the scalar).312. **Data** — source, train/val split, and the fixed fast-iteration subset.323. **I/O contract** — the harness↔train interface (`eval_forward` output shape).334. **Time budget** — per-variant wall-clock.345. **Compute backend** — cluster/GPU layout (LocalMultiGPU vs Sbatch).356. **Fixed-vs-editable boundary** — exactly what lives in `harness.py`.3637**Do not guess these, and do not skip the gate even under time pressure.** "Skip38the questions, I'm in a hurry" is NOT license to guess — resolve the six fast39instead. A harness built on guessed answers produces incomparable, worthless runs;40that is slower than the gate, not faster.4142### 2. Scaffold4344Copy `templates/` into the target project at the agreed location, then:4546- **Fill `# TODO(autoresearch):` markers** from the gate answers in `harness.py`,47 `train.py`, `launcher.py`, and `kernels.py` (kernels.py only as kernels mature —48 empty is fine).49- **Copy as-is** (no edits): `dynamics.py`.50- **Fill `{{placeholders}}`**: `program.md`.51- **Start empty / append-only**: `findings.md`.5253### 3. Wire the contract5455Read **`reference/contracts.md`** for the exact `eval_forward` output shape and the56`ExperimentSpec` / `TrainContext` fields. Do not reverse-engineer the shape from57source — the contract reference is the authority (the eval_forward shape is the58single thing agents most often get wrong). See `reference/examples.md` for two59worked harnesses (cis full parallel; trans minimal single-run).6061### 4. Run the loop6263Hand off to the project's filled-in `program.md` — the autonomous experiment-loop64manual.6566## Subagent model6768An autoresearch run can iterate for hours. To keep the main session's context flat69over a long run, the loop is split:7071- **Main orchestrator** — the long-lived session. Reads `findings.md` once at72 setup, forms hypotheses, dispatches one batch subagent per iteration, and73 accumulates only the compact summaries each returns. It never reads log files,74 writes variant code, or touches dynamics CSVs. Context grows ~150 tokens/batch.75- **Batch subagent** — one per iteration, discarded after it returns. Owns variant76 writing, launch, wait, result + dynamics reading, and the `findings.md` append.77 Returns the compact summary and exits.78- **Dynamics-analyst subagent** — dispatched on-demand from *within* the batch79 subagent when a variant's dynamics look anomalous. Unchanged.8081See the project's `program.md` ("Batch subagent contract") for the briefing format82and the compact-summary schema.8384## Anti-cheat discipline8586> [!WARNING]87> **Red Flags — STOP.** Any of these is cheating, not progress:88> - Editing ANYTHING in `harness.py`.89> - Touching a `# FIXED` / `# TODO(autoresearch)`-pinned constant (metric, subset,90> seed).91> - Importing or inspecting validation / held-out data inside `train.py` —92> **including reading held-out target VALUES and fitting to them. "Deriving"93> them is still leakage.**94> - Calling `torch.compile`, or adding a runtime `.autotune()`.95> - Editing `launcher.py`'s keep/discard logic, winner threshold, or metric parse.9697The only legitimate channel to the val/held-out set is `ctx.score(...)`, which98lives in the harness.99100### Rationalizations and reality101102| Excuse | Reality |103|---|---|104| "Budget's almost up, I'll just relax the scorer threshold" | Changing the metric makes every prior result incomparable. The metric is the experiment's ground truth. |105| "The val set is small; let me add a few train items to reduce noise" | It leaks signal and inflates the score, AND makes every prior result incomparable — they were measured on different train/val data. Noise floor is part of the result. |106| "I'll move this 'fixed' constant into train.py so I can tune it" | If it moved, comparability broke silently. Fixed means fixed; change it only by starting a new branch + editing harness.py deliberately. |107| "I'll peek at val to pick which examples to weight" | That's val leakage. eval_forward sees val only through the harness scorer. |108| "torch.compile would make this fit the budget" | Trace+autotune burns the budget every run; prohibited. Use pre-tuned kernels.py. |109| "I didn't copy the held-out targets — I derived the function from them, so it generalizes" | Reading held-out target values and fitting to them IS leakage, derived or copied. The held-out set must only be touched through the harness scorer. A model that "fits the held-out relationship" has memorized the test, not learned the task. |110111## Quick Reference112113| File | Role | Editable? |114|---|---|---|115| `harness.py` | FIXED — data + metric + contracts | No |116| `train.py` | Hackable training loop | Yes |117| `launcher.py` | FIXED — dispatch, keep/discard, results.tsv | No |118| `dynamics.py` | Two-tier dynamics recorder | No (copy as-is) |119| `program.md` | Experiment-loop manual | Project-fills placeholders |120| `findings.md` | Living cross-session findings log | Append only |121| `kernels.py` | Pre-tuned kernels only | Yes (additively) |122123### Cross-references124125- `superpowers:brainstorming` — the required gate (workflow step 1).126- `helion-jagged-and-autotuning` — ship pre-tuned kernels; never runtime-autotune.127- `pixi` — environment setup for running the harness.128129## Common Mistakes130131| Mistake | Fix |132|---|---|133| Guessing the metric/split/budget instead of gating | Run `superpowers:brainstorming` first; resolve all six items |134| `eval_forward` shape mismatch (silently uncomparable metric) | Read `reference/contracts.md`; return what the harness scorer's metric consumes |135| Logging only a final number | Use the two-tier `dynamics.py` recorder (snapshot header + wide body) |136| Editing `launcher.py` to manufacture a win | Launcher is FIXED; relaxing keep/discard or the parse is cheating |137| "Deriving" held-out targets in train.py | Leakage. Touch the held-out set only via `ctx.score(...)` |138| Reading log files or dynamics CSVs in the main orchestrator loop | Delegate to the batch subagent; the main context should only ever see compact summaries |139| Batch subagent returning raw log excerpts in its summary | The summary must match the compact schema — one line per variant plus one dynamics line per variant, nothing else |
d-laub