Autoresearch

Autoresearch

Autonomous experimentation. Agent and human design the strategy together, then the agent runs the loop.

Core Idea

while budget remains:
  hypothesize → modify → run → measure → keep or revert → log

Autoresearch explores an open-ended space. There's no "done" — only "better than before."

Setup

When a user invokes autoresearch, collaborate with them to set up the experiment. Walk through each of these steps interactively:

1. Understand the Codebase

Read the in-scope files for full context. Understand what's being optimized and how the system works before proposing anything.

2. Agree on the Program

The agent drafts the program based on conversation with the human. A good program defines:

• Metric: One number. Automated. Comparable across experiments. Which direction is better (lower/higher). How to measure it (command to run, output to parse).
• Scope: What files/areas the agent may modify. What's off-limits (measurement code, test fixtures, data prep, etc.).
• Constraints: Invariants that must hold (e.g., "all tests must pass", "don't add dependencies", "don't change the API surface"). Soft constraints too (e.g., "some VRAM increase is acceptable for meaningful gains").
• Simplicity criterion: All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Removing something and getting equal or better results is a great outcome.
• Strategy hints: What directions to explore first, what's been tried, what's promising. Rough priority order.
• Run command: The exact command to run an experiment and how to extract the metric from the output.

See references/program-guide.md for writing guidance and references/examples/ml-training.md for a complete real-world program (ML training optimization, from karpathy/autoresearch).

3. Set Up the Branch

Create a dedicated experiment branch: git checkout -b autoresearch/<tag> from the current main branch. Propose a tag based on context (e.g., date, metric name).

4. Initialize Results Tracking

Create a results.tsv (tab-separated) with header row. Leave it untracked by git — it's a local log, not part of the branch history.

commit	metric	status	description

5. Run Baseline

The first experiment is always the baseline — run the code as-is to establish the starting metric. Record it in results.tsv.

6. Confirm and Go

Confirm the setup looks good with the human, then begin the autonomous loop.

The Loop

Once setup is confirmed, the agent runs autonomously:

LOOP FOREVER:
1. Look at current state: branch, last results, journal history
2. Hypothesize a change based on strategy + what's been tried
3. Modify in-scope files
4. git commit
5. Run experiment (redirect output to run.log — do NOT flood context)
6. Extract metric from output
7. If metric improved → KEEP (advance branch)
   If metric regressed or lateral → REVERT (git reset to previous commit)
   If crashed → diagnose, maybe fix and retry, or log as crash and move on
8. Log to results.tsv (commit hash, metric, status, description)
9. Goto 1

Crash Handling

If a run crashes (OOM, bug, timeout, etc.):

• Easy fix (typo, missing import): fix and re-run
• Fundamentally broken idea: log as crash, revert, move on
• Stuck after a few attempts: give up on that direction, move on

Timeout

If an experiment exceeds 2x the expected duration, kill it and treat as a failure.

Never Stop

Once the loop begins, do NOT pause to ask the human if you should continue. The human may be away and expects autonomous operation. If you run out of ideas, think harder — re-read in-scope files for new angles, try combining previous near-misses, try more radical changes. The loop runs until the human interrupts you.

Tip: Use /loop 10m to periodically check on the experiment — read results.tsv, verify the branch is advancing, and flag if the loop appears stuck.

Decision Rules

• Keep: metric strictly improved
• Revert: metric regressed OR stayed the same (avoid drift)
• Flag for human: metric improved but change is architecturally risky or surprising
• Stop exploring direction: 3 consecutive reverts in same direction → try something else

Results Log

The results log is the primary deliverable — not just the final code, but the full exploration history.

commit	metric	status	description
a1b2c3d	0.9979	keep	baseline
b2c3d4e	0.9932	keep	increase LR to 0.04
c3d4e5f	1.0050	discard	switch to GeLU activation
d4e5f6g	0.0000	crash	double model width (OOM)

Key Principles

1. One change per experiment — if you change two things and it improves, you don't know which helped
2. Always revert on regression — never accumulate failed experiments
3. Fixed environment = fair comparison — same machine, same data, same conditions across experiments
4. Log everything — the results log is how the human reviews what happened
5. Strategy is collaborative, execution is autonomous — human and agent design the program together; the agent provides tireless iteration
6. Exploit AND explore — don't just hill-climb the first thing that works; periodically try orthogonal directions
7. Simplicity wins — a small improvement that adds ugly complexity is not worth it; deleting code for equal results is a win

When to Use This

Autoresearch is best for open-ended optimization — when you have a metric to improve but no single correct answer. The agent explores many directions, keeps what works, reverts what doesn't.

Good fit: "Make API response 2x faster", "reduce bundle size", "improve throughput", "get the lowest val_bpb" Bad fit: Tasks with a single known target or pass/fail gate (e.g., "port this to a new platform") — those are convergence problems, not exploration problems.