Autoresearch Fleet

Autoresearch Fleet

Autonomous research loop inspired by karpathy/autoresearch. One mutable file, one immutable eval harness, git as state machine, and a "NEVER STOP" directive. The agent modifies code, evaluates the result, keeps improvements, discards regressions, and repeats indefinitely.

Open-world extension: when the agent plateaus (N consecutive discards), the orchestrator injects a web-search prompt, breaking through knowledge ceilings the LLM can't cross alone.

When to use

• Optimizing a single metric (latency, accuracy, loss, score)
• The problem has a fast, deterministic eval harness
• You want autonomous overnight runs (100+ experiments while you sleep)
• The search space is too large for manual exploration

How it works

┌─────────────────────────────────────────────────┐
│                 orchestrator.sh                  │
│                                                  │
│  for each iteration:                             │
│    1. Count trailing discards in results.tsv     │
│    2. If >= plateau_threshold → search prompt    │
│    3. Spawn agent (claude -p, codex exec, or pi -p)      │
│    4. Agent reads program.md, edits file, evals  │
│    5. Agent updates results.tsv, keeps/reverts   │
│    6. Check stop conditions (iter/cost/plateau)  │
│    7. Loop                                       │
└─────────────────────────────────────────────────┘

The agent handles everything: reading files, editing code, running eval, committing, updating results.tsv, and reverting on failure. The orchestrator just loops, switches prompts on plateau, and enforces stop conditions.

Directory structure

$FLEET_ROOT/                    # The problem directory
  fleet.json                    # Fleet config
  program.md                    # Agent instructions (you write this)
  eval.py                       # Immutable eval harness (you write this)
  solution.py                   # Mutable file (agent edits this)
  results.tsv                   # Experiment log (agent updates, git-untracked)
  orchestrator.sh               # Generated by launch.sh
  .orch-state.json              # Iteration state
  .paused                       # Sentinel (touch to pause)
  logs/
    session-iter-1.jsonl         # Per-iteration session logs
    session-iter-2.jsonl
    ...

fleet.json schema

{
  "fleet_name": "optimize-api-latency",
  "type": "autoresearch",
  "config": {
    "model": "sonnet",
    "fallback_model": "haiku",
    "provider": "claude",
    "budget_per_iter": 2.00,
    "max_turns": 0
  },
  "problem": {
    "workdir": "/home/user/my-project",
    "eval_command": "make benchmark",
    "metric_regex": "^p99_latency_ms:\\s*([0-9.]+)",
    "metric_direction": "minimize",
    "results_file": "results.tsv",
    "program_md": "program.md"
  },
  "stop_when": {
    "max_iterations": 30,
    "cost_cap_usd": 15.0
  },
  "search": {
    "enabled": true,
    "plateau_threshold": 3
  }
}

Config fields

Required inputs

You need 3 things in a fleet root directory:

1. fleet.json — points problem.workdir at the target repo, sets eval_command
2. program.md — agent instructions. Must say NEVER STOP.
3. An eval command — anything that outputs a metric

fleet-root/              ← fleet.json + program.md + logs
  fleet.json
  program.md
  logs/                  ← created automatically
your-repo/               ← workdir (agent operates here)
  src/...
  results.tsv            ← created automatically

When the user doesn't specify a benchmark

If the user gives you a repo and a goal but no eval command:

1. Check for existing benchmarks: look for Makefile targets (make benchmark, make perf), package.json scripts (npm run bench, yarn test), pytest markers (pytest -m benchmark), or bench/ directories.
2. If found: use it as eval_command. Set metric_regex if it doesn't print a single number.
3. If not found: write a benchmark script (bench.sh or bench.py) in the workdir that:
   • Runs the relevant operation (API call, function invocation, build, test suite)
   • Measures the metric the user cares about (latency, pass rate, bundle size, etc.)
   • Prints a single number to stdout
   • Exits 0 on success, non-zero on crash
4. Set eval_command to run this script.

Example: user says "optimize API latency in my Express app":

#!/usr/bin/env bash
# bench.sh — measure p99 latency
npm start &>/dev/null &
PID=$!
sleep 3
RESULT=$(curl -s -o /dev/null -w '%{time_total}' http://localhost:3000/api/health)
kill $PID 2>/dev/null
echo "$RESULT"

Then set "eval_command": "bash bench.sh" in fleet.json.

Setup

1. Create fleet root with fleet.json + program.md (+ bench.sh if you wrote one)
2. bash ${AGENTS_SKILLS_DIR}/scripts/launch.sh <fleet-root> (git init + results.tsv auto-created)
3. bash ${AGENTS_SKILLS_DIR}/scripts/status.sh <fleet-root> to monitor

Available scripts

program.md template

Your program.md should follow this structure (adapt to your problem):

# autoresearch: <problem description>

## Setup
1. Explore the codebase to understand the architecture.
2. Read `results.tsv` for prior experiment history.
3. Run `<eval_command>` to establish a baseline.

## Rules
- Goal: <minimize|maximize> the metric.
- Make ONE change per experiment. Keep changes focused.
- <any constraints: don't touch tests, don't modify config, etc.>

## The experiment loop
LOOP FOREVER:
1. Read results.tsv for context on what's been tried.
2. Make ONE change to the codebase.
3. `git add -A && git commit -m "short description"`
4. Run: `<eval_command>`
5. Record in results.tsv (tab-separated): `commit  metric  status  description`
6. If metric improved: keep the commit.
7. If worse or crash: `git reset --hard HEAD~1` and log as discard/crash.
8. Go to step 1.

**NEVER STOP.** Run until manually interrupted.

Key design principles (from Karpathy)

1. Git as state machine — improvement = advance branch; regression = reset
2. Fixed eval — makes all experiments comparable
3. results.tsv as shared memory — agent reads history to avoid repeating failures
4. NEVER STOP — agent runs autonomously until killed
5. Simplicity criterion — a small gain with ugly complexity is not worth it

Open-world search (the extension)

When search.enabled is true, the orchestrator counts trailing discards in results.tsv. If the count exceeds search.plateau_threshold, the next iteration gets a search-augmented prompt telling the agent to use WebSearch before coding.

This is validated: in experiment 009, search found Winograd's Strassen variant (15 additions vs 18) — a technique not in the LLM's training data — breaking through a plateau where vanilla autoresearch was stuck.

Critical: plateau detection is done in bash (deterministic), not by the LLM. The agent miscounted consecutive discards in early experiments, hallucinating plateaus. External counting is mandatory.

Rationalizations to reject

$ARGUMENTS