# Autoresearch

> Runs an autonomous experiment loop that optimizes any measurable metric. Proposes hypotheses, modifies code, measures results, and keeps only improvements — running indefinitely until stopped. Suitable for automated optimization of code performance, ML metrics, build speed, bundle size, test speed, Lighthouse scores, or any numeric target.

- Skill: `neethanwu/autoresearch` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add neethanwu/autoresearch`
- Raw SKILL.md: https://api.skillmd.com/api/skills/neethanwu/autoresearch/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: neethanwu (https://skillmd.com/u/neethanwu)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/neethanwu/autoresearch

---


# Autoresearch — Autonomous Experiment Loop

Run an autonomous experiment loop: hypothesize, modify code, run experiment,
measure metric, keep if improved or revert, repeat. Inspired by
[karpathy/autoresearch](https://github.com/karpathy/autoresearch).

## Quick Start

If `.autoresearch/config.json` exists in the project, this is a **resume**.
Skip to [Resume Protocol](#resume-protocol).

Otherwise, proceed with [Setup](#setup).

---

## Setup

Collect configuration via **inline parameters** or **interactive questions**.

### Parameters

Collect via **inline parameters** (see `argument-hint` in frontmatter) or ask
these questions one at a time:

1. **Goal** — What metric to optimize? (free text)
2. **Command** — Shell command to run the experiment?
3. **Metric extraction** — Choose one:
   - `metric_pattern`: Regex with capture group applied to run.log (e.g., `"time: (\d+\.?\d*)s"`)
   - `metric_command`: Shell command that outputs a single number (e.g., `"jq '.score' report.json"`)
4. **Direction** — `minimize` or `maximize`?
5. **Files in scope** — Which files may be modified? (comma-separated paths)
6. **Timeout** — Max seconds per experiment run? (default: 300)
7. **Validation command** — Optional command that must exit 0 for a "keep" (e.g., `npm test`)
8. **Runs per experiment** — For noisy metrics, how many runs to median-average? (default: 1)
9. **Max experiments** — Safety limit before auto-stop? (default: 200, null for infinite)

**Note:** Experiment commands may optionally output `METRIC name=value` lines
(one per line, numeric values) to track secondary metrics like VRAM usage,
duration, error rate, etc. These are logged but do not affect keep/discard.

### Setup Execution

After collecting parameters:

1. Verify the working tree is clean. If dirty, abort with a warning.
2. Generate a `session_id` from the goal (kebab-case, e.g., `minimize-bundle-size-mar12`).
   Validate against `^[a-zA-Z0-9_-]+$`. Reject invalid characters.
3. Create branch `autoresearch/<session_id>` from current HEAD.
4. Add `.autoresearch/` to `.gitignore` if not already present. Commit the change.
5. Create `.autoresearch/` directory.
6. Write `.autoresearch/config.json` — see [Config Schema](#config-schema).
   Compute `config_hash` as SHA-256 of the config content (excluding the hash field itself).
   Make the file read-only: `chmod 444 .autoresearch/config.json`.
7. Initialize `.autoresearch/results.tsv` with header row:
   ```
   experiment	commit	metric	status	duration_s	description
   ```
8. Write initial `.autoresearch/session.md` with goal and configuration summary.
   Copy `references/analysis-template.py` to `.autoresearch/analyze.py` if not present.
   (User can run `python .autoresearch/analyze.py` for post-hoc analysis.)
9. Read all files in scope for context.
10. Run baseline experiment (no modifications). Record as experiment 0 with status `keep`.
11. Announce: "Baseline: [metric]. Starting autonomous experiment loop."
12. Enter the [Experiment Loop](#experiment-loop).

### Config Schema

Write to `.autoresearch/config.json` (immutable after creation):

```json
{
  "goal": "minimize bundle size",
  "command": "npm run build 2>&1",
  "metric_pattern": "size: (\\d+\\.?\\d*) KB",
  "metric_command": null,
  "direction": "minimize",
  "files_in_scope": ["webpack.config.js", "src/index.ts"],
  "timeout_seconds": 120,
  "validation_command": "npm test",
  "session_id": "minimize-bundle-size-mar12",
  "max_experiments": 200,
  "runs_per_experiment": 1,
  "created_at": "2026-03-12T10:00:00Z",
  "config_hash": "sha256:..."
}
```

One of `metric_pattern` or `metric_command` must be non-null.

---

## Experiment Loop

```
REPEAT:
  0. Check for STOP sentinel and max_experiments limit
  1. Derive state from results.tsv (best metric, experiment count)
  2. Formulate hypothesis (avoid repeating recent discards)
  3. Edit in-scope files to implement hypothesis
  4. Run experiment with timeout
  5. Extract and validate metric
  6. Run validation command (if configured)
  7. Compare metric against current best
  8. Keep (commit) or discard (checkout)
  9. Append result to results.tsv
  10. Every 10 experiments: update session.md, back up results.tsv
```

### Step 0: Check Stop Conditions

Before each iteration:

```bash
# Graceful stop — user or outer agent creates this file anytime
if [ -f .autoresearch/STOP ]; then
  # Append "stopped" entry to results.tsv
  # Write final summary to session.md
  # Remove the STOP file
  # Exit the loop
fi
```

Also check if experiment count has reached `max_experiments`. If so, stop gracefully.

### Step 1: Derive State

All state comes from `results.tsv`. Do NOT maintain a separate state file.
Parse the TSV to find: best metric value, best commit, experiment count.
For `maximize`, sort descending; for `minimize`, sort ascending.

### Step 2: Formulate Hypothesis

Generate an experiment idea based on:
- The current state of in-scope files
- Recent experiment history (read last 15 lines of results.tsv)
- Learnings and **Ideas Backlog** from session.md (see [session.md Structure](#sessionmd-structure))
- Avoid ideas similar to recent discards

### Step 3: Edit Files

Modify only files listed in `files_in_scope`. Never touch other files.

### Step 4: Run Experiment

Run `timeout <timeout_seconds> <command> > .autoresearch/run.log 2>&1`.
If `runs_per_experiment > 1`: run N times, take the **median** metric.

### Step 5: Extract and Validate Metric

Extract primary metric from `.autoresearch/run.log` using `metric_pattern`
(regex with capture group) or `metric_command` (shell command outputting a
number). Validate the result is numeric — if not, treat as a **crash**.

**Secondary metrics** (optional, best-effort):
```bash
grep '^METRIC ' .autoresearch/run.log
# Parses lines matching: METRIC <name>=<value>  (value must be numeric)
# Multiple lines with same name: last occurrence wins
# Malformed values: silently ignored
```

Append parsed secondary metrics to the description field in results.tsv:
```
Reduced dependencies [loss=0.5, vram_mb=4200]
```

Secondary metrics are informational — they do NOT affect keep/discard decisions.
The experiment command can output `METRIC` lines alongside normal output.
If no `METRIC` lines are found, the description has no bracketed suffix.

### Step 6: Validation Command

If `validation_command` is configured, run it. Non-zero exit → treat as discard.

### Step 7: Compare Metric

```
if direction == "minimize": improved = (new_metric < best_metric)
if direction == "maximize": improved = (new_metric > best_metric)
```

### Step 8: Keep or Discard

**Confidence scoring** (after 3+ keeps): Compute MAD (Median Absolute Deviation)
over the last 5 kept metrics. Report `confidence: high` (<1% relative MAD),
`medium` (<5%), or `low` (≥5%). Purely informational — does NOT change
keep/discard. When `low`, note in session.md (consider `runs_per_experiment`).

**If improved AND validation passed (KEEP):**
```bash
git add <files_in_scope>
git commit -m "autoresearch(N): <description> [metric: old → new]"
```

**If NOT improved or validation failed (DISCARD):**
```bash
git checkout -- <file1> <file2> ...   # Scoped to in-scope files only
```

**If CRASH (non-zero exit, timeout, or metric extraction failure):**
1. Read `tail -30 .autoresearch/run.log` for diagnosis.
2. If trivial fix (typo, import, syntax): fix, re-run. Max 2 fix attempts.
3. After 2 failed attempts: `git checkout -- <files_in_scope>`, log as crash.

### Step 9: Log Result

Append one line to `.autoresearch/results.tsv`:

```
<experiment_number>	<commit_hash_or_dash>	<metric_or_dash>	<keep|discard|crash>	<duration_seconds>	<1-2 sentence description>
```

Use `-` for commit hash on discard/crash. Use `-` for metric on crash.

### Step 10: Periodic Maintenance

Every 10 experiments:
- Update `session.md` with new learnings, dead ends, and promising directions.
- Back up results: `cp .autoresearch/results.tsv .autoresearch/results.tsv.bak`
- Regenerate the dashboard (see below).

After experiment 25 in a session: write all learnings to `session.md` and
prepare for potential context reset (flush all state to disk).

### Dashboard Generation

Generate `.autoresearch/dashboard.html` every 10 experiments, on session end
(STOP or max_experiments), and on resume. Uses a template-copy approach:

1. Read `references/dashboard-template.html` from the skill directory.
2. Serialize `results.tsv` + `config.json` into a JSON blob:
   ```json
   const DATA = {
     goal: "<goal>", direction: "<direction>",
     best: <best_metric>, baseline: <baseline_metric>,
     experiments: [
       { experiment: 0, metric: 1.023, status: "keep", duration_s: 45, description: "Baseline" },
       ...
     ]
   };
   ```
   Include the last 50 experiments. Use `metric: null` for crashes.
3. Replace everything between `/* __DATA_BLOCK_START__ */` and
   `/* __DATA_BLOCK_END__ */` with the serialized `const DATA = {...};`.
4. Write the result to `.autoresearch/dashboard.html`.

Dashboard generation is **non-critical**. If it fails, log a warning and
continue. Never treat a dashboard error as a crash.

### Post-Experiment Scope Verification

After each experiment, check `git diff --name-only` against `files_in_scope`.
Silently revert any out-of-scope changes with `git checkout -- <file>`.

---

## Loop Rules

1. **NEVER STOP.** Never ask "should I continue?" The user may be asleep.
   Run until STOP sentinel exists or max_experiments is reached.
2. **Primary metric is king.** The configured metric determines keep/discard.
3. **Simpler is better.** A tiny improvement adding 20 lines of hacky code?
   Probably not worth it. A tiny improvement from deleting code? Definitely keep.
4. **No new dependencies** unless the user explicitly permits it.
5. **Only modify files in scope.** Never touch files outside the list.
6. **Redirect ALL output to run.log.** Never let output flood the context.
   Do NOT use `tee`. Do NOT read the full log — use `tail` and `grep`.
7. **Read only what you need.** Never `cat` entire files repeatedly.
   Only re-read in-scope files after a "keep" to see the new state.
8. **When stuck, think harder.** Re-read in-scope files for new angles,
   combine previous near-misses, try more radical changes.
9. **Avoid repeating failures.** Scan recent results.tsv before proposing.
   `grep 'discard' .autoresearch/results.tsv | tail -15`
10. **Crash budget: 2 fix attempts.** Fix trivial errors and retry.
    After 2 failed fixes, log as crash and move on to a new idea.
11. **Strategy rotation.** After 10 consecutive discards, re-read all
    in-scope files and try a fundamentally different approach category.
12. **Context checkpoint.** Every 10 experiments, write learnings to
    `session.md`. After experiment 25, prepare for imminent context reset.

---

## Resume Protocol

Triggered when `.autoresearch/config.json` exists at skill invocation.

### Integrity Checks

1. Read `.autoresearch/config.json`. Verify it is valid JSON.
2. Recompute SHA-256 of config (excluding `config_hash` field).
   Compare to stored `config_hash`. If mismatch: **halt and warn user**
   that config may have been tampered with.
3. Verify git branch `autoresearch/<session_id>` exists.
   If not on it: `git checkout autoresearch/<session_id>`.
4. Verify `.autoresearch/results.tsv` exists and has at least 1 data row.
   - If results.tsv missing but config exists: re-run baseline only.
   - Validate last line has correct field count. Remove if malformed.
5. If `metric_command` references a `.py` file that does not exist, **halt
   and warn user** rather than letting the first experiment crash silently.
6. If working tree is dirty: `git checkout -- <files_in_scope>` to clean up.
   (Scoped to in-scope files only — never revert unrelated work.)

### Reconstruct State

1. Derive best metric, best commit, experiment count from `results.tsv`.
2. Read `.autoresearch/session.md` for qualitative learnings.
3. Read last 20 lines of `results.tsv` for recent experiment history.
4. Re-read in-scope files for current code state.

### Resume

1. Regenerate `.autoresearch/dashboard.html` from current state
   (see [Dashboard Generation](#dashboard-generation)).
2. Enter the [Experiment Loop](#experiment-loop). Continue from the current
   experiment count. All loop rules apply.

---

## session.md Structure

Updated every 10 experiments. Contains ONLY qualitative insights, not numeric
status (all numbers derive from results.tsv).

```markdown
# Autoresearch Session: <goal>

## Learnings
- <Insight from experiment 3: reducing X helps>
- <Insight from experiment 7: approach Y is a dead end>

## Promising Directions
- <High-level strategies worth pursuing>

## Dead Ends (do not retry)
- <Approaches that consistently fail>

## Ideas Backlog
- [ ] <Specific, actionable hypothesis to test>
- [x] ~~<Tried idea>~~ → exp N, status, metric result
```

**Ideas Backlog rules:**
- Add ideas during hypothesis generation (Step 2) when you generate multiple
  but only test one. Record the untested ideas for later.
- Mark ideas as tried after each experiment with result annotation.
- On strategy rotation (10 consecutive discards), review the backlog before
  re-reading files — untried ideas are your first resort.

Keep session.md under 100 lines. Summarize older learnings and prune completed
backlog items when the file grows. On resume, read this file for context.

---

## Auto-Resume

Resume is automatic — re-invoking `/autoresearch` detects `config.json`,
reconstructs state, and re-enters the loop. No data is lost between sessions.

To run continuously (overnight), configure your agent to re-invoke
`/autoresearch` on context exhaustion. See
[references/claude-code-auto-resume.md](references/claude-code-auto-resume.md)
for Claude Code hooks and a universal shell wrapper.

---

## Composite Scoring

When a single extracted number isn't enough — e.g., you need to penalize
drawdown alongside Sharpe ratio, or balance accuracy against latency — use
a **scoring script** as your `metric_command`.

A scoring script reads `.autoresearch/run.log`, extracts multiple values,
computes a composite score, and prints it to stdout. It can also emit
`METRIC name=value` lines so sub-metrics appear in results.tsv and the
dashboard. On failure (missing patterns, threshold violations), it exits
with code 1, which triggers the normal crash protocol.

**Ready-to-use templates** in `references/scoring/`:
- `penalty_scorer.py` — primary metric minus soft penalties, with hard cutoffs
- `weighted_scorer.py` — weighted sum of normalized metrics (multi-objective)
- `gated_scorer.py` — primary metric gated by pass/fail threshold checks

**Setup:** During interactive setup, if `metric_command` references a `.py`
file that doesn't exist, offer to copy a scoring template. The user picks
one and customizes the `# === CUSTOMIZE BELOW ===` section (regex patterns,
thresholds, weights).

**Important:** Scoring scripts must NOT be listed in `files_in_scope`.
If the agent modifies the scorer during experiments, it changes the
definition of success, invalidating all prior results.

---

## Domain Examples

For example configurations across different optimization domains (ML training,
build performance, test speed, web performance, code quality, composite
scoring), see [references/examples.md](references/examples.md).