Autoresearch

Autoresearch — Autonomous Experiment Loop

Run an autonomous experiment loop: hypothesize, modify code, run experiment, measure metric, keep if improved or revert, repeat. Inspired by karpathy/autoresearch.

Quick Start

If .autoresearch/config.json exists in the project, this is a resume. Skip to Resume Protocol.

Otherwise, proceed with Setup.

Setup

Collect configuration via inline parameters or interactive questions.

Parameters

Collect via inline parameters (see argument-hint in frontmatter) or ask these questions one at a time:

1. Goal — What metric to optimize? (free text)
2. Command — Shell command to run the experiment?
3. Metric extraction — Choose one:
   • metric_pattern: Regex with capture group applied to run.log (e.g., "time: (\d+\.?\d*)s")
   • metric_command: Shell command that outputs a single number (e.g., "jq '.score' report.json")
4. Direction — minimize or maximize?
5. Files in scope — Which files may be modified? (comma-separated paths)
6. Timeout — Max seconds per experiment run? (default: 300)
7. Validation command — Optional command that must exit 0 for a "keep" (e.g., npm test)
8. Runs per experiment — For noisy metrics, how many runs to median-average? (default: 1)
9. Max experiments — Safety limit before auto-stop? (default: 200, null for infinite)

Note: Experiment commands may optionally output METRIC name=value lines (one per line, numeric values) to track secondary metrics like VRAM usage, duration, error rate, etc. These are logged but do not affect keep/discard.

Setup Execution

After collecting parameters:

1. Verify the working tree is clean. If dirty, abort with a warning.
2. Generate a session_id from the goal (kebab-case, e.g., minimize-bundle-size-mar12). Validate against ^[a-zA-Z0-9_-]+$. Reject invalid characters.
3. Create branch autoresearch/<session_id> from current HEAD.
4. Add .autoresearch/ to .gitignore if not already present. Commit the change.
5. Create .autoresearch/ directory.
6. Write .autoresearch/config.json — see Config Schema. Compute config_hash as SHA-256 of the config content (excluding the hash field itself). Make the file read-only: chmod 444 .autoresearch/config.json.
7. Initialize .autoresearch/results.tsv with header row:

   experiment    commit    metric    status    duration_s    description
   

8. Write initial .autoresearch/session.md with goal and configuration summary. Copy references/analysis-template.py to .autoresearch/analyze.py if not present. (User can run python .autoresearch/analyze.py for post-hoc analysis.)
9. Read all files in scope for context.
10. Run baseline experiment (no modifications). Record as experiment 0 with status keep.
11. Announce: "Baseline: [metric]. Starting autonomous experiment loop."
12. Enter the Experiment Loop.

Config Schema

Write to .autoresearch/config.json (immutable after creation):

{
  "goal": "minimize bundle size",
  "command": "npm run build 2>&1",
  "metric_pattern": "size: (\\d+\\.?\\d*) KB",
  "metric_command": null,
  "direction": "minimize",
  "files_in_scope": ["webpack.config.js", "src/index.ts"],
  "timeout_seconds": 120,
  "validation_command": "npm test",
  "session_id": "minimize-bundle-size-mar12",
  "max_experiments": 200,
  "runs_per_experiment": 1,
  "created_at": "2026-03-12T10:00:00Z",
  "config_hash": "sha256:..."
}

One of metric_pattern or metric_command must be non-null.

Experiment Loop

REPEAT:
  0. Check for STOP sentinel and max_experiments limit
  1. Derive state from results.tsv (best metric, experiment count)
  2. Formulate hypothesis (avoid repeating recent discards)
  3. Edit in-scope files to implement hypothesis
  4. Run experiment with timeout
  5. Extract and validate metric
  6. Run validation command (if configured)
  7. Compare metric against current best
  8. Keep (commit) or discard (checkout)
  9. Append result to results.tsv
  10. Every 10 experiments: update session.md, back up results.tsv

Step 0: Check Stop Conditions

Before each iteration:

# Graceful stop — user or outer agent creates this file anytime
if [ -f .autoresearch/STOP ]; then
  # Append "stopped" entry to results.tsv
  # Write final summary to session.md
  # Remove the STOP file
  # Exit the loop
fi

Also check if experiment count has reached max_experiments. If so, stop gracefully.

Step 1: Derive State

All state comes from results.tsv. Do NOT maintain a separate state file. Parse the TSV to find: best metric value, best commit, experiment count. For maximize, sort descending; for minimize, sort ascending.

Step 2: Formulate Hypothesis

Generate an experiment idea based on:

• The current state of in-scope files
• Recent experiment history (read last 15 lines of results.tsv)
• Learnings and Ideas Backlog from session.md (see session.md Structure)
• Avoid ideas similar to recent discards

Step 3: Edit Files

Modify only files listed in files_in_scope. Never touch other files.

Step 4: Run Experiment

Run timeout <timeout_seconds> <command> > .autoresearch/run.log 2>&1. If runs_per_experiment > 1: run N times, take the median metric.

Step 5: Extract and Validate Metric

Extract primary metric from .autoresearch/run.log using metric_pattern (regex with capture group) or metric_command (shell command outputting a number). Validate the result is numeric — if not, treat as a crash.

Secondary metrics (optional, best-effort):

grep '^METRIC ' .autoresearch/run.log
# Parses lines matching: METRIC <name>=<value>  (value must be numeric)
# Multiple lines with same name: last occurrence wins
# Malformed values: silently ignored

Append parsed secondary metrics to the description field in results.tsv:

Reduced dependencies [loss=0.5, vram_mb=4200]

Secondary metrics are informational — they do NOT affect keep/discard decisions. The experiment command can output METRIC lines alongside normal output. If no METRIC lines are found, the description has no bracketed suffix.

Step 6: Validation Command

If validation_command is configured, run it. Non-zero exit → treat as discard.

Step 7: Compare Metric

if direction == "minimize": improved = (new_metric < best_metric)
if direction == "maximize": improved = (new_metric > best_metric)

Step 8: Keep or Discard

Confidence scoring (after 3+ keeps): Compute MAD (Median Absolute Deviation) over the last 5 kept metrics. Report confidence: high (<1% relative MAD), medium (<5%), or low (≥5%). Purely informational — does NOT change keep/discard. When low, note in session.md (consider runs_per_experiment).

If improved AND validation passed (KEEP):

git add <files_in_scope>
git commit -m "autoresearch(N): <description> [metric: old → new]"

If NOT improved or validation failed (DISCARD):

git checkout -- <file1> <file2> ...   # Scoped to in-scope files only

If CRASH (non-zero exit, timeout, or metric extraction failure):

1. Read tail -30 .autoresearch/run.log for diagnosis.
2. If trivial fix (typo, import, syntax): fix, re-run. Max 2 fix attempts.
3. After 2 failed attempts: git checkout -- <files_in_scope>, log as crash.

Step 9: Log Result

Append one line to .autoresearch/results.tsv:

<experiment_number>	<commit_hash_or_dash>	<metric_or_dash>	<keep|discard|crash>	<duration_seconds>	<1-2 sentence description>

Use - for commit hash on discard/crash. Use - for metric on crash.

Step 10: Periodic Maintenance

Every 10 experiments:

• Update session.md with new learnings, dead ends, and promising directions.
• Back up results: cp .autoresearch/results.tsv .autoresearch/results.tsv.bak
• Regenerate the dashboard (see below).

After experiment 25 in a session: write all learnings to session.md and prepare for potential context reset (flush all state to disk).

Dashboard Generation

Generate .autoresearch/dashboard.html every 10 experiments, on session end (STOP or max_experiments), and on resume. Uses a template-copy approach:

1. Read references/dashboard-template.html from the skill directory.
2. Serialize results.tsv + config.json into a JSON blob:

   const DATA = {
     goal: "<goal>", direction: "<direction>",
     best: <best_metric>, baseline: <baseline_metric>,
     experiments: [
       { experiment: 0, metric: 1.023, status: "keep", duration_s: 45, description: "Baseline" },
       ...
     ]
   };
   

   Include the last 50 experiments. Use metric: null for crashes.
3. Replace everything between /* __DATA_BLOCK_START__ */ and /* __DATA_BLOCK_END__ */ with the serialized const DATA = {...};.
4. Write the result to .autoresearch/dashboard.html.

Dashboard generation is non-critical. If it fails, log a warning and continue. Never treat a dashboard error as a crash.

Post-Experiment Scope Verification

After each experiment, check git diff --name-only against files_in_scope. Silently revert any out-of-scope changes with git checkout -- <file>.

Loop Rules

1. NEVER STOP. Never ask "should I continue?" The user may be asleep. Run until STOP sentinel exists or max_experiments is reached.
2. Primary metric is king. The configured metric determines keep/discard.
3. Simpler is better. A tiny improvement adding 20 lines of hacky code? Probably not worth it. A tiny improvement from deleting code? Definitely keep.
4. No new dependencies unless the user explicitly permits it.
5. Only modify files in scope. Never touch files outside the list.
6. Redirect ALL output to run.log. Never let output flood the context. Do NOT use tee. Do NOT read the full log — use tail and grep.
7. Read only what you need. Never cat entire files repeatedly. Only re-read in-scope files after a "keep" to see the new state.
8. When stuck, think harder. Re-read in-scope files for new angles, combine previous near-misses, try more radical changes.
9. Avoid repeating failures. Scan recent results.tsv before proposing. grep 'discard' .autoresearch/results.tsv | tail -15
10. Crash budget: 2 fix attempts. Fix trivial errors and retry. After 2 failed fixes, log as crash and move on to a new idea.
11. Strategy rotation. After 10 consecutive discards, re-read all in-scope files and try a fundamentally different approach category.
12. Context checkpoint. Every 10 experiments, write learnings to session.md. After experiment 25, prepare for imminent context reset.

Resume Protocol

Triggered when .autoresearch/config.json exists at skill invocation.

Integrity Checks

1. Read .autoresearch/config.json. Verify it is valid JSON.
2. Recompute SHA-256 of config (excluding config_hash field). Compare to stored config_hash. If mismatch: halt and warn user that config may have been tampered with.
3. Verify git branch autoresearch/<session_id> exists. If not on it: git checkout autoresearch/<session_id>.
4. Verify .autoresearch/results.tsv exists and has at least 1 data row.
   • If results.tsv missing but config exists: re-run baseline only.
   • Validate last line has correct field count. Remove if malformed.
5. If metric_command references a .py file that does not exist, halt and warn user rather than letting the first experiment crash silently.
6. If working tree is dirty: git checkout -- <files_in_scope> to clean up. (Scoped to in-scope files only — never revert unrelated work.)

Reconstruct State

1. Derive best metric, best commit, experiment count from results.tsv.
2. Read .autoresearch/session.md for qualitative learnings.
3. Read last 20 lines of results.tsv for recent experiment history.
4. Re-read in-scope files for current code state.

Resume

1. Regenerate .autoresearch/dashboard.html from current state (see Dashboard Generation).
2. Enter the Experiment Loop. Continue from the current experiment count. All loop rules apply.

session.md Structure

Updated every 10 experiments. Contains ONLY qualitative insights, not numeric status (all numbers derive from results.tsv).

# Autoresearch Session: <goal>

## Learnings
- <Insight from experiment 3: reducing X helps>
- <Insight from experiment 7: approach Y is a dead end>

## Promising Directions
- <High-level strategies worth pursuing>

## Dead Ends (do not retry)
- <Approaches that consistently fail>

## Ideas Backlog
- [ ] <Specific, actionable hypothesis to test>
- [x] ~~<Tried idea>~~ → exp N, status, metric result

Ideas Backlog rules:

• Add ideas during hypothesis generation (Step 2) when you generate multiple but only test one. Record the untested ideas for later.
• Mark ideas as tried after each experiment with result annotation.
• On strategy rotation (10 consecutive discards), review the backlog before re-reading files — untried ideas are your first resort.

Keep session.md under 100 lines. Summarize older learnings and prune completed backlog items when the file grows. On resume, read this file for context.

Auto-Resume

Resume is automatic — re-invoking /autoresearch detects config.json, reconstructs state, and re-enters the loop. No data is lost between sessions.

To run continuously (overnight), configure your agent to re-invoke /autoresearch on context exhaustion. See references/claude-code-auto-resume.md for Claude Code hooks and a universal shell wrapper.

Composite Scoring

When a single extracted number isn't enough — e.g., you need to penalize drawdown alongside Sharpe ratio, or balance accuracy against latency — use a scoring script as your metric_command.

A scoring script reads .autoresearch/run.log, extracts multiple values, computes a composite score, and prints it to stdout. It can also emit METRIC name=value lines so sub-metrics appear in results.tsv and the dashboard. On failure (missing patterns, threshold violations), it exits with code 1, which triggers the normal crash protocol.

Ready-to-use templates in references/scoring/:

• penalty_scorer.py — primary metric minus soft penalties, with hard cutoffs
• weighted_scorer.py — weighted sum of normalized metrics (multi-objective)
• gated_scorer.py — primary metric gated by pass/fail threshold checks

Setup: During interactive setup, if metric_command references a .py file that doesn't exist, offer to copy a scoring template. The user picks one and customizes the # === CUSTOMIZE BELOW === section (regex patterns, thresholds, weights).

Important: Scoring scripts must NOT be listed in files_in_scope. If the agent modifies the scorer during experiments, it changes the definition of success, invalidating all prior results.

Domain Examples

For example configurations across different optimization domains (ML training, build performance, test speed, web performance, code quality, composite scoring), see references/examples.md.