# Autoresearch > Autonomous experimentation loop. Given a codebase and a metric, run bounded experiments autonomously — modify code, measure, keep or revert, repeat. Inspired by Karpathy's autoresearch: agent modifies code within constraints, evaluates via a single metric, logs everything. Agent collaborates with the human to write the strategy (program), then executes the loop. Use for: performance optimization, bundle size reduction, latency tuning, algorithm improvement, architecture exploration, or any open-ended 'make this number better' task. Triggers on: 'autoresearch', 'experiment loop', 'optimize this metric', 'run experiments overnight', 'autonomous optimization', 'tune this', 'make this faster/smaller/better'. - Skill: `arjunkmrm/autoresearch` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add arjunkmrm/autoresearch` - Raw SKILL.md: https://api.skillmd.com/api/skills/arjunkmrm/autoresearch/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: arjunkmrm (https://skillmd.com/u/arjunkmrm) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/arjunkmrm/autoresearch --- # Autoresearch Autonomous experimentation. Agent and human design the strategy together, then the agent runs the loop. ## Core Idea ``` while budget remains: hypothesize → modify → run → measure → keep or revert → log ``` Autoresearch explores an open-ended space. There's no "done" — only "better than before." ## Setup When a user invokes autoresearch, collaborate with them to set up the experiment. Walk through each of these steps interactively: ### 1. Understand the Codebase Read the in-scope files for full context. Understand what's being optimized and how the system works before proposing anything. ### 2. Agree on the Program The agent drafts the program based on conversation with the human. A good program defines: - **Metric**: One number. Automated. Comparable across experiments. Which direction is better (lower/higher). How to measure it (command to run, output to parse). - **Scope**: What files/areas the agent may modify. What's off-limits (measurement code, test fixtures, data prep, etc.). - **Constraints**: Invariants that must hold (e.g., "all tests must pass", "don't add dependencies", "don't change the API surface"). Soft constraints too (e.g., "some VRAM increase is acceptable for meaningful gains"). - **Simplicity criterion**: All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Removing something and getting equal or better results is a great outcome. - **Strategy hints**: What directions to explore first, what's been tried, what's promising. Rough priority order. - **Run command**: The exact command to run an experiment and how to extract the metric from the output. See [references/program-guide.md](references/program-guide.md) for writing guidance and [references/examples/ml-training.md](references/examples/ml-training.md) for a complete real-world program (ML training optimization, from [karpathy/autoresearch](https://github.com/karpathy/autoresearch)). ### 3. Set Up the Branch Create a dedicated experiment branch: `git checkout -b autoresearch/` from the current main branch. Propose a tag based on context (e.g., date, metric name). ### 4. Initialize Results Tracking Create a `results.tsv` (tab-separated) with header row. Leave it untracked by git — it's a local log, not part of the branch history. ``` commit metric status description ``` ### 5. Run Baseline The first experiment is always the baseline — run the code as-is to establish the starting metric. Record it in results.tsv. ### 6. Confirm and Go Confirm the setup looks good with the human, then begin the autonomous loop. ## The Loop Once setup is confirmed, the agent runs autonomously: ``` LOOP FOREVER: 1. Look at current state: branch, last results, journal history 2. Hypothesize a change based on strategy + what's been tried 3. Modify in-scope files 4. git commit 5. Run experiment (redirect output to run.log — do NOT flood context) 6. Extract metric from output 7. If metric improved → KEEP (advance branch) If metric regressed or lateral → REVERT (git reset to previous commit) If crashed → diagnose, maybe fix and retry, or log as crash and move on 8. Log to results.tsv (commit hash, metric, status, description) 9. Goto 1 ``` ### Crash Handling If a run crashes (OOM, bug, timeout, etc.): - **Easy fix** (typo, missing import): fix and re-run - **Fundamentally broken idea**: log as crash, revert, move on - **Stuck after a few attempts**: give up on that direction, move on ### Timeout If an experiment exceeds 2x the expected duration, kill it and treat as a failure. ### Never Stop Once the loop begins, do NOT pause to ask the human if you should continue. The human may be away and expects autonomous operation. If you run out of ideas, think harder — re-read in-scope files for new angles, try combining previous near-misses, try more radical changes. The loop runs until the human interrupts you. **Tip**: Use `/loop 10m` to periodically check on the experiment — read `results.tsv`, verify the branch is advancing, and flag if the loop appears stuck. ### Decision Rules - **Keep**: metric strictly improved - **Revert**: metric regressed OR stayed the same (avoid drift) - **Flag for human**: metric improved but change is architecturally risky or surprising - **Stop exploring direction**: 3 consecutive reverts in same direction → try something else ### Results Log The results log is the primary deliverable — not just the final code, but the full exploration history. ``` commit metric status description a1b2c3d 0.9979 keep baseline b2c3d4e 0.9932 keep increase LR to 0.04 c3d4e5f 1.0050 discard switch to GeLU activation d4e5f6g 0.0000 crash double model width (OOM) ``` ## Key Principles 1. **One change per experiment** — if you change two things and it improves, you don't know which helped 2. **Always revert on regression** — never accumulate failed experiments 3. **Fixed environment = fair comparison** — same machine, same data, same conditions across experiments 4. **Log everything** — the results log is how the human reviews what happened 5. **Strategy is collaborative, execution is autonomous** — human and agent design the program together; the agent provides tireless iteration 6. **Exploit AND explore** — don't just hill-climb the first thing that works; periodically try orthogonal directions 7. **Simplicity wins** — a small improvement that adds ugly complexity is not worth it; deleting code for equal results is a win ## When to Use This Autoresearch is best for **open-ended optimization** — when you have a metric to improve but no single correct answer. The agent explores many directions, keeps what works, reverts what doesn't. **Good fit**: "Make API response 2x faster", "reduce bundle size", "improve throughput", "get the lowest val_bpb" **Bad fit**: Tasks with a single known target or pass/fail gate (e.g., "port this to a new platform") — those are convergence problems, not exploration problems.