Autoresearch
Use this skill when the goal is to improve a specific, measurable outcome through repeated automated experiments — write a change, measure it, keep it or revert it, repeat. You define the metric; the agent runs the loop autonomously until interrupted or the budget is exhausted.
This is distinct from test-driven development (writing tests for behavior) and systematic debugging (diagnosing a known failure). Use this skill when optimization through iteration is the work.
Use this skill when
- You want to iteratively improve a measurable outcome: execution time, bundle size, test pass rate, build time, latency, coverage, complexity, or a custom benchmark score.
- You have a metric command that produces a numeric result and you want the agent to run experiments autonomously until interrupted or a budget is reached.
- The right solution is not obvious upfront and you want to explore the space empirically.
- You want to hill-climb: keep changes that help, discard changes that do not, and log everything.
Do not use this skill when
- The task is writing tests for a new feature or behavior (use
test-driven-development).
- The goal is diagnosing why something is broken (use
systematic-debugging).
- There is no measurable metric — the outcome is purely qualitative or subjective.
- The codebase does not have a git repository or the user has not confirmed a dedicated experiment branch is acceptable.
Routing boundary
| Situation |
Use this skill? |
Route instead |
| Writing failing tests before implementing a feature |
No |
test-driven-development |
| Diagnosing why a test or build is failing |
No |
systematic-debugging |
| Iteratively optimizing a benchmark or metric |
Yes |
— |
| Improving code quality with no numeric signal |
No |
systematic-debugging or verification-before-completion |
| Running a single targeted experiment to validate a hypothesis |
No |
Implement directly; this skill is for multi-experiment autonomous loops |
Inputs to gather
Required before editing
- Goal: what you are trying to improve (e.g., execution time, pass rate, bundle size).
- Metric command: the exact shell command that produces the metric (e.g.,
npm run benchmark, go test -bench=. ./..., hyperfine './build.sh').
- Metric extraction: how to pull the numeric value from the output (regex, JSON field, specific line).
- Direction:
lower_is_better or higher_is_better.
- In-scope files/directories: what the agent is allowed to edit.
- Out-of-scope files/directories: what must not be touched.
Helpful if present
- Max experiments: a count or
unlimited (default: unlimited, stop only on interrupt).
- Constraints: time budget per run, dependency policy, test-passing requirement, API compatibility, memory limits.
- Simplicity policy: default is "simpler is better — weigh complexity cost against improvement magnitude."
Only investigate if encountered
- Existing benchmark history or prior result logs.
- CI-enforced test gates that must stay green throughout the loop.
First move
- Confirm all required inputs are known. If the metric command or extraction method is unclear, ask before creating the branch.
- Create a dedicated experiment branch:
git checkout -b autoresearch/<tag> (use today's date as the tag, e.g., autoresearch/2025-07-09).
- Run the metric command on the unmodified code and record the baseline value before any experiments begin.
Workflow
Phase 1: Setup confirmation
Summarize all parameters in a table before proceeding:
| Parameter |
Value |
| Goal |
… |
| Metric command |
… |
| Metric extraction |
… |
| Direction |
lower is better / higher is better |
| In-scope files |
… |
| Out-of-scope files |
… |
| Constraints |
… |
| Max experiments |
… |
| Simplicity policy |
… |
Do not start the loop until confirmed.
Phase 2: Branch and baseline
- Create branch:
git checkout -b autoresearch/<tag>.
- Read all in-scope files to build full context of the current state.
- Initialize
results.tsv in the repo root with the header row (see references/experiment-guide.md for the TSV format). Add results.tsv and run.log to .git/info/exclude so they stay untracked without modifying any tracked file.
- Run the metric command on unmodified code. Record as experiment
0 with status baseline.
- Report baseline to the user before starting the loop.
Phase 3: Experiment loop
Run until MAX_EXPERIMENTS is reached or the user interrupts. Do not stop to ask permission between iterations.
For each experiment:
- Think — analyze previous results and current code. Generate a hypothesis. Consult
references/experiment-guide.md for strategy ordering (low-hanging fruit first, diversify after plateaus, combine winners, etc.).
- Edit — modify only in-scope files. Keep changes focused and minimal per experiment.
- Commit —
git add <changed files> && git commit -m "experiment: <short description>".
- Run — execute the metric command. Redirect to
run.log (<command> > run.log 2>&1).
- Measure — extract the metric from
run.log. If extraction fails, read the last 50 lines for the error.
- Decide:
- Improved: keep the commit. Update the running best.
- Same or worse: revert with a revert commit —
git revert HEAD --no-edit. Log status discard.
- Crash/error: if a quick single-line fix is obvious, apply it in a new commit and re-run once. If still broken after one fix attempt, revert with
git revert HEAD --no-edit (or revert both commits if two were made) and log status crash. Do not use --amend or reset --hard.
- Log — append one row to
results.tsv: experiment_number commit_hash metric_value status description.
- Continue — go to step 1.
Phase 4: Report
When the loop ends:
- Print the full
results.tsv as a formatted table.
- Summarize: total experiments, kept / discarded / crashed counts, baseline vs. final metric, improvement percentage, top 3 most impactful changes.
- Show the kept-commit log:
git log --oneline <start_commit>..HEAD.
- Recommend next steps: ideas that were too risky or complex for the automated loop.
Outputs
autoresearch/<tag> branch containing only kept (improving) experiment commits.
results.tsv (untracked) — full experiment journal.
run.log (untracked) — output from the most recent metric run.
- A written Phase 4 summary report in the chat.
Guardrails
- Never modify out-of-scope files, even to make a failing metric run succeed.
- Never skip the measurement step. Every experiment must be measured before the keep/discard decision.
- Never use
git reset --hard as a revert strategy. Always revert with a new commit (git revert HEAD --no-edit) to preserve history and avoid data loss.
- Never use
git commit --amend in the loop. Amended history makes the results log unreliable.
- Never install new dependencies or change environment configuration without explicit user approval.
- Keep changes per experiment small. Large batched changes hide which modification drove the metric.
- Do not keep a change that improves the metric but regresses another metric the user named as a constraint.
Validation
Smoke tests:
- should trigger: "I want you to keep running experiments on my Go benchmark until you find a 10% speedup"
- should not trigger: "Write a failing test for the
parseUser function before I implement it" (→ test-driven-development)
- should not trigger: "My CI pipeline is failing and I don't know why" (→
systematic-debugging)
Before each reporting step, apply verification-before-completion: confirm the results.tsv row count matches the experiment count and the final metric was measured from the last run.
Examples
- "Run experiments on
src/compute.ts to reduce the npm run benchmark p95 latency. Keep going until I stop you."
- "I want to minimize
go test -bench=. ./... ns/op for the BenchmarkSort function. File a result log and tell me what you tried."
- "Hill-climb my Rust build time. The metric is
time cargo build --release, lower is better. Only touch Cargo.toml and src/."
Reference files
references/experiment-guide.md — experiment strategy order, TSV format spec, git safety patterns, and constraint handling
assets/setup-template.md — fillable setup table to confirm inputs before the loop starts
1---2name: autoresearch3description: "Use when running autonomous iterative experiments to optimize a measurable metric on an existing codebase — not for writing tests or diagnosing a specific failure."4license: GNU GPL v35metadata:6 source: https://skillmd.com/skills/matt-riley/autoresearch7 version: 2.0.1 # x-release-please-version8 category: workflow9 audience: general-coding-agent10 maturity: draft11 kind: task12---1314# Autoresearch1516Use this skill when the goal is to improve a specific, measurable outcome through repeated automated experiments — write a change, measure it, keep it or revert it, repeat. You define the metric; the agent runs the loop autonomously until interrupted or the budget is exhausted.1718This is distinct from test-driven development (writing tests for behavior) and systematic debugging (diagnosing a known failure). Use this skill when optimization through iteration is the work.1920## Use this skill when2122- You want to iteratively improve a measurable outcome: execution time, bundle size, test pass rate, build time, latency, coverage, complexity, or a custom benchmark score.23- You have a metric command that produces a numeric result and you want the agent to run experiments autonomously until interrupted or a budget is reached.24- The right solution is not obvious upfront and you want to explore the space empirically.25- You want to hill-climb: keep changes that help, discard changes that do not, and log everything.2627## Do not use this skill when2829- The task is writing tests for a new feature or behavior (use `test-driven-development`).30- The goal is diagnosing why something is broken (use `systematic-debugging`).31- There is no measurable metric — the outcome is purely qualitative or subjective.32- The codebase does not have a git repository or the user has not confirmed a dedicated experiment branch is acceptable.3334## Routing boundary3536| Situation | Use this skill? | Route instead |37| --- | --- | --- |38| Writing failing tests before implementing a feature | No | `test-driven-development` |39| Diagnosing why a test or build is failing | No | `systematic-debugging` |40| Iteratively optimizing a benchmark or metric | Yes | — |41| Improving code quality with no numeric signal | No | `systematic-debugging` or `verification-before-completion` |42| Running a single targeted experiment to validate a hypothesis | No | Implement directly; this skill is for multi-experiment autonomous loops |4344## Inputs to gather4546**Required before editing**4748- **Goal**: what you are trying to improve (e.g., execution time, pass rate, bundle size).49- **Metric command**: the exact shell command that produces the metric (e.g., `npm run benchmark`, `go test -bench=. ./...`, `hyperfine './build.sh'`).50- **Metric extraction**: how to pull the numeric value from the output (regex, JSON field, specific line).51- **Direction**: `lower_is_better` or `higher_is_better`.52- **In-scope files/directories**: what the agent is allowed to edit.53- **Out-of-scope files/directories**: what must not be touched.5455**Helpful if present**5657- **Max experiments**: a count or `unlimited` (default: unlimited, stop only on interrupt).58- **Constraints**: time budget per run, dependency policy, test-passing requirement, API compatibility, memory limits.59- **Simplicity policy**: default is "simpler is better — weigh complexity cost against improvement magnitude."6061**Only investigate if encountered**6263- Existing benchmark history or prior result logs.64- CI-enforced test gates that must stay green throughout the loop.6566## First move67681. Confirm all required inputs are known. If the metric command or extraction method is unclear, ask before creating the branch.692. Create a dedicated experiment branch: `git checkout -b autoresearch/<tag>` (use today's date as the tag, e.g., `autoresearch/2025-07-09`).703. Run the metric command on the unmodified code and record the baseline value before any experiments begin.7172## Workflow7374### Phase 1: Setup confirmation7576Summarize all parameters in a table before proceeding:7778| Parameter | Value |79| --- | --- |80| Goal | … |81| Metric command | … |82| Metric extraction | … |83| Direction | lower is better / higher is better |84| In-scope files | … |85| Out-of-scope files | … |86| Constraints | … |87| Max experiments | … |88| Simplicity policy | … |8990Do not start the loop until confirmed.9192### Phase 2: Branch and baseline93941. Create branch: `git checkout -b autoresearch/<tag>`.952. Read all in-scope files to build full context of the current state.963. Initialize `results.tsv` in the repo root with the header row (see `references/experiment-guide.md` for the TSV format). Add `results.tsv` and `run.log` to `.git/info/exclude` so they stay untracked without modifying any tracked file.974. Run the metric command on unmodified code. Record as experiment `0` with status `baseline`.985. Report baseline to the user before starting the loop.99100### Phase 3: Experiment loop101102Run until `MAX_EXPERIMENTS` is reached or the user interrupts. Do not stop to ask permission between iterations.103104For each experiment:1051061. **Think** — analyze previous results and current code. Generate a hypothesis. Consult `references/experiment-guide.md` for strategy ordering (low-hanging fruit first, diversify after plateaus, combine winners, etc.).1072. **Edit** — modify only in-scope files. Keep changes focused and minimal per experiment.1083. **Commit** — `git add <changed files> && git commit -m "experiment: <short description>"`.1094. **Run** — execute the metric command. Redirect to `run.log` (`<command> > run.log 2>&1`).1105. **Measure** — extract the metric from `run.log`. If extraction fails, read the last 50 lines for the error.1116. **Decide**:112 - **Improved**: keep the commit. Update the running best.113 - **Same or worse**: revert with a revert commit — `git revert HEAD --no-edit`. Log status `discard`.114 - **Crash/error**: if a quick single-line fix is obvious, apply it in a new commit and re-run once. If still broken after one fix attempt, revert with `git revert HEAD --no-edit` (or revert both commits if two were made) and log status `crash`. Do not use `--amend` or `reset --hard`.1157. **Log** — append one row to `results.tsv`: `experiment_number commit_hash metric_value status description`.1168. **Continue** — go to step 1.117118### Phase 4: Report119120When the loop ends:1211221. Print the full `results.tsv` as a formatted table.1232. Summarize: total experiments, kept / discarded / crashed counts, baseline vs. final metric, improvement percentage, top 3 most impactful changes.1243. Show the kept-commit log: `git log --oneline <start_commit>..HEAD`.1254. Recommend next steps: ideas that were too risky or complex for the automated loop.126127## Outputs128129- `autoresearch/<tag>` branch containing only kept (improving) experiment commits.130- `results.tsv` (untracked) — full experiment journal.131- `run.log` (untracked) — output from the most recent metric run.132- A written Phase 4 summary report in the chat.133134## Guardrails135136- Never modify out-of-scope files, even to make a failing metric run succeed.137- Never skip the measurement step. Every experiment must be measured before the keep/discard decision.138- Never use `git reset --hard` as a revert strategy. Always revert with a new commit (`git revert HEAD --no-edit`) to preserve history and avoid data loss.139- Never use `git commit --amend` in the loop. Amended history makes the results log unreliable.140- Never install new dependencies or change environment configuration without explicit user approval.141- Keep changes per experiment small. Large batched changes hide which modification drove the metric.142- Do not keep a change that improves the metric but regresses another metric the user named as a constraint.143144## Validation145146Smoke tests:147148- should trigger: "I want you to keep running experiments on my Go benchmark until you find a 10% speedup"149- should not trigger: "Write a failing test for the `parseUser` function before I implement it" (→ `test-driven-development`)150- should not trigger: "My CI pipeline is failing and I don't know why" (→ `systematic-debugging`)151152Before each reporting step, apply `verification-before-completion`: confirm the `results.tsv` row count matches the experiment count and the final metric was measured from the last run.153154## Examples155156- "Run experiments on `src/compute.ts` to reduce the `npm run benchmark` p95 latency. Keep going until I stop you."157- "I want to minimize `go test -bench=. ./...` ns/op for the `BenchmarkSort` function. File a result log and tell me what you tried."158- "Hill-climb my Rust build time. The metric is `time cargo build --release`, lower is better. Only touch `Cargo.toml` and `src/`."159160## Reference files161162- `references/experiment-guide.md` — experiment strategy order, TSV format spec, git safety patterns, and constraint handling163- `assets/setup-template.md` — fillable setup table to confirm inputs before the loop starts