Autoresearch

Autoresearch experimentation loop

Use this skill to turn a measurable programming goal into a git-backed autonomous loop that creates experiments, runs the metric command, keeps improvements, reverts regressions, and reports a research journal.

When to invoke

• "Run autonomous experiments to improve this metric."
• "Optimize code by trying changes automatically."
• "Use an autoresearch loop for performance tuning, coverage, latency, throughput, memory, or build time."
• "Hill climb this benchmark or test score."
• "Keep iterating until the experiment budget is reached."

Prerequisites and context

• Requires git, a git repository, and terminal access.
• Requires a measurable metric with an exact command, extraction rule, and direction.
• Inspired by Karpathy's autoresearch: https://github.com/karpathy/autoresearch.
• Do not use for one-shot tasks, simple bug fixes, code review, or tasks without a measurable metric.

Setup parameters

Before any experiment, collect and confirm every parameter. Do not assume or skip any item.

Confirm setup in a table and do not proceed until the user confirms.

| Parameter | Value |
| --- | --- |
| Goal | <goal> |
| Metric command | `<METRIC_COMMAND>` |
| Metric extraction | `<METRIC_EXTRACTION>` |
| Direction | `lower_is_better` | `higher_is_better` |
| In-scope files | `<IN_SCOPE_FILES>` |
| Out-of-scope files | `<OUT_OF_SCOPE_FILES>` |
| Constraints | `<CONSTRAINTS>` |
| Max experiments | `<MAX_EXPERIMENTS>` |
| Simplicity policy | `<SIMPLICITY_POLICY>` |

Procedure

1. Create a branch with a date tag such as autoresearch/mar17: git checkout -b autoresearch/<tag>.
2. Read all in-scope files to understand the current state.
3. Create results.tsv in the repo root with header experiment commit metric status description.
4. Append results.tsv and run.log to .git/info/exclude if not already present, so they stay untracked without modifying tracked files.
5. Run the baseline metric command on unmodified code and record experiment 0 with status baseline.
6. Report the baseline metric and start the loop.
7. For each experiment, THINK, EDIT, COMMIT, RUN, MEASURE, DECIDE, LOG, and CONTINUE until MAX_EXPERIMENTS is reached or the user interrupts.
8. When the loop ends, print results.tsv, summarize kept/discarded/crashed experiments, compare baseline and final metrics, show git log --oneline <start_commit>..HEAD, and recommend next steps.

Experiment loop

Crash handling: attempt a quick fix for typo, import, or simple error, amend the experiment commit with git commit --amend, and rerun. If unfixable after 2 attempts, revert the experiment with git reset --hard HEAD~1 and log status crash.

Constraint handling:

• If a run exceeds 2x the expected duration, kill it and treat it as a crash.
• If constraints require tests to pass, run them before and after each kept change and revert failures.
• Monitor memory and resource limits; revert when usage exceeds stated limits.
• Do not install new dependencies or make environment changes unless the user approved it.
• Do not keep regressions unless the user explicitly allowed a trade-off.

Results log

results.tsv is the research journal and has exactly five tab-separated columns:

experiment	commit	metric	status	description
0	a1b2c3d	0.997900	baseline	unmodified code
1	b2c3d4e	0.993200	keep	increase learning rate to 0.04
2	c3d4e5f	1.005000	discard	switch to GeLU activation
3	d4e5f6g	0.000000	crash	double model width (OOM)

All experiments happen on autoresearch/<tag>. Failed experiments are reverted with git reset --hard HEAD~1; successful experiments advance the branch. results.tsv and run.log remain untracked through .git/info/exclude.

Gotchas

• No metric means no autoresearch: every experiment must produce a comparable numeric result.
• Commit before run: committing each attempt makes clean reverts possible.
• Do not skip the baseline: without experiment 0, improvement percentage is meaningless.
• Do not ask mid-loop: once confirmed, run autonomously until budget or interruption.
• Complexity is a cost: a small improvement that adds ugly complexity may not be worth keeping under the simplicity policy.

Autoresearch vocabulary and examples

Preserve the original loop vocabulary because users paste it into requests and logs: LOOP, IMPROVED, SAME, WORSE, CRASH, LIMITS, JSON, keeping/discarding, out-of-scope, read-only, files/dirs, code/complexity, trade-offs, risky/complex, before/after, crash/error, shell-appropriate, Memory/resources**, metric_name, and in. Example metric commands include dotnet test, dotnet test --logger trx, npm run benchmark, time ./build.sh, pytest --tb=short, and hyperfine './my-program' for my-program.

Output template

### Autoresearch result

**Status:** running | complete | interrupted | blocked
**Branch:** `autoresearch/<tag>`
**Goal:** <goal>
**Metric:** `<METRIC_COMMAND>` using `<METRIC_EXTRACTION>`; `lower_is_better` | `higher_is_better`
**Scope:** `<IN_SCOPE_FILES>`; off-limits `<OUT_OF_SCOPE_FILES>`

| Experiment | Commit | Metric | Status | Description |
| --- | --- | --- | --- | --- |
| 0 | `<hash>` | `<value>` | baseline | unmodified code |
| 1 | `<hash>` | `<value>` | keep | <description> |
| 2 | `<hash>` | `<value>` | discard | <description> |
| 3 | `<hash>` | `0.000000` | crash | double model width (OOM) |

**Summary**
- Total experiments: <count>
- Kept / discarded / crashed: <counts>
- Starting metric: <value>
- Final metric: <value>
- Improvement: <percent>
- Top 3 changes: <list>

**Kept commits**
- `<git log --oneline <start_commit>..HEAD>`

**Recommended next steps**
- <human research idea or next experiment family>

Quality gate

• Goal, METRIC_COMMAND, METRIC_EXTRACTION, METRIC_DIRECTION, IN_SCOPE_FILES, OUT_OF_SCOPE_FILES, CONSTRAINTS, MAX_EXPERIMENTS, and SIMPLICITY_POLICY are collected and confirmed.
• Baseline is measured before any code change and logged as experiment 0.
• The branch name follows autoresearch/<tag>.
• results.tsv and run.log are added to .git/info/exclude rather than tracked.
• Every experiment is committed before running and measured afterward.
• Same or worse results are reverted with git reset --hard HEAD~1 unless an approved trade-off exists.
• Crashes are fixed at most twice, amended with git commit --amend, or reverted and logged as crash.
• The final report includes the full results table, metric comparison, kept git log, and recommended next steps.

References

• Karpathy autoresearch