Autoresearch

Autoresearch: Autonomous Improvement Loop

An AI-driven methodology where you iteratively modify a single target, evaluate changes against an objective scoring function, and compound improvements. Based on Karpathy's autoresearch pattern, proven on 103 experiments achieving 7.6x score improvement.

When to Use

• Optimizing trading strategies against a backtest harness
• Improving code quality metrics (test pass rate, coverage, performance)
• Tuning configuration parameters against benchmarks
• Prompt engineering with measurable evaluation
• Any domain where you have: a mutable target + an objective score

The 5 Pillars

1. Single Mutable Surface

Only ONE file (or narrow set of files) can change per experiment. Everything else is frozen. This constrains the search space and makes experiments reproducible.

2. Immutable Evaluation Harness

The test/scoring system NEVER changes during the loop. If you change both the target and the evaluator, you can't tell what improved.

3. Composite Scoring Function

Reduce multi-dimensional quality to a single number. The scoring function should:

• Reward the primary objective (Sharpe ratio, test pass rate, accuracy)
• Penalize failure modes (drawdown, flakiness, latency)
• Have hard cutoffs for degenerate outputs (crashes, zero results)

score = primary_metric * quality_factor - penalty_1 - penalty_2

4. Git as Experiment Ledger

Every experiment is one commit. If the score improves, keep the commit. If not, git reset --hard HEAD~1. The git history becomes a complete, reproducible record of every hypothesis tested.

5. Autonomous Loop

Once started, the loop runs until interrupted. No human approval per iteration. The scoring function IS the approval.

The Loop Protocol

SETUP:
1. Identify target file(s) — what you'll modify
2. Identify eval command — what you'll run to test
3. Identify score extraction — how to parse the score from output
4. Create experiment branch: git checkout -b autoresearch/<tag>
5. Run baseline: execute eval, record initial score
6. Initialize results log

LOOP (repeat until interrupted):
1. READ current state of target file
2. ANALYZE what has been tried (check results log)
3. PROPOSE a single, focused modification with a clear hypothesis
4. IMPLEMENT the modification in the target file
5. COMMIT: git add <target> && git commit -m "exp<N>: <hypothesis>"
6. TEST: run eval command, capture output
7. EXTRACT score from output
8. EVALUATE:
   - If score > best_score: KEEP (update best_score, log as "KEEP")
   - If score <= best_score: REVERT (git reset --hard HEAD~1, log as "DISCARD")
   - If crashed: REVERT, log error, try different approach
9. LOG result to results.tsv
10. CONTINUE — do NOT stop to ask the user

NEVER:
- Modify the eval harness
- Skip the scoring step
- Keep a change that didn't improve the score
- Stop the loop without being interrupted

Results Tracking

results.tsv format

experiment	score	delta	status	hypothesis
exp0	2.724	+0.000	BASELINE	Initial baseline measurement
exp1	2.962	+0.238	KEEP	Multi-timeframe momentum confirmation
exp2	2.541	-0.421	DISCARD	Reduced position sizing
exp3	3.292	+0.330	KEEP	EMA crossover signal addition

Evolution Log (EXPERIMENTS.md)

For significant experiments, document:

### exp<N> — <title> (<status>, score <X>)
**Hypothesis**: What we expected to happen and why
**Change**: What was actually modified
**Result**: Score delta, why it worked/failed
**Learning**: What this teaches us for future experiments

Research Direction Tiers

Always prioritize experiments by expected impact:

Tier 1 — High probability, proven patterns

• Parameter tuning within known-good ranges
• Adding signals that have worked in related domains
• Simplification (removing features that don't contribute)

Tier 2 — Worth exploring

• New signal types or indicators
• Regime-adaptive behavior
• Cross-asset or cross-domain signals

Tier 3 — Novel / speculative

• Fundamentally different approaches
• Complex feature interactions
• Machine learning components

The Simplicity Principle

From 103 experiments on Nunchi's auto-researchtrading:

The biggest gains came from REMOVING complexity, not adding it. "The Great Simplification" produced +2.0 score by eliminating pyramiding, funding boost, BTC filter, and correlation weighting.

Apply this: After every 10-20 experiments, run a "simplification pass" — systematically remove features one at a time and test if the score improves without them.

Domain Adaptations

Trading Strategy Optimization

• Target: strategy.py
• Eval: uv run backtest.py or equivalent
• Score: grep "^score:" output
• See: ~/.claude/skills/autoresearch/templates/aster-trading.md

Code Quality Improvement

• Target: source file(s) under test
• Eval: pytest --tb=short -q or project test command
• Score: Parse pass/fail counts, compute pass_rate * coverage_factor
• Hard cutoffs: any test that previously passed now fails = -999

Prompt Engineering

• Target: prompt template file
• Eval: Run prompt against test cases, score responses
• Score: accuracy * consistency - latency_penalty - cost_penalty
• Log: Track each prompt version with its scores

Configuration Tuning

• Target: config file (YAML, JSON, TOML)
• Eval: Integration test or benchmark suite
• Score: success_rate * performance_factor - error_penalty

Agent Definition Improvement

• Target: agent .md file
• Eval: Run agent against benchmark prompts, score outputs
• Score: relevance * quality - hallucination_penalty

Integration with Existing Skills

Anti-Patterns

• Changing the evaluator mid-loop — Invalidates all prior comparisons
• Keeping changes without score improvement — "It looks better" is not evidence
• Multiple changes per experiment — Can't attribute improvement to any single change
• Stopping too early — The best improvements often come after 50+ experiments
• Ignoring the simplification pass — Complexity accumulates; periodically prune
• Over-fitting to the scoring function — If score seems too good, check for degenerate behavior