# Autonomous Experimentation > Autonomous Experimentation - Skill: `elophanto/autonomous-experimentation` (Agent Skill) - Install (CLI): `npx skillmds@latest add elophanto/autonomous-experimentation` - Raw SKILL.md: https://api.skillmd.com/api/skills/elophanto/autonomous-experimentation/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: elophanto (https://skillmd.com/u/elophanto) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/elophanto/autonomous-experimentation --- # Autonomous Experimentation ## Description A structured loop for autonomous, metric-driven experimentation. The agent modifies code, measures a metric, keeps improvements, discards regressions, logs everything, and repeats indefinitely. Inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch), which uses this pattern to let an AI agent run ML training experiments overnight — modifying a training script, running 5-minute experiments, keeping what improves val_bpb, discarding what doesn't. This skill generalizes that pattern beyond ML: any measurable optimization target works — code performance, binary size, test coverage, response latency, memory usage, benchmark scores, or any custom metric the user defines. ## Triggers - experiment - optimize - benchmark - "run experiments overnight" - "try things and keep what works" - "improve performance" - autoresearch - "experiment loop" - "try different approaches" - ablation ## Instructions ### Setup Phase Before starting the experiment loop, establish these with the user: 1. **Metric** — What number are you optimizing? Must be machine-readable from command output. - Examples: `val_bpb`, `pytest --tb=no | grep passed`, `time ./benchmark`, `wc -c binary` - Must have a clear direction: lower is better, or higher is better 2. **Target file(s)** — What can the agent modify? - Keep scope narrow. One file is ideal, two or three is acceptable. - Everything else is read-only context. 3. **Run command** — How to measure the metric after each change. - Must be deterministic (or averaged over multiple runs) - Must complete in a bounded time (set a timeout) 4. **Constraints** — What must NOT break? - Tests that must still pass - Resource limits (memory, disk, time) - Code style / complexity preferences 5. **Run tag** — A label for this experiment session (e.g., `mar7-perf`). - Create a git branch: `experiment/` 6. **Experiment journal** — Create `experiments.tsv` with header: ``` commit metric status description ``` Record the baseline as the first entry. ### The Experiment Loop LOOP FOREVER: 1. **Baseline check** — Read the current best metric from the journal. 2. **Hypothesis** — Formulate ONE specific change to try. Write it down before coding. - Draw from: literature in code comments, prior near-misses in the journal, combining successful changes, architectural alternatives, hyperparameter sweeps, simplification (removing code that doesn't help). 3. **Implement** — Modify only the in-scope file(s). Keep changes minimal and focused. - One idea per experiment. Never combine multiple hypotheses. 4. **Commit** — `git commit` the change with a descriptive message. 5. **Run** — Execute the run command, redirecting output to `run.log`: ``` > run.log 2>&1 ``` Do NOT let output flood the context window. 6. **Extract metric** — Parse the metric from `run.log`. - If the run crashed: `tail -n 50 run.log` to diagnose. Try a quick fix (typo, import). If fundamentally broken, log as `crash` and revert. 7. **Decide** — Compare to baseline: - **Improved**: Keep the commit. Update baseline. Log as `keep`. - **Equal or worse**: `git reset --hard HEAD~1`. Log as `discard`. - **Crash**: Revert. Log as `crash`. 8. **Log** — Append to `experiments.tsv`: ``` ``` 9. **Repeat** — Go to step 2. Never stop. Never ask "should I continue?" ### Simplicity Criterion Borrowed from autoresearch: all else being equal, simpler is better. - A small metric improvement that adds ugly complexity? Probably not worth it. - A small metric improvement from *deleting* code? Definitely keep. - Equal metric but much simpler code? Keep. - Weigh complexity cost against improvement magnitude. ### When Stuck If the last 5+ experiments all got discarded: 1. Re-read the in-scope files for new angles 2. Review the journal for patterns (what kinds of changes worked?) 3. Try more radical changes (architectural, not just parameter tweaks) 4. Try combining two previous near-misses 5. Try the opposite of what you've been doing ### Timeout and Crash Handling - Set a timeout for each run (e.g., 2x the expected duration) - If a run exceeds the timeout, kill it and treat as a crash - If a crash is a simple bug (typo, missing import), fix and re-run - If the idea itself is broken, log as crash and move on - After 3 consecutive crashes, pause and re-evaluate approach ### Integration with EloPhanto - Use `self_modify_source` for changes to EloPhanto's own code - Use `self_run_tests` as the constraint checker (tests must still pass) - Use `goal_create` for long-running experiment sessions with checkpoints - Use the autonomous mind to run experiments in the background - Use `knowledge_write` to save insights from successful experiments ## Examples ### Good: Performance Optimization ``` User: "Optimize the response time of the LLM router. Run experiments overnight." Agent: 1. Establishes metric: average latency from `pytest tests/test_core/test_router.py -v --tb=no` 2. Target file: core/router.py 3. Creates branch: experiment/mar7-router-perf 4. Records baseline: 145ms average 5. Experiment 1: cache provider configs → 138ms → keep 6. Experiment 2: async DNS resolution → 135ms → keep 7. Experiment 3: connection pooling → 131ms → keep 8. Experiment 4: remove fallback retries → 130ms but tests fail → discard 9. ... continues overnight ``` ### Good: ML Training (autoresearch-style) ``` User: "Run autoresearch on train.py. Optimize val_bpb." Agent: 1. Metric: val_bpb (lower is better), extracted via `grep "^val_bpb:" run.log` 2. Target: train.py 3. Run command: `uv run train.py > run.log 2>&1` (5-min budget) 4. Experiments: architecture changes, hyperparameter sweeps, optimizer tweaks 5. Each experiment is a 5-minute training run 6. ~12 experiments/hour, ~100 overnight ``` ### Bad: No Clear Metric ``` User: "Make the code better" Agent should ask: - Better how? Faster? Smaller? More readable? More test coverage? - What specific metric can we measure? - What file(s) should I modify? ``` ### Bad: Too Many Changes at Once ``` Experiment: "Change the optimizer, double the learning rate, and switch to a different architecture" This combines 3 hypotheses. If it improves, you don't know which change helped. If it regresses, you don't know which change hurt. One change at a time. ``` ## Verify - Hypothesis is stated in 'if X then Y because Z' form before the experiment runs - Sample size, duration, and primary metric are committed to in writing before reading any results - Control and treatment are specified concretely (config diff, feature flag, audience filter), not described abstractly - The experiment record stores raw outcome data, not just the conclusion, so it can be re-analyzed later - Results report effect size and a confidence interval (or equivalent uncertainty), not only a point estimate - A 'no decision' or 'inconclusive' branch is allowed in the analysis plan; the agent does not force a winner ## Notes - The experiment journal (`experiments.tsv`) is the ground truth. Always log. - Git branches keep the main branch clean. All experiments happen on `experiment/`. - The agent should never stop to ask if it should continue. The human will interrupt when they want it to stop. - This pattern works best with `full_auto` permission mode and the autonomous mind enabled. - For EloPhanto self-optimization, combine with `self_run_tests` to ensure experiments don't break existing functionality. - Original inspiration: [karpathy/autoresearch](https://github.com/karpathy/autoresearch) — autonomous ML research where an AI agent iterates on a training script overnight, keeping what improves the loss metric and discarding what doesn't.