Computational Experiments

Computational Experiments

Lifecycle skill for algorithmic research projects where the code IS the scientific contribution.

Modes

Default: Full. Detect mode from user request or ask if ambiguous.

--scaffold Flag

Sets a stage progression template for the experiment campaign. Templates provide structured checklists and exit criteria for each stage.

Templates live in templates/experiments/. When a scaffold is active:

1. Present the current stage's checklist before starting work
2. Gate progression: don't move to the next stage until exit criteria are met
3. Log stage transitions in the experiment breadcrumb

Default (no flag): user-defined stages (current behavior). Scaffold is a guide, not a cage — users can skip stages or reorder with explicit acknowledgment.

--budget Flag

Sets a campaign-level time budget in minutes for the entire experiment run. When set:

1. Record start time at the beginning of Phase 3 (any variant)
2. Check remaining budget before launching each new config, sweep batch, or explore iteration
3. Soft stop when budget is exhausted: finish the current run, save all results collected so far, skip remaining configs
4. Never hard-kill a running experiment mid-execution — always let the current run complete

Reporting: When a budget triggers early stop, append to the breadcrumb:

- **Budget:** Stopped after <N>/<M> configs (budget: <X> min, elapsed: <Y> min)

Default: No budget (run all configs to completion). Typical values: --budget 30 for quick exploration, --budget 120 for overnight sweeps.

Use Experiment when the design space is known upfront (grid/random sweep). Use Explore when the design space is unknown and the agent should adaptively search (inspired by Karpathy's autoresearch).

When to Use

• "Set up experiments for my algorithm" / "Scaffold the project"
• "Run the benchmark" / "Design the sweep"
• "Explore what auction parameters reduce collusion" / "Try different architectures"
• "Generate convergence plots" / "Make publication figures from results"
• Any project where algorithms, simulations, or optimisation loops are the contribution

When NOT to Use

• Empirical data analysis (observational/survey data) → data-analysis
• Experimental design for human subjects → experiment-design
• Causal inference strategy → causal-design
• LaTeX compilation → latex

Workflow

Phase 1: Detect & Configure

1. Read project context: CLAUDE.md, MEMORY.md, .context/project-recap.md if they exist
2. Detect structure: Scan for src/, experiments/, tests/, pyproject.toml, setup.py
3. Detect language: Check existing files. Default: Python with uv. Ask if ambiguous.
4. Read MEMORY.md: Look for [LEARN:code] tags, notation registry, key decisions — apply established conventions
5. Determine mode: From user request or ask. If existing package exists, skip scaffold.
6. Inventory existing code: Map modules, entry points, config objects, result files

Read references/package-scaffold.md if scaffold mode is active.

Phase 2: Scaffold

Skip if: Package structure already exists and user requested experiment/figures mode.

1. Package structure: Create src/<pkg>/, tests/, experiments/configs/, scripts/
2. Build config: pyproject.toml with hatchling, dev dependencies (pytest, matplotlib, numpy)
3. Algorithm skeleton: Base classes (Algorithm, Experiment, Metric) with # TODO: markers. For multi-agent projects, use BaseAgent, Environment, MultiAgentSimulation instead — see references/multi-agent-patterns.md
4. Test skeleton: Unit test stubs, convergence test template, smoke test
5. Experiment infrastructure: Config dataclass template, runner script template
6. Gitignore: Add results/, *.pkl, wandb/, __pycache__/

Read references/algorithm-templates.md for skeleton code. Read references/package-scaffold.md for directory layout.

Gate: Verify package installs with uv pip install -e ".[dev]" before proceeding.

Phase 3: Experiment Design

Prerequisite: Working package (Phase 2 or pre-existing).

1. Config schema: Python dataclass with validation, defaults, serialization to/from YAML
2. Config hashing: SHA-256 fingerprint for reproducibility tracking (see references/experiment-patterns.md)
3. Sweep definitions: Grid sweep (all combinations), random sweep (budget-limited), manual configs
4. Seed management: np.random.default_rng(seed) everywhere. Seeds passed through config, never global state.
5. Runner script: Config → initialize → loop → collect → save. Reads config, runs n_seeds repetitions, saves per-seed results.
6. Baseline implementations: Same interface as main algorithm, registered in config
7. Result aggregation: Per-seed CSV → aggregated stats (mean ± std). Canonical column naming.
8. Parallelization: concurrent.futures.ProcessPoolExecutor for independent seeds/configs. For large sweeps (10+ configs × 10+ seeds, GPU-bound, or >30-min runs): move to [HPC cluster] HPC — see docs/guides/hpc.md in Task Management and copy templates/slurm/{array,gpu}.sbatch into hpc/ with sync-up.sh / sync-down.sh. Recent reference implementations: Projects/NLP/{example-project-a,benchmark-gaming-llm-safety}/hpc/.
9. Checkpointing: Save intermediate results to allow resume on crash
10. Dual output: Dated archive + "latest" symlink for quick access (see references/experiment-patterns.md)

For multi-agent simulations, also include:

11. Agent composition: Heterogeneous population specs with distribution presets (uniform, weighted, custom)
12. Feature toggles: Opt-in capabilities per agent type (memory, messaging) — enables ablation studies
13. Messaging service: Round-based broadcast channel for inter-agent communication
14. Multi-level metrics: Agent-level (individual performance) + system-level (aggregate outcomes) + emergent-level (herding, convergence)

See references/multi-agent-patterns.md for all multi-agent patterns.

Read references/experiment-patterns.md for config, sweep, and runner patterns.

Gate: Run a smoke test — single config, single seed, verify output files are created.

Phase 3E: Explore Loop (Adaptive Experiments)

Use instead of Phase 3 when: The design space is unknown, the user wants to adaptively search rather than run a pre-specified sweep, or the user says "explore", "try things", "see what works".

Read references/explore-loop.md for the full protocol. Summary:

1. Branch: Create experiments/<tag> branch (e.g. experiments/mar14-collusion-params)
2. Baseline: Run the code as-is, record baseline metric in results.tsv
3. Loop (autonomous until interrupted or budget exhausted):
   • Propose a modification based on results so far (informed by prior keep/discard outcomes)
   • Apply the change to the target file(s)
   • git commit the change
   • Run with time budget: timeout <budget>s uv run python <script> > run.log 2>&1
   • Extract metric from output
   • Log to results.tsv: commit hash, metric, status (keep/discard/crash), description
   • If improved → keep (advance branch)
   • If not improved → discard (git reset --hard HEAD~1)
   • If crashed → attempt fix once, else discard and move on
4. Stop when: user interrupts, --budget time exhausted (campaign-level), per-run timeout exceeded, or N consecutive discards with no progress

Key principles:

• One change per iteration — never stack multiple untested changes
• Log every attempt, including discards and crashes
• The TSV is the ground truth — it captures the full exploration history
• Simplicity criterion: prefer changes that simplify the code for equal or better metrics

Gate: results.tsv exists with at least a baseline entry before entering the loop.

Phase 3A: Autonomous Sweep (Parallel Self-Correcting)

Use instead of Phase 3 when: 10+ configs, known failure-prone experiments, or user says "autonomous", "hands-off", "self-correcting sweep".

Read references/autonomous-sweep.md for the full protocol. Summary:

1. Batch configs into groups of max 4 parallel agents
2. Spawn sub-agents — each runs its batch, detects failures by signature (shape mismatch, NaN, empty array, timeout), diagnoses, fixes, retries (max 3 cycles)
3. Aggregate results — coordinator collects batch results, reports success/fixed/unresolved counts
4. Generate figures for all successful configs

Key constraints:

• Max 4 parallel agents
• Max 3 retry cycles per config
• Agents must not modify shared code simultaneously
• Each fix is committed for traceability

Graceful degradation: If some batches fail while others succeed, collect all successful results and report failures. Only stop entirely if ALL batches fail. See shared/skill-design-patterns.md (Graceful Degradation section).

Gate: At least one config must succeed. If all fail, report the unresolved errors and stop.

Breadcrumb: After any Phase 3 variant completes, append to .planning/state.md (if exists) or .context/current-focus.md:

### [computational-experiments] Experiments complete [YYYY-MM-DD HH:MM]
- **Done:** [N configs run, N seeds, mode: experiment/explore/autonomous]
- **Outputs:** [result files at <path>, N successful / N total]
- **Next:** Publication output (figures/tables)

Phase 4: Publication Output

Prerequisite: Result files exist in results/ or experiments/results/.

1. Convergence plots: Metric vs iteration/budget (mean ± std across seeds, multiple methods)
2. Comparison charts: Bar chart or table comparing methods across problems
3. Ablation plots: Performance with/without each component (for multi-agent: use feature toggle ablation configs)
4. Sensitivity analysis: Metric vs parameter (line plot with shaded confidence band)
5. Tables: Performance tables exported as .tex via \input{} — never hard-code results
6. Figure script: scripts/make_all_figures.py that regenerates all figures from saved results

Read references/figure-recipes.md for matplotlib recipes.

Output routing:

• Figures → paper/figures/ as PDF (per overleaf-separation rule)
• Tables → paper/tables/ as .tex (per no-hardcoded-results rule)
• Figure scripts → scripts/ (never inside paper/)

Phase 5: Audit & Save

1. Reproducibility check:
   • Seed determinism: same config + seed → same output?
   • Config completeness: can results be regenerated from saved configs?
   • Output freshness: are figures newer than their source data?
   • Dependency pinning: are versions locked in pyproject.toml or uv.lock?
2. Code review: Invoke the code-review agent on all generated scripts (via skill-routing mechanism)
3. Record learnings: Write [LEARN:code] tags for project-specific conventions discovered
4. Suggest next steps: compilation with latex, additional experiments, replication-package

Breadcrumb: After Phase 5 completes, append to .planning/state.md (if exists) or .context/current-focus.md:

### [computational-experiments] Phase 5 complete [YYYY-MM-DD HH:MM]
- **Done:** [reproducibility check, code review score, N learn tags recorded]
- **Outputs:** [figures at <path>, tables at <path>]
- **Next:** [suggested next steps]

Evolution Protocol

This skill improves with each invocation on a project:

1. Before work: Read MEMORY.md for existing [LEARN:code] entries and Key Decisions
2. During work: Note any convention, naming pattern, or structural choice that should persist
3. After work: Record [LEARN:code] tags and update Key Decisions table

Examples of learnings to capture:

• [LEARN:code] This project uses ElicitationConfig dataclass, not YAML files
• [LEARN:code] Metrics are in src/utils/metrics.py, not a separate package
• [LEARN:code] Seeds are managed via utils/seeds.py with MASTER_SEED + offset

Cross-References