Experiment Suite

Experiment Suite

Overview

End-to-end experiment package builder. Single stage, full quality from the start. The agent (Claude Code / Cursor / Aider / Codex / …) writes everything directly using its own tools (Write, Bash, WebFetch, …). This skill contains procedure + reference playbooks + figure-example scripts — no Python runtime, no LLM SDK.

The substantive work is decomposed into reference playbooks under references/:

Also: figure_examples/ — publication-style matplotlib scripts plus a shared style kit the agent can use as starting points.

Read the relevant reference before writing, not after. The full pass does not fit in a single turn — references/00-incremental-execution.md is the only execution mode that completes.

When to Use

• User wants to "design an experiment" for a research question.
• User needs runnable code for a specific task (classification / forecasting / detection / …).
• User wants to compare methods and have a structured report at the end.
• User needs publication-quality figures of experimental results.

When NOT to Use

• User only wants a quick code snippet (write code directly).
• User wants a full paper → paper-writer.
• User wants a literature survey → literature-survey.

Workflow

Step 1 — Understand the question and operating mode

Confirm with the user:

• Research question — what we are trying to answer.
• Task type — classification / regression / forecasting / detection / generation / …
• Mode
  • measured — user has real data or will run code themselves; provide a path to a measured results.json or run train.py against real data later.
  • simulated (default) — agent generates a plausible-shaped, deterministic results.json as a placeholder. Every figure/table caption must say "simulated".
• Framework preference — PyTorch (default), JAX, TensorFlow, or sklearn.
• Compute budget — hours / GPUs available; constrains the code skeleton and hyperparameter plan.

If the user has data and time, push toward measured mode. If not, simulated is acceptable provided disclosures are honest in every artefact.

Step 2 — Set up the run directory

QUESTION="<research_question>"
SLUG=$(python3 -c "import re,hashlib,sys; t=sys.argv[1]; n=re.sub(r'[\\s_]+','-',re.sub(r'[^\\w\\s-]','',t.lower().strip())).strip('-')[:40].rstrip('-'); h=hashlib.sha1(t.encode()).hexdigest()[:8]; print(f'{n}-{h}')" "$QUESTION")
TS=$(date +%Y-%m-%d_%H%M%S)
RUN=output/experiment-suite/$SLUG/$TS

mkdir -p "$RUN/experiment" "$RUN/figures"
ln -sfn "$TS" "output/experiment-suite/$SLUG/latest"

In commands below $RUN = output/experiment-suite/<slug>/latest.

The agent will create five top-level files inside $RUN/:

• experiment_design.md
• data_contract.md
• experiment/{model.py,data.py,train.py,evaluate.py,config.yaml,requirements.txt,README.md}
• results.json
• figures/*.pdf plus their make_*.py source and a manifest.json
• experiment_report.md

Step 3 — Build the package (REQUIRED — this is the whole job)

Open references/00-incremental-execution.md first. Then carry out the six tracks below across many turns, persisting state to $RUN/ after every batch.

3.1 Design — full justification document

Open: references/01-design-depth.md and references/01a-data-contract.md. First write $RUN/data_contract.md as the dataset contract for this run. It must say whether the data are user-supplied, agent-discovered, reused public, controlled, or synthetic fallback. Then write $RUN/experiment_design.md as a real design (≥ 700 words): motivation → hypothesis → datasets → baselines → metrics → ablations → compute budget. Justify every choice.

3.2 Code — actually runnable

Open: references/02-code-quality.md. Fill $RUN/experiment/ with code that an engineer could launch with python train.py --config config.yaml. Real (if minimal) model class, real data loader, real train loop, real eval. The generated data.py and config.yaml are runtime products of this run and should bind to $RUN/data_contract.md, not to a repository-wide hard-coded benchmark. Add a README.md with run instructions.

3.3 Results — honest provenance

Open: references/03-results-protocol.md. Produce $RUN/results.json with a well-formed schema: per-seed entries, per-method per-metric mean & std, ablation block, and a provenance field that names the source.

• measured mode — the user runs experiment/train.py (or supplies a results JSON) and the agent loads it into $RUN/results.json, setting "simulated": false and "provenance": "loaded from <path>".
• measured mode, agent-discovered data — the agent may search for and bind an open dataset itself, but the chosen source, split, and access route must first be written into $RUN/data_contract.md; results.json provenance must point back to that binding.
• simulated mode — the agent writes a deterministic seeded JSON of plausible shape, setting "simulated": true.

3.4 Figures — 3–6 publication-grade

Open: references/04-publication-figures.md, references/04a-figure-contract.md, references/04b-figure-qa.md, and figure_examples/. Before writing plotting code, define the figure contract in a small working note under $RUN/figures/figure_contract.md:

• one-sentence conclusion,
• figure archetype,
• panel map,
• evidence hierarchy,
• statistics needed,
• reviewer risk.

Then plan and generate at minimum:

• 1 method-comparison chart (bar or line).
• 1 ablation breakdown.
• Optionally training curves, scaling plot, heatmap.

Save each figure into $RUN/figures/<basename>.pdf with its make_*.py source alongside. Prefer saving an editable .svg and print-grade .tiff alongside the PDF when the environment supports it. Append entries to $RUN/figures/manifest.json storing basenames only (never absolute paths) so paper-writer can copy them in directly. Apply the shared publication style (embedded fonts, explicit palette, panel labels, simulated watermark when applicable).

If simulated, watermark the figures or always note "simulated" in their captions in the report.

3.5 Report — structured

Open: references/05-report-structure.md. Write $RUN/experiment_report.md with sections: problem statement → design rationale → method → setup → results → analysis → limitations. Reference figures by filename. This report is the primary deliverable for users who want only the experiment package (no paper-writer follow-up).

3.6 Quality gate

Open: references/06-quality-gate.md. Targets: design ≥ 700 words, code imports cleanly (python -c "import experiment.model" from inside $RUN), results.json passes schema check, ≥ 3 figures, report ≥ 6 sections.

Step 4 — Deliver

Report:

1. output/experiment-suite/<slug>/latest/experiment_design.md
2. output/experiment-suite/<slug>/latest/experiment/ — runnable code package.
3. output/experiment-suite/<slug>/latest/results.json — with provenance.
4. output/experiment-suite/<slug>/latest/figures/ — publication-grade charts + manifest.json.
5. output/experiment-suite/<slug>/latest/experiment_report.md — structured report.
6. Stats per the report format in references/06-quality-gate.md.

Cross-skill data flow (path convention)

The paper-writer skill computing the same slug for the same topic will look here:

• output/experiment-suite/<slug>/latest/results.json — source of the numbers and the "simulated" flag (drives the disclosure clause in the paper).
• output/experiment-suite/<slug>/latest/figures/*.pdf (+ manifest.json) — figures to reuse rather than redraw.

Always store basenames in manifest.json. Absolute paths in the manifest break paper-writer's \includegraphics{figures/<basename>}.

Important rules

• No LLM SDK in this skill. No import anthropic / import openai. The skill is SKILL.md + references + figure examples only.
• Simulated results must always remain visibly labelled — in results.json ("simulated": true), in figure captions, in the report's top-of-page disclosure, and in any downstream paper's \thanks footnote.
• Never present simulated results as measured. When in doubt, treat as simulated and disclose.
• The runnable code is a starting point, not a SOTA reproduction. Be honest about its scope in experiment/README.md.
• A real experiment package would normally take days of compute for real numbers; the simulated path lets the workflow proceed when that's not possible, with honest disclosure throughout.