# Ml Experiment Design

> Use when turning a rough ML idea into a runnable experiment - define the experiment card (question, hypothesis, baseline, primary metric, budget, cheapest verification rung) before writing training code.

- Skill: `pengzhangzhi/ml-experiment-design` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pengzhangzhi/ml-experiment-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pengzhangzhi/ml-experiment-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Finance & Business
- Author: pengzhangzhi (https://skillmd.com/u/pengzhangzhi)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pengzhangzhi/ml-experiment-design

---


# ML Experiment Design

## Overview

An ML idea is not runnable until it is a CARD. This skill turns a rough idea into a compact, one-screen experiment card — rung R0, the precondition for planning or launching anything. Fill it WITH your human partner; do not write training code until the card is settled.

**Core principle:** One change under test, one locked metric, one decision rule. If you cannot state in advance what result would change your mind, you are not ready to run.

**Upstream:** the idea comes from `superpowers-ml:brainstorming`.
**Downstream:** the card drives `superpowers-ml:ml-feedback-ladder` (how to verify it cheaply) and `superpowers-ml:writing-plans` (the tasks).

## The Experiment Card

Settle every field. Keep each to a line or two — a card, not a document.

- **Research question** — the one question this run answers.
- **Hypothesis** — specific and falsifiable: what changes, and the direction you expect.
- **Baseline** — the exact, runnable comparison (config / commit, not "the usual setup").
- **Variant(s)** — the SINGLE change under test. Anything else is a separate experiment.
- **Primary metric** — one metric, locked now, that decides the outcome.
- **Guardrail metrics** — what must NOT regress while the primary metric moves (cost, latency, memory, a quality/safety metric).
- **Dataset / split** — train / val / test, and the leakage risk you checked for.
- **Seed policy** — how many seeds, fixed or swept; how you tell seed noise from a real effect.
- **Budget** — the compute / wall-clock ceiling you will spend before stopping to decide.
- **Cheapest useful rung** — the smallest rung that tells you something real, to start at (see `superpowers-ml:ml-feedback-ladder`).
- **Success criterion** — the threshold on the primary metric that CONFIRMS the hypothesis, set before running.
- **Exploratory-only** — what this run may suggest but can never confirm (anything the locked primary metric does not measure).

## One Change At A Time

A variant is ONE change, or you cannot attribute the result — two changes is two experiments. If your human partner wants to vary many things, that is a study: write a card per change, then design a ladder for each.

## Template

```
Question:         ...
Hypothesis:       changing X will <increase|decrease> <metric> because ...
Baseline:         <config / commit>
Variant:          <the single change>
Primary metric:   <one, locked>     Success: <threshold set in advance>
Guardrails:       <must-not-regress>
Data / split:     <train / val / test>     Leakage check: <...>
Seeds:            <n, fixed | swept>
Budget:           <GPU-hours / wall-clock ceiling>
Start rung:       <cheapest useful rung>
Exploratory-only: <what will not count as confirmation>
```

## Red Flags - STOP

- No locked primary metric, or the metric is chosen after seeing results.
- Success criterion not written down before the run.
- More than one change under test in a single variant.
- "We'll know it when we see it" — no decision rule.

A card you can fill in five minutes saves a burned cluster job. Write it first.