# Experiment Rigor > Structure an experiment and its write-up so the Methods reconstruct without the conversation and the uncertainty is declared, not hidden — a typed record.yaml across a probe / measurement / decision tier ladder, every load-bearing rule a script that exits non-zero rather than a line of prose. Use when you pre-register an A/B, freeze a plan before running it, write up an experiment or a comparison, add error bars or a confidence interval to a rate, reconcile declared cells against the runs that happened, separate confirmatory from exploratory outcomes, name the threats to validity, design the 2x2, or ask whether a skill, tier, model, or strategy actually helps and how you would show it rigorously. Covers the small-n refusal (no CLT below 30 — Wilson, Clopper-Pearson, or a within-experiment Beta-Binomial), the design-arithmetic reconciliation (declared cells == disposition == denominators), the plan freeze and its drift gate, the rate that needs both a numerator and a denominator, probe self-labeling, and recor - Skill: `grimaldost/experiment-rigor` (Agent Skill, multi-file: 31 files) - Install (CLI): `npx skillmds@latest add grimaldost/experiment-rigor` - Raw SKILL.md: https://api.skillmd.com/api/skills/grimaldost/experiment-rigor/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: grimaldost (https://skillmd.com/u/grimaldost) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/grimaldost/experiment-rigor --- # Experiment Rigor Turn an experiment into a typed `record.yaml` whose `report.md` is derived, not written. This is a **rigid** skill: the scripts under `scripts/` are the mechanism, and each bright line below is a gate that exits non-zero with a stable error code (`validate.py`), not advice to remember. The dividing line the discipline draws: **methods uncertainty is disqualifying, effect uncertainty is declarable** — a reader who cannot reconstruct what was manipulated, where, and how it was measured has found a defect; a wide confidence interval is an honest result. Paths below are relative to `${CLAUDE_PLUGIN_ROOT}/skills/experiment-rigor/`, and PyYAML is the scripts' only non-stdlib dependency — run them as the gates do: ```bash uv run --no-project --with pyyaml -- python \ "${CLAUDE_PLUGIN_ROOT}/skills/experiment-rigor/scripts/validate.py" ``` The same two gates ship as pre-commit hooks (`experiment-rigor-validate`, `experiment-rigor-render-check`); the README names the install surface. ## The tier ladder `tier` is a field, not a second skill. It selects the required fields and which gates apply. Tier-0 is the exception: it names no artifact, so it never appears as a `tier:` value. - **check** (tier-0) — the structured check. An evaluation act answered inline in the response: no file, no record, no gate. An *evaluation act* asks a question whose correct answer is a valuative claim ("is this effective", "which is better"); an *execution or lookup request* asks for an action or a fact and owes nothing here. The five-element shape — method, metric, result(s) with denominators, conclusion, and a one-line "what this updates" — the boundary, and two worked examples live in `references/report-skeleton.md`. This rung is guidance, not a bright line, and entry above it is unchanged: a record is owed when a decision rides on the result. - **probe** — cheap and exploratory. Refuses a confirmatory verdict or any posterior; a probe that wants either graduates to measurement. - **measurement** — a frozen pre-registration and a reported interval on every rate. - **decision** — adds the comprehension gate: fresh-context readers reconstruct the Methods before the result is allowed to move a decision. ## The loop 1. Copy the tier skeleton from `templates/.yaml`; the field guide is `templates/SCHEMA.md` (generated from `templates/schema.json`, the canonical schema). 2. Author the record. While drafting, `validate.py --schema-only` checks shape without the context gates (they are skipped and listed). 3. Freeze the plan: commit the pre-registration and record `plan_frozen_at.commit` before the first run. 4. Run the experiment; fill results. `from_fathom.py` maps a fathom ledger into the run-derived fields. 5. `validate.py ` runs the full gate. `render.py ` derives `report.md`; `render.py --check` is the drift gate over a committed pair. Never hand-edit `report.md`. ## The activation line Whenever the frame engages, one plain line opens the work product, naming the tier and the artifact behind it. At `probe` and above it is generated, not typed: `render.py --activation-line ` prints it, and `--check-activation-line "" ` exits non-zero when the tier or the path disagrees — the line is a claim tied to the artifact, not a badge. At tier-0 the artifact reference is the literal `inline`, which nothing resolves. ```text [experiment-rigor | check -> inline] [experiment-rigor | measurement -> experiments/retry-backoff/record.yaml] ``` ## Bright lines Each is a gate in `validate.py`; the error code is named so a failure points at the rule. - **The freeze (`ER-PREREG`).** The pre-registration subset — `design.cells`, each outcome's `role`, operationalization, and verifier hash, and `analysis_plan` — is reconstructed with `git show` on `plan_frozen_at.commit` and compared to the analyzed record. Any drift fails; a cut forced after the freeze is an amendment, not a drift — `design.amendments[]` re-plans declared cells' `planned_n` under `analysis_plan.amendments[]`'s chronology. A confirmatory verdict is legal only on an outcome whose frozen `role` is `confirmatory`. The two guards the freeze buys — a null bank, and a pre-committed rule for what a finding must survive — are worked, with the day they both fired, in `references/two-guards.md`. - **Declared-cells reconciliation (`ER-RECON`).** `N_expected` is the sum of `design.cells[].planned_n` (as amended), and it must equal the disposition total and every outcome's sum of arm denominators. The model tier is one named factor level, not a separate multiplier. - **The confirmatory / exploratory partition.** `outcomes[].role` belongs to the frozen plan. An outcome added after the freeze carries `added_after_freeze: true` and `role: exploratory` — the quarantine — and may report only an `exploratory_signal` or `inconclusive` verdict, never a confirmatory one. - **The small-n CI refusal (`ER-STATS`).** No CLT / normal method below a cell denominator of 30. Allowed: `wilson`, `clopper_pearson`, `beta_binomial`. Every stated interval is recomputed from `stats.py` and must match to four decimals. `references/small-n-stats.md` covers the rest. - **A rate needs both a numerator and a denominator (`ER-SCHEMA`).** A rate lives under `results..arms.` with both; a loose rate at the outcome level fails. - **Record is source, report is derived (`ER-PARITY`, `render.py --check`).** The record is the single source of truth; `report.md` is regenerated and must not contradict it. - **Probe self-labeling (`ER-PROBE`).** A probe carrying a confirmatory verdict or a posterior fails; the message names the graduation path to measurement. - **Threat coverage (`ER-THREAT`).** Every core threat in the closed enum carries a row with a status and a statement; silence on one fails. The enum is in `references/threats-catalog.md`. ## What binds, role-generically - **Readiness before build.** When a spec-readiness gate is installed (for example keel's Definition of Ready), a decision-tier experiment that will inform a build defers spec readiness to it rather than re-deciding it here. Absent one, the experiment stands alone. - **Fresh-context readers.** The decision-tier comprehension block records independent readers who reconstruct the Methods with no access to this conversation. Any fresh-context reader tooling that is installed can produce those reads; the block records the transcript path and the four verbatim answers, and their genuineness is ceded to review. The gate checks presence, resolution, and unanimity, not sincerity. ## Boundaries Running the scenario matrix, scoring a bank, or driving the paid run is fathom's `fathom-eval`; this skill structures and gates the record that results. Measuring one skill's trigger activation is `evaluate-skill`. Judging whether a spec is ready to decompose is a readiness gate (keel's Definition of Ready). A throwaway spot-check you will decide nothing on needs none of this. ## Correct-usage check - [ ] The record passes `validate.py` at its declared tier (exit 0). - [ ] The Methods reconstruct from the record alone — what was manipulated, where it was placed, how each outcome was operationalized, whether execution was real — without the conversation. - [ ] Every reported rate carries a recomputable interval; no CLT below 30. - [ ] `report.md` is derived (`render.py`), not hand-edited (`render.py --check` clean). - [ ] The work product opens with the activation line — `inline` at tier-0, and at `probe` and above the generated line, `--check-activation-line` clean.