Using Counterfactual Statistics

Using Counterfactual Statistics

Overview

When you fork an experiment into matched branches and compare them, you have bought precision, not independence. The branches are repeated measures of one thing; the thing is the run you forked. Nearly every statistical failure in this domain is a consequence of forgetting that — and the rest are consequences of choosing what to measure after seeing the results.

Counterfactual evaluation is unusually powerful and unusually easy to get wrong. Matching branches against a no-op control can cut required sample size by 6× (common-random-numbers-and-matching); counting those branches as independent samples understates your uncertainty by 2.6× and turns a null into a headline (statistical-units-and-clustering). The two errors live one line of code apart.

This pack covers the statistical discipline that paired counterfactual experiments demand and that general ML evaluation guidance does not: what the unit is, what makes the pairing valid, which data may inform which decision, how much of a "best-of-K" result is selection, and what a result must contain before anyone acts on it.

When to Use

Use this pack when:

• You fork branches from a shared snapshot, seed, or checkpoint and compare them against a control.
• Your evaluation includes a no-intervention / no-op branch and you need to anchor utility against it.
• You are about to write n = in a results table and the rows outnumber the runs you launched.
• A screening stage picks the best of K candidates and you need to report the winner's effect honestly.
• A judge may decline to act, and you need to grade abstention rather than accuracy.
• Someone asks "is this significant?", "how many seeds do we need?", or "why didn't it replicate?"
• You are designing an evaluation harness and want the statistics to be correct by construction rather than by review.

Do not use this pack when:

• You need general A/B or multivariate experiment design for user-facing products, randomisation-unit and assignment mechanics, observational causal inference (difference-in-differences, instrumental variables, propensity scores, RDD), or variance reduction outside CRN (CUPED, stratification) → yzmir-experimentation (planned — not yet in the marketplace). This pack owns the paired counterfactual slice: the unit is a trajectory, the control is a matched no-op, and the branches are clones of a shared state.
• You are choosing which baselines a growth or morphogenesis experiment must run (off-switch, static-initial, static-final, fixed-schedule), or comparing checkpoints of different shapes → yzmir-morphogenetic-rl/evaluation-under-topology-change. That sheet answers which comparisons to run; this pack answers what the independent unit is and how to compute a valid interval over it.
• You need bit-exact reproducibility, cross-machine determinism, floating-point or GPU nondeterminism, or replay divergence localisation → axiom-determinism-and-replay. This pack presupposes that contract: common random numbers is not achievable in a system that cannot replay.
• You are debugging an RL reward function or algorithm choice → yzmir-deep-rl.
• You need production monitoring, drift detection, or online evaluation → yzmir-ml-production.
• You need to construct the dataset or eval set rather than reason about it — curation, coverage/slice design, deduplication tooling, label quality and annotator agreement, synthetic-data provenance, the dataset release contract and its documentation standards → yzmir-ml-production/dataset-curation-and-quality. This pack owns the inference drawn from a split; that sheet owns how the data was built. grouped-splits-and-leakage.md remains authoritative for the leakage taxonomy itself.
• You need general test strategy or flaky-test triage → ordis-quality-engineering.

The Running Example

Every sheet uses one worked system. Substitute your own nouns:

A training run is periodically snapshotted. At each snapshot the harness forks matched branches from that exact state: K candidate interventions plus one mandatory no-op branch that changes nothing. Every branch resumes from the same snapshot and consumes the same future minibatches. After H steps each branch reports a metric. A candidate's measured effect is metric(no-op) − metric(candidate), so the no-op scores exactly zero by construction.

The fleet: G = 24 independent runs, D = 3 decision points each, K = 8 candidates each — 576 candidate-vs-no-op differences and 24 independent observations.

The same structure covers matched-seed ablation forests, A/B rollouts forked from a shared checkpoint, prompt/config sweeps evaluated on common inputs, and any "clone the world, change one thing, replay" design.

Start Here

Greenfield, in order. Steps 1–3 are the spike: if the unit, the pairing, and the splits are wrong, nothing downstream can be repaired by analysis.

1. statistical-units-and-clustering — what one observation is. Everything else takes its n from here.
2. common-random-numbers-and-matching — what must be identical for the pairing to be worth anything. Worth 6× the fleet.
3. grouped-splits-and-leakage — the four data roles and the walls between them, assigned by whole units.
4. effect-sizes-and-cost-charged-utility — define the quantity you will act on, with a real zero.
5. power-and-sample-size-for-paired-designs — how many units the claim needs. Do this before buying compute.
6. preregistration-and-exploratory-vs-confirmatory — freeze the plan; commit it before the first confirmatory unit runs.
7. frontier-and-reliability-reporting — know what you will report before you run, so you collect it.

Auditing someone else's work instead? Go straight to anti-pattern-catalogue and work down it in order, or dispatch experiment-statistics-reviewer.

How to Access Reference Sheets

All sheets live in the same directory as this SKILL.md. A link like [paired-comparison-methods.md](paired-comparison-methods.md) means: read that file from this directory.

Routing

Reference Sheets

Foundations — get these wrong and no later sheet can save the result:

1. statistical-units-and-clustering.md — the independent unit, repeated measures, ICC and design effect, cluster-robust inference, pseudo-replication
2. paired-comparison-methods.md — zero-anchored controls, aggregate-then-test, paired t / Wilcoxon / bootstrap, when pairing breaks
3. common-random-numbers-and-matching.md — the matching contract, RNG-stream discipline, CRN as 6× variance reduction

Data discipline:

4. grouped-splits-and-leakage.md — support / screen / audit / report roles, whole-unit splits, the five-class leakage taxonomy, cross-fitting
5. selection-bias-and-best-of-k.md — the winner's curse quantified, independent audit data, what analytic corrections assume
6. abstention-and-calibration.md — no-op precision/recall, false-intervention rate, regret as the threshold objective, reliability diagrams and ECE

Inference:

7. multiple-comparisons-and-sequential-testing.md — family definition, Holm vs BH, alpha spending for interim looks
8. power-and-sample-size-for-paired-designs.md — power from unit-level sd_d, MDE, pilot-variance uncertainty, Type-M exaggeration
9. preregistration-and-exploratory-vs-confirmatory.md — the pre-registration artifact, researcher degrees of freedom, honest amendment

Measurement and reporting:

10. effect-sizes-and-cost-charged-utility.md — utility with a real zero, admission vs retention weights, practical vs statistical significance
11. frontier-and-reliability-reporting.md — the six-row reliability report, quality–cost–stability frontiers, negatives as output
12. horizon-choice-and-divergence-noise.md — signal vs divergence noise, the interior optimum, multi-horizon endpoints without p-hacking

Audit:

13. anti-pattern-catalogue.md — twenty-one anti-patterns with symptom, mechanism, severity, detector, and fix

Specialist Commands

• /design-counterfactual-experiment — from a research question and constraints, emit a complete pre-registered analysis plan: unit, pairing and matching contract, split roles, endpoints, horizon, N, tests, correction method, and abort/success criteria.
• /analyze-paired-trial — given branch outcomes keyed by unit, run the correct clustered/paired analysis and emit the reliability report.
• /audit-experiment-statistics — adversarially review an existing analysis, harness, or paper against the anti-pattern catalogue; severity-rated findings, each citing the sheet that fixes it.

Specialist Agents

• agent: counterfactual-statistician — forward-design SME. Turns a research question into the unit definition, split plan, and pre-registration. Dispatch when designing.
• agent: experiment-statistics-reviewer — critic SME. Hunts pseudo-replication, leakage, selection bias, and calibration-on-test in designs, code, or papers. Refuses to rubber-stamp: zero findings is treated as an audit defect. Dispatch when reviewing.

Skills vs agents: sheets design the experiment; agents audit or produce an artifact. Load a sheet when deciding; dispatch an agent when reviewing someone's work or when you want the pre-registration written.

Common Scenarios

Greenfield counterfactual experiment

1. statistical-units-and-clustering — name the unit, add unit_id everywhere
2. common-random-numbers-and-matching — write the matching contract, wire the stream assertion into CI
3. grouped-splits-and-leakage — assign whole units to four roles by stable hash
4. effect-sizes-and-cost-charged-utility — define utility with a zero-anchored control
5. horizon-choice-and-divergence-noise — pilot the d_z curve, pick the plateau centre
6. power-and-sample-size-for-paired-designs — size the fleet at the UCL of sd_d
7. preregistration-and-exploratory-vs-confirmatory — commit the plan, then launch

"Is this result real?"

1. statistical-units-and-clustering — recount n; this alone resolves most cases
2. paired-comparison-methods — was the pairing used?
3. selection-bias-and-best-of-k — how many candidates were tried?
4. multiple-comparisons-and-sequential-testing — how big was the family; was it stopped early?
5. power-and-sample-size-for-paired-designs — at this n, is a significant result exaggerated?

"It didn't replicate"

1. power-and-sample-size-for-paired-designs — Type-M: was the original underpowered and therefore inflated?
2. selection-bias-and-best-of-k — was the original a selected maximum?
3. preregistration-and-exploratory-vs-confirmatory — were the two analyses even the same procedure?
4. grouped-splits-and-leakage — did the original leak?

Designing a screener / admission gate

1. effect-sizes-and-cost-charged-utility — the utility it thresholds
2. abstention-and-calibration — calibrate, then choose the threshold by regret
3. grouped-splits-and-leakage — fit on screen units, freeze, never touch report units
4. selection-bias-and-best-of-k — give the audit stage its own data

Writing up a fleet result

1. frontier-and-reliability-reporting — the six-row report over units
2. power-and-sample-size-for-paired-designs — MDE, mandatory if the result is null
3. anti-pattern-catalogue — self-audit before publishing

Decision Tree

Designing an experiment?
├─ Yes → units → matching → splits → utility → horizon → power → pre-register
└─ No  → continue

Auditing an existing analysis, harness, or paper?
└─ anti-pattern-catalogue, in order: units/pairing → walls → selection/testing
   → design adequacy → utility → reporting.  Or dispatch
   experiment-statistics-reviewer.

Reported n far exceeds runs launched?            → statistical-units-and-clustering
Two-sample test on matched arms?                 → paired-comparison-methods
sd_d larger than the design predicted?           → common-random-numbers-and-matching
Held-out number implausibly good?                → grouped-splits-and-leakage
"Best of K" reported without K?                  → selection-bias-and-best-of-k
Gate can decline, graded on accuracy?            → abstention-and-calibration
Many p-values, or a fleet stopped early?         → multiple-comparisons-and-sequential-testing
Null with no MDE, or a fleet sized from a pilot? → power-and-sample-size-for-paired-designs
Analysis choices made after seeing results?      → preregistration-and-exploratory-vs-confirmatory
Winner is the most expensive candidate?          → effect-sizes-and-cost-charged-utility
Single mean standing in for a distribution?      → frontier-and-reliability-reporting
Reported horizon differs from the design's?      → horizon-choice-and-divergence-noise

Rationalization Resistance

Red Flags Checklist

Any one of these means stop and route before believing the number:

• n exceeds the number of independent runs launched
• No no-op / no-intervention branch — the counterfactual was never run
• A two-sample test on matched arms, or per-arm error bars compared by eye
• Additive seeding (base_seed + branch_id) or per-branch dataloaders
• A random row-level train_test_split on derived rows
• An audit stage with a 0% historical rejection rate
• "Best of K" reported without K
• A threshold, temperature, or cost weight whose fit provenance nobody can state
• A results table of many p-values with the smallest one in the summary
• A fleet stopped early, or a dashboard recomputing significance continuously
• A null reported without an MDE
• An analysis script whose endpoint or exclusions changed after the fleet finished
• A leaderboard ranked on the uncharged metric
• A single mean per configuration, with no n_units, IQR, or failure rate
• No stored rejections, abstentions, or failures

Integration with Other Skillpacks

Quick Reference

The Bottom Line

Forking branches from a shared state buys you precision, not sample size. The independent unit is the thing you would have to launch again — the trajectory, not the branch — and every interval, every power calculation, and every split is denominated in those. Match the branches so the difference contains only the intervention; anchor against a control that scores exactly zero so "do nothing" can win; keep the data that chose a candidate away from the data that judges it; declare the plan before the data can influence it; and report the distribution over units, including the third of runs that got worse. Skip any of these and the experiment will still produce a number — it just will not be a measurement.