# Experimentation Causal Inference

> Experimentation & Causal Inference

- Skill: `sanjeevrg89/experimentation-causal-inference` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add sanjeevrg89/experimentation-causal-inference`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sanjeevrg89/experimentation-causal-inference/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: sanjeevrg89 (https://skillmd.com/u/sanjeevrg89)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/sanjeevrg89/experimentation-causal-inference

---


# Experimentation & Causal Inference

Apply the judgment of someone who has run a large-scale experimentation platform and an applied causal
inference practice for years. The core belief: **correlation is cheap and usually wrong about
direction; the only reliable way to know whether a change helped is a trustworthy randomized
experiment — and when you cannot randomize, you must explicitly argue identification, not wave at a
chart.** An experiment that is run wrong is worse than no experiment, because it launders a guess into
a "result." Trust comes first; cleverness second.

## How to use this skill

1. **Read `experimentation-causal-inference-guide.md`** in this directory — the full reference
   (potential outcomes & ATE, designing a trustworthy experiment, statistics done right, the pitfalls
   and Twyman's law, experimentation at scale, quasi-experiments & causal inference, uplift/CATE,
   application to ML/recsys). Apply it to the task at hand.
2. For concrete artifacts to imitate — a full experiment design doc (OEC + guardrails + power +
   randomization unit), an SRM check with a CUPED note, and a difference-in-differences sketch for a
   non-randomizable rollout — read **`examples.md`**.
3. Match the surrounding org's metric definitions and platform conventions; apply the trust rules
   (SRM check, pre-registration, correct randomization unit, no peeking without sequential methods)
   regardless of local habit.

## The essentials (full rationale in the guide)

- **Define the OEC before you look at data.** One Overall Evaluation Criterion that is sensitive,
  hard to game, and a leading indicator of long-term value (not raw revenue, not a single-day click).
  Pre-register hypothesis, OEC, guardrails, segments, and decision rule. Anything decided after seeing
  results is HARKing.
- **Guardrail metrics protect the business** while you optimize the OEC: latency, crash rate, revenue,
  unsubscribes. A move that wins on OEC but breaks a guardrail does not ship.
- **Pick the randomization unit deliberately** — almost always the *user* (stable, captures the full
  experience, avoids carryover). Session/request units leak treatment across the same user and
  understate variance. The analysis unit must match (or be nested in) the randomization unit.
- **Power the test first.** Compute the Minimum Detectable Effect, choose α and power (commonly 0.05 /
  0.80), and derive sample size and runtime *before* launch. Underpowered tests produce noise and
  inflated "winners" (Type M/S errors).
- **SRM is the #1 trust check.** If the observed treatment/control split deviates from the designed
  ratio (chi-square p < ~0.001), the experiment is broken — stop and debug; do not read the metrics.
- **A p-value is not P(no effect) and a 95% CI is not "95% probability the true value is inside."**
  Report effect sizes with CIs; correct for multiple comparisons (FDR/Benjamini-Hochberg).
- **Do not peek at fixed-horizon tests.** Continuous monitoring with fixed-α inflates false positives
  badly. If you need to stop early, use **sequential / always-valid inference (mSPRT, group-sequential,
  confidence sequences)**.
- **Reduce variance with CUPED** (use pre-period covariates) to cut required sample size — often a large
  reduction — without bias. Trigger-based analysis (dilution) also sharpens sensitivity.
- **Watch for interference.** In marketplaces, social graphs, and shared-resource systems, SUTVA is
  violated; use **cluster / switchback / budget-split** designs and interpret naive A/B with suspicion.
- **Run A/A tests** to validate the platform (false-positive rate ~ α, no SRM, correct CIs) before
  trusting A/B results. Ramp up exposure (1% → 5% → 50%) to limit blast radius.
- **Twyman's law:** any figure that looks too good or too surprising is probably wrong — investigate
  instrumentation, logging, and SRM before celebrating.
- **When you cannot randomize, name the identification strategy** (DiD, RD, IV, PSM, synthetic control)
  and its assumptions explicitly. Observational "lift" without an identification argument is not causal.

## Related skills

- `[[ml-evaluation-evals]]` — offline/online quality measurement of models; this skill is the
  decision/causality layer above it (does the better-on-eval model actually move the business metric?).
- `[[recsys-ranking]]` — ranking/model changes are validated by online experiments; the offline-online
  gap and interleaving live here.
- `[[ml-observability-monitoring]]` — near-real-time experiment monitoring, metric pipelines, and
  auto-shutoff hook into the same telemetry.
- `[[ml-system-design]]` — experimentation platform as a system; metric stores, assignment service.
- `[[data-engineering-feature-stores]]` — pre-period covariates for CUPED and metric computation come
  from the same feature/metric data plane.
- `[[responsible-ai-governance]]` — guardrails, fairness slices, and decision accountability for
  launches.