Ac Online Experiments

Running trustworthy online controlled experiments: the experiment as a decision with a hypothesis and a success metric, statistical power and minimum detectable effect before launch, randomisation units and stable bucketing, ramping percentages and the peeking problem, guardrail metrics that stop an experiment early, novelty and primacy effects that fade, multiple comparisons across a fleet, and the written experiment record. It explains how to design, run, and read a controlled experiment; it never certifies that a variant won. Use when asking "A/B test", "controlled experiment", "treatment arm", "randomisation unit", "minimum detectable effect", "statistical power", "peeking", "sequential testing", "novelty effect", "guardrail metric", "experiment ramp", "hash bucketing", "why did our A/B test flip sign". Not for trading strategy backtests (ac-backtest-integrity), not for prompt and model evaluation sets (ac-prompt-engineer), not for feature deployment mechanics like canary and rollback (ac-ci-and-release).

jiayaoqijia Updated 81 repo stars

File contents

jiayaoqijia/cryptoskill commit 82b88c1d7e

Frequently asked questions

npx get-skill add jiayaoqijia/ac-online-experiments