Contract
- Input: feature change, user segment, metric of interest.
- Output: test design with power analysis and rollback rules.
- Side effects: none (design only; execution requires deployment and user exposure).
- Dependencies: analytics platform (for measurement), experiment platform (for assignment).
- Stop condition: design complete; power computed; rollback rules defined.
- Risk: medium — bad test design leads to false conclusions; exposure to users requires ethical review.
- Boundary: designs experiment; does not expose users to unapproved changes.
A/B Testing Design
Design an A/B test — hypothesis, randomisation, metrics, statistical power — with valid inference.
Process
1. Hypothesis
State a clear, testable statement:
- "Changing the checkout button from grey to green increases conversion rate by ≥ 2%."
- Must specify: metric, direction, magnitude, user segment.
Completion criterion: hypothesis saved with metric, direction, magnitude.
2. Selection
- Control: current experience.
- Treatment: new experience.
- Randomisation: user-level assignment (cookie / account / session); must be independent of behaviour.
- Segmentation: if testing on a subset (e.g. new users only), state why.
- Exclusions: users with special conditions (e.g. VIP, internal, disabled users) must be handled ethically.
Completion criterion: assignment method saved; exclusions noted.
3. Metrics
- Primary: the metric that determines success (e.g. conversion rate, click-through rate, retention).
- Secondary: supportive metrics (e.g. revenue per user, time on task, error rate).
- Guardrail: metrics that must not degrade (e.g. page load time, accessibility, error rate, support tickets).
Completion criterion: metrics defined with direction (increase / decrease / maintain).
4. Duration and sample size
Compute:
- Minimum detectable effect (MDE): smallest improvement that is practically meaningful.
- Power: 1 — β (typically 0.8 or 0.9).
- Significance level α: 0.05 (or 0.01 for critical decisions).
- Sample size per variant: derived from baseline rate, MDE, α, power.
- Duration: sample size / daily traffic per variant; account for weekly seasonality; run at least one full week.
Completion criterion: sample size and duration computed.
5. Analysis plan
- Statistical test: z-test for proportions; t-test for means; Mann-Whitney for non-normal; bootstrap for complex metrics.
- Segmentation: analyse by subgroups (device, region, user type) — but do not over-segment (risk of false positives).
- Interaction: check if treatment effect varies by subgroup (interaction test).
- Multiple testing correction: Bonferroni or FDR if testing many metrics or segments.
Completion criterion: analysis plan saved.
6. Rollback / escalation rules
- Early stopping rules: if guardrail metric degrades beyond threshold during first N% of planned duration, stop.
- Success criteria: primary metric improves by ≥ MDE with p < α; guardrail metrics not degraded.
- Rollback: revert treatment for all users if success criteria not met by end of planned duration, or if guardrail fails.
Completion criterion: rollback and escalation rules saved.
1---2name: prod-ab-testing3description: Design A/B tests — hypothesis, control/treatment, randomization, metrics, statistical power, duration — with valid inference and rollback rules.4---56## Contract78- **Input:** feature change, user segment, metric of interest.9- **Output:** test design with power analysis and rollback rules.10- **Side effects:** none (design only; execution requires deployment and user exposure).11- **Dependencies:** analytics platform (for measurement), experiment platform (for assignment).12- **Stop condition:** design complete; power computed; rollback rules defined.13- **Risk:** medium — bad test design leads to false conclusions; exposure to users requires ethical review.14- **Boundary:** designs experiment; does not expose users to unapproved changes.1516# A/B Testing Design1718Design an **A/B test** — hypothesis, randomisation, metrics, statistical power — with valid inference.1920## Process2122### 1. Hypothesis23State a clear, testable statement:24- "Changing the checkout button from grey to green increases conversion rate by ≥ 2%."25- Must specify: metric, direction, magnitude, user segment.2627**Completion criterion:** hypothesis saved with metric, direction, magnitude.2829### 2. Selection30- **Control:** current experience.31- **Treatment:** new experience.32- **Randomisation:** user-level assignment (cookie / account / session); must be independent of behaviour.33- **Segmentation:** if testing on a subset (e.g. new users only), state why.34- **Exclusions:** users with special conditions (e.g. VIP, internal, disabled users) must be handled ethically.3536**Completion criterion:** assignment method saved; exclusions noted.3738### 3. Metrics39- **Primary:** the metric that determines success (e.g. conversion rate, click-through rate, retention).40- **Secondary:** supportive metrics (e.g. revenue per user, time on task, error rate).41- **Guardrail:** metrics that must not degrade (e.g. page load time, accessibility, error rate, support tickets).4243**Completion criterion:** metrics defined with direction (increase / decrease / maintain).4445### 4. Duration and sample size46Compute:47- **Minimum detectable effect (MDE):** smallest improvement that is practically meaningful.48- **Power:** 1 — β (typically 0.8 or 0.9).49- **Significance level α:** 0.05 (or 0.01 for critical decisions).50- **Sample size per variant:** derived from baseline rate, MDE, α, power.51- **Duration:** sample size / daily traffic per variant; account for weekly seasonality; run at least one full week.5253**Completion criterion:** sample size and duration computed.5455### 5. Analysis plan56- **Statistical test:** z-test for proportions; t-test for means; Mann-Whitney for non-normal; bootstrap for complex metrics.57- **Segmentation:** analyse by subgroups (device, region, user type) — but do not over-segment (risk of false positives).58- **Interaction:** check if treatment effect varies by subgroup (interaction test).59- **Multiple testing correction:** Bonferroni or FDR if testing many metrics or segments.6061**Completion criterion:** analysis plan saved.6263### 6. Rollback / escalation rules64- **Early stopping rules:** if guardrail metric degrades beyond threshold during first N% of planned duration, stop.65- **Success criteria:** primary metric improves by ≥ MDE with p < α; guardrail metrics not degraded.66- **Rollback:** revert treatment for all users if success criteria not met by end of planned duration, or if guardrail fails.6768**Completion criterion:** rollback and escalation rules saved.
quantumquirkxyz