Contract
- Input: feature change, user segment, metric of interest.
- Output: test design with power analysis and rollback rules.
- Scope: designs experiment; does not expose users to unapproved changes.
- Rule: designs experiment; does not expose users to unapproved changes.
- Rule: documented standards override defaults; explicit project rules take precedence.
- Rule: if blocked by missing context or dependencies, surface the blocker before proceeding.
Provenance
| Question |
Answer |
| What is the source of truth? |
The user's request, originating spec/issue, and the skill's declared outputs |
| What is in scope? |
Work covered by the skill's acceptance criteria and completion rules |
| What is explicitly out of scope? |
Files, behaviors, and decisions outside the skill's declared boundary |
| Who or what consumes this artifact afterward? |
The next skill in the workflow or the user |
| What evidence proves it is done? |
Completion criteria met, artifact saved, validation passed |
| What risk remains? |
Subjective judgment calls, missing context, or external dependency failures |
Artifact
Emit ProdAbTestingArtifact as both:
- JSON:
.agents/skills/platform/artifacts/prod-ab-testing/{identifier}.json
- Markdown view: same filename with
.md extension
A/B Testing Design
Design an A/B test — hypothesis, randomisation, metrics, statistical power — with valid inference.
Process
1. Hypothesis
State a clear, testable statement:
- "Changing the checkout button from grey to green increases conversion rate by ≥ 2%."
- Must specify: metric, direction, magnitude, user segment.
Completion criterion: hypothesis saved with metric, direction, magnitude.
2. Selection
- Control: current experience.
- Treatment: new experience.
- Randomisation: user-level assignment (cookie / account / session); must be independent of behaviour.
- Segmentation: if testing on a subset (e.g. new users only), state why.
- Exclusions: users with special conditions (e.g. VIP, internal, disabled users) must be handled ethically.
Completion criterion: assignment method saved; exclusions noted.
3. Metrics
- Primary: the metric that determines success (e.g. conversion rate, click-through rate, retention).
- Secondary: supportive metrics (e.g. revenue per user, time on task, error rate).
- Guardrail: metrics that must not degrade (e.g. page load time, accessibility, error rate, support tickets).
Completion criterion: metrics defined with direction (increase / decrease / maintain).
4. Duration and sample size
Compute:
- Minimum detectable effect (MDE): smallest improvement that is practically meaningful.
- Power: 1 — β (typically 0.8 or 0.9).
- Significance level α: 0.05 (or 0.01 for critical decisions).
- Sample size per variant: derived from baseline rate, MDE, α, power.
- Duration: sample size / daily traffic per variant; account for weekly seasonality; run at least one full week.
Completion criterion: sample size and duration computed.
5. Analysis plan
- Statistical test: z-test for proportions; t-test for means; Mann-Whitney for non-normal; bootstrap for complex metrics.
- Segmentation: analyse by subgroups (device, region, user type) — but do not over-segment (risk of false positives).
- Interaction: check if treatment effect varies by subgroup (interaction test).
- Multiple testing correction: Bonferroni or FDR if testing many metrics or segments.
Completion criterion: analysis plan saved.
6. Rollback / escalation rules
- Early stopping rules: if guardrail metric degrades beyond threshold during first N% of planned duration, stop.
- Success criteria: primary metric improves by ≥ MDE with p < α; guardrail metrics not degraded.
- Rollback: revert treatment for all users if success criteria not met by end of planned duration, or if guardrail fails.
Completion criterion: rollback and escalation rules saved.
Completion
- the skill's completion criteria are explicitly checked
- the artifact is saved and validated
- any blockers or skipped validations are documented
- the next consumer is identified or the work is handed off
@include .agents/skills/platform/contract-base.xml
1---2name: prod-ab-testing-33category: "product"4maturity: "stable"5version: "1"6description: "Design A/B tests — hypothesis, control/treatment, randomization, metrics, statistical power, duration — with valid inference and rollback rules."7license: "MIT"8capabilities:9 - apply prod ab testing methods within explicit product boundaries10 - analyze design a/b tests — hypothesis, control/treatment, randomization, metrics, statistical power, duration — with valid inference and rollback rules with assumptions, constraints, and failure modes11 - produce and validate a product artifact with evidence, decisions, and next steps12outputs: "test design with power analysis and rollback rules."13sideEffects: []14dependencies: []15stopCondition: "Test design saved; analysis plan saved; rollback rules defined; statistical power computed."16risk: "medium"17trustTier: "3"18maxIterations: "5"19promptVersion: "2.0"20artifactType: "pull-request"21modelTier: "reasoning"22evaluators: ["behavioral", "regression"]23fixturesPath: .agents/skills/platform/fixtures/behavioral/prod-ab-testing.json24diataxis: "how-to"25tags: ["product"]26compatibility: []27approvalRequired: "true"28approvalFor: []29metadata:30 source: https://skillmd.com/skills/quantumquirkxyz/prod-ab-testing-331---3233## Contract3435- Input: feature change, user segment, metric of interest.36- Output: test design with power analysis and rollback rules.37- Scope: designs experiment; does not expose users to unapproved changes.38- Rule: designs experiment; does not expose users to unapproved changes.39- Rule: documented standards override defaults; explicit project rules take precedence.40- Rule: if blocked by missing context or dependencies, surface the blocker before proceeding.4142## Provenance4344| Question | Answer |45|---|---|46| What is the source of truth? | The user's request, originating spec/issue, and the skill's declared outputs |47| What is in scope? | Work covered by the skill's acceptance criteria and completion rules |48| What is explicitly out of scope? | Files, behaviors, and decisions outside the skill's declared boundary |49| Who or what consumes this artifact afterward? | The next skill in the workflow or the user |50| What evidence proves it is done? | Completion criteria met, artifact saved, validation passed |51| What risk remains? | Subjective judgment calls, missing context, or external dependency failures525354## Artifact5556Emit `ProdAbTestingArtifact` as both:57- JSON: `.agents/skills/platform/artifacts/prod-ab-testing/{identifier}.json`58- Markdown view: same filename with `.md` extension596061# 62# 63# A/B Testing Design6465Design an **A/B test** — hypothesis, randomisation, metrics, statistical power — with valid inference.6667## Process6869### 1. Hypothesis70State a clear, testable statement:71- "Changing the checkout button from grey to green increases conversion rate by ≥ 2%."72- Must specify: metric, direction, magnitude, user segment.7374**Completion criterion:** hypothesis saved with metric, direction, magnitude.7576### 2. Selection77- **Control:** current experience.78- **Treatment:** new experience.79- **Randomisation:** user-level assignment (cookie / account / session); must be independent of behaviour.80- **Segmentation:** if testing on a subset (e.g. new users only), state why.81- **Exclusions:** users with special conditions (e.g. VIP, internal, disabled users) must be handled ethically.8283**Completion criterion:** assignment method saved; exclusions noted.8485### 3. Metrics86- **Primary:** the metric that determines success (e.g. conversion rate, click-through rate, retention).87- **Secondary:** supportive metrics (e.g. revenue per user, time on task, error rate).88- **Guardrail:** metrics that must not degrade (e.g. page load time, accessibility, error rate, support tickets).8990**Completion criterion:** metrics defined with direction (increase / decrease / maintain).9192### 4. Duration and sample size93Compute:94- **Minimum detectable effect (MDE):** smallest improvement that is practically meaningful.95- **Power:** 1 — β (typically 0.8 or 0.9).96- **Significance level α:** 0.05 (or 0.01 for critical decisions).97- **Sample size per variant:** derived from baseline rate, MDE, α, power.98- **Duration:** sample size / daily traffic per variant; account for weekly seasonality; run at least one full week.99100**Completion criterion:** sample size and duration computed.101102### 5. Analysis plan103- **Statistical test:** z-test for proportions; t-test for means; Mann-Whitney for non-normal; bootstrap for complex metrics.104- **Segmentation:** analyse by subgroups (device, region, user type) — but do not over-segment (risk of false positives).105- **Interaction:** check if treatment effect varies by subgroup (interaction test).106- **Multiple testing correction:** Bonferroni or FDR if testing many metrics or segments.107108**Completion criterion:** analysis plan saved.109110### 6. Rollback / escalation rules111- **Early stopping rules:** if guardrail metric degrades beyond threshold during first N% of planned duration, stop.112- **Success criteria:** primary metric improves by ≥ MDE with p < α; guardrail metrics not degraded.113- **Rollback:** revert treatment for all users if success criteria not met by end of planned duration, or if guardrail fails.114115**Completion criterion:** rollback and escalation rules saved.116117## Completion118119- the skill's completion criteria are explicitly checked120- the artifact is saved and validated121- any blockers or skipped validations are documented122- the next consumer is identified or the work is handed off123---124@include .agents/skills/platform/contract-base.xml