# Ab Experimentation

> Plans and interprets A/B tests with clear hypotheses, primary metrics, guardrails, and statistical conclusions. Use when designing, analyzing, or stopping controlled experiments.

- Skill: `poly-gents/ab-experimentation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add poly-gents/ab-experimentation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/poly-gents/ab-experimentation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: poly-gents (https://skillmd.com/u/poly-gents)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/poly-gents/ab-experimentation

---


# A/B experimentation

## Purpose

Help design and interpret controlled experiments: hypothesis, primary metric, guardrails, sample size/runtime, and statistically sound conclusions—avoiding peeking bias and vague "try again" advice.

## When to Use

- User asks for A/B test design, analysis, or whether to ship variant B.
- Choosing primary metric, guardrails, and runtime.
- Interpreting inconclusive or negative results.
- Linking experiment data in sheets/BI when tools exist.

## When NOT to Use

- Feature flag rollout without experiment framing → **feature-flagging**.
- Product roadmap prioritization without experiment → **prioritize-roadmap**.
- Casual UX opinions without measurement plan.

## Expected Outcome

- Experiment brief (hypothesis, metrics, population, duration).
- Analysis with effect size, interval, and decision (ship, extend, stop, redesign).
- Concrete next steps if underpowered—not vague retry language.

## Inputs to Gather

- Hypothesis and user-facing change.
- Primary metric and guardrails (latency, errors, revenue).
- Assignment unit (user, session, account) and integrity checks.
- Runtime, sample size, and peeking/sequential rules if any.
- Novelty/seasonality concerns.

## Workflow

1. Lock hypothesis and primary metric before viewing results (or note breach if peeked).
2. Validate assignment integrity and sample ratio mismatch.
3. Report effect size, CI, and runtime; compare guardrails.
4. If inconclusive: compute power or recommend duration/design change with numbers.
5. Document ship/kill/iterate recommendation.

## Domain guidance

- **Hypothesis** and **primary metric** first; guardrails (latency, errors) second.
- Check assignment integrity, novelty, and seasonality.
- Report effect size, confidence interval, and runtime; avoid peeking bias unless using sequential methods.
- If inconclusive: recommend power, duration, or design change—not vague "try again."

## Tool Availability Rules

| Access | Behavior |
|--------|----------|
| Full tool access | Pull experiment metrics from sheets/BI tools when connected. |
| Read-only | Analyze user-supplied tables; show calculations. |
| No data | Provide design template; do not invent significance. |

### Related tool sets

- `google-sheets`
- `jira`

## Review / Decision / Execution Criteria

- One primary metric; guardrails explicit.
- Call out peeking, SRM, and low power explicitly.
- Decisions tied to pre-registered success criteria when available.

## Output Format

1. Experiment summary.
2. Results (primary + guardrails) with effect and CI.
3. Decision and rationale.
4. Next steps (extend, redesign, ship).

## Quality Bar

- Statistically literate but readable for PMs.
- No fabricated p-values or lift percentages.

## Safety and Boundaries

- Do not recommend shipping on guardrail regressions without explicit acceptance.
- Ethical/user harm review for sensitive experiments.

## Escalation / Dispatch Rules

- Rollout mechanics → **feature-flagging**.
- Metric definitions in dashboards → observability skills.

## References

- `skills/old_skills.json` (`ab-experimentation`).
- `skills/skill.instruction.md`, `skills/meta.instructions.md`