poly-gents A/B experimentation
Purpose
Help design and interpret controlled experiments: hypothesis, primary metric, guardrails, sample size/runtime, and statistically sound conclusions—avoiding peeking bias and vague "try again" advice.
When to Use
- User asks for A/B test design, analysis, or whether to ship variant B.
- Choosing primary metric, guardrails, and runtime.
- Interpreting inconclusive or negative results.
- Linking experiment data in sheets/BI when tools exist.
When NOT to Use
- Feature flag rollout without experiment framing → feature-flagging.
- Product roadmap prioritization without experiment → prioritize-roadmap.
- Casual UX opinions without measurement plan.
Expected Outcome
- Experiment brief (hypothesis, metrics, population, duration).
- Analysis with effect size, interval, and decision (ship, extend, stop, redesign).
- Concrete next steps if underpowered—not vague retry language.
Inputs to Gather
- Hypothesis and user-facing change.
- Primary metric and guardrails (latency, errors, revenue).
- Assignment unit (user, session, account) and integrity checks.
- Runtime, sample size, and peeking/sequential rules if any.
- Novelty/seasonality concerns.
Workflow
- Lock hypothesis and primary metric before viewing results (or note breach if peeked).
- Validate assignment integrity and sample ratio mismatch.
- Report effect size, CI, and runtime; compare guardrails.
- If inconclusive: compute power or recommend duration/design change with numbers.
- Document ship/kill/iterate recommendation.
Domain guidance
- Hypothesis and primary metric first; guardrails (latency, errors) second.
- Check assignment integrity, novelty, and seasonality.
- Report effect size, confidence interval, and runtime; avoid peeking bias unless using sequential methods.
- If inconclusive: recommend power, duration, or design change—not vague "try again."
Tool Availability Rules
| Access | Behavior |
|---|---|
| Full tool access | Pull experiment metrics from sheets/BI tools when connected. |
| Read-only | Analyze user-supplied tables; show calculations. |
| No data | Provide design template; do not invent significance. |
Related tool sets
google-sheetsjira
Review / Decision / Execution Criteria
- One primary metric; guardrails explicit.
- Call out peeking, SRM, and low power explicitly.
- Decisions tied to pre-registered success criteria when available.
Output Format
- Experiment summary.
- Results (primary + guardrails) with effect and CI.
- Decision and rationale.
- Next steps (extend, redesign, ship).
Quality Bar
- Statistically literate but readable for PMs.
- No fabricated p-values or lift percentages.
Safety and Boundaries
- Do not recommend shipping on guardrail regressions without explicit acceptance.
- Ethical/user harm review for sensitive experiments.
Escalation / Dispatch Rules
- Rollout mechanics → feature-flagging.
- Metric definitions in dashboards → observability skills.
References
skills/old_skills.json(ab-experimentation).skills/skill.instruction.md,skills/meta.instructions.md