UitbreidenOS Experiment Tracker Skill
When to activate
- Running an A/B test and need a structured hypothesis and success metrics before launch
- Calculating sample size and test duration before starting an experiment
- Analysing test results and determining if you've reached statistical significance
- Documenting experiment learnings for the team or experiment log
- Prioritising which experiments to run next based on ICE or RICE scoring
- A test has concluded and you need to decide: ship, kill, or iterate
When NOT to use
- Full experiment design from scratch — use
/experiment-designerfor that - Analytics setup and event tracking — use
/analytics-tracking - Interpreting qualitative research or user interviews — different methodology
- When your sample size is too small to run any valid test (< 100 conversions per variant expected)
Instructions
Hypothesis writing framework
Write a structured experiment hypothesis for my A/B test.
Test idea: [describe the change you want to make]
Page / feature: [where in the product or funnel]
Current state: [what exists today]
Proposed change: [what you want to test]
Produce a hypothesis in this format:
We believe that [CHANGE]
for [AUDIENCE SEGMENT]
will result in [EXPECTED OUTCOME]
because [MECHANISM / REASONING]
We'll know this is true when [MEASURABLE SUCCESS CRITERIA]
and the test has reached [MINIMUM SAMPLE SIZE] conversions per variant
with [95%] statistical confidence.
Also produce:
- Primary metric: [the one metric that determines win/loss]
- Secondary metrics: [guardrail metrics — must not regress]
- Minimum detectable effect (MDE): [smallest improvement worth shipping]
- Risk: [what could go wrong — novelty effect, segment interaction, etc.]
Sample size calculator
Calculate the required sample size for my A/B test.
Test type: [conversion rate / revenue per user / retention / engagement]
Current baseline rate: [X%] (e.g., current conversion rate)
Minimum detectable effect (MDE): [X%] (relative improvement worth detecting)
— Conservative: 5-10% relative lift (need large sample)
— Moderate: 15-20% relative lift (typical)
— Aggressive: 30%+ relative lift (small sample, only detect large changes)
Statistical significance: [95%] (standard) or [90%] (acceptable for low-stakes tests)
Statistical power: [80%] (standard) or [90%] (high-stakes tests)
Number of variants: [2] (A vs. B) or [3+] (multi-variant — divide by n-1)
Calculation:
For conversion rate tests, use the two-proportion z-test:
Required n per variant = (z_α/2 + z_β)² × [p1(1-p1) + p2(1-p2)] / (p1 - p2)²
Where:
- p1 = baseline rate
- p2 = baseline rate × (1 + MDE)
- z_α/2 = 1.96 (95% significance)
- z_β = 0.842 (80% power)
Provide:
- Required conversions per variant: [N]
- Required visitors per variant (at current conversion rate): [N]
- Expected test duration at [current traffic] per day: [X days / weeks]
- Warning if duration > 8 weeks (seasonal effects will contaminate results)
- Warning if conversions per variant < 100 (test is underpowered — increase MDE or wait)
Show me the numbers for my test.
Pre-launch checklist
Run a pre-launch check on my A/B test before I start it.
Test name: [name]
Tool: [Optimizely / VWO / LaunchDarkly / GrowthBook / custom]
Hypothesis: [from hypothesis framework above]
Sample size needed: [N per variant]
Expected traffic per day: [N visitors]
Expected test duration: [X days]
Pre-launch checklist:
TRACKING
□ Primary metric is tracked correctly (event fires on conversion, not page load)
□ Secondary/guardrail metrics are tracked (revenue, session length, error rate)
□ Test assignment event is tracked (so you can segment by variant in analytics)
□ No existing funnel breaks or bugs on the control — test a broken baseline = invalid results
□ QA in staging: confirm variant displays correctly across browsers + mobile
SETUP
□ Traffic split confirmed: [50/50 or X/Y — document the split]
□ Targeting rules documented: [who is included / excluded]
□ Mutual exclusivity: does this test conflict with any other running test?
□ Holdback if needed: if test affects revenue significantly, keep 5-10% out of all tests
DURATION
□ Run for at least 2 full business cycles (minimum 2 weeks — never stop at first significance)
□ Do not peek at results daily and stop early — this inflates false positive rate
□ Set a hard stop date: [date] — do not extend without a documented reason
RISK
□ Can you roll back the variant instantly if a guardrail metric crashes?
□ Is there a novelty effect risk? (new UI = short-term lift that doesn't persist)
□ Will this test segment interact with another test? Map your test matrix.
Sign off when all boxes checked.
Results analysis
Analyse my A/B test results and tell me what to do.
Test: [name]
Duration: [X days]
Tool: [analytics platform]
Results:
Control (A):
- Visitors: [N]
- Conversions: [N]
- Conversion rate: [X%]
- Revenue per visitor (if applicable): $[X]
Variant (B):
- Visitors: [N]
- Conversions: [N]
- Conversion rate: [X%]
- Revenue per visitor (if applicable): $[X]
Relative lift: [(B-A)/A × 100]%
P-value: [X] (from your test tool)
Confidence: [X%]
Statistical significance reached: [Yes / No]
Analysis:
DECISION FRAMEWORK:
1. Is the result statistically significant at 95%?
YES → proceed to business impact analysis
NO → check: did you reach your required sample size?
- If yes + no significance: the effect is smaller than MDE → likely not worth shipping
- If no: extend the test or accept you cannot detect an effect this small
2. Is the lift meaningful in dollar terms?
Annual revenue impact of this lift = [calculation]:
Lift × daily conversions × average order value × 365 = $[X]/year
If annual impact < cost to implement permanently, reconsider.
3. Did any guardrail metrics regress?
Revenue per visitor, session length, error rate, support contacts?
If yes: do NOT ship even if primary metric is positive. A lift in signups that doubles support contacts is not a win.
4. Segment analysis — does the lift hold across:
- Mobile vs. desktop?
- New vs. returning users?
- Traffic source (paid vs. organic)?
- Geography?
Significant interaction effects suggest the variant works for a segment, not universally.
DECISION: [SHIP / KILL / SEGMENT-SHIP / ITERATE]
Reasoning: [specific, based on the numbers]
Next experiment: [what to test next based on these results]
Experiment log template
Document this experiment for the team experiment log.
Experiment: [name — searchable, descriptive]
Date: [start] → [end]
Owner: [name]
Team: [growth / product / marketing]
Status: [running / concluded]
## Hypothesis
[From hypothesis framework]
## Setup
- Tool: [Optimizely / VWO / custom]
- Traffic split: [50/50]
- Audience: [all users / new users / mobile / etc.]
- Targeting: [URL, segment, feature flag]
## Results
| Metric | Control | Variant | Lift | Significance |
|---|---|---|---|---|
| Primary: [metric] | [X%] | [X%] | [+X%] | [95%] |
| Guardrail: [metric] | [X] | [X] | [+/-X%] | [N/A] |
| Guardrail: [metric] | [X] | [X] | [+/-X%] | [N/A] |
Sample: [N] per variant | Duration: [X days] | P-value: [X]
## Decision
[SHIPPED / KILLED / ITERATED]
Rationale: [why]
## Learning
[What this tells us about user behaviour — not just "variant won"]
[What to test next]
## Annual impact (if shipped)
$[X] incremental revenue or [X%] metric improvement
Experiment prioritisation
Prioritise my experiment backlog using ICE scoring.
My experiment ideas:
1. [Idea 1]
2. [Idea 2]
3. [Idea 3]
[add as many as needed]
Score each on ICE:
IMPACT (1-10): If this works at the expected lift, how big is the revenue/metric impact?
- 10: >$100K annual impact or >20% lift on a key metric
- 7: $20-100K or 10-20% lift
- 4: $5-20K or 5-10% lift
- 1: <$5K or <5% lift
CONFIDENCE (1-10): How sure are we this will work?
- 10: Prior data, user research, and proven elsewhere all point to this
- 7: Strong qualitative signal or similar test worked in adjacent area
- 4: Theory-based, no direct evidence
- 1: Pure gut feeling
EASE (1-10): How fast can we run this test?
- 10: Copywriting change only, no dev needed, live in 1 day
- 7: Minor frontend change, 1-2 days dev
- 4: Backend or logic change, 1 week dev
- 1: Architecture change, 4+ weeks dev
ICE Score = (Impact + Confidence + Ease) / 3
Rank all ideas by ICE score. Flag any that require >2 weeks of engineering as High Effort — evaluate if they should go in the product backlog instead.
Produce ranked experiment calendar for next 4 weeks.
Example
User: I want to test changing our pricing page CTA from "Start Free Trial" to "See Pricing" because I think users want to see numbers before committing. Baseline conversion is 3.2%. What sample size do I need and how do I set this up?
Claude's output:
Hypothesis: We believe that changing the primary CTA from "Start Free Trial" to "See Pricing" for all visitors landing on /pricing will result in a higher click-through rate to the checkout or plan selection page because users in a price-comparison mindset want to see numbers before committing to a "trial" framing. We'll know this is true when the CTA click rate increases by ≥ 15% (relative) with 95% statistical confidence and 80% power.
Sample size calculation:
- Baseline: 3.2% CTA click rate
- MDE: 15% relative = 3.68% target rate
- Required: 2,843 visitors per variant (5,686 total)
- At 500 visitors/day on the pricing page: 12-day test
- At 200 visitors/day: 29-day test — run it, worth the wait
Pre-launch:
- Confirm CTA click event fires in analytics (not page load)
- Track: plan selection page views (guardrail — should not drop)
- Track: trial signups (the real downstream metric — segment by variant in your analytics tool)
- Do not stop early — even if you hit significance in week 1, run the full 2 weeks
What to watch: If "See Pricing" wins on clicks but trial signups don't increase, users are more curious but not more committed. That's a pricing clarity problem, not a CTA problem — next test is the pricing page itself.