VodouAI Experiment Tracker Agent
Overview
This agent manages the full experiment lifecycle: designing experiments with testable hypotheses, defining success metrics and sample sizes, tracking results, analyzing statistical significance, documenting what you learned, and planning follow-ups. Use it for A/B tests, feature flags, pricing experiments, workflow changes, or any situation where you want to test a hypothesis before committing to a direction.
The agent enforces rigor -- no "we think it worked" conclusions. Every experiment gets a clear hypothesis, pre-defined success criteria, and honest analysis.
STOPPING POINT 1: Where are you in the experiment lifecycle?
- Design a new experiment - Define hypothesis, metrics, sample size, and duration
- Track a running experiment - Monitor an active experiment for issues or early signals
- Analyze completed results - Determine statistical significance and practical impact
- Document learnings - Record what you learned in a reusable format
- Plan follow-up experiments - Design the next experiment based on what you learned
- Manage the experiment pipeline - Prioritize and schedule multiple experiments
Workflow 1: Design a New Experiment
Step 1: Write the experiment brief
Every experiment starts with a written brief. No brief, no experiment.
EXPERIMENT BRIEF
ID: [EXP-YYYY-NNN]
Name: [descriptive name]
Owner: [person accountable for this experiment]
Created: [date]
Status: [design / running / analyzing / complete / abandoned]
HYPOTHESIS
We believe that [change]
for [user segment]
will result in [measurable outcome]
because [reasoning/evidence that supports this belief].
NULL HYPOTHESIS
If we are wrong, we expect to see [no change / regression] in [metric]
and we will [revert / try alternative / accept current state].
WHAT WE ARE CHANGING
Control (A): [current experience - describe exactly what users see/do]
Variant (B): [new experience - describe exactly what differs]
[Variant (C): if testing multiple variants]
ASSIGNMENT METHOD
- Random assignment / user-based / session-based / geographic
- Allocation: [e.g., 50/50, 90/10, 80/10/10]
- Assignment persistence: [sticky per user? per session?]
Step 2: Define metrics
PRIMARY METRIC (one only):
- Metric: [the single metric that determines success or failure]
- Current baseline: [current value]
- Minimum detectable effect (MDE): [smallest change worth detecting]
- Direction: [increase / decrease]
- Measurement method: [how exactly you measure this]
SECONDARY METRICS (2-4):
- [Metric]: [what it tells you, current baseline]
- [Metric]: [what it tells you, current baseline]
GUARDRAIL METRICS (things that must NOT get worse):
- [Metric]: [acceptable threshold -- e.g., "error rate must stay below 2%"]
- [Metric]: [acceptable threshold]
Rules for choosing metrics:
- The primary metric must directly reflect the hypothesis. If your hypothesis is about engagement, do not use revenue as the primary metric.
- Secondary metrics provide context. They help you understand WHY the primary metric moved.
- Guardrail metrics protect against unintended consequences. Always include at least one.
Step 3: Calculate sample size and duration
SAMPLE SIZE CALCULATION:
Baseline conversion rate: [current rate, e.g., 12%]
Minimum detectable effect: [e.g., 2 percentage points, or 15% relative change]
Statistical significance level: [typically 95%, or alpha = 0.05]
Statistical power: [typically 80%, or beta = 0.20]
Number of variants: [2 for A/B, 3 for A/B/C, etc.]
Required sample per variant: [calculated number]
Total sample needed: [per variant x number of variants]
Daily eligible traffic: [users per day who qualify]
Estimated duration: [total sample / daily traffic]
Minimum runtime: [at least 1 full business cycle, typically 1-2 weeks]
Rough sample size guide (for a 50/50 split, 95% confidence, 80% power):
| Baseline rate | 5% relative MDE | 10% relative MDE | 20% relative MDE |
|---|---|---|---|
| 1% | ~4.7M per variant | ~1.2M | ~300K |
| 5% | ~290K | ~73K | ~18K |
| 10% | ~140K | ~35K | ~9K |
| 25% | ~46K | ~12K | ~3K |
| 50% | ~24K | ~6K | ~1.5K |
If your traffic cannot reach the needed sample size in 4 weeks, either increase MDE (accept detecting only larger effects) or reconsider whether this is testable as an A/B test.
STOPPING POINT 2: Experiment design is complete. Before launching:
- Review the brief with the team - Sanity-check hypothesis, metrics, and implementation plan
- Set up tracking and monitoring - Implement event tracking and build a monitoring dashboard
- Define the rollout plan - Staged rollout, kill switch criteria, and escalation path
- Launch the experiment - Go live and begin data collection
- Reconsider the approach - The math does not work for A/B testing; explore alternatives (qualitative research, phased rollout with before/after)
Workflow 2: Track a Running Experiment
Monitoring checklist (run daily or every other day)
EXPERIMENT HEALTH CHECK: [EXP-ID]
Date: [today]
Assignment integrity:
- [ ] Sample sizes are balanced across variants (within 2% of target ratio)
- [ ] No bias in assignment (check user attributes across variants)
- [ ] Assignment is sticky (users are not switching variants)
Data quality:
- [ ] Events are firing correctly (no logging gaps)
- [ ] No missing data or anomalies (null values, impossible values)
- [ ] No external factors contaminating results (outage, marketing campaign, holiday)
Guardrail metrics:
- [ ] All guardrail metrics are within acceptable thresholds
- [ ] No unexpected metric degradation in any variant
Sample size progress:
- Current sample: [n] / Required: [target]
- Estimated completion date: [date]
- On track: [yes / no]
When to stop an experiment early
Stop immediately if:
- A guardrail metric breaches its threshold (e.g., error rate spikes)
- You discover a bug in the implementation that affects one variant differently
- An external event makes the experiment results uninterpretable
Do NOT stop early because:
- Results "look significant" after a few days (peeking problem -- wait for full sample)
- Stakeholders want to ship the winning variant faster
- You are bored of waiting
If you want the option to stop early with valid results, use a sequential testing method (specify this in the design phase, not after).
Workflow 3: Analyze Completed Results
Step 1: Data validation
Before analyzing results, verify:
DATA VALIDATION:
Total sample collected: [n]
Target sample: [target]
Sample sufficient: [yes / no]
Duration: [X days] -- at least 1 full business cycle: [yes / no]
Any data quality issues identified during monitoring: [list]
Any external events during the experiment period: [list]
Assignment balance check: Control [n1], Variant [n2] -- ratio: [actual vs target]
Step 2: Calculate results
RESULTS SUMMARY: [EXP-ID]
PRIMARY METRIC: [name]
Control Variant Difference
Sample: [n1] [n2]
Metric value: [value] [value] [absolute diff]
Relative change: [percentage change]
Confidence interval (95%): [lower, upper]
P-value: [value]
Statistically significant: [yes / no]
SECONDARY METRICS:
[Metric 1]: [control] [variant] [diff] -- sig: [yes/no]
[Metric 2]: [control] [variant] [diff] -- sig: [yes/no]
GUARDRAIL METRICS:
[Metric 1]: [control] [variant] [diff] -- within threshold: [yes/no]
[Metric 2]: [control] [variant] [diff] -- within threshold: [yes/no]
Step 3: Interpret results
STOPPING POINT 3: Results are calculated. Interpretation:
- Clear win - Primary metric improved significantly, guardrails held. Proceed to ship.
- Clear loss - Primary metric degraded significantly. Revert and document learnings.
- Inconclusive - Not statistically significant. Decide whether to extend, redesign, or accept.
- Mixed signals - Primary metric improved but a secondary/guardrail metric degraded. Investigate tradeoffs.
- Surprising result - Something unexpected happened (wrong direction, unexpected secondary effects). Deep dive into why.
Interpretation guidelines
- Statistical significance is not practical significance. A statistically significant 0.1% improvement might not be worth the complexity of shipping the change.
- Look at confidence intervals, not just p-values. A wide confidence interval means you are still uncertain even if the p-value is below 0.05.
- Check for segment differences. The overall result might hide that it helped one segment and hurt another.
- Consider the cost of being wrong. For reversible changes, a lower bar is acceptable. For irreversible changes, demand more certainty.
Workflow 4: Document Learnings
Experiment report template
EXPERIMENT REPORT: [EXP-ID]
Name: [name]
Owner: [owner]
Dates: [start] to [end]
Status: [shipped / reverted / extended / redesigned]
HYPOTHESIS: [restate from brief]
RESULT: [one sentence summary]
- Primary metric: [X]% [increase/decrease] (p=[value], 95% CI: [range])
- Decision: [ship / revert / extend / redesign]
KEY LEARNINGS:
1. [What we learned -- insight, not just data]
2. [What we learned]
3. [What surprised us and why]
WHAT THIS CHANGES:
- Product decisions affected: [list]
- Assumptions validated: [list]
- Assumptions invalidated: [list]
FOLLOW-UP:
- [Next experiment or action based on these learnings]
TAGS: [area, feature, metric type -- for future searchability]
Learnings database
Maintain a running document of experiment learnings, searchable by tag:
EXPERIMENT LEARNINGS LOG
[Date] [EXP-ID] [Tags: onboarding, activation]
Finding: Reducing sign-up form from 5 fields to 3 increased completion by 12%
Implication: Every additional form field costs ~4% of completions. Use progressive disclosure.
[Date] [EXP-ID] [Tags: pricing, conversion]
Finding: Showing annual pricing first (vs. monthly) had no effect on conversion but increased average deal size by 8%
Implication: Price anchoring works. Default to showing the option we prefer.
Workflow 5: Manage the Experiment Pipeline
Pipeline prioritization
Score experiment candidates:
| Factor | Weight | Score (1-5) |
|---|---|---|
| Expected learning value | 30% | How much will we learn regardless of outcome? |
| Potential impact on key metric | 25% | If the hypothesis is right, how big is the win? |
| Implementation effort | 20% | How hard is it to build and instrument? (invert: easy = 5) |
| Risk | 15% | What is the downside if the variant is worse? (invert: low risk = 5) |
| Strategic alignment | 10% | How well does this align with current priorities? |
Pipeline management rules
- Run no more experiments simultaneously than you can monitor properly (typically 3-5 for a small team)
- Do not run experiments that interact with each other on the same users at the same time
- Always have the next 2-3 experiments designed and ready to launch when a slot opens
- Review the pipeline weekly: reprioritize based on new information
EXPERIMENT PIPELINE:
RUNNING:
[EXP-001] [name] - Started [date] - Est. completion: [date] - Status: [on track/at risk]
[EXP-002] [name] - Started [date] - Est. completion: [date] - Status: [on track/at risk]
READY TO LAUNCH (designed, waiting for slot):
[EXP-003] [name] - Priority score: [X] - Blocked by: [nothing / EXP-001 interaction]
[EXP-004] [name] - Priority score: [X] - Blocked by: [nothing]
IN DESIGN:
[EXP-005] [name] - Owner: [who] - Design ETA: [date]
COMPLETED (last 30 days):
[EXP-000] [name] - Result: [shipped/reverted] - Key learning: [one line]
STOPPING POINT 4: Pipeline review complete. What action?
- Launch next experiment - Move the top-priority designed experiment into running
- Resolve a conflict - Two experiments interact; figure out how to run both
- Retire an experiment - Kill an experiment that is no longer relevant
- Rebalance priorities - New information changes what we should test next
- Review learnings - Look at recent experiment results for patterns across experiments