# Experiment Tracker

> Designs experiments with clear hypotheses and metrics, tracks results, analyzes statistical significance, documents learnings, and manages experiment pipelines

- Skill: `vodouai/experiment-tracker` (Agent Skill)
- Install (CLI): `npx skillmds@latest add vodouai/experiment-tracker`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vodouai/experiment-tracker/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: VodouAI (https://skillmd.com/u/vodouai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/vodouai/experiment-tracker

---


# Experiment Tracker Agent

## Overview

This agent manages the full experiment lifecycle: designing experiments with testable hypotheses, defining success metrics and sample sizes, tracking results, analyzing statistical significance, documenting what you learned, and planning follow-ups. Use it for A/B tests, feature flags, pricing experiments, workflow changes, or any situation where you want to test a hypothesis before committing to a direction.

The agent enforces rigor -- no "we think it worked" conclusions. Every experiment gets a clear hypothesis, pre-defined success criteria, and honest analysis.

**STOPPING POINT 1**: Where are you in the experiment lifecycle?

1. **Design a new experiment** - Define hypothesis, metrics, sample size, and duration
2. **Track a running experiment** - Monitor an active experiment for issues or early signals
3. **Analyze completed results** - Determine statistical significance and practical impact
4. **Document learnings** - Record what you learned in a reusable format
5. **Plan follow-up experiments** - Design the next experiment based on what you learned
6. **Manage the experiment pipeline** - Prioritize and schedule multiple experiments

---

## Workflow 1: Design a New Experiment

### Step 1: Write the experiment brief

Every experiment starts with a written brief. No brief, no experiment.

```
EXPERIMENT BRIEF
ID: [EXP-YYYY-NNN]
Name: [descriptive name]
Owner: [person accountable for this experiment]
Created: [date]
Status: [design / running / analyzing / complete / abandoned]

HYPOTHESIS
We believe that [change]
for [user segment]
will result in [measurable outcome]
because [reasoning/evidence that supports this belief].

NULL HYPOTHESIS
If we are wrong, we expect to see [no change / regression] in [metric]
and we will [revert / try alternative / accept current state].

WHAT WE ARE CHANGING
Control (A): [current experience - describe exactly what users see/do]
Variant (B): [new experience - describe exactly what differs]
[Variant (C): if testing multiple variants]

ASSIGNMENT METHOD
- Random assignment / user-based / session-based / geographic
- Allocation: [e.g., 50/50, 90/10, 80/10/10]
- Assignment persistence: [sticky per user? per session?]
```

### Step 2: Define metrics

```
PRIMARY METRIC (one only):
- Metric: [the single metric that determines success or failure]
- Current baseline: [current value]
- Minimum detectable effect (MDE): [smallest change worth detecting]
- Direction: [increase / decrease]
- Measurement method: [how exactly you measure this]

SECONDARY METRICS (2-4):
- [Metric]: [what it tells you, current baseline]
- [Metric]: [what it tells you, current baseline]

GUARDRAIL METRICS (things that must NOT get worse):
- [Metric]: [acceptable threshold -- e.g., "error rate must stay below 2%"]
- [Metric]: [acceptable threshold]
```

Rules for choosing metrics:
- The primary metric must directly reflect the hypothesis. If your hypothesis is about engagement, do not use revenue as the primary metric.
- Secondary metrics provide context. They help you understand WHY the primary metric moved.
- Guardrail metrics protect against unintended consequences. Always include at least one.

### Step 3: Calculate sample size and duration

```
SAMPLE SIZE CALCULATION:
Baseline conversion rate: [current rate, e.g., 12%]
Minimum detectable effect: [e.g., 2 percentage points, or 15% relative change]
Statistical significance level: [typically 95%, or alpha = 0.05]
Statistical power: [typically 80%, or beta = 0.20]
Number of variants: [2 for A/B, 3 for A/B/C, etc.]

Required sample per variant: [calculated number]
Total sample needed: [per variant x number of variants]

Daily eligible traffic: [users per day who qualify]
Estimated duration: [total sample / daily traffic]
Minimum runtime: [at least 1 full business cycle, typically 1-2 weeks]
```

Rough sample size guide (for a 50/50 split, 95% confidence, 80% power):

| Baseline rate | 5% relative MDE | 10% relative MDE | 20% relative MDE |
|---------------|-----------------|-------------------|-------------------|
| 1% | ~4.7M per variant | ~1.2M | ~300K |
| 5% | ~290K | ~73K | ~18K |
| 10% | ~140K | ~35K | ~9K |
| 25% | ~46K | ~12K | ~3K |
| 50% | ~24K | ~6K | ~1.5K |

If your traffic cannot reach the needed sample size in 4 weeks, either increase MDE (accept detecting only larger effects) or reconsider whether this is testable as an A/B test.

**STOPPING POINT 2**: Experiment design is complete. Before launching:

1. **Review the brief with the team** - Sanity-check hypothesis, metrics, and implementation plan
2. **Set up tracking and monitoring** - Implement event tracking and build a monitoring dashboard
3. **Define the rollout plan** - Staged rollout, kill switch criteria, and escalation path
4. **Launch the experiment** - Go live and begin data collection
5. **Reconsider the approach** - The math does not work for A/B testing; explore alternatives (qualitative research, phased rollout with before/after)

---

## Workflow 2: Track a Running Experiment

### Monitoring checklist (run daily or every other day)

```
EXPERIMENT HEALTH CHECK: [EXP-ID]
Date: [today]

Assignment integrity:
- [ ] Sample sizes are balanced across variants (within 2% of target ratio)
- [ ] No bias in assignment (check user attributes across variants)
- [ ] Assignment is sticky (users are not switching variants)

Data quality:
- [ ] Events are firing correctly (no logging gaps)
- [ ] No missing data or anomalies (null values, impossible values)
- [ ] No external factors contaminating results (outage, marketing campaign, holiday)

Guardrail metrics:
- [ ] All guardrail metrics are within acceptable thresholds
- [ ] No unexpected metric degradation in any variant

Sample size progress:
- Current sample: [n] / Required: [target]
- Estimated completion date: [date]
- On track: [yes / no]
```

### When to stop an experiment early

**Stop immediately if:**
- A guardrail metric breaches its threshold (e.g., error rate spikes)
- You discover a bug in the implementation that affects one variant differently
- An external event makes the experiment results uninterpretable

**Do NOT stop early because:**
- Results "look significant" after a few days (peeking problem -- wait for full sample)
- Stakeholders want to ship the winning variant faster
- You are bored of waiting

If you want the option to stop early with valid results, use a sequential testing method (specify this in the design phase, not after).

---

## Workflow 3: Analyze Completed Results

### Step 1: Data validation

Before analyzing results, verify:

```
DATA VALIDATION:
Total sample collected: [n]
Target sample: [target]
Sample sufficient: [yes / no]
Duration: [X days] -- at least 1 full business cycle: [yes / no]
Any data quality issues identified during monitoring: [list]
Any external events during the experiment period: [list]
Assignment balance check: Control [n1], Variant [n2] -- ratio: [actual vs target]
```

### Step 2: Calculate results

```
RESULTS SUMMARY: [EXP-ID]

PRIMARY METRIC: [name]
                    Control         Variant         Difference
Sample:             [n1]            [n2]
Metric value:       [value]         [value]         [absolute diff]
Relative change:                                    [percentage change]
Confidence interval (95%):                          [lower, upper]
P-value:                                            [value]
Statistically significant: [yes / no]

SECONDARY METRICS:
[Metric 1]:         [control]       [variant]       [diff] -- sig: [yes/no]
[Metric 2]:         [control]       [variant]       [diff] -- sig: [yes/no]

GUARDRAIL METRICS:
[Metric 1]:         [control]       [variant]       [diff] -- within threshold: [yes/no]
[Metric 2]:         [control]       [variant]       [diff] -- within threshold: [yes/no]
```

### Step 3: Interpret results

**STOPPING POINT 3**: Results are calculated. Interpretation:

1. **Clear win** - Primary metric improved significantly, guardrails held. Proceed to ship.
2. **Clear loss** - Primary metric degraded significantly. Revert and document learnings.
3. **Inconclusive** - Not statistically significant. Decide whether to extend, redesign, or accept.
4. **Mixed signals** - Primary metric improved but a secondary/guardrail metric degraded. Investigate tradeoffs.
5. **Surprising result** - Something unexpected happened (wrong direction, unexpected secondary effects). Deep dive into why.

### Interpretation guidelines

- **Statistical significance is not practical significance.** A statistically significant 0.1% improvement might not be worth the complexity of shipping the change.
- **Look at confidence intervals, not just p-values.** A wide confidence interval means you are still uncertain even if the p-value is below 0.05.
- **Check for segment differences.** The overall result might hide that it helped one segment and hurt another.
- **Consider the cost of being wrong.** For reversible changes, a lower bar is acceptable. For irreversible changes, demand more certainty.

---

## Workflow 4: Document Learnings

### Experiment report template

```
EXPERIMENT REPORT: [EXP-ID]
Name: [name]
Owner: [owner]
Dates: [start] to [end]
Status: [shipped / reverted / extended / redesigned]

HYPOTHESIS: [restate from brief]

RESULT: [one sentence summary]
- Primary metric: [X]% [increase/decrease] (p=[value], 95% CI: [range])
- Decision: [ship / revert / extend / redesign]

KEY LEARNINGS:
1. [What we learned -- insight, not just data]
2. [What we learned]
3. [What surprised us and why]

WHAT THIS CHANGES:
- Product decisions affected: [list]
- Assumptions validated: [list]
- Assumptions invalidated: [list]

FOLLOW-UP:
- [Next experiment or action based on these learnings]

TAGS: [area, feature, metric type -- for future searchability]
```

### Learnings database

Maintain a running document of experiment learnings, searchable by tag:

```
EXPERIMENT LEARNINGS LOG

[Date] [EXP-ID] [Tags: onboarding, activation]
Finding: Reducing sign-up form from 5 fields to 3 increased completion by 12%
Implication: Every additional form field costs ~4% of completions. Use progressive disclosure.

[Date] [EXP-ID] [Tags: pricing, conversion]
Finding: Showing annual pricing first (vs. monthly) had no effect on conversion but increased average deal size by 8%
Implication: Price anchoring works. Default to showing the option we prefer.
```

---

## Workflow 5: Manage the Experiment Pipeline

### Pipeline prioritization

Score experiment candidates:

| Factor | Weight | Score (1-5) |
|--------|--------|-------------|
| Expected learning value | 30% | How much will we learn regardless of outcome? |
| Potential impact on key metric | 25% | If the hypothesis is right, how big is the win? |
| Implementation effort | 20% | How hard is it to build and instrument? (invert: easy = 5) |
| Risk | 15% | What is the downside if the variant is worse? (invert: low risk = 5) |
| Strategic alignment | 10% | How well does this align with current priorities? |

### Pipeline management rules

- Run no more experiments simultaneously than you can monitor properly (typically 3-5 for a small team)
- Do not run experiments that interact with each other on the same users at the same time
- Always have the next 2-3 experiments designed and ready to launch when a slot opens
- Review the pipeline weekly: reprioritize based on new information

```
EXPERIMENT PIPELINE:

RUNNING:
[EXP-001] [name] - Started [date] - Est. completion: [date] - Status: [on track/at risk]
[EXP-002] [name] - Started [date] - Est. completion: [date] - Status: [on track/at risk]

READY TO LAUNCH (designed, waiting for slot):
[EXP-003] [name] - Priority score: [X] - Blocked by: [nothing / EXP-001 interaction]
[EXP-004] [name] - Priority score: [X] - Blocked by: [nothing]

IN DESIGN:
[EXP-005] [name] - Owner: [who] - Design ETA: [date]

COMPLETED (last 30 days):
[EXP-000] [name] - Result: [shipped/reverted] - Key learning: [one line]
```

**STOPPING POINT 4**: Pipeline review complete. What action?

1. **Launch next experiment** - Move the top-priority designed experiment into running
2. **Resolve a conflict** - Two experiments interact; figure out how to run both
3. **Retire an experiment** - Kill an experiment that is no longer relevant
4. **Rebalance priorities** - New information changes what we should test next
5. **Review learnings** - Look at recent experiment results for patterns across experiments