Product Experimentation

Product Experimentation

End-to-end product experimentation: from assumption mapping through method selection, instrumentation, guardrail enforcement, and decision-readout that updates the product roadmap. Owns the complete experiment workflow; routes statistical design and rollout mechanics to specialist skills.

Pipeline

ASSUMPTIONS → [HYPOTHESIS] → [METHOD SELECT] → [INSTRUMENT] → [RUN] → [DECIDE] → [RECORD]
                  |                |                |           |          |           |
             Experiment       Qualitative       Tracking     Guardrail   Decision    Readout
               brief          Prototype          plan         monitor     rules      learning
                              Operational
                              Quantitative

Loading Guide

Load only the reference or template relevant to the task. Do not load every file at once.

Working Method

1. Map assumptions

Surface the assumptions driving the proposed change. Classify each by risk (what breaks if it is wrong), evidence strength (what evidence already exists), and testability (can it be tested, and how cheaply). Use templates/assumption-map.md.

2. Translate into hypotheses

Convert the riskiest, least-evidenced assumptions into falsifiable hypotheses. Each hypothesis names the independent variable (what changes), the dependent variable (what outcome is measured), the predicted direction, and the smallest effect that matters. Use templates/experiment-brief.md.

3. Select the appropriate method

Choose the lightest-weight method that can falsify the hypothesis with sufficient confidence. The method ladder, from lightest to heaviest:

Do not default to A/B testing. Start at the top of the ladder and only move down when the question cannot be answered at the current level. A qualitative interview or prototype test is often the right answer. Full method selection guidance is in references/method-selection.md.

4. Define instrumentation, guardrails, and ethics

Before running the experiment, define:

• Instrumentation: what metrics are tracked, how they are computed, and that they are measurable with the available tooling. Route measurement contracts to product-analytics-and-measurement.
• Guardrails: mandatory safety metrics that can stop the experiment regardless of the primary outcome. At minimum: error rate, latency/degradation, and any domain-specific harm metric. Every experiment must name at least one guardrail metric. See references/guardrails-and-ethics.md.
• Ethical boundaries: user consent, data minimization, vulnerable-population considerations, and institutional-review alignment. Record all ethical decisions.
• Stopping rules: when the experiment stops early — guardrail breach, sufficient evidence reached, or time cap reached.
• Decision ownership: who makes the ship/no-ship call and what inputs they consider (statistical evidence, guardrail evidence, qualitative signal, practical constraints).

Use templates/guardrail-and-decision-rule.md to record these.

5. Determine exposure and duration

Define the target population, allocation, and minimum detectable effect. Route statistical design (power analysis, sample-size calculation, estimator selection) to ../data-scientist/SKILL.md. An underpowered experiment — one that cannot detect the smallest effect that matters — is a validity failure; do not ship based on a null result from an underpowered test.

6. Run and monitor

Execute the experiment. Monitor guardrails continuously. Route production rollout mechanics (feature flags, canary stages, progressive delivery) to ../release-engineering/SKILL.md.

7. Decide

Make the ship/no-ship decision using multiple criteria, never statistical significance alone:

A statistically significant result with a failing guardrail is a no-ship. A statistically significant result that exceeds authority boundaries (e.g., safety, compliance, ethics) is a no-ship. Record the decision and its rationale.

8. Record the readout

Document what was learned and what changed as a result. The readout updates the product roadmap, backlog, decision log, or lifecycle evidence. Routing: feeds product-roadmapping-and-portfolio (roadmap updates), product-adoption (adoption evidence), and product-lifecycle-learning (retained learning). Use templates/readout-learning-entry.md and references/experiment-readout.md.

Trigger Conditions

Load this skill when:

• The task involves designing, running, or deciding on a product experiment
• You need to choose between qualitative, prototype, operational, and quantitative test methods
• You have assumptions that need to be tested before committing to build
• You need to define guardrails, stopping rules, or decision criteria for an experiment
• You need to interpret experiment results and make a ship/no-ship decision
• You need to record experiment outcomes that update product direction

When Not to Use

• Statistical design (power analysis, estimator selection, significance testing in depth) — route to data-scientist. This skill frames the question and selects the method; data-scientist owns the statistical machinery.
• Production rollout mechanics (feature-flag infrastructure, canary stages, CD pipeline integration) — route to release-engineering. This skill defines the experiment design; release-engineering owns the safe delivery.
• User research and usability testing — route to product-design-and-ux for interaction-focused studies.
• Pricing-specific tests — route to financial-modeling for elasticity, willingness-to-pay, and pricing-page experiments.
• Opportunity-solution tree construction — route to product-methodology for connecting customer needs to build decisions before experimentation.
• Pure analytics instrumentation (tracking-plan design, event taxonomy, metric definitions) — route to product-analytics-and-measurement for measurement contracts.

Portability

This skill is intentionally host-neutral. It requires no profile system, output format, scripts, or external services. Load references and templates directly by path using the host agent's normal file-loading mechanism.