Long Term Impact Evaluation

Long-Term Impact Evaluation

Use this skill to choose a practical method for measuring product impact beyond the initial experiment window. It compares long-term holdbacks, post-period analysis, continuous monitoring, and customer lifetime value models.

Source Traceability

Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 8 on long-term impact, short-term and long-term metric relationships, holdbacks, post-period analysis, continuous monitoring, and CLV models.

Related skills:

• holdback-experiment-design for detailed holdback planning.
• trustworthy-experiment-insights for result credibility and false positives.
• experimentation-strategy-roadmap for cost, quality, and complexity tradeoffs.

Reference Routing

Workflow

1. State why short-term experiment metrics are insufficient.
2. Identify the expected time horizon and delayed mechanisms.
3. Compare holdback, post-period analysis, continuous monitoring, and CLV model options.
4. Weigh accuracy against user cost, business cost, complexity, and confounding.
5. Choose the simplest method that answers the long-term question.
6. Define metric cadence, interpretation limits, and follow-up decisions.

Output Format

# Long-Term Impact Evaluation Plan

## Long-Term Question
[What delayed or sustained effect must be measured.]

## Recommended Method
[Holdback | Post-period analysis | Continuous monitoring | CLV model | Hybrid]

## Tradeoffs
| Method | Benefit | Cost/Risk | Fit |
|--------|---------|-----------|-----|

## Measurement Plan
- Short-term metric:
- Long-term metric:
- Time horizon:
- Confounders:

## Decision Rules
- Continue measuring if:
- End or revise if:
- Escalate if:

Quality Bar

• Do not create a long-term holdback when a lower-cost method can answer the decision well enough.
• Do not use post-period analysis without accounting for external factors such as seasonality or campaigns.
• Do not rely on CLV models without naming model drift and behavior-change risk.
• Do not confuse continuous monitoring with causal long-term measurement.