LVTD-LLC Inclusive Experiment Analysis
Use this skill to make sure experiment design and readouts consider the range of users affected by a product change. It focuses on subgroup impact, accessibility, representation, data dimensions, and unintended harm.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 1 lines 719-912 and related subgroup analysis context from lines 639-718. Metric and eligibility context comes from chapter 2 lines 1564-1735.
Related Advanced Skills
trustworthy-experiment-insights: use when subgroup findings may be underpowered, false positives, or false negatives.experiment-verification-monitoring: use when inclusion risks depend on assignment, exposure, device, geography, accessibility, or segment monitoring.adaptive-experimentation-strategy: use cautiously when contextual bandits or personalization could create uneven user impact across groups.
Reference Routing
| Need | Read |
|---|---|
| Inclusive experiment concepts | references/core/knowledge.md |
| Design and analysis rules | references/core/rules.md |
| Segment examples | references/core/examples.md |
| Review workflow | workflows/review-inclusive-impact.md |
Workflow
- Identify which user groups could experience the change differently.
- Choose dimensions that are relevant, ethical, and available.
- Check test/control balance for important dimensions when possible.
- Include accessibility, bandwidth, device, privacy, geography, and usage-level concerns where relevant.
- Analyze subgroup outcomes without cherry-picking.
- Recommend launch, mitigation, follow-up testing, or deeper research.
Output Format
# Inclusive Experiment Review
## Change Under Review
[What is changing and who may be affected.]
## User Dimensions
| Dimension | Why It Matters | Data Available? | Use In Analysis? |
|-----------|----------------|-----------------|------------------|
## Balance And Impact
| Segment | Control | Test | Result | Concern |
|---------|---------|------|--------|---------|
## Risks
- Accessibility:
- Device or bandwidth:
- Privacy or consent:
- Representation:
- Data limitations:
## Recommendation
[Ship | Ship with mitigation | Do not ship | Investigate] because [reason].
Quality Bar
- Do not use sensitive attributes casually; explain why a dimension is needed.
- Do not claim inclusive impact when the data lacks relevant representation.
- Do not average away harm to a meaningful subgroup.
- Pair quantitative subgroup analysis with qualitative or accessibility review when metrics cannot capture the risk.