Optimization.experiment Analysis

Purpose

Accelerate experiment readouts by combining statistical rigor with storytelling tailored to executive stakeholders.

Pre-run Checklist

• ✅ Export experiment results (variant metrics, significance, sample sizes).
• ✅ Gather qualitative feedback or session notes if applicable.
• ✅ Align on rollout decisions pending the analysis.

Invocation Guidance

codex run --skill optimization.experiment_analysis \
  --input data/{{experiment_name}}-results.csv \
  --vars "experiment_name={{experiment_name}}" \
         "primary_metric={{primary_metric}}" \
         "secondary_metrics={{secondary_metrics}}" \
         "audience={{audience}}"

Recommended Input Attachments

• Experiment tracking sheet or stats engine export.
• Screenshots of variants.
• Customer feedback related to the experiment.

Claude Workflow Outline

1. Summarize experiment purpose, setup, and success criteria.
2. Present results for primary and secondary metrics with statistical significance.
3. Interpret findings, including customer behavior shifts and operational considerations.
4. Recommend decisions (ship, iterate, stop) with supporting rationale.
5. Highlight next steps, follow-up analyses, and knowledge base updates.

Output Template

# Experiment Analysis — {{experiment_name}}

## Overview
- Objective:
- Dates:
- Audience:

## Results Summary
| Metric | Control | Variant | Δ | Significance | Notes |
| --- | --- | --- | --- | --- | --- |

## Interpretation
- Customer Impact:
- Business Impact:
- Operational Considerations:

## Recommendation
- Decision:
- Rationale:
- Dependencies:

## Next Steps
- Action:
- Owner:
- Timeline:

Follow-up Actions

• Present findings in the growth or optimization forum.
• Update experiment backlog with learnings and links to artifacts.
• Coordinate rollout or rollback actions per recommendation.