sairam0424 Skill — Experiment Platform
When this skill activates
This skill activates when building experimentation infrastructure, implementing statistical testing frameworks, or designing A/B testing platforms. Use when organizations need to scale testing velocity while maintaining statistical rigor.
Mandatory actions when this skill is active
Before writing any code
- Define experiment framework components: randomization service, exposure logging, metric computation pipeline, and statistical analysis engine
- Establish statistical rigor standards: minimum sample size, power (80%+), significance level (5%), minimum detectable effect, and multiple comparison corrections
- Design guardrail metrics framework: business health (revenue, retention), user experience (latency, errors), and ecosystem health (partner impact)
- Plan experiment lifecycle states: draft, review, running, paused, completed, archived with transition criteria and approval gates
During implementation
- Implement consistent randomization using stable hashing (user_id + experiment_id) ensuring users see same variant across sessions
- Build exposure logging capturing: timestamp, user_id, experiment_id, variant, context for accurate sample size and covariate adjustment
- Create metric computation pipeline with: numerator/denominator structure, winsorization for outliers, delta method for ratios, bootstrap for confidence intervals
- Design sequential testing capability for early stopping: alpha spending functions, futility boundaries, and minimum runtime requirements
- Implement stratified analysis for heterogeneous treatment effects: by platform, user segment, geography with interaction effect testing
- Build automated guardrail checks: alert on significant negative movement in critical metrics with experiment auto-pause capability
- Create experiment metadata repository: hypothesis, success criteria, related experiments, learnings, and decision outcome for institutional knowledge
After implementation
- Generate automated experiment scorecards: primary metric movement, guardrail status, statistical significance, practical significance, recommendation
- Build experiment catalog with search and discovery: hypothesis library, metric glossary, analysis templates, and historical results
- Create experimentation health dashboard: velocity (experiments/week), quality (statistical power distribution), impact (significant wins), and coverage (features tested)
- Document statistical methodology: test selection, variance reduction techniques, multiple comparison approach, and sequential testing procedures
Self-check before task completion
- Randomization service ensures stable assignment and balanced allocation across variants
- Exposure logging captures all necessary context for accurate analysis and debugging
- Statistical analysis engine implements proper corrections for multiple comparisons and peeking
- Guardrail metrics monitored automatically with alerting and experiment pause capability
- Experiment lifecycle enforces minimum runtime and sample size before declaring results