# Experiment Tracker > Valida features y decisiones con datos. Usa este skill para diseñar A/B tests, feature flags, métricas de experimentos, análisis de resultados, y tomar decisiones data-driven sobre qué features escalar o matar. - Skill: `leandroomargarcia/experiment-tracker` (Agent Skill) - Install (CLI): `npx skillmds@latest add leandroomargarcia/experiment-tracker` - Raw SKILL.md: https://api.skillmd.com/api/skills/leandroomargarcia/experiment-tracker/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: leandroomargarcia (https://skillmd.com/u/leandroomargarcia) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/leandroomargarcia/experiment-tracker --- # Experiment Tracker Especialista en validación basada en datos de features y decisiones de producto. Diseña, ejecuta, y analiza experimentos para reducir el riesgo de construir cosas que no funcionan. Se activa automáticamente cuando se añaden feature flags. ## Cuándo Usar Este Skill - Diseñar A/B tests - Configurar feature flags - Definir métricas de éxito - Analizar resultados de experimentos - Decidir ship/kill/iterate - Documentar learnings ## Responsabilidades Principales ### 1. Experiment Design - Define hipótesis claras - Elige métricas apropiadas - Calcula sample size necesario - Diseña control y variantes - Planifica duración ### 2. Implementation Support - Configura feature flags - Asegura tracking correcto - Valida data collection - Monitors experiment health - Handles edge cases ### 3. Analysis & Insights - Analiza resultados estadísticamente - Identifica segmentos afectados - Detecta efectos secundarios - Genera recommendations - Documenta learnings ### 4. Decision Making - Presenta resultados objetivamente - Facilita go/no-go decisions - Trackea decisions y outcomes - Builds institutional knowledge ## Experiment Design Template ```markdown ## Experiment: [Nombre] ### Hypothesis Si [hacemos X], entonces [métrica Y] mejorará en [Z%] porque [insight/razón]. ### Background - Problem: [qué estamos tratando de resolver] - Evidence: [por qué creemos que esto funcionará] - Risk: [qué podría salir mal] ### Metrics **Primary:** [la métrica que decide success] **Secondary:** [métricas de soporte] **Guardrail:** [métricas que NO deben empeorar] ### Test Design - **Control:** [experiencia actual] - **Variant:** [cambio propuesto] - **Population:** [quién ve el experimento] - **Allocation:** [% en cada grupo] ### Sample Size - Baseline: [métrica actual] - MDE (Minimum Detectable Effect): [X%] - Power: 80% - Significance: 95% - **Required N:** [calculado] ### Duration - Estimated: [X días/semanas] - Start date: [fecha] - End date: [fecha] ### Risks & Mitigations - [Risk 1]: [Mitigation] ### Success Criteria - Ship if: [condición] - Kill if: [condición] - Iterate if: [condición] ``` ## Statistical Significance ``` ENTENDER RESULTADOS: p-value < 0.05 = Estadísticamente significativo - Hay <5% chance de que el resultado sea casualidad Confidence Interval: - Rango donde probablemente está el efecto real - No incluye 0 = significant Efecto práctico vs estadístico: - Puede ser significativo estadísticamente pero el efecto es muy pequeño para importar - Siempre evaluar impacto de negocio ERRORES COMUNES: ❌ Parar experimento temprano cuando "se ve bien" ❌ Ignorar guardrail metrics ❌ No esperar sample size suficiente ❌ Correr múltiples tests sin corrección ``` ## Feature Flag Best Practices ``` NAMING: - use-kebab-case - include-date: new-checkout-2024-01 - be-descriptive: show-ai-suggestions TYPES: - Release: on/off para nuevas features - Experiment: A/B con % allocation - Ops: Kill switches para emergencies - Permission: User-level access LIFECYCLE: 1. Create flag (default: off) 2. Implement behind flag 3. QA with flag on 4. Gradual rollout (1% → 10% → 50% → 100%) 5. Clean up flag after full rollout HYGIENE: - Review flags monthly - Remove after 30 days of 100% - Document all active flags - Limit concurrent experiments ``` ## Sample Size Calculator ``` INPUTS NECESARIOS: - Baseline conversion rate - Minimum detectable effect (MDE) - Statistical power (usually 80%) - Significance level (usually 95%) EJEMPLO: Baseline: 10% conversion MDE: 10% lift (to 11%) Power: 80% Significance: 95% Required: ~14,700 per variant Total: ~29,400 users RULE OF THUMB: - Bigger effect → smaller sample - Smaller baseline → larger sample - More power → larger sample ``` ## Results Analysis Template ```markdown ## Experiment Results: [Nombre] ### Summary **Decision:** 🚀 Ship / 🔪 Kill / 🔄 Iterate ### Metrics Performance | Metric | Control | Variant | Lift | p-value | |--------|---------|---------|------|---------| | Primary: [X] | Y% | Z% | +A% | 0.0X | | Secondary: [X] | Y% | Z% | +A% | 0.0X | | Guardrail: [X] | Y% | Z% | +A% | 0.XX | ### Statistical Summary - Sample size: [N] - Duration: [X days] - Power achieved: [X%] - Confidence interval: [X% to Y%] ### Segmentation Analysis | Segment | Control | Variant | Notes | |---------|---------|---------|-------| | New users | X% | Y% | [insight] | | Power users | X% | Y% | [insight] | ### Interpretation [What do these results mean in plain language] ### Recommendation [Clear action with rationale] ### Learnings - [Insight 1] - [Insight 2] ### Follow-up Experiments - [If applicable] ``` ## Experiment Hygiene ``` BEFORE LAUNCHING: ☐ Hypothesis documented ☐ Metrics defined & tracking verified ☐ Sample size calculated ☐ Duration estimated ☐ Guardrails set ☐ Rollback plan ready DURING: ☐ Daily health check ☐ No peeking at results (avoid bias) ☐ Monitor for bugs/issues ☐ Watch guardrail metrics AFTER: ☐ Wait for full duration ☐ Analyze with statistics ☐ Document learnings ☐ Clean up feature flags ☐ Share with team ``` ## Experiment Catalog ```markdown ## Active Experiments | Name | Start | End | Status | Owner | |------|-------|-----|--------|-------| | [Exp 1] | [date] | [date] | Running | @person | | [Exp 2] | [date] | [date] | Analysis | @person | ## Completed Experiments | Name | Result | Lift | Learning | |------|--------|------|----------| | [Exp 1] | Shipped | +X% | [insight] | | [Exp 2] | Killed | -X% | [insight] | ``` ## Common Pitfalls ``` ❌ PEEKING Looking at results before sufficient data → Increases false positive rate → Wait for pre-calculated duration ❌ MULTIPLE COMPARISONS Testing many metrics without correction → 5% of tests will be false positive by chance → Define primary metric upfront ❌ NOVELTY EFFECT New things get attention initially → Results may regress → Run for at least 2 weeks ❌ SEGMENT HUNTING Finding random segments that "won" → Post-hoc segmentation is suspicious → Pre-define segments ❌ STOPPING EARLY "It's clearly winning/losing" → Statistical significance needs sample size → Commit to duration upfront ``` ## Mejores Prácticas 1. **Document before testing** - Hypothesis first 2. **Don't peek** - Commits to duration 3. **One primary metric** - Others are supporting 4. **Clean up flags** - Technical debt builds 5. **Share learnings** - Even failed experiments teach 6. **Trust the data** - Even when it's surprising ## Filosofía > "The goal of experimentation isn't to prove you're right—it's to learn what's true. Failed experiments are just as valuable as successful ones." El objetivo es reducir el riesgo de construir features que no funcionan, tomando decisiones basadas en data real de usuarios reales.