# Ds Randomized Experiments > "Provides Designs and analyzes randomized controlled trials (RCTs) A/B tests, experimental blocking, and sample size calculations" - Skill: `paulpas/ds-randomized-experiments` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-randomized-experiments` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-randomized-experiments/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-randomized-experiments --- # Randomized Experiments Comprehensive guide to randomized experiments in machine learning and data science workflows. ## When to Use This Skill - Solving real-world causal inference problems - Building machine learning pipelines with randomized experiments - Implementing best practices for randomized experiments - Optimizing model performance using randomized experiments techniques - Learning industry-standard approaches to randomized experiments ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require randomized experiments rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Randomized Experiments is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Randomized Experiments ```python import pandas as pd import numpy as np from scipy import stats from typing import Dict, Optional def basic_rct_analysis( data: pd.DataFrame, treatment_col: str, outcome_col: str, block_col: Optional[str] = None ) -> Dict[str, float]: """ Perform basic two-sample analysis on randomized experiment data. Handles optional blocking and returns statistical results. """ if outcome_col not in data.columns or treatment_col not in data.columns: raise ValueError("Missing required columns in data") treatment_data = data[data[treatment_col] == 1][outcome_col] control_data = data[data[treatment_col] == 0][outcome_col] if len(treatment_data) < 2 or len(control_data) < 2: raise ValueError("Insufficient samples in treatment or control groups") t_stat, p_value = stats.ttest_ind(treatment_data, control_data, equal_var=False) effect_size = float(np.mean(treatment_data) - np.mean(control_data)) results: Dict[str, float] = { "t_statistic": float(t_stat) "p_value": float(p_value) "effect_size": effect_size "significant": float(1.0 if p_value < 0.05 else 0.0) } if block_col and block_col in data.columns: blocked_means: Dict[str, float] = {} for block in data[block_col].unique(): block_data = data[data[block_col] == block] t_b, p_b = stats.ttest_ind( block_data[block_data[treatment_col] == 1][outcome_col] block_data[block_data[treatment_col] == 0][outcome_col] equal_var=False ) blocked_means[f"block_{block}_p"] = float(p_b) results["blocked_analysis"] = blocked_means return results ``` ### Pattern 2: Production-Ready Randomized Experiments ```python import logging import pandas as pd import numpy as np from typing import Any, Dict, List from scipy import stats from statsmodels.stats.power import TTestIndPower logger = logging.getLogger(__name__) class RandomizedExperiment: """Production-grade implementation for designing and analyzing randomized experiments.""" def __init__(self, alpha: float = 0.05, power: float = 0.8, effect_size: float = 0.5) -> None: self.alpha = alpha self.power = power self.effect_size = effect_size self._analysis_results: Dict[str, Any] = {} def calculate_sample_size(self, n_control: int = 100) -> int: """Calculate required treatment group size for desired power.""" analysis = TTestIndPower() n_treatment = analysis.solve_power( effect_size=self.effect_size, power=self.power, alpha=self.alpha, nobs1=n_control, ratio=1.0 ) logger.info(f"Calculated required treatment size: {int(np.ceil(n_treatment))}") return int(np.ceil(n_treatment)) def assign_treatment(self, data: pd.DataFrame, block_col: str = None) -> pd.DataFrame: """Randomly assign treatment/control with optional blocking.""" if block_col is None: data = data.copy() data['treatment'] = np.random.binomial(1, 0.5, size=len(data)) else: data = data.copy() data['treatment'] = 0 for block in data[block_col].unique(): mask = data[block_col] == block block_size = mask.sum() data.loc[mask, 'treatment'] = np.random.binomial(1, 0.5, size=block_size) return data def analyze(self, data: pd.DataFrame, outcome_col: str) -> Dict[str, Any]: """Run statistical analysis on assigned experiment data.""" if 'treatment' not in data.columns: raise ValueError("Treatment assignment missing. Run assign_treatment first.") treatment = data[data['treatment'] == 1][outcome_col] control = data[data['treatment'] == 0][outcome_col] t_stat, p_value = stats.ttest_ind(treatment, control, equal_var=False) ci = stats.t.interval(0.95, len(treatment) + len(control) - 2, loc=np.mean(treatment) - np.mean(control) scale=np.sqrt(np.var(treatment)/len(treatment) + np.var(control)/len(control))) self._analysis_results = { "t_statistic": float(t_stat) "p_value": float(p_value) "confidence_interval": [float(ci[0]), float(ci[1])] "significant": bool(p_value < self.alpha) "n_treatment": len(treatment) "n_control": len(control) } return self._analysis_results ``` ### Pattern 3: Anti-Patterns and Best Practices ```python # BAD: Magic numbers, no error handling, violates DRY principle def bad_experiment(data): t = data[data['group'] == 1]['value'] c = data[data['group'] == 0]['value'] return np.mean(t) - np.mean(c) # No significance testing, hardcoded assumptions # GOOD: Validated inputs, statistical rigor, follows DRY and SOLID principles def good_experiment(data: pd.DataFrame, group_col: str, value_col: str) -> Dict[str, float]: """Analyze experiment with proper validation and statistical testing.""" if group_col not in data.columns or value_col not in data.columns: raise ValueError("Invalid column names provided") groups = data[group_col].unique() if len(groups) != 2: raise ValueError("Exactly two groups required for comparison") t_stat, p_val = stats.ttest_ind( data[data[group_col] == groups[0]][value_col] data[data[group_col] == groups[1]][value_col] equal_var=False ) return { "effect_size": float(np.mean(data[data[group_col] == groups[1]][value_col]) - np.mean(data[data[group_col] == groups[0]][value_col])) "p_value": float(p_val) "significant": bool(p_val < 0.05) } ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Randomized Experiment — Wikipedia](https://en.wikipedia.org/wiki/Randomized_experiment) - [Causal Inference & Control Experiments (Microsoft)](https://www.microsoft.com/en-us/research/uploads/prod/2018/03/online-a-b-testing.pdf) - [Experimental Design — NIST Engineering Handbook](https://www.itl.nist.gov/div898/handbook/index.htm) - [A/B Testing Best Practices (Google Analytics)](https://marketingplatform.google.com/about/analytics/) - [Randomization in Clinical Trials (FDA Guidance)](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/statistical-guidance-clinical-trials)