# Ds Model Selection > "Provides Compares and selects models using AIC, BIC, validation curves learning curves, and model comparison techniques" - Skill: `paulpas/ds-model-selection` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-model-selection` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-model-selection/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-model-selection --- # Model Selection Comprehensive guide to model selection in machine learning and data science workflows. ## When to Use This Skill - Solving real-world model evaluation & selection problems - Building machine learning pipelines with model selection - Implementing best practices for model selection - Optimizing model performance using model selection techniques - Learning industry-standard approaches to model selection ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require model selection rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Model Selection is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Model Selection ```python import numpy as np import pandas as pd from typing import Dict, Any from sklearn.model_selection import train_test_split, cross_val_score from sklearn.linear_model import LinearRegression, Ridge from sklearn.metrics import mean_squared_error, r2_score from sklearn.datasets import make_regression def basic_model_selection() -> Dict[str, Any]: """ Demonstrates basic model selection by comparing Linear and Ridge regression. Follows DRY principle by centralizing evaluation logic. """ X, y = make_regression(n_samples=500, n_features=10, noise=0.1, random_state=42) df = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(X.shape[1])]) df['target'] = y X_train, X_test, y_train, y_test = train_test_split( df.drop('target', axis=1), df['target'], test_size=0.2, random_state=42 ) models: Dict[str, Any] = { 'Linear Regression': LinearRegression() 'Ridge Regression': Ridge(alpha=1.0) } results: Dict[str, Any] = {} for name, model in models.items(): cv_scores = cross_val_score(model, X_train, y_train, cv=5, scoring='r2') model.fit(X_train, y_train) y_pred = model.predict(X_test) r2 = r2_score(y_test, y_pred) mse = mean_squared_error(y_test, y_pred) results[name] = { 'cv_r2_mean': float(np.mean(cv_scores)) 'cv_r2_std': float(np.std(cv_scores)) 'test_r2': float(r2) 'test_mse': float(mse) } print(f"{name} - CV R2: {np.mean(cv_scores):.4f} (+/- {np.std(cv_scores):.4f}), Test R2: {r2:.4f}") return results if __name__ == "__main__": basic_model_selection() ``` ### Pattern 2: Production-Ready Model Selection ```python import logging import numpy as np import pandas as pd from typing import Any, Dict, List from sklearn.model_selection import train_test_split, cross_val_score from sklearn.linear_model import LogisticRegression from sklearn.ensemble import RandomForestClassifier from sklearn.svm import SVC from sklearn.metrics import accuracy_score, classification_report from sklearn.datasets import make_classification logger = logging.getLogger(__name__) class ModelSelector: """Production implementation of Model Selection following SOLID principles.""" def __init__(self, cv_folds: int = 5, random_state: int = 42) -> None: self.cv_folds = cv_folds self.random_state = random_state self.selected_model: Any = None self.results: Dict[str, Any] = {} def _prepare_data(self, data: pd.DataFrame, target_col: str) -> tuple: if target_col not in data.columns: raise ValueError(f"Target column '{target_col}' not found in data") X = data.drop(columns=[target_col]) y = data[target_col] if X.isnull().any().any() or y.isnull().any(): logger.warning("Data contains missing values. Dropping rows with NaN.") X, y = X.dropna(), y.dropna() return train_test_split(X, y, test_size=0.2, random_state=self.random_state, stratify=y) def execute(self, data: pd.DataFrame, target_col: str = 'target') -> Dict[str, Any]: """Execute Model Selection on data""" try: X_train, X_test, y_train, y_test = self._prepare_data(data, target_col) models = { 'Logistic Regression': LogisticRegression(max_iter=1000, random_state=self.random_state) 'Random Forest': RandomForestClassifier(n_estimators=100, random_state=self.random_state) 'SVM': SVC(kernel='rbf', probability=True, random_state=self.random_state) } best_score = -np.inf best_name = None for name, model in models.items(): cv_scores = cross_val_score(model, X_train, y_train, cv=self.cv_folds, scoring='accuracy') model.fit(X_train, y_train) y_pred = model.predict(X_test) acc = accuracy_score(y_test, y_pred) report = classification_report(y_test, y_pred, output_dict=True) self.results[name] = { 'cv_accuracy_mean': float(np.mean(cv_scores)) 'cv_accuracy_std': float(np.std(cv_scores)) 'test_accuracy': float(acc) 'classification_report': report } if np.mean(cv_scores) > best_score: best_score = np.mean(cv_scores) best_name = name self.selected_model = model logger.info(f"Selected model: {best_name} with CV accuracy: {best_score:.4f}") return self.results except Exception as e: logger.error(f"Model selection failed: {str(e)}") raise if __name__ == "__main__": logging.basicConfig(level=logging.INFO) X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42) df = pd.DataFrame(X, columns=[f'feat_{i}' for i in range(X.shape[1])]) df['target'] = y selector = ModelSelector(cv_folds=5) results = selector.execute(df, target_col='target') print(f"Best model selected: {selector.selected_model}") ``` ### BAD vs GOOD Example ```python # BAD: Hardcoded values, no error handling, mixes data prep and evaluation def bad_selection(data): model = LinearRegression() model.fit(data[:, :-1], data[:, -1]) return model.score(data[:, :-1], data[:, -1]) # GOOD: Type hints, validation, separation of concerns, proper metrics def good_selection(X: np.ndarray, y: np.ndarray) -> float: if X.shape[0] != y.shape[0]: raise ValueError("X and y must have the same number of samples") X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) model = LinearRegression() model.fit(X_train, y_train) return float(r2_score(y_test, model.predict(X_test))) ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Scikit-learn Model Evaluation](https://scikit-learn.org/stable/modules/model_evaluation.html) - [Model Selection — Scikit-learn docs](https://scikit-learn.org/stable/modules/model_selection.html) - [Cross-Validation (Kaggle Learn)](https://www.kaggle.com/learn/cross-validation) - [AIC vs BIC (NIST Handbook)](https://www.itl.nist.gov/div898/handbook/tq/section4/tq_2.htm) - [Model Selection and Regularization (ESL Chapter 7)](https://web.stanford.edu/~hastie/ElemStatLearn/)