# Ds Feature Selection > "Selects relevant features using univariate selection, recursive elimination stability selection, and importance-based methods" - Skill: `paulpas/ds-feature-selection` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-feature-selection` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-feature-selection/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-feature-selection --- # Feature Selection Comprehensive guide to feature selection in machine learning and data science workflows. ## When to Use This Skill - Solving real-world feature engineering problems - Building machine learning pipelines with feature selection - Implementing best practices for feature selection - Optimizing model performance using feature selection techniques - Learning industry-standard approaches to feature selection ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require feature selection rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Feature Selection is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Feature Selection ```python # BAD: Fitting selector on full dataset before splitting causes data leakage import pandas as pd import numpy as np from sklearn.feature_selection import SelectKBest, f_classif from sklearn.model_selection import train_test_split X = pd.DataFrame(np.random.randn(100, 10), columns=[f"feat_{i}" for i in range(10)]) y = np.random.randint(0, 2, 100) selector = SelectKBest(f_classif, k=5) X_selected = selector.fit_transform(X, y) # Data leakage: information from test set leaks into training X_train, X_test, y_train, y_test = train_test_split(X_selected, y, random_state=42) # GOOD: Using Pipeline ensures proper train/test separation and prevents leakage from sklearn.pipeline import Pipeline from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import cross_val_score pipeline = Pipeline([ ('selector', SelectKBest(f_classif, k=5)) ('classifier', RandomForestClassifier(n_estimators=100, random_state=42)) ]) scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy') print(f"Cross-validated accuracy: {scores.mean():.3f} (+/- {scores.std() * 2:.3f})") ``` ### Pattern 2: Production-Ready Feature Selection ```python import logging import pandas as pd import numpy as np from typing import Any, Dict, List from sklearn.feature_selection import SelectFromModel from sklearn.ensemble import RandomForestClassifier from sklearn.exceptions import ConvergenceWarning import warnings logger = logging.getLogger(__name__) warnings.filterwarnings("ignore", category=ConvergenceWarning) class FeatureSelectionPipeline: """Production-grade feature selection with importance-based filtering. Follows SOLID principles for maintainability and testability.""" def __init__(self, threshold: float = 0.01, max_features: int = None) -> None: self.threshold = threshold self.max_features = max_features self.selector = SelectFromModel( RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1) threshold=threshold ) self.selected_features: List[str] = [] def execute(self, data: pd.DataFrame, target_col: str) -> Dict[str, Any]: """Execute feature selection and return structured results.""" if target_col not in data.columns: raise ValueError(f"Target column '{target_col}' not found in data.") X = data.drop(columns=[target_col]) y = data[target_col] if X.shape[1] == 0: raise ValueError("No features available for selection.") self.selector.fit(X, y) self.selected_features = X.columns[self.selector.get_support()].tolist() X_selected = self.selector.transform(X) result = { 'status': 'success' 'original_features': X.shape[1] 'selected_features': len(self.selected_features) 'feature_names': self.selected_features 'transformed_data': X_selected 'importance_scores': dict(zip(X.columns, self.selector.estimator_.feature_importances_)) } logger.info(f"Selected {len(self.selected_features)} features from {X.shape[1]}") return result ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Scikit-learn Feature Selection](https://scikit-learn.org/stable/modules/feature_selection.html) - [SelectKBest, RFE — Scikit-learn docs](https://scikit-learn.org/stable/modules/feature_selection.html#feature-selection) - [Mutual Information Classifier (Scikit-learn)](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html) - [Feature Importance with Tree Ensembles](https://scikit-learn.org/stable/modules/tree.html#decisions-and-feature-importance) - [Recursive Feature Elimination Tutorial (Kaggle)](https://www.kaggle.com/code/leandro0421/feature-selection-with-recursive-feature-elimination)