# Ds Classification Metrics > "Evaluates classification models using precision, recall, F1-score ROC-AUC, confusion matrix, and other classification metrics" - Skill: `paulpas/ds-classification-metrics` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-classification-metrics` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-classification-metrics/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-classification-metrics --- # Classification Metrics Comprehensive guide to classification metrics in machine learning and data science workflows. ## When to Use This Skill - Solving real-world model evaluation & selection problems - Building machine learning pipelines with classification metrics - Implementing best practices for classification metrics - Optimizing model performance using classification metrics techniques - Learning industry-standard approaches to classification metrics ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require classification metrics rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Classification Metrics is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Classification Metrics ```python import numpy as np import pandas as pd from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import ( precision_score, recall_score, f1_score roc_auc_score, confusion_matrix, classification_report ) # Generate synthetic binary classification dataset X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=42) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Train a baseline classifier model = LogisticRegression(random_state=42) model.fit(X_train, y_train) y_pred = model.predict(X_test) y_prob = model.predict_proba(X_test)[:, 1] # Calculate core classification metrics metrics = { 'precision': precision_score(y_test, y_pred) 'recall': recall_score(y_test, y_pred) 'f1_score': f1_score(y_test, y_pred) 'roc_auc': roc_auc_score(y_test, y_prob) 'confusion_matrix': confusion_matrix(y_test, y_pred).tolist() } # Output results print(classification_report(y_test, y_pred)) print(f"ROC-AUC: {metrics['roc_auc']:.4f}") ``` ### Pattern 2: Production-Ready Classification Metrics ```python import logging from typing import Any, Dict, List, Optional import numpy as np from sklearn.metrics import accuracy_score, precision_recall_fscore_support, roc_auc_score logger = logging.getLogger(__name__) class ClassificationMetrics: """Production implementation of Classification Metrics""" def __init__(self, threshold: float = 0.5, metrics_list: Optional[List[str]] = None): self.threshold = threshold self.metrics_list = metrics_list or ['accuracy', 'precision', 'recall', 'f1', 'roc_auc'] def execute(self, y_true: np.ndarray, y_pred: np.ndarray, y_prob: Optional[np.ndarray] = None) -> Dict[str, Any]: """Execute Classification Metrics on predictions""" if y_true.shape != y_pred.shape: raise ValueError("y_true and y_pred must have the same shape") results: Dict[str, Any] = {'status': 'success', 'metrics': {}} for metric_name in self.metrics_list: if metric_name == 'accuracy': results['metrics']['accuracy'] = float(accuracy_score(y_true, y_pred)) elif metric_name == 'precision': results['metrics']['precision'] = float(precision_recall_fscore_support(y_true, y_pred, average='weighted')[0]) elif metric_name == 'recall': results['metrics']['recall'] = float(precision_recall_fscore_support(y_true, y_pred, average='weighted')[1]) elif metric_name == 'f1': results['metrics']['f1'] = float(precision_recall_fscore_support(y_true, y_pred, average='weighted')[2]) elif metric_name == 'roc_auc' and y_prob is not None: results['metrics']['roc_auc'] = float(roc_auc_score(y_true, y_prob)) else: logger.warning(f"Metric {metric_name} not implemented or skipped") return results ``` ### BAD vs GOOD: Metric Calculation ```python # BAD: Hardcoded thresholds, duplicated logic, and no type safety def bad_metrics(y_true, y_pred): p = sum((y_pred == 1) & (y_true == 1)) / sum(y_pred == 1) r = sum((y_pred == 1) & (y_true == 1)) / sum(y_true == 1) f1 = 2 * p * r / (p + r) return {'precision': p, 'recall': r, 'f1': f1} # GOOD: Vectorized operations, reusable components, and proper error handling def good_metrics(y_true: np.ndarray, y_pred: np.ndarray) -> Dict[str, float]: """Calculate precision, recall, and F1 using vectorized numpy operations.""" if not isinstance(y_true, np.ndarray) or not isinstance(y_pred, np.ndarray): raise TypeError("Inputs must be numpy arrays") tp = np.sum((y_pred == 1) & (y_true == 1)) fp = np.sum((y_pred == 1) & (y_true == 0)) fn = np.sum((y_pred == 0) & (y_true == 1)) precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0 recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0 f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0 return {'precision': float(precision), 'recall': float(recall), 'f1': float(f1)} ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging - ✅ Follow DRY and SOLID principles to keep metric calculations modular and maintainable - ✅ Reference the scikit-learn API design standard for consistent estimator interfaces ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Scikit-learn Model Evaluation — Classification Metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics) - [Precision, Recall, F1-Score Explained (Scikit-learn docs)](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html) - [ROC Curves — Scikit-learn](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html) - [Imbalanced Classification Metrics (Kaggle Learn)](https://www.kaggle.com/learn/metrics-for-machine-learning-education) - [Classification Report — MLflow metrics](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.classification_report)