Model Evaluation
Overview
The wrong metric on the wrong split produces confident, wrong conclusions. Evaluation is about choosing a metric that matches the business cost, validating it on a split that mirrors production, and reporting it honestly with uncertainty.
When to use
- Selecting how to score a model.
- A model "looks great" but you're unsure it's real.
- Comparing candidate models for promotion.
Metric selection
| Problem |
Default metric |
Use when |
| Balanced classification |
ROC-AUC, accuracy |
classes ~balanced |
| Imbalanced classification |
PR-AUC, F1, recall@k |
rare positives (fraud, disease) |
| Probabilistic output |
Log loss, Brier, calibration |
you need trustworthy probabilities |
| Ranking |
NDCG, MAP, MRR |
recommendation/search |
| Regression |
MAE (robust), RMSE (penalize big errors) |
match error cost |
| Regression, multiplicative |
MAPE / RMSLE |
errors scale with magnitude |
Cross-validation strategy
- Default:
StratifiedKFold for classification.
- Time series:
TimeSeriesSplit — never shuffle; train on past, validate on future.
- Grouped data (multiple rows per user):
GroupKFold so the same group never spans train and test.
- Small data: repeated CV; report mean ± std.
Honest reporting
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np
cv = StratifiedKFold(5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring="average_precision")
print(f"PR-AUC: {scores.mean():.3f} ± {scores.std():.3f}") # always report spread
Beyond a single number
- Confusion matrix / classification report at the chosen threshold — accuracy hides per-class failure.
- Threshold tuning — default 0.5 is rarely optimal; pick it from the PR curve to match precision/recall needs.
- Calibration —
CalibratedClassifierCV or reliability curves when probabilities feed decisions.
- Slice metrics — evaluate on subgroups to catch fairness/robustness gaps.
Pitfalls
- Accuracy on imbalanced data — 99% accuracy by predicting the majority class. Use PR-AUC/recall.
- Tuning the threshold on the test set — do it on validation.
- Shuffling time-series CV — leaks the future, inflates scores, collapses in production.
- Reporting a point estimate with no variance — a 0.2% gain inside ±1.5% noise is not a gain.
- Evaluating on the data you tuned on — keep a final untouched test set.
Hand-off
A defensible metric + validated score with uncertainty, plus a chosen decision threshold, for experiment-tracking to log and stakeholders to trust.
1---2name: model-evaluation3description: Use when choosing metrics, validating models, or interpreting results. Covers metric selection by problem type, cross-validation strategy, calibration, confusion-matrix analysis, and avoiding misleading scores.4---56# Model Evaluation78## Overview910The wrong metric on the wrong split produces confident, wrong conclusions. Evaluation is about choosing a metric that matches the business cost, validating it on a split that mirrors production, and reporting it honestly with uncertainty.1112## When to use1314- Selecting how to score a model.15- A model "looks great" but you're unsure it's real.16- Comparing candidate models for promotion.1718## Metric selection1920| Problem | Default metric | Use when |21|---------|---------------|----------|22| Balanced classification | ROC-AUC, accuracy | classes ~balanced |23| Imbalanced classification | PR-AUC, F1, recall@k | rare positives (fraud, disease) |24| Probabilistic output | Log loss, Brier, calibration | you need trustworthy probabilities |25| Ranking | NDCG, MAP, MRR | recommendation/search |26| Regression | MAE (robust), RMSE (penalize big errors) | match error cost |27| Regression, multiplicative | MAPE / RMSLE | errors scale with magnitude |2829## Cross-validation strategy3031- **Default:** `StratifiedKFold` for classification.32- **Time series:** `TimeSeriesSplit` — never shuffle; train on past, validate on future.33- **Grouped data** (multiple rows per user): `GroupKFold` so the same group never spans train and test.34- **Small data:** repeated CV; report mean ± std.3536## Honest reporting3738```python39from sklearn.model_selection import cross_val_score, StratifiedKFold40import numpy as np4142cv = StratifiedKFold(5, shuffle=True, random_state=42)43scores = cross_val_score(model, X, y, cv=cv, scoring="average_precision")44print(f"PR-AUC: {scores.mean():.3f} ± {scores.std():.3f}") # always report spread45```4647## Beyond a single number4849- **Confusion matrix / classification report** at the chosen threshold — accuracy hides per-class failure.50- **Threshold tuning** — default 0.5 is rarely optimal; pick it from the PR curve to match precision/recall needs.51- **Calibration** — `CalibratedClassifierCV` or reliability curves when probabilities feed decisions.52- **Slice metrics** — evaluate on subgroups to catch fairness/robustness gaps.5354## Pitfalls5556- **Accuracy on imbalanced data** — 99% accuracy by predicting the majority class. Use PR-AUC/recall.57- **Tuning the threshold on the test set** — do it on validation.58- **Shuffling time-series CV** — leaks the future, inflates scores, collapses in production.59- **Reporting a point estimate** with no variance — a 0.2% gain inside ±1.5% noise is not a gain.60- **Evaluating on the data you tuned on** — keep a final untouched test set.6162## Hand-off6364A defensible metric + validated score with uncertainty, plus a chosen decision threshold, for experiment-tracking to log and stakeholders to trust.
param087