ML Evaluation
Overview
Use this skill for evaluating machine learning systems before, during, and after deployment. A valid evaluation answers: what decision will use the model, what data distribution matters, what failure costs exist, which metric encodes those costs, and whether the measured improvement is real rather than leakage, variance, or overfitting to validation.
Evaluation Design Checklist
- Define the unit of prediction and the unit of independence. Split by user, account, patient, device, document, session, or time when examples are correlated.
- Establish baseline performance using a simple heuristic, prior model, or linear/tree baseline.
- Pick metrics aligned to the decision, not just convenient defaults.
- Freeze test data until final assessment; use validation data for tuning.
- Evaluate slices: labels, geography, device, language, demographic groups when appropriate, data source, time period, and known hard cases.
- Report uncertainty with confidence intervals or repeated CV when sample size is limited.
- Perform qualitative error analysis before scaling model complexity.
Split Strategies
Use random stratified splits only for iid data where examples are independent. Use grouped splits when multiple rows share an entity. Use time-based splits for forecasting, recommendation, fraud, logs, and any deployment where future data differs from past data. Use nested cross-validation when both hyperparameters and performance estimates must be unbiased. For small datasets, repeated stratified CV can reduce variance, but keep a final untouched test set when stakes are high.
Comprehensive Data Leakage Prevention Checklist
- Split Before Fit: Apply
scaler.fit(), imputer.fit(), and encoder fits only on the training subset. Never fit on the entire dataset.
- Temporal Monotonicity: Ensure that for time-series splits, the validation/test indices are chronologically strictly after the training indices.
- Grouped Isolations: Verify that data points sharing common grouping factors (e.g., patient IDs, company IDs, session IDs) do not span both the training and testing sets.
- Target Leakage: Check for columns representing events downstream of the prediction time (e.g., "refunded_date" in a churn model).
- No Pre-split Augmentation: Apply minority class oversampling (e.g., SMOTE) or data augmentations after splitting. Oversampling before splitting places synthetic twins across the boundary.
Metric Selection
| Task |
Prefer |
Watch for |
| Balanced classification |
Accuracy, macro F1, log loss |
Accuracy hides calibration and minority errors |
| Imbalanced classification |
PR-AUC, recall at precision, F-beta, cost utility |
ROC-AUC can look strong despite poor positive precision |
| Probabilistic classification |
Log loss, Brier score, calibration curves |
Threshold metrics ignore probability quality |
| Regression |
MAE, RMSE, RMSLE, pinball loss, R2 |
RMSE overweights outliers; MAPE fails near zero |
| Ranking/retrieval |
NDCG, MAP, MRR, recall@k, precision@k |
Offline negatives may not match production candidates |
| Forecasting |
sMAPE, MASE, pinball loss, coverage |
Random splits leak future patterns |
| Generation |
Task-specific human/eval model rubrics plus automated checks |
BLEU/ROUGE alone often miss usefulness and safety |
| Embeddings/RAG |
Recall@k, MRR, answer faithfulness, groundedness, latency |
Retrieval and generation failures must be separated |
For high-stakes decisions, evaluate operating thresholds with confusion matrices and cost curves. Optimize thresholds on validation, then report locked-threshold performance on test. Calibrate probabilities when decisions consume probabilities.
Statistical Comparisons
Do not treat a simple metric difference as proof. Use paired comparisons when models are evaluated on the same examples.
Bootstrap Confidence Intervals (Python Recipe)
import numpy as np
from sklearn.metrics import f1_score
def bootstrap_metric_ci(
y_true: np.ndarray,
y_pred: np.ndarray,
metric_fn = f1_score,
n_bootstrap: int = 1000,
confidence_level: float = 0.95
):
bootstrapped_scores = []
rng = np.random.default_rng(seed=42)
n_samples = len(y_true)
for _ in range(n_bootstrap):
# Sample with replacement
indices = rng.choice(n_samples, size=n_samples, replace=True)
score = metric_fn(y_true[indices], y_pred[indices])
bootstrapped_scores.append(score)
sorted_scores = np.sort(bootstrapped_scores)
alpha = 1.0 - confidence_level
lower_idx = int(np.floor(alpha / 2.0 * n_bootstrap))
upper_idx = int(np.ceil((1.0 - alpha / 2.0) * n_bootstrap))
return sorted_scores[lower_idx], sorted_scores[upper_idx]
McNemar's Test for Comparing Paired Classifiers
McNemar's test assesses if two models disagree significantly.
import numpy as np
from scipy.stats import chi2_contingency
def mcnemar_test(y_true: np.ndarray, pred_model_a: np.ndarray, pred_model_b: np.ndarray):
# Contingency Table:
# Model B Correct Model B Incorrect
# Model A Correct n00 n01
# Model A Incorrect n10 n11
correct_a = (pred_model_a == y_true)
correct_b = (pred_model_b == y_true)
n01 = np.sum(correct_a & ~correct_b) # A correct, B incorrect
n10 = np.sum(~correct_a & correct_b) # A incorrect, B correct
# Calculate chi-squared with continuity correction
if n01 + n10 == 0:
return 1.0 # Perfect agreement
statistic = (abs(n01 - n10) - 1)**2 / (n01 + n10)
# 1 degree of freedom
from scipy.stats import chi2
p_value = 1 - chi2.cdf(statistic, df=1)
return p_value
Temperature Scaling for Logits Calibration (PyTorch)
import torch
import torch.nn as nn
import torch.optim as optim
class TemperatureScaler(nn.Module):
def __init__(self):
super().__init__()
# Initial temperature parameter of 1.0 (no scaling)
self.temperature = nn.Parameter(torch.ones(1))
def forward(self, logits: torch.Tensor):
return logits / self.temperature
def fit(self, val_logits: torch.Tensor, val_labels: torch.Tensor, max_iter: int = 50):
# Optimizes temperature parameter on validation set
nll_criterion = nn.CrossEntropyLoss()
optimizer = optim.LBFGS([self.temperature], lr=0.01, max_iter=max_iter)
def eval_loss():
optimizer.zero_grad()
loss = nll_criterion(self.forward(val_logits), val_labels)
loss.backward()
return loss
optimizer.step(eval_loss)
return self.temperature.item()
Error Analysis
Create a structured error taxonomy rather than scanning random failures. Segment false positives, false negatives, high-confidence errors, low-confidence correct cases, out-of-distribution examples, missing values, rare labels, long-tail entities, prompt categories, and latency/timeouts.
Error analysis should feed concrete remediation: collect data, relabel, change split, alter loss, add features, improve retrieval, calibrate threshold, simplify model, or adjust serving pipeline.
Responsible AI and Robustness
Evaluate model behavior across relevant demographic, geographic, linguistic, accessibility, and operational slices. Choose fairness metrics that match the domain: demographic parity, equalized odds, equal opportunity, calibration by group, or counterfactual consistency. Explain trade-offs; fairness metrics can conflict. Use interpretable models or explainability tools when users need reasons, but validate explanations for stability and plausibility.
Test robustness to missing fields, schema changes, adversarial or noisy inputs, prompt injection for LLM/RAG systems, image corruptions, text typos, distribution shift, and low-resource slices. Do not deploy without a monitoring plan for data drift, performance drift, calibration drift, and safety incidents.
Production Evaluation
Offline metrics are necessary but not sufficient. Before launch, run shadow evaluation, backtests, batch replay, or canaries where feasible. Compare training/serving preprocessing outputs on the same examples. Define rollback thresholds, alert destinations, retraining criteria, and owners. For A/B tests, include guardrail metrics such as latency, cost, user harm, fairness slices, business constraints, and support burden.
Common Evaluation Failures
- Great validation, poor production: leakage, distribution shift, serving skew, threshold mismatch, or non-representative validation.
- Strong ROC-AUC, poor user value: wrong metric for imbalance or thresholded decision.
- Improving mean metric, harming key users: missing slice evaluation.
- Test set repeatedly reused: hidden overfitting to benchmark.
- LLM evaluation looks good but answers hallucinate: automated lexical metrics not measuring groundedness, citation quality, or refusal behavior.
Sources
1---
2name: ml-evaluation
3description: |
4 This skill should be used when the user asks to evaluate, validate, compare, explain, or debug model performance. PROACTIVELY activate for: (1) metrics selection for classification, regression, ranking, NLP, CV, recommender, forecasting, and generative AI, (2) train/validation/test splits, cross-validation, grouped or time-series validation, (3) confusion matrices, ROC/PR curves, calibration, thresholds, error analysis, (4) ablation studies, statistical significance, confidence intervals, bootstrap tests, (5) bias, fairness, explainability, robustness, leakage detection. Provides: rigorous evaluation methodology and production-readiness checks.
5metadata:
6 source: https://skillmd.com/skills/josiahsiegel/ml-evaluation
7---
8
9# ML Evaluation
10
11## Overview
12
13Use this skill for evaluating machine learning systems before, during, and after deployment. A valid evaluation answers: what decision will use the model, what data distribution matters, what failure costs exist, which metric encodes those costs, and whether the measured improvement is real rather than leakage, variance, or overfitting to validation.
14
15## Evaluation Design Checklist
16
171. Define the unit of prediction and the unit of independence. Split by user, account, patient, device, document, session, or time when examples are correlated.
182. Establish baseline performance using a simple heuristic, prior model, or linear/tree baseline.
193. Pick metrics aligned to the decision, not just convenient defaults.
204. Freeze test data until final assessment; use validation data for tuning.
215. Evaluate slices: labels, geography, device, language, demographic groups when appropriate, data source, time period, and known hard cases.
226. Report uncertainty with confidence intervals or repeated CV when sample size is limited.
237. Perform qualitative error analysis before scaling model complexity.
24
25## Split Strategies
26
27Use random stratified splits only for iid data where examples are independent. Use grouped splits when multiple rows share an entity. Use time-based splits for forecasting, recommendation, fraud, logs, and any deployment where future data differs from past data. Use nested cross-validation when both hyperparameters and performance estimates must be unbiased. For small datasets, repeated stratified CV can reduce variance, but keep a final untouched test set when stakes are high.
28
29### Comprehensive Data Leakage Prevention Checklist
30* **Split Before Fit**: Apply `scaler.fit()`, `imputer.fit()`, and encoder fits **only** on the training subset. Never fit on the entire dataset.
31* **Temporal Monotonicity**: Ensure that for time-series splits, the validation/test indices are chronologically strictly after the training indices.
32* **Grouped Isolations**: Verify that data points sharing common grouping factors (e.g., patient IDs, company IDs, session IDs) do not span both the training and testing sets.
33* **Target Leakage**: Check for columns representing events downstream of the prediction time (e.g., "refunded_date" in a churn model).
34* **No Pre-split Augmentation**: Apply minority class oversampling (e.g., SMOTE) or data augmentations **after** splitting. Oversampling before splitting places synthetic twins across the boundary.
35
36## Metric Selection
37
38| Task | Prefer | Watch for |
39|---|---|---|
40| Balanced classification | Accuracy, macro F1, log loss | Accuracy hides calibration and minority errors |
41| Imbalanced classification | PR-AUC, recall at precision, F-beta, cost utility | ROC-AUC can look strong despite poor positive precision |
42| Probabilistic classification | Log loss, Brier score, calibration curves | Threshold metrics ignore probability quality |
43| Regression | MAE, RMSE, RMSLE, pinball loss, R2 | RMSE overweights outliers; MAPE fails near zero |
44| Ranking/retrieval | NDCG, MAP, MRR, recall@k, precision@k | Offline negatives may not match production candidates |
45| Forecasting | sMAPE, MASE, pinball loss, coverage | Random splits leak future patterns |
46| Generation | Task-specific human/eval model rubrics plus automated checks | BLEU/ROUGE alone often miss usefulness and safety |
47| Embeddings/RAG | Recall@k, MRR, answer faithfulness, groundedness, latency | Retrieval and generation failures must be separated |
48
49For high-stakes decisions, evaluate operating thresholds with confusion matrices and cost curves. Optimize thresholds on validation, then report locked-threshold performance on test. Calibrate probabilities when decisions consume probabilities.
50
51## Statistical Comparisons
52
53Do not treat a simple metric difference as proof. Use paired comparisons when models are evaluated on the same examples.
54
55### Bootstrap Confidence Intervals (Python Recipe)
56```python
57import numpy as np
58from sklearn.metrics import f1_score
59
60def bootstrap_metric_ci(
61 y_true: np.ndarray,
62 y_pred: np.ndarray,
63 metric_fn = f1_score,
64 n_bootstrap: int = 1000,
65 confidence_level: float = 0.95
66):
67 bootstrapped_scores = []
68 rng = np.random.default_rng(seed=42)
69 n_samples = len(y_true)
70
71 for _ in range(n_bootstrap):
72 # Sample with replacement
73 indices = rng.choice(n_samples, size=n_samples, replace=True)
74 score = metric_fn(y_true[indices], y_pred[indices])
75 bootstrapped_scores.append(score)
76
77 sorted_scores = np.sort(bootstrapped_scores)
78 alpha = 1.0 - confidence_level
79 lower_idx = int(np.floor(alpha / 2.0 * n_bootstrap))
80 upper_idx = int(np.ceil((1.0 - alpha / 2.0) * n_bootstrap))
81
82 return sorted_scores[lower_idx], sorted_scores[upper_idx]
83```
84
85### McNemar's Test for Comparing Paired Classifiers
86McNemar's test assesses if two models disagree significantly.
87```python
88import numpy as np
89from scipy.stats import chi2_contingency
90
91def mcnemar_test(y_true: np.ndarray, pred_model_a: np.ndarray, pred_model_b: np.ndarray):
92 # Contingency Table:
93 # Model B Correct Model B Incorrect
94 # Model A Correct n00 n01
95 # Model A Incorrect n10 n11
96
97 correct_a = (pred_model_a == y_true)
98 correct_b = (pred_model_b == y_true)
99
100 n01 = np.sum(correct_a & ~correct_b) # A correct, B incorrect
101 n10 = np.sum(~correct_a & correct_b) # A incorrect, B correct
102
103 # Calculate chi-squared with continuity correction
104 if n01 + n10 == 0:
105 return 1.0 # Perfect agreement
106
107 statistic = (abs(n01 - n10) - 1)**2 / (n01 + n10)
108 # 1 degree of freedom
109 from scipy.stats import chi2
110 p_value = 1 - chi2.cdf(statistic, df=1)
111
112 return p_value
113```
114
115### Temperature Scaling for Logits Calibration (PyTorch)
116```python
117import torch
118import torch.nn as nn
119import torch.optim as optim
120
121class TemperatureScaler(nn.Module):
122 def __init__(self):
123 super().__init__()
124 # Initial temperature parameter of 1.0 (no scaling)
125 self.temperature = nn.Parameter(torch.ones(1))
126
127 def forward(self, logits: torch.Tensor):
128 return logits / self.temperature
129
130 def fit(self, val_logits: torch.Tensor, val_labels: torch.Tensor, max_iter: int = 50):
131 # Optimizes temperature parameter on validation set
132 nll_criterion = nn.CrossEntropyLoss()
133 optimizer = optim.LBFGS([self.temperature], lr=0.01, max_iter=max_iter)
134
135 def eval_loss():
136 optimizer.zero_grad()
137 loss = nll_criterion(self.forward(val_logits), val_labels)
138 loss.backward()
139 return loss
140
141 optimizer.step(eval_loss)
142 return self.temperature.item()
143```
144
145## Error Analysis
146
147Create a structured error taxonomy rather than scanning random failures. Segment false positives, false negatives, high-confidence errors, low-confidence correct cases, out-of-distribution examples, missing values, rare labels, long-tail entities, prompt categories, and latency/timeouts.
148
149Error analysis should feed concrete remediation: collect data, relabel, change split, alter loss, add features, improve retrieval, calibrate threshold, simplify model, or adjust serving pipeline.
150
151## Responsible AI and Robustness
152
153Evaluate model behavior across relevant demographic, geographic, linguistic, accessibility, and operational slices. Choose fairness metrics that match the domain: demographic parity, equalized odds, equal opportunity, calibration by group, or counterfactual consistency. Explain trade-offs; fairness metrics can conflict. Use interpretable models or explainability tools when users need reasons, but validate explanations for stability and plausibility.
154
155Test robustness to missing fields, schema changes, adversarial or noisy inputs, prompt injection for LLM/RAG systems, image corruptions, text typos, distribution shift, and low-resource slices. Do not deploy without a monitoring plan for data drift, performance drift, calibration drift, and safety incidents.
156
157## Production Evaluation
158
159Offline metrics are necessary but not sufficient. Before launch, run shadow evaluation, backtests, batch replay, or canaries where feasible. Compare training/serving preprocessing outputs on the same examples. Define rollback thresholds, alert destinations, retraining criteria, and owners. For A/B tests, include guardrail metrics such as latency, cost, user harm, fairness slices, business constraints, and support burden.
160
161## Common Evaluation Failures
162
163- Great validation, poor production: leakage, distribution shift, serving skew, threshold mismatch, or non-representative validation.
164- Strong ROC-AUC, poor user value: wrong metric for imbalance or thresholded decision.
165- Improving mean metric, harming key users: missing slice evaluation.
166- Test set repeatedly reused: hidden overfitting to benchmark.
167- LLM evaluation looks good but answers hallucinate: automated lexical metrics not measuring groundedness, citation quality, or refusal behavior.
168
169## Sources
170
171- scikit-learn model evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html
172- TensorFlow Model Analysis: https://www.tensorflow.org/tfx/model_analysis
173- Google ML crash course evaluation guidance: https://developers.google.com/machine-learning/crash-course/classification
174- Fairlearn documentation: https://fairlearn.org/
175- Evidently AI ML monitoring and evaluation concepts: https://docs.evidentlyai.com/