Amey-Thakur Train-test discipline
An evaluation is a simulation of deployment. Every way your split lets training peek at test conditions is a way production will disappoint you by exactly that margin.
Method
- Split first, explore second. Carve out the test set before EDA, feature design, or imputation decisions; explore only train. Every choice made after seeing test data (features that "help", removed outliers, tuned thresholds) leaks your judgment into the score even when no rows leak.
- Match the split to the deployment boundary. Predicting the future: temporal split (train on past, test on later period), never random: random splits let the model interpolate a future it will not have. Entities appearing in many rows (users, patients, devices): group split so no entity spans sets, or you measure memorization. Both concerns: group-aware temporal split.
- Keep three sets with distinct jobs. Train fits; validation selects (features, hyperparameters, early stopping, thresholds); test estimates final performance, touched once per project milestone. When validation gets overfit by many decisions (see hyperparameter-tuning), refresh it from train; the test set has no such rescue, which is why it stays locked.
- Audit for contamination mechanically. Duplicate/near-duplicate rows across sets (exact hashes, then fuzzy on text), the same entity under different keys, preprocessing fit on all data (scalers, encoders, imputers must fit train only; see feature-engineering), and augmented copies of a test item landing in train. Run these checks in CI on the dataset build, not once by hand.
- Mirror production's information state. Test features computed point-in-time (as-of joins), test distribution matching the serving population (not the convenient historical slice), and the label definition frozen to the frame (see ml-problem-framing). If production sees 1% fraud, a balanced test set inflates every metric you report.
- Report with uncertainty and slices. Bootstrap confidence intervals on the headline metric, plus per-slice results (see model-evaluation); a point estimate on one random split is a coin flip with decimals. For small data, nested cross-validation replaces the single validation set (see cross-validation).
Boundaries
- Public benchmark test sets are contaminated for any model trained on web-scale data; treat leaderboard deltas there accordingly.
- Time-series with one history (one market, one machine) cannot give independent test periods; walk-forward evaluation estimates stability, not true generalization, and honesty requires saying so.
- Discipline bounds optimism; it cannot fix a label or population that shifted after your data ended (see drift-monitoring).