Amey-Thakur Cross-validation
CV answers "how much does my estimate depend on which rows landed where" by averaging over K splits. It only works when every fold respects the same boundaries deployment will face; a leaky fold structure launders leakage into confidence.
Method
- Choose the scheme from the data's dependence structure. IID rows: stratified K-fold (K=5-10; stratify on the label, especially when imbalanced; see imbalanced-data). Repeated measures per entity: GroupKFold on the entity so no user/patient/device spans folds (see train-test-discipline). Temporal data: walk-forward (expanding or sliding window, train always before test); shuffled K-fold on time series is the canonical way to publish a model that cannot predict anything.
- Run the entire pipeline inside each fold. Scaling, imputation, encoding, feature selection, resampling: all fit on the fold's training portion only (pipeline objects make this automatic; see feature-engineering). Preprocessing fit on the full dataset before CV is the most common silent inflation in applied work.
- Nest when CV both tunes and evaluates. Inner loop selects hyperparameters (see hyperparameter-tuning), outer loop estimates generalization of the whole selection procedure; the outer score is the honest one. Tuning and reporting on the same folds overstates performance by exactly the amount the search exploited them.
- Report the spread with the mean. Per-fold scores, mean, and standard deviation; overlapping spreads between model A and B is a tie (see model-evaluation on uncertainty). One anomalous fold is information: inspect it (a segment? a time period?), do not average it away (see ml-error-analysis).
- Keep a final untouched test set anyway. CV replaces the validation role at small scale, not the test role: model and protocol decisions accumulate against the CV estimate, so the last word comes from data no fold ever saw (see train-test-discipline).
- Match the final refit to the protocol. Standard practice: select config by CV, refit on all training data with that config, confirm on test. For iterative learners early-stopped per fold, set the final iteration count from the folds' median rather than re-early-stopping on test.
Boundaries
- At large data scale, a single well-constructed temporal/group split estimates as well as CV at a tenth the compute; CV's value concentrates where data is scarce.
- CV variance understates true uncertainty when folds share an entity, a time regime, or collection artifacts; the scheme can only respect boundaries you know to encode.
- Leave-one-out is high-variance and expensive for most learners; prefer K=5-10 unless a statistical reason says otherwise.