Amey-Thakur Overfitting diagnosis
Poor validation performance has several causes that look identical from a single number. Treating leakage as overfitting, or shift as capacity, wastes weeks. Diagnosis is comparing the right curves before choosing a treatment.
Method
- Compare training and validation curves together. A widening gap is overfitting; both high and flat is underfitting; validation improving then worsening is the classic overfit turning point.
- Check for leakage first. Validation performance that is suspiciously good, or better than training, usually means information from validation reached training (see train-test-discipline).
- Verify the split is honest. Random splits leak across time, groups, and duplicates, and the split must respect the structure of the data (see cross-validation).
- Test whether the model can overfit at all. If it cannot drive training loss down on a small subset, the problem is capacity, bugs, or the learning rate rather than generalisation.
- Compare validation with deployment distribution. A gap that appears only in production is shift rather than overfitting (see drift-monitoring).
- Inspect the worst validation examples. They usually show a pattern, and often the pattern is label noise rather than model failure.
- Treat the diagnosed cause. More data and regularisation for overfitting, more capacity or training for underfitting, and a fixed pipeline for leakage (see regularization-techniques).
Boundaries
Diagnosis requires a validation set that represents deployment, and if it does not, every conclusion is unreliable. Small validation sets give noisy estimates that look like real differences. Repeatedly tuning against one validation set overfits it (see agent-eval-design).