Amey-Thakur Imbalanced data
With 1% positives, "always predict negative" scores 99% accuracy. The cure is rarely synthetic data; it is evaluating correctly, weighting costs, and choosing the operating threshold deliberately.
Method
- Fix the evaluation first. PR curves and PR-AUC, precision and recall at the deployment threshold, per the cost table from ml-problem-framing; never accuracy, and treat ROC-AUC with suspicion at heavy skew (it barely moves when false positives grow by thousands against a huge negative pool). Keep the test set at the natural imbalance whatever you do to train (see train-test-discipline).
- Move the threshold before touching the data. A well-calibrated model at natural imbalance plus a cost-derived threshold (minimize expected cost on validation) solves many "imbalance problems" outright. Sweep the threshold, plot cost vs threshold, pick the minimum; revisit when costs or base rates change.
- Weight before you resample. Class weights
(inverse-frequency or cost-ratio) in the loss are equivalent to
oversampling without duplicating rows, supported natively by most
learners (
class_weight,scale_pos_weight). This is the default second move; it changes nothing about the data pipeline. - Resample only with care, inside the fold. Random undersampling of negatives works at massive scale (and speeds training); SMOTE-class synthesis helps sometimes on tabular data and harms whenever synthetic points cross the true boundary. All resampling fits inside cross-validation folds on train only; resampling before splitting leaks duplicated/synthetic neighbors into test (see cross-validation).
- Recalibrate after any reweighting. Weights and resampling distort predicted probabilities away from true base rates; if probabilities feed thresholds or expected values, recalibrate on an untouched validation slice (see model-evaluation calibration).
- Buy more positives where it is cheapest. Longer time windows, related-event labels, hard-negative mining to sharpen the boundary, or targeted labeling of model-uncertain cases (active learning); a hundred real positives beat ten thousand synthetic ones. At extreme rarity (<0.01%), reconsider the frame: anomaly detection or tiered rules-then-model funnels (see ml-problem-framing).
Boundaries
- Imbalance is not the problem when both classes are plentiful in absolute terms; a 1:100 ratio with a million positives needs nothing special.
- If the minority class is undersampled in collection (label bias, censoring), no algorithm recovers what was never observed; that is a data acquisition problem.
- Base rates drift; a threshold tuned at 1% positives misfires at 3% (see drift-monitoring), so monitor the rate itself.