Imbalanced Classification

Imbalanced Classification (Verbose)

Core Patterns

Diagnose Before You Treat

Imbalance is not itself a problem — it is a symptom that the default loss and the default threshold are misaligned with the cost structure. Establish three numbers before touching a sampler:

1. Prevalence. 1:10 rarely needs intervention. 1:1000 changes the whole approach.
2. Absolute positive count. 500 positives out of 500,000 is workable; 20 positives out of 20,000 is a problem of data volume, not of balance, and no resampler fixes it.
3. Relative cost. A missed fraud costing $400 against a review costing $5 gives an 80:1 ratio that directly implies the operating threshold.

import numpy as np
counts = np.bincount(y)
print(counts, counts[1] / counts.sum())   # e.g. [99000  1000] 0.01

Metric Selection

from sklearn.metrics import (average_precision_score, roc_auc_score,
                             precision_recall_curve, confusion_matrix)

p = model.predict_proba(X_test)[:, 1]
print("PR-AUC ", average_precision_score(y_test, p))   # baseline = prevalence
print("ROC-AUC", roc_auc_score(y_test, p))             # baseline = 0.5

The baselines differ in a way that matters. A random classifier scores 0.5 ROC-AUC but only 0.01 PR-AUC at 1% prevalence, so a PR-AUC of 0.25 — unimpressive-looking — is a 25× lift over chance. Report PR-AUC against prevalence, not against 0.5.

Always print the confusion matrix at the deployed threshold. Aggregate scores hide whether the model produces 40 false positives or 4,000.

Cost-Sensitive Learning

The cheapest effective intervention is reweighting the loss, which leaves the data untouched and therefore cannot leak.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
import xgboost as xgb

LogisticRegression(class_weight="balanced", max_iter=1000)
RandomForestClassifier(class_weight="balanced_subsample")
xgb.XGBClassifier(scale_pos_weight=(y == 0).sum() / (y == 1).sum())

"balanced" sets w_c = n_samples / (n_classes * n_c). For explicit control pass a dict — class_weight={0: 1, 1: 80} — derived from the actual cost ratio rather than from the frequency ratio. Those two ratios are different quantities and are only accidentally equal.

Resampling

SMOTE assumes Euclidean neighbourhoods are meaningful. On one-hot encoded high-cardinality categoricals, on unscaled features, or in more than ~50 dimensions, interpolation produces points that lie off the data manifold — synthetic records that no real process could generate. Scale first, or use SMOTENC, or skip it.

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler
from sklearn.model_selection import cross_val_score, StratifiedKFold

pipe = Pipeline([
    ("over", SMOTE(sampling_strategy=0.1, random_state=0)),   # minority to 10% of majority
    ("under", RandomUnderSampler(sampling_strategy=0.5, random_state=0)),
    ("clf", RandomForestClassifier(n_estimators=300, n_jobs=-1)),
])
scores = cross_val_score(pipe, X, y, cv=StratifiedKFold(5), scoring="average_precision")

Note sampling_strategy=0.1 rather than full balance. Resampling all the way to 1:1 is a habit, not a requirement; moderate rebalancing frequently outperforms it and distorts probabilities less. Treat the ratio as a hyperparameter.

Threshold Optimization

import numpy as np
from sklearn.metrics import precision_recall_curve

prec, rec, thr = precision_recall_curve(y_val, p_val)

# Highest recall subject to precision >= 0.30
ok = prec[:-1] >= 0.30
threshold = thr[ok][np.argmax(rec[:-1][ok])]

Or optimize expected cost directly, which is the honest version when the numbers exist:

def expected_cost(t, cost_fn=400, cost_fp=5):
    pred = (p_val >= t).astype(int)
    fn = ((pred == 0) & (y_val == 1)).sum()
    fp = ((pred == 1) & (y_val == 0)).sum()
    return fn * cost_fn + fp * cost_fp

grid = np.linspace(0.01, 0.99, 99)
best = grid[np.argmin([expected_cost(t) for t in grid])]

Choose the threshold on validation data and freeze it. Selecting it on the test set is the same leakage as tuning any other hyperparameter there.

Calibration

Resampling and class weighting shift the predicted probability distribution: a model trained on artificially balanced data outputs values near a 50% base rate rather than the true 1%. Ranking is preserved, so AUC is unaffected — but any downstream expected-value calculation is now wrong.

from sklearn.calibration import CalibratedClassifierCV

clf = CalibratedClassifierCV(base_model, method="isotonic", cv=5)
clf.fit(X_train, y_train)     # fit on the real, unresampled distribution

Use method="sigmoid" (Platt scaling) when calibration data is scarce — isotonic needs roughly 1,000+ samples and will overfit below that.

Common Anti-Patterns

❌ Reporting accuracy on 99:1 data. The trivial constant classifier wins. ✅ PR-AUC, plus precision/recall at the deployed threshold.

❌ X_res, y_res = SMOTE().fit_resample(X, y) before splitting — synthetic points interpolated from validation rows land in training. ✅ imblearn.pipeline.Pipeline, which resamples only during fit.

❌ Resampling the test set to "make evaluation fair" — the test set must reflect production prevalence or precision is meaningless. ✅ Resample training only; evaluate at the real base rate.

❌ Shipping the 0.5 threshold because predict() uses it. ✅ Tune the threshold on validation against the cost structure.

❌ Optimizing ROC-AUC at 1:1000 and being surprised that 96% of alerts are false. ✅ Optimize PR-AUC or precision@k, which react to false positives.

❌ Always resampling to exactly 1:1. ✅ Sweep sampling_strategy; compare against plain class_weight="balanced" first.

❌ SMOTE on one-hot categoricals or unscaled features — interpolation invents impossible records. ✅ SMOTENC for mixed types, scaling before any distance-based sampler.

❌ Treating 20 positives as an imbalance problem. ✅ It is a data-collection problem — gather more labels, or use anomaly detection.

Imbalanced Classification Checklist

• Prevalence, absolute positive count, and cost ratio all measured
• Primary metric is PR-AUC / precision@k, never accuracy
• PR-AUC compared against the prevalence baseline
• Class weighting tried before any resampler
• Resamplers live inside imblearn.pipeline.Pipeline
• Validation and test sets left at production prevalence
• sampling_strategy tuned, not fixed at 1:1
• SMOTENC used when categorical features are present
• Decision threshold chosen on validation from the cost structure
• Confusion matrix at the deployed threshold reviewed
• Probabilities calibrated if consumed as probabilities
• Stratified splits used throughout
• Alert volume checked against downstream review capacity