Gdro Tabular Imbalance Eval

gdro-tabular-imbalance-eval

Investigating Group Distributionally Robust Optimization for Deep Imbalanced Learning: A Case Study of Binary Tabular Data Classification — Ismail B. Mustapha et al. (2023) (arXiv:2303.02505, 2023)

What this evaluates

Assesses deep learning models' ability to classify highly imbalanced binary tabular data by comparing standard empirical risk minimization against group distributionally robust optimization.

Datasets

• Multiple benchmark imbalanced tabular datasets — total ?; splits: train (-1), validation (-1)

Metrics

• g-mean (primary) — range: [0, 1]
  • Geometric mean of sensitivity and specificity, balancing performance across majority and minority classes.
• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's ability to discriminate between classes across all classification thresholds.

Input / output format

Input: Tabular feature vectors representing binary classification instances.

Output: Binary class predictions or class probabilities used to compute g-mean and ROC-AUC.

Scoring recipe

metrics_per_run = []
for rep in range(5):
    model = train(method, train_split)
    preds = model.predict(val_split)
    g_mean = geometric_mean(sensitivity(preds), specificity(preds))
    roc_auc = compute_roc_auc(preds, labels)
    metrics_per_run.append((g_mean, roc_auc))
mean_g_mean = np.mean([m[0] for m in metrics_per_run])
std_g_mean = np.std([m[0] for m in metrics_per_run])
# Repeat for ROC-AUC
# Rank methods per dataset per metric, then average ranks across datasets

Common pitfalls

• Relying solely on overall accuracy, which is misleading for imbalanced data.
• Reporting single-run results instead of the required 5-repetition mean ± standard deviation.
• Averaging performance across datasets without tracking per-dataset rankings, which obscures method consistency.

Evidence (verbatim from paper)

Experiments on benchmark imbalanced tabular datasets show that gDRO outperforms ERM and classical methods (e.g., reweighting) in g-mean and ROC-AUC metrics, demonstrating superior generalization to minority classes in highly skewed data. The mean (±standard deviation) of the validation performance of each method across the respective evaluation metrics for each dataset over five repetitions of the experiment are presented in Fig 2.

Citation

@misc{mustapha2023gdro,
  title={Investigating Group Distributionally Robust Optimization for Deep Imbalanced Learning: A Case Study of Binary Tabular Data Classification},
  author={Ismail B. Mustapha et al. (2023)},
  year={2023},
  note={arXiv:2303.02505}
}

• arXiv: 2303.02505