Pmlbmini Eval

pmlbmini-eval

PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applications — Knauer et al. (2024) (arXiv:2409.01635, 2024)

What this evaluates

This benchmark evaluates the discriminative performance of machine learning models on tabular classification tasks under data-scarce conditions (sample sizes ≤500). It specifically probes whether complex AutoML and deep learning approaches can consistently outperform simple baselines like logistic regression when training data is limited.

Datasets

• PMLBmini — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RicardoKnauer/TabMini

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Computed as the mean test AUC across folds in a stratified 3-fold cross-validation procedure to assess discriminative performance.

Input / output format

Input: Tabular feature matrix with binary class labels.

Output: Predicted class probabilities or decision scores for each instance.

Scoring recipe

def compute_auc(y_true, y_pred_scores):
    fpr, tpr, _ = roc_curve(y_true, y_pred_scores)
    return auc(fpr, tpr)

fold_aucs = []
for train_idx, test_idx in StratifiedKFold(n_splits=3).split(X, y):
    model.fit(X[train_idx], y[train_idx])
    y_scores = model.predict_proba(X[test_idx])[:, 1]
    fold_aucs.append(compute_auc(y[test_idx], y_scores))
mean_test_auc = np.mean(fold_aucs)

Common pitfalls

• Complex models (AutoML/DL) often overfit in low-data regimes, making simple logistic regression a strong baseline that matches or exceeds them on many datasets.
• TabPFN's pretraining data overlaps with 45% of the benchmark datasets, which inflates its reported performance and requires careful interpretation.
• Strict runtime limits (1h for training AUC, 3h for test AUC) must be enforced to maintain comparability across methods; exceeding them breaks the evaluation protocol.

Evidence (verbatim from paper)

We measured the discriminative performance in terms of the AUC. The training AUC was recorded to assess overfitting and evaluated with a 1h runtime limit for each method per benchmark dataset, the mean test AUC with a 3h runtime limit via a stratified, 3-fold cross-validation procedure (i.e., 1h per fold).

Citation

@misc{knauer2024pmlbmini,
  title={PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applications},
  author={Knauer et al. (2024)},
  year={2024},
  note={arXiv:2409.01635}
}

• arXiv: 2409.01635