Tabarena Eval

tabarena-eval

TabArena: A Living Benchmark for Machine Learning on Tabular Data — Nick Erickson et al. (2025) (arXiv:2506.16791, 2025)

What this evaluates

Evaluates the predictive performance of tabular machine learning models across 51 real-world datasets under standardized, reproducible protocols. It probes how hyperparameter tuning, nested cross-validation, and post-hoc ensembling affect peak performance and efficiency trade-offs.

Datasets

• TabArena — total 51; splits: train (-1), test (-1)

Metrics

• predictive performance (primary) — range: other
  • Task-dependent standard tabular metric: AUC-ROC for classification, R² or RMSE for regression. Aggregated across all datasets.
• median inference time per 1000 samples — range: other
  • Wall-clock time to generate predictions on 1000 test samples, reported as median across datasets.

Input / output format

Input: Tabular feature matrix (numerical/categorical) and target column.

Output: Predicted target values or class probabilities per instance.

Scoring recipe

scores = []
for dataset in datasets:
    train, test = split(dataset)
    model = train_with_nested_cv(train)
    preds = model.predict(test)
    scores.append(compute_metric(preds, test.target))
return mean(scores)

Common pitfalls

• Using holdout validation instead of nested cross-validation significantly underestimates model performance and biases rankings toward ensembling methods.
• Evaluating models with default parameters only misrepresents peak performance; post-hoc ensembling is required to reveal true capabilities.
• Ignoring compute and hardware constraints leads to inaccurate assessments of efficiency trade-offs and inference costs.

Evidence (verbatim from paper)

Predictive performance of a model with tuning and ensembling when using holdout or cross-validation.

Citation

@misc{erickson2025tabarena,
  title={TabArena: A Living Benchmark for Machine Learning on Tabular Data},
  author={Nick Erickson et al. (2025)},
  year={2025},
  note={arXiv:2506.16791}
}

• arXiv: 2506.16791