Omnitabbench Eval

omnitabbench-eval

OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale — Dihong Jiang et al. (2026) (arXiv:2604.06814, 2026)

What this evaluates

Evaluates the out-of-the-box predictive performance of tree-based models, neural networks, and foundation models on a large-scale collection of real-world tabular datasets. It also analyzes how dataset metafeatures (e.g., size, feature distribution, target skewness) correlate with model success to identify which model category excels under specific data conditions.

Datasets

• OmniTabBench — total 3030; splits: test (3030)

Metrics

• performance score (primary) — range: other
  • Standard tabular metrics: accuracy for classification tasks and root mean squared error (RMSE) for regression tasks. Exact formulas and conventions are detailed in Section 3.3 of the paper.

Input / output format

Input: Tabular data containing numerical and categorical columns, with missing values imputed and numericals quantile-transformed. Categoricals are label-encoded or one-hot encoded depending on the model.

Output: Predicted class labels for classification or continuous values for regression.

Scoring recipe

if task == 'classification':
    return np.mean(preds == gold)
else:
    return np.sqrt(np.mean((preds - gold) ** 2))

Common pitfalls

• Models are evaluated with default/standard configurations only; tuning is explicitly excluded to test out-of-the-box capability.
• TabPFN has hard constraints (≤10k samples, ≤500 features); evaluating it on larger datasets violates its design.
• Using a small number of datasets for evaluation causes rank oscillation and can lead to misleading model rankings.

Evidence (verbatim from paper)

We use the same metrics that quantify the difficulty of each dataset as described in Section 3.3 to evaluate the performance of selected models.

Citation

@misc{jiang2026omnitabbench,
  title={OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale},
  author={Dihong Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.06814}
}

• arXiv: 2604.06814