Texttabbench Eval

texttabbench-eval

Towards Benchmarking Foundation Models for Tabular Data With Text — Martin Mráz et al. (2025) (arXiv:2507.07829, 2025)

What this evaluates

Evaluates foundation models on tabular prediction tasks that require leveraging mixed categorical, numerical, and free-text features across diverse real-world domains. The benchmark tests whether models can maintain predictive performance when text features contain semantic ambiguity, synonym variation, or noise, while preserving structural tabular signals.

Datasets

• fraud — total 17880; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• kick — total 94125; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• osha — total 4847; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• cards — total 2810; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• complaints — total 96935; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• spotify — total 10000; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• airbnb — total 3818; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• beer — total 2914; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• houses — total 44913; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• laptops — total 984; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• mercari — total 100000; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• permits — total 90876; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench
• wine — total 1281; splits: train (-1), test (-1); repo https://github.com/mrazmartin/TextTabBench

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification metric: proportion of correctly predicted class labels out of total instances.
• MSE — range: [0, inf)
  • Standard regression metric: mean squared error between predicted and true continuous values.

Input / output format

Input: Tabular instances containing categorical, numerical, and free-text columns.

Output: Predicted class labels (binary or multi-class) or continuous values (regression).

Scoring recipe

if task in ['b-clf', 'm-clf']:
    score = sum(pred == gold) / len(gold)
elif task == 'reg':
    score = mean((pred - gold) ** 2)
return score

Common pitfalls

• Heavy preprocessing or category collapsing that destroys original free-text variation.
• Target leakage where target information inadvertently appears in input features.
• Datasets lacking dual-signal, collapsing into pure-NLP or pure-tabular tests.

Evidence (verbatim from paper)

Existing collections of datasets for benchmarking tabular data with text do not yet suffice for a robust evaluation. Therefore, we curate a new corpus according to five rules: (i) Real (free) text features. We select datasets that are not merely short categorical codes. Instead, we include datasets with real text features that can contain any free text. (ii) Dual-signal requirement. Both textual and structural features must carry predictive information; otherwise, a task collapses into either a pure-NLP or pure-tabular test. (iii) Tabular predictive task. The dataset must be primarily a tabular predictive task. Thus, we exclude datasets that are, for example, recommender systems or text retrieval/look-up tasks. (iv) Accessibility. Restricted data, such as real-world patient records, are omitted so that the benchmark remains easily accessible for all users. (v) Domain and target diversity. We do not reuse the same source datasets for multiple benchmark tasks. Instead, we cover commerce, reviews, finance, and sensor-augmented data, spanning regression and (binary/multi-class) classification tasks.

Citation

@misc{mraz2025texttabbench,
  title={Towards Benchmarking Foundation Models for Tabular Data With Text},
  author={Martin Mráz et al. (2025)},
  year={2025},
  note={arXiv:2507.07829}
}

• arXiv: 2507.07829