Tabularmath Eval

tabularmath-eval

TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis — Cheng et al. (2026) (arXiv:2602.02523, 2026)

What this evaluates

Evaluates computational extrapolation and algorithmic generalization in tabular learning models by testing their ability to predict target values outside the training distribution. It probes whether models learn statistical interpolation versus deterministic computation on program-verified synthetic math problems.

Datasets

• TabularMath — total 233472; splits: context (-1), query (-1); repo https://github.com/Marco-Cheng/TabularMath

Metrics

• rounded consistency (primary) — range: [0, 1]
  • Exact-match accuracy after rounding predictions and ground truth to the nearest integer. Formula: Accuracy = (1/|D_test|) * sum_{i in D_test} 1[round(y_hat_i) == round(y_i)].

Input / output format

Input: Tabular rows with numeric features and a target variable y. Models receive a context set (training rows) to learn the mapping, followed by a query set (test rows) for prediction. For ICL, rows are serialized as structured text prompts.

Output: Continuous predicted values for each query row. For ICL models, predictions must be returned as a JSON list of floats corresponding to query rows in order.

Scoring recipe

def score_rounded_consistency(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if round(p) == round(g))
    return correct / len(gold)

Common pitfalls

• Data leakage occurs if preprocessing (standardization/imputation) is fit on the full dataset instead of strictly on the context set.
• Confusing input extrapolation with output extrapolation; the OOD split specifically tests output extrapolation where test targets exceed the maximum training target.
• Evaluating ICL models with row caps larger than 128 violates the specified context-length constraints.

Evidence (verbatim from paper)

Our primary metric in the paper is rounded consistency, defined as exact-match accuracy after rounding predictions to integer, reflecting whether or not the underlying math problem is correctly solved by the model’s prediction: | Accuracy=\frac{1}{|\mathcal{D}{\text{test}}|}\sum{i\in\mathcal{D}{\text{test}}}\mathbf{1}[\mathrm{round}(\hat{y}{i})=\mathrm{round}(y_{i})]. | The rounded consistency metric reflects the discrete correctness requirement of mathematical problems.

Citation

@misc{cheng2026tabularmath,
  title={TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis},
  author={Cheng et al. (2026)},
  year={2026},
  note={arXiv:2602.02523}
}

• arXiv: 2602.02523